We Keep Coding

async / await TCP listener saving images from cameras

Hi guys I have a big problem: I have createad a windows form that worked as a Server listening for TCP messages sended by some cameras connected to the same private network.
Each camera send a tcp message to the server when the camera detects something, the server has to work continuosly without freezing the gui interface and has to process every TCP message. The processing of this data includes the saving of an image shooted by the sender camera. The server saves these images into a folder and here there is my problem:
the server is not able to save correclty each image: it looks like that some bytes have been lost during the trasmission but I think that all the bytes have riched the server, something deeper has happend. Maybe it could be the way that I have programmed the async / await server?
I have a list of TcpListener because I have to use more server, one for each tab.
Here how the saved images look like
https://imgur.com/xtlgHPk
https://imgur.com/CcvWbDH
As you can see is not completely saved, except someone for some unkown reason
https://imgur.com/G25UPSS
public void TcpServer(int port)
{
IPAddress ipAddress = null;
string hostName = Dns.GetHostName();
IPHostEntry ipHostInfo = Dns.GetHostEntry(hostName);
for (int i = 0; i < ipHostInfo.AddressList.Length; ++i)
{
if (ipHostInfo.AddressList[i].AddressFamily ==
AddressFamily.InterNetwork)
{
ipAddress = ipHostInfo.AddressList[i];
_listener.Add( new TcpListener(ipAddress, port));
ReceiveDataAsync();
break;
}
}
if (ipAddress == null)
throw new Exception("No IPv4 address for server");
}
private async void ReceiveDataAsync()
{
try
{
_listener[tbServer.SelectedIndex].Start();
while (true)
{
var tcpClient = await _listener[tbServer.SelectedIndex].AcceptTcpClientAsync();
ReadDataFromClientAsync(tcpClient);
}
}
catch (Exception e)
{
MessageBox.Show("Errore: ", e.Message.ToString());
}
}
private async Task ReadDataFromClientAsync(TcpClient client)
{
try
{
using (NetworkStream stream=client.GetStream())
{
int selectedTabIndex, indexOfPort;
string endPoint;
while (client.Connected)
{
int count = 0;
var countBytes = new byte[4];
for (int i = 0; i < 6; i++)
{
count = await stream.ReadAsync(countBytes, 0, countBytes.Length);
}
//The data dimension of the TCP message is into his header, 24th byte.
if (count == 0)
{
break;
}
byte[] bytes = new byte[BitConverter.ToUInt32(countBytes, 0)];
await stream.ReadAsync(bytes, 0, bytes.Length);
indexOfPort = Settings.getIndexOfPort(client.Client.LocalEndPoint.ToString());
endPoint = client.Client.RemoteEndPoint.ToString();
selectedTabIndex = Settings.getIndexOfPort(client.Client.LocalEndPoint.ToString());
updateGui("entry", indexOfPort,endPoint);
BufferData bufferService = new BufferData();
CameraMessage cameraMessage = await bufferService.writeBufferData(bytes.ToList());
if (cameraMessage == null)
return;
//doing some stuff
msgToShow = await bufferService.SaveImageFromCamera(cameraMessage);
//doing other stuff
}
client.Close();
closeCommunication(selectedTabIndex,indexOfPort,endPoint);
}
}
}
catch (IOException e)
{
updateGui("stop");
}
}
The BufferData class:
public class BufferData
{
public async Task<CameraMessage> writeBufferData(List<byte> bytesList)
{
CameraMessage cameraMessage = new CameraMessage();
try
{
int num = 0;
int startSubData = 0;
for (int startData = 0; startData < bytesList.Count-8; startData = startSubData + num)
{
int codeDataMessage = BitConverter.ToInt32(bytesList.GetRange(startData, 4).ToArray(), 0);
int indexDataSize = startData + 4;
int SizeDataMessage = BitConverter.ToInt32(bytesList.GetRange(indexDataSize, 4).ToArray(), 0);
startSubData = indexDataSize + 4;
byte[] array = bytesList.GetRange(startSubData, SizeDataMessage).ToArray();
num = 0;
switch (codeDataMessage)
{
case 14020:
cameraMessage.image = await this.Base64ToImage(array);
num = this.OffSetStringType(SizeDataMessage);
break;
}
}
return cameraMessage;
}
catch (Exception ex)
{
Console.WriteLine(ex.Message.ToString());
return null;
}
}
public async Task<string> SaveImageFromCamera( CameraMessage cameraMessage)
{
string path = Settings.pathFolder + cameraMessage.ld_I_SN + #"\Images\";
string res;
if (!Directory.Exists(Settings.pathFolder+cameraMessage.ld_I_SN))
{
Directory.CreateDirectory(Settings.pathFolder + cameraMessage.ld_I_SN + #"\Images");
}
try
{
await Task.Run(() => { cameraMessage.image.Save(path + cameraMessage.ld_I_FILENAME, ImageFormat.Jpeg); });
res = "#3 - OK - IMMAGINE SALVATA CON SUCCESSO";
cameraMessage.image.Dispose();
return res;
}
catch (Exception ex)
{
res = "#3 - ERR - IMMAGINE NON SALVATA";
return res;
}
}
public async static Task<Image> Base64ToImage(byte[] imageBytes)
{
MemoryStream ms = new MemoryStream(imageBytes, 0, imageBytes.Length);
ms.Write(imageBytes, 0, imageBytes.Length);
return System.Drawing.Image.FromStream(ms, true);
}
}

Your core problem is that you're assuming a socket read will read all the bytes you ask it to. That's not how socket reads work.
When you call await stream.ReadAsync(bytes, 0, bytes.Length), the stream will read any number of bytes from 1 to bytes.Length. It will return the number of bytes actually read. You need to take this into consideration and read again as long as your buffer hasn't been read completely.
Note that this has to be done both for the header bytes and the payload bytes.

how to receive data from a Socket in .NET Core?

I'm currently porting a hardware library to .NET Core. The communication works over TCP. I have problems with the 'Socket.BeginReceive' method. MSDN
It seems there is no equivalent method in .NET Core. How can I receive data from a TCP Socket?
private void InternalDataReceived(IAsyncResult ar)
{
int dataCount = 0;
byte[] buffer;
try
{
if (_client != null && _client.Client != null)
{
dataCount = _client.Client.EndReceive(ar);
}
if (dataCount > 0)
{
try
{
buffer = new byte[dataCount];
Array.Copy(_inBuffer, buffer, dataCount);
if (DataReceived != null)
{
DataReceived(buffer);
}
}
catch (Exception exc)
{
if (exc is System.Net.Sockets.SocketException)
{
Disconnect();
return;
}
}
_client.Client.BeginReceive(_inBuffer, 0, _inBuffer.Length, SocketFlags.None, InternalDataReceived, null);
}
}
catch
{
Disconnect();
}
}

I found another way to do it. Hope this helps someone else.
Basically I just ended up using the NetworkStream class. You can get an instance by calling TcpClient.GetStream(). If you use a using block with GetStream your connection will get closed after the using. This is why I'm not using it in my example because I need the connection to stay alive.
MSDN NetworkStream.Read
My example code:
static void Main(string[] args)
{
TcpClient client = new TcpClient();
client.Client.Connect(IPAddress.Parse("192.168.100.5"), 8000);
//Task.Run(() => ReadData(client));
Task.Run(() => ReadDataLoop(client));
client.Client.Send(Encoding.ASCII.GetBytes("{\"TID\":1111,\"blabla\":{}}"));
while (true)
{
}
}
private static void ReadDataLoop(TcpClient client)
{
while (true)
{
if (!client.Connected)
break;
string xxx = "";
xxx = ReadData(client);
Console.WriteLine(xxx);
}
}
private static string ReadData(TcpClient client)
{
string retVal;
byte[] data = new byte[1024];
NetworkStream stream = client.GetStream();
byte[] myReadBuffer = new byte[1024];
StringBuilder myCompleteMessage = new StringBuilder();
int numberOfBytesRead = 0;
do
{
numberOfBytesRead = stream.Read(myReadBuffer, 0, myReadBuffer.Length);
myCompleteMessage.AppendFormat("{0}", Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));
}
while (stream.DataAvailable);
retVal = myCompleteMessage.ToString();
return retVal;
}

Read InputStream asynchronously

I'm trying to receive packets and if no bytes are received continue with the code below. await ReadAsync blocks until a new packet is received. Is there any way to just read the current bytes received?
If I don't use await messages aren't received.
byte[] data = new byte[BufferSize];
IInputStream inputStream = socket.InputStream;
IBuffer buffer = data.AsBuffer();
socketInformation.GetStopwatchPingIdleTime().Start();
while (socketInformation.open)
{
try
{
inputStream.ReadAsync(buffer, BufferSize, InputStreamOptions.Partial);
data = buffer.ToArray();
}
catch (Exception)
{
break;
}
while (true)
{
// Wait for payload size
if (buffer.Length >= 4)
{
int commandType = (int)BitConverter.ToInt16(data, 0);
int payloadSize = (int)BitConverter.ToInt16(data, 2);
int packetSize = PacketHeaderSize + payloadSize;
// Wait for a full message
if (buffer.Length >= packetSize)
{
byte[] packet = new byte[packetSize];
System.Buffer.BlockCopy(data, 0, packet, 0, packetSize);
ParsePacketSequence(socket, socketInformation, packet);
if (buffer.Length > packetSize)
{
int bufferLength = (int)buffer.Length - packetSize;
byte[] newData = new byte[BufferSize];
System.Buffer.BlockCopy(data, packetSize, newData, 0, bufferLength);
data = newData;
buffer.Length = (uint)bufferLength;
}
else if (buffer.Length == packetSize)
{
break;
}
else
{
break;
}
}
else if (buffer.Length == packetSize)
{
break;
}
}
else
{
break;
}
}
if (host)
{
// Send ping to player
if (socketInformation.waitingPong == false &&
socketInformation.GetStopwatchPingIdleTime().ElapsedMilliseconds > 5000)
{
byte[] pingPacket = CreatePacket(6, null);
SendPacket(socket, socketInformation, pingPacket);
socketInformation.waitingPong = true;
}
}
await Task.Delay(33, tokenSource.Token);
}
inputStream.Dispose();
socket.Dispose();
tokenSource.Cancel();

It looks to me you are receiving a stream of messages. When a message is there you want to process it potentially at some later time or at a different place in the code.
A good approach for that is to have one Task continuously reading messages from the socket and putting them into a queue. You can then pull complete messages from the queue whenever you like.
That way you can get rid of most of the logic here. You never have to abort a read request and you never need to check for timeouts.

Sending and receiving files keeps hanging

my program should be able to send and receive files but for some reason whenever i click on the send (button1) and receive (button2) buttons, it keeps hanging. Not sure if its something wrong my my codes? Also, I feel like my codes are pretty long as compared to other examples i found online but I'm not sure how to rectify.
Client codes
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
IPEndPoint remoteEP = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 9000);
private void Form1_Load(object sender, EventArgs e)
{
try
{
socket.Connect(remoteEP);
textBox2.Text = "Connected to Server";
}
catch (Exception ex)
{
textBox2.Text = "Unable to connect to Server";
textBox2.Text = ex.Message;
}
}
public const string SEND = "[SEND]";
public const string RECEIVE = "[RECEIVE]";
public const string QUIT = "[QUIT]";
private void button1_Click(object sender, EventArgs e)
{
textBox1.Clear();
textBox2.Clear();
NetworkStream stream = new NetworkStream(socket);
StreamReader reader = new StreamReader(stream);
StreamWriter writer = new StreamWriter(stream);
try
{
writer.WriteLine(RECEIVE);
writer.Flush();
writer.WriteLine(textBox1.Text);
writer.Flush();
Bitmap bmp = new Bitmap(#"C:\Users\Y400\Desktop\Lectures\Year 3\WAD\Week 11" + textBox1.Text);
MemoryStream ms = new MemoryStream();
bmp.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
byte[] bmpBytes = ms.GetBuffer();
bmp.Dispose();
ms.Close();
int sent;
sent = sendData(socket, bmpBytes);
textBox1.Text = "Transferring file complete\r\n";
textBox1.Text += bmpBytes.Length + " bytes sent to Server.";
}
catch (Exception ex)
{
textBox2.Text = ex.Message;
}
}
public static int sendData (Socket s, byte[] data)
{
int total = 0;
int size = data.Length;
int left = size;
int sent;
byte[] datasize = new byte[4];
datasize = BitConverter.GetBytes(size);
sent = s.Send(datasize);
while(total<size)
{
sent = s.Send(data, total, left, SocketFlags.None);
total += sent;
left -= sent;
}
return total;
}
private void button2_Click(object sender, EventArgs e)
{
textBox2.Clear();
textBox1.Clear();
byte[] data = new byte[1024];
string fileN = textBox2.Text.Trim();
NetworkStream ns = new NetworkStream(socket);
StreamReader reader = new StreamReader(ns);
StreamWriter writer = new StreamWriter(ns);
writer.WriteLine(SEND);
writer.Flush();
writer.WriteLine(fileN);
writer.Flush();
try
{
while (true)
{
data = receiveData(socket);
MemoryStream ms = new MemoryStream(data);
break;
}
textBox2.Text = ("Receiving file from server ...\r\n" + data.Length + " bytes copied");
}
catch (Exception ex)
{
textBox2.Text = ex.Message;
}
}
public static byte[] receiveData (Socket s)
{
int total = 0;
int recv;
byte[] datasize = new byte[4];
recv = s.Receive(datasize, 0, 4, 0);
int size = BitConverter.ToInt32(datasize, 0);
int dataleft = size;
byte[] data = new byte[size];
while (total < size)
{
recv = s.Receive(data, total, dataleft, 0);
if (recv == 0)
{
break;
}
total += recv;
dataleft -= recv;
}
return data;
}
private void button3_Click(object sender, EventArgs e)
{
textBox1.Clear();
textBox2.Clear();
textBox2.Text = "Connection closed";
socket.Shutdown(SocketShutdown.Both);
socket.Close();
}
Server codes
class Program
{
public const string SEND = "[SEND]";
public const string RECV = "[RECV]";
public const string QUIT = "[QUIT]";
static void Main(string[] args)
{
runServer();
}
static void runServer()
{
try
{
byte[] data = new byte[1024];
Socket server = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
IPEndPoint localEP = new IPEndPoint(IPAddress.Any, 9000);
server.Bind(localEP);
server.Listen(10);
Console.WriteLine("Waiting for Client ...");
Socket client = server.Accept();
Console.WriteLine("Client connected");
NetworkStream stream = new NetworkStream(client);
StreamReader reader = new StreamReader(stream);
StreamWriter writer = new StreamWriter(stream);
try
{
while(true)
{
string request = reader.ReadLine();
string filename = reader.ReadLine();
if (request == QUIT)
{
Console.WriteLine("Client disconnected");
break;
}
else if (request == SEND)
{
getFileFromClient(filename, client);
}
else if (request == RECV)
{
receiveFileFromClient(filename, client);
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
public static void getFileFromClient(string filename, Socket client)
{
try
{
FileStream output = File.OpenWrite(filename);
Console.WriteLine(filename + " created");
int count = 0;
while(true)
{
byte[] data = new byte[1024];
int size = client.Receive(data);
output.Write(data, 0, size);
count += size;
if(size<1024)
{
break;
}
}
output.Close();
Console.WriteLine(count + " bytes read from client");
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
public static void receiveFileFromClient(string filename, Socket client)
{
int count = 0;
FileStream input = File.OpenRead(filename);
Console.WriteLine("Reading " + filename);
while(true)
{
byte[] data = new byte[1024];
int bytesRead = input.Read(data, 0, 1024);
client.Send(data, bytesRead, SocketFlags.None);
count += bytesRead;
if(bytesRead < 1024)
{
break;
}
}
Console.WriteLine("Transferring file completed\r\n" + count + " bytes sent to Client");
input.Close();
}
}

Generally speaking
...try to approach the problem differently.
You can't just copy paste stuff from the Internet and hope for the best. You need to understand what you're doing thoroughly.
Regarding your exact problem
Take a look at the button2_Click method.
It contains a while loop which apparently never finishes.
while (true)
{
data = receiveData(socket);
MemoryStream ms = new MemoryStream(data);
break;
}
It does finish because of the break command. But this is all very hard to read.
When you copy paste code around and then apply quick fixes you end up with a pile of code which is very hard to debug.
It took me about 10 minutes to notice the fact that the client defines it's "message verbs" like so:
public const string SEND = "[SEND]";
public const string RECEIVE = "[RECEIVE]";
public const string QUIT = "[QUIT]";
while the server defines them like so:
public const string SEND = "[SEND]";
public const string RECV = "[RECV]";
public const string QUIT = "[QUIT]";
This is maybe not the only problem, but it is sufficient to create a deadlock,
because the server never executes the positive branch of this if statement:
else if (request == RECV)
{
receiveFileFromClient(filename, client);
}
so the client believes it is about to receive something, which proves to be false.
Also make sure you send the "SEND" and "RECEIVE" message verbs when you should and not mix them up.
Good luck!
PS: I would suggest you take a look at more simpler to use techniques for sending and receiving data, such as:
WCF
ASP.NET Web Services
Web API

Ignoring any logic errors that may occur in your programs, the way you are handling things in your client whenever it is doing an action it is doing it on the GUI thread. This will make your application seem like it is locking but instead it is executing your logic on the GUI thread.
The same problem is occurring on the server. It accepts a connection, and then goes on to receive the file. It will not be able to receive any other connection until it finished receiving the file.
The server is not without problems either because it is never checking if it receives 0 bytes from the socket. Which means that the client closed its end of the connection. You are simply assuming that if you receive less than 1024 you are receiving your last part of the file. This is simply not true for TCP. You only know you received the last part if you receive 0 bytes. TCP is a byte streaming protocol you cannot assume you will be receiving blocks of 1024 bytes. It is likely that this will be in fact the case, but you should not code it like that. Check for reception of 0 bytes. On the client you did check for 0 bytes, I am puzzled why you did not do the same on the server. The problematic part is this :
byte[] data = new byte[1024];
int size = client.Receive(data);
output.Write(data, 0, size);
count += size;
if(size<1024) //you can only break if the size is 0
{
break;
}
There are probably more bugs. As the other answer also indicated some other issues.

Task asynchronous pattern with sockets

I am trying to implement a class that can listen for incoming TCP data. I am trying to do so using tasks.
I have two central methods
private async void ReceiveAsync()
{
while (true)
{
int bytesRead = await Receive();
if (bytesRead > 0)
{
byte[] result = new byte[bytesRead];
Buffer.BlockCopy(_buffer, 0, result, 0, bytesRead);
Console.WriteLine(Encoding.UTF8.GetString(result));
}
}
}
and
private Task<int> Receive()
{
return Task.Run(() =>
{
if (sock.Poll(-1, SelectMode.SelectRead))
if (sock.Available > 0)
return sock.Receive(_buffer, _buffer.Length, SocketFlags.None);
return -1;
}
);
}
In my main program, I call ReceiveAsync() and then send some data down the TCP pipeline, to which the receiver responds. I do get this reply, but now I am caught in an endless loop inside the while(true), and further data sent from the "receiver" is not being received.
There is something completely wrong. What is it?

Try:
int bytesRead;
do {
bytesRead = await Receive();
if(bytesRead > 0) {...}
} while (bytesRead > 0);
i.e. use bytesRead as an exit condition.
Also: you can probably use async here very nicely:
// for illustration ONLY
TaskCompletionSource<int> source = new TaskCompletionSource<int>();
sock.BeginReceive(_buffer, 0, _buffer.Length, SocketFlags.None, ar =>
{
try
{
int val = ((Socket)ar.AsyncState).EndReceive(ar);
source.SetResult(val);
}
catch (Exception ex)
{
source.SetException(ex);
}
}, sock);
return source.Task;