I'm trying to receive packets and if no bytes are received continue with the code below. await ReadAsync blocks until a new packet is received. Is there any way to just read the current bytes received?
If I don't use await messages aren't received.
byte[] data = new byte[BufferSize];
IInputStream inputStream = socket.InputStream;
IBuffer buffer = data.AsBuffer();
socketInformation.GetStopwatchPingIdleTime().Start();
while (socketInformation.open)
{
try
{
inputStream.ReadAsync(buffer, BufferSize, InputStreamOptions.Partial);
data = buffer.ToArray();
}
catch (Exception)
{
break;
}
while (true)
{
// Wait for payload size
if (buffer.Length >= 4)
{
int commandType = (int)BitConverter.ToInt16(data, 0);
int payloadSize = (int)BitConverter.ToInt16(data, 2);
int packetSize = PacketHeaderSize + payloadSize;
// Wait for a full message
if (buffer.Length >= packetSize)
{
byte[] packet = new byte[packetSize];
System.Buffer.BlockCopy(data, 0, packet, 0, packetSize);
ParsePacketSequence(socket, socketInformation, packet);
if (buffer.Length > packetSize)
{
int bufferLength = (int)buffer.Length - packetSize;
byte[] newData = new byte[BufferSize];
System.Buffer.BlockCopy(data, packetSize, newData, 0, bufferLength);
data = newData;
buffer.Length = (uint)bufferLength;
}
else if (buffer.Length == packetSize)
{
break;
}
else
{
break;
}
}
else if (buffer.Length == packetSize)
{
break;
}
}
else
{
break;
}
}
if (host)
{
// Send ping to player
if (socketInformation.waitingPong == false &&
socketInformation.GetStopwatchPingIdleTime().ElapsedMilliseconds > 5000)
{
byte[] pingPacket = CreatePacket(6, null);
SendPacket(socket, socketInformation, pingPacket);
socketInformation.waitingPong = true;
}
}
await Task.Delay(33, tokenSource.Token);
}
inputStream.Dispose();
socket.Dispose();
tokenSource.Cancel();
It looks to me you are receiving a stream of messages. When a message is there you want to process it potentially at some later time or at a different place in the code.
A good approach for that is to have one Task continuously reading messages from the socket and putting them into a queue. You can then pull complete messages from the queue whenever you like.
That way you can get rid of most of the logic here. You never have to abort a read request and you never need to check for timeouts.
Related
I am working on a .NET application where the server sends JPG compressed images from a webcam to the client via TCP/IP. For serialization/deserialization I am using the BinaryFormatter class. When testing between my Desktop computer (client/Windows 10) and my Laptop (server/Windows 10), everything works fine in the long run. When using a LattePanda (server/Windows 7), I get the following error after approximately 3-5 minutes runtime (I send/receive 30 frames per second):
An unhandled exception of type 'System.Runtime.Serialization.SerializationException' occurred in mscorlib.dll
Additional information: The input stream is not a valid binary format. The starting contents (in bytes) are: 00-00-00-01-00-00-00-FF-FF-FF-FF-01-00-00-00-00-00 ...
Here's a snippet from the server code:
private void distribute(Camera camera, Mat frame) {
if(clientSockets!=null) {
if(clientSockets.Count > 0) {
if(camera.Streaming) {
// compress and encapsulate raw image with jpg algorithm
CameraImage packet = new CameraImage(camera.Id, frame, camera.CodecInfo, camera.EncoderParams);
packet.SystemId = Program.Controller.Identity.Id;
packet.SequenceNumber = curSeqNum;
byte[] content;
using(MemoryStream ms = new MemoryStream()) {
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(ms, packet);
content = ms.ToArray();
}
byte[] payload = new byte[content.Length+4];
// prefix with packet length
Array.Copy(BitConverter.GetBytes(content.Length), 0, payload, 0, 4);
// append payload after length header
Array.Copy(content, 0, payload, 4, content.Length);
// distribute to connected clients
this.distribute(payload);
}
}
}
}
private void distribute(byte[] bytes) {
if(Program.Launched) {
lock(syncobj) {
// distribute to connected clients
for(int i=clientSockets.Count-1; i>=0; i--) {
try {
clientSockets[i].Send(bytes, bytes.Length, SocketFlags.None);
} catch(SocketException) {
clientSockets.RemoveAt(i);
}
}
}
}
}
Here's a snippet from the client code:
private void receive() {
try {
while(running) {
if((available = clientSocket.Receive(buffer, 4, SocketFlags.None)) > 0) {
// receive bytes from tcp stream
int offset = 0;
int bytesToRead = BitConverter.ToInt32(buffer, 0);
byte[] data = new byte[bytesToRead];
while(bytesToRead > 0) {
int bytesReceived = clientSocket.Receive(data, offset, bytesToRead, SocketFlags.None);
offset += bytesReceived;
bytesToRead -= bytesReceived;
}
// deserialize byte array to network packet
NetworkPacket packet = null;
using(MemoryStream ms = new MemoryStream(data)) {
BinaryFormatter bf = new BinaryFormatter();
packet = (NetworkPacket)bf.Deserialize(ms);
}
// deliver network packet to listeners
if(packet!=null) {
this.onNetworkPacketReceived?.Invoke(packet);
}
// update network statistics
NetworkStatistics.getInstance().TotalBytesRtpIn += data.Length;
}
}
} catch(SocketException ex) {
onNetworkClientDisconnected?.Invoke(agent.SystemId);
} catch(ObjectDisposedException ex) {
onNetworkClientDisconnected?.Invoke(agent.SystemId);
} catch(ThreadAbortException ex) {
// allows your thread to terminate gracefully
if(ex!=null) Thread.ResetAbort();
}
}
Any ideas why I get the SerializationException only on one machine and not on the other? Different mscorlib.dll libraries installed? How to check the version of the relevant (?) libraries?
Here is a tweaked version of your answer. Right now if clientSocket.Available < 4 and running == true you have a empty while(true) { } loop. This is going to take up 100% of one cpu core doing no useful work.
Instead of looping doing nothing till you have 4 bytes in the system I/O buffer use the same kind of loop you did for the payload for your message and load it to your byte array for the header. (I have also simplified the loop of reading the payload data to the loop I unusually use.)
private void receive() {
try {
while(running) {
int offset = 0;
byte[] header = new byte[4];
// receive header bytes from tcp stream
while (offset < header.Length) {
offset += clientSocket.Receive(header, offset, header.Length - offset, SocketFlags.None);
}
int bytesToRead = BitConverter.ToInt32(header, 0);
// receive body bytes from tcp stream
offset = 0;
byte[] data = new byte[bytesToRead];
while(offset < data.Length) {
offset += clientSocket.Receive(data, offset, data.Length - offset, SocketFlags.None);
}
// deserialize byte array to network packet
NetworkPacket packet = null;
using(MemoryStream ms = new MemoryStream(data)) {
BinaryFormatter bf = new BinaryFormatter();
packet = (NetworkPacket)bf.Deserialize(ms);
}
// deliver network packet to listeners
if(packet!=null) {
this.onNetworkPacketReceived?.Invoke(packet);
}
// update network statistics
NetworkStatistics.getInstance().TotalBytesRtpIn += data.Length;
}
}
} catch(SocketException ex) {
onNetworkClientDisconnected?.Invoke(agent.SystemId);
} catch(ObjectDisposedException ex) {
onNetworkClientDisconnected?.Invoke(agent.SystemId);
} catch(ThreadAbortException ex) {
// allows your thread to terminate gracefully
if(ex!=null) Thread.ResetAbort();
}
}
However, if you switched from the System.Net.Socket class to the System.Net.TcpClient class you could simplify your code a lot. First, if you don't need TotalBytesRtpIn to be updating you can stop sending the header, it is not needed for the deserialization as BinaryFormatter already has it's length stored as a internal field of the payload. Then all you need to do is get the NetworkStream from the TcpClient and process the packets as they come in.
private TcpClient _client; // Set this wherever you had your original Socket set up.
private void receive() {
try {
using(var stream = _client.GetStream()) {
BinaryFormatter bf = new BinaryFormatter();
while(running) {
#region This part is not needed if you are only going to deserialize the stream and not update TotalBytesRtpIn, make sure the server stops sending the header too!
int offset = 0;
byte[] header = new byte[4];
// receive header bytes from tcp stream
while (offset < header.Length) {
offset += stream.Read(header, offset, header.Length - offset);
}
int bytesToRead = BitConverter.ToInt32(header, 0);
#endregion
packet = (NetworkPacket)bf.Deserialize(stream);
// deliver network packet to listeners
if(packet!=null) {
this.onNetworkPacketReceived?.Invoke(packet);
}
// update network statistics
NetworkStatistics.getInstance().TotalBytesRtpIn += bytesToRead;
}
}
}
//These may need to get changed.
} catch(SocketException ex) {
onNetworkClientDisconnected?.Invoke(agent.SystemId);
} catch(ObjectDisposedException ex) {
onNetworkClientDisconnected?.Invoke(agent.SystemId);
} catch(ThreadAbortException ex) {
// allows your thread to terminate gracefully
if(ex!=null) Thread.ResetAbort();
}
}
As Scott suggested, I replaced the unsafe line that reads the header of the messages with a safer approach:
private void receive() {
try {
while(running) {
if(clientSocket.Available>=4) {
// receive header bytes from tcp stream
byte[] header = new byte[4];
clientSocket.Receive(header, 4, SocketFlags.None);
int bytesToRead = BitConverter.ToInt32(header, 0);
// receive body bytes from tcp stream
int offset = 0;
byte[] data = new byte[bytesToRead];
while(bytesToRead > 0) {
int bytesReceived = clientSocket.Receive(data, offset, bytesToRead, SocketFlags.None);
offset += bytesReceived;
bytesToRead -= bytesReceived;
}
// deserialize byte array to network packet
NetworkPacket packet = null;
using(MemoryStream ms = new MemoryStream(data)) {
BinaryFormatter bf = new BinaryFormatter();
packet = (NetworkPacket)bf.Deserialize(ms);
}
// deliver network packet to listeners
if(packet!=null) {
this.onNetworkPacketReceived?.Invoke(packet);
}
// update network statistics
NetworkStatistics.getInstance().TotalBytesRtpIn += data.Length;
}
}
} catch(SocketException ex) {
onNetworkClientDisconnected?.Invoke(agent.SystemId);
} catch(ObjectDisposedException ex) {
onNetworkClientDisconnected?.Invoke(agent.SystemId);
} catch(ThreadAbortException ex) {
// allows your thread to terminate gracefully
if(ex!=null) Thread.ResetAbort();
}
}
Now it's running flawlessly :) Thanks a lot for your help!
I'm looking to read data off the wire and send it to a function to parse the contents. Due to the potential varying size of the message (xml), I could read a whole message, more than one message, or a message portion.
I am trying to implement code using the BlockingCollection where I would call TryAdd when I read the data off the wire and use a consumer thread to pull the data off the BlockingCollection to parse. The examples seem pretty straight-forward, but they only seem to work once, then exit. I want the consumer to continuously parse as messages come in. See code below for what i am currently doing.
handling of messages:
private static BlockingCollection<byte[]> queue = new BlockingCollection<byte[]>();
public XmlHandler()
{
CancellationTokenSource cts = new CancellationTokenSource();
Task.Factory.StartNew(() =>
{
if (Console.ReadKey().KeyChar == 'c')
cts.Cancel();
});
Task.Factory.StartNew(() => ParserWorker(queue, cts.Token));
}
//run producer
public void AddData(byte[] data, int bytesRead)
{
bool success = false;
try
{
success = queue.TryAdd(data);
}
catch (OperationCanceledException)
{
Console.WriteLine("Add loop canceled.");
queue.CompleteAdding();
}
if (success)
{
Console.WriteLine(" Add:{0}", data);
}
else
{
Console.Write(" AddBlocked");
}
System.Console.WriteLine("queue count = " + queue.Count);
}
private static void ParserWorker(BlockingCollection<byte[]> bc, CancellationToken ct)
{
ASCIIEncoding encoder = new ASCIIEncoding();
String xmlString;
while (!bc.IsCompleted)
{
byte[] nextItem;
try
{
if (!bc.TryTake(out nextItem, 0, ct))
{
Console.WriteLine(" Take Blocked");
}
else
{
xmlString = encoder.GetString(nextItem, 0, nextItem.Length);
System.Console.WriteLine(xmlString);
}
}
catch (OperationCanceledException)
{
Console.WriteLine("Taking canceled.");
break;
}
}
}
Reading off the wire (this runs in a thread):
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
byte[] message = new byte[8192];
int bytesRead;
while (true)
{
bytesRead = 0;
try
{
bytesRead = clientStream.Read(message, 0, 4096);
byte[] temp = new byte[bytesRead];
Array.Copy(message, temp, bytesRead);
/*CODE WILL HANG HERE...*/
ASCIIEncoding encoder = new ASCIIEncoding();
String xmlString = encoder.GetString(message, 0, message.Length);
System.Console.WriteLine(xmlString);
/*DOES NOT GO BEYOND LINE ABOVE */
handler.AddData(message, bytesRead); //xmlhandler
}
catch (Exception e)
{
System.Console.WriteLine(e.ToString());
break;
}
if (bytesRead == 0)
{
break;
}
}
}
So can anyone tell me what I am doing wrong here?
I will get 3mins of delay at the start of the below program.but where as in hyper terminal it shows every data for every 2 secs.
Her is my below code.please let me know where i went wrong .How i can rectify this delay. any suggestion?
private void StartReceiving()
{
// coding part of receiving changed.
try
{
string IPStr = textBox1_IPaddress.Text.Trim();
string portStr = textBox2_Port.Text.Trim();
int port = Convert.ToInt32(portStr);
int bytesRead = 0;
byte[] buffer = new byte[9];
//------------------------------------------------------------------
IPAddress ipAddress = System.Net.IPAddress.Parse(IPStr);
//create server's tcp listener for incoming connection
try
{
textBox3.Visible = true;
client = new TcpClient();
client.Connect(IPStr, port);
ns = client.GetStream();
while (true)
{
if (ns.DataAvailable)
{
byte[] data = ReadNumberBytes(ns, 9);
ASCIIEncoding encoder = new ASCIIEncoding();
msg = encoder.GetString(data);
textBox3.AppendText(msg);
textBox3.AppendText("\n");
GetData(msg);
}
else
{
Thread.Sleep(4000);
byte[] data = ReadNumberBytes(ns, 9);
ASCIIEncoding encoder = new ASCIIEncoding();
msg = encoder.GetString(data);
GetData(msg);
}
}
client.Close();
}
catch (SocketException se)
{
//message box;
}
}
}
public static byte[] ReadNumberBytes(NetworkStream stream, int n)
{
byte[] buffer = new byte[n];
int bytesRead = 0;
int chunk;
while (bytesRead < n)
{
chunk = stream.Read(buffer, (int)bytesRead, buffer.Length - int)bytesRead);
if (chunk == 0)
{
// error out.
throw new Exception("Unexpected disconnect");
}
bytesRead += chunk;
}
return buffer;
}
I am writing what is essentially an image backup server to store images. It is a one way service that will not return anything beyond a basic success or failure message to the client.
The issue that I am experienceing is that when I send a byte array through the network stream, it is being cut-off before the end of the stream at random locations. I do not have this issue when I run the server on my development machine and connect locally, but rather it only occurs when the server is deployed on a remote server.
When I send very small arrays ( < 512 bytes) the server recieves the entire stream successfully, but on streams larger than 2000 bytes I experience issues. The code for the client is as follows:
try
{
TcpClient Voice = new System.Net.Sockets.TcpClient();
//Obviously I use the remote IP when it is deployed - but have altered it for privacy.
IPEndPoint BackupServer = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 57000);
Voice.Connect(BackupServer);
NetworkStream DataStream = Voice.GetStream();
byte[] buffer = new ASCIIEncoding().GetBytes(ImageData.GetXml());
DataStream.Write(buffer, 0, buffer.Length);
DataStream.Flush();
}
catch
{
}
try
{
buffer = new byte[4096];
int read = DataStream.Read(buffer, 0, buffer.Length);
MessageBox.Show(new ASCIIEncoding().GetString(buffer) + " : " + read.ToString());
}
catch
{
}
The client code executes without any errors or problems regardless of the size of data I send.
And the code for the server side is as follows:
private void BackMeUp(object voice)
{
TcpClient Voice = (TcpClient)voice;
Voice.ReceiveTimeout = 30000;
NetworkStream DataStream = Voice.GetStream();
try
{
bool ShouldLoop = true;
//int loops = 0;
int loops = -1;
byte[] input = new byte[2048];
byte[] buffer = new byte[0];
//while (ShouldLoop)
while(loops != 0)
{
loops = DataStream.Read(input, 0, 2048);
for (int x = 0; x < loops; x++)
{
Array.Resize(ref buffer, buffer.Length + 1);
buffer[buffer.Length - 1] = input[x];
}
//if (loops < 2048)
//{
//ShouldLoop = false;
//break;
//}
}
while (true)
{
StringReader Reader = new StringReader(new ASCIIEncoding().GetString(buffer, 0, buffer.Length));
DataSet DS = new DataSet();
DS.ReadXml(Reader);
if (DS.Tables.Count > 0)
{
if (DS.Tables["Images"].Rows.Count > 0)
{
foreach (DataRow row in DS.Tables["Images"].Rows)
{
//
}
}
}
string response = "Got it!";
DataStream.Write(new ASCIIEncoding().GetBytes(response), 0, response.Length);
DataStream.Flush();
Voice.Close();
break;
}
}
catch (Exception Ex)
{
File.WriteAllText("Elog.txt", Ex.Message + " " + (Ex.InnerException != null ? Ex.InnerException.ToString() : " no Inner"));
Voice.Close();
}
}
The server recieves the data fine, and closes the stream when it reaches the end, however the data is cut-off and I get an error when I try to rebuild the dataset.
I have the impression this has to do with the time it takes to send the stream, and I have played around with the Close and Flush commands but I feel like I'm just shooting in the dark. Any help would be appreciated.
Concise version of question: What factors are involved with a TcpListener that could cause a) the truncation of the stream. or b) premature closing of the stream prior to all bytes being read. When the listener in question is on a remote host rather than a local server.
The Read method doesn't have to return the number of bytes that you requested, or the entire stream at once. Especially if the stream is slow, it will be returned in small chunks.
Call the Read method repeatedly, and handle the data for each block that you get. The Read method returns zero when the stream is read to the end:
buffer = new byte[4096];
do {
int read = DataStream.Read(buffer, 0, buffer.Length);
if (read != 0) {
// handle the first "read" bytes of the buffer (index 0 to read-1)
}
} while (read != 0);
If you know that your buffer is enough for any stream, you can fill up the buffer and handle it afterwards:
buffer = new byte[4096];
int offset = 0;
do {
int read = DataStream.Read(buffer, offset, buffer.Length - offset);
offset += read;
} while (read != 0);
// handle the first "offset" bytes of the buffer (index 0 to offset-1)
First i am n00b in socket programming. So i decided to write simple data over lan tcp server
My server code that handles incomming data is
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
clientStream.ReadTimeout = 10;
int size = 4096 * 1000;
byte[] message = new byte[size];
byte[] All = new byte[0];
int bytesRead;
string error = "";
lock (this)
{
while (true)
{
All = new byte[0];
while (true)
{
bytesRead = 0;
try
{
bytesRead = clientStream.Read(message, 0, size);
All = AddBArrays(All, message, bytesRead);
}
catch
{
break;
}
if (bytesRead == 0)
{
break;
}
}
if (All.Length > 0)
{
Message m = (Message)Tools.ByteArrayToObject(All);
OnRecived(new RecivedArgs("localhost", (Message)Tools.ByteArrayToObject(All)));
}
}
tcpClient.Close();
}
}
byte[] AddBArrays(byte[] ar1, byte[] ar2, int read)
{
byte[] concat = new byte[ar1.Length + read];
if (ar1.Length != 0)
System.Buffer.BlockCopy(ar1, 0, concat, 0, ar1.Length);
System.Buffer.BlockCopy(ar2, 0, concat, ar1.Length, read);
return concat;
}
it works but have some issues. It fales receiving files bigger then 100 mbs or smthng and also if i send data very often interval < 800 then data is lost. how should i improve my code? The large file issue is not so important the primary issue is the data loss in fast data sending.
tnx for help
Ok i now updated the code by the suggestions
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
clientStream.ReadTimeout = 10;
int size = 4096 * 1000;
List<byte> Test = new List<byte>();
byte[] message = new byte[size];
byte[] All = new byte[0];
int bytesRead;
while (true)
{
//All = new byte[0];
while (true)
{
bytesRead = 0;
try
{
bytesRead = clientStream.Read(message, 0, size);
for (int i = 0; i < bytesRead; i++)
{
Test.Add(message[i]);
}
}
catch
{
break;
}
if (bytesRead == 0)
{
break;
}
}
if (Test.Count > 0)
{
Message m = (Message)Tools.ByteArrayToObject(Test.ToArray());
OnRecived(new RecivedArgs("localhost", m));
Test = new List<byte>();
}
}
tcpClient.Close();
}
but the issues still there
Edit--> large file issue fixed it was just a 'System.OutOfMemoryException' but it didn't throw a error.
The All byte array you should change to a List<byte>. You are creating instances like crazy right now. The GC is probably working a lot more than it needs to. This might be slowing it down so much that it can't keep up.
Not really related to sockets:
Make size a const
NEVER lock this. Create a private field that you can lock. In fact, I don't even think you need a lock here.
remove the error string.
OK i solved the problem.
I simple send to much data to fast so data loss was unavoidable.
My optimized code is
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
int size = 1;
byte[] message = new byte[1];
int bytesRead;
while (true)
{
bytesRead = 0;
if (clientStream.DataAvailable)
bytesRead = clientStream.Read(message, 0, 1);
if (bytesRead > 0)
{
OnRecived(new RecivedArgs("tick", null));
}
Thread.Sleep(1);
}
}
i have tested intervals as low as 1 ms and no data loss :)
thanks for your help