We Keep Coding

C# Serial Port Buffer Looking for Specific Binary Patterns

I have spent a lot of team researching the proper ways to use the serial port in C# such that you don't have problems reading in data. I think I have a solution which is pretty close to functional, but I have some glitches every once in a while which I cannot seem to figure out.
My goal: Read formatted binary messages from the serial port, and pass them along to a processor.
The message format looks something like this:
(MSG-HEADER)(MSG-ID)(MSG-LENGTH)(DATA0)(DATA1)(DATA2)...(DATA-N)
Each "word" in the data is 16 bits (2-bytes). My basic approach is to start in a "read message header" state, where each time the serial data received event occurs, I read from the serial port, store the data in a buffer, and then check to see if I detect the message header. If I detect the message header, I move into a "read data" state, where I keep reading data into a data buffer until I have read bytes.
This seems to work pretty well, except occasionally I see "data glitches". Where I end up storing a message that looks something like this:
(MSG1-HEADER)(MSG1-ID)(MSG1-LENGTH)(DATA0)(DATA1)(DATA2)(MSG2-HEADER)(MSG2-ID)..etc
Basically, every so often I get a proper message header, message ID, message length, then the data starts (typically around 200 bytes), and right in the middle of that data I see another message header, message id, and message length, and presumably the start of another message data section. And I can't seem to figure out why.
Here is the code in the serial port data received event I am using:
public byte[] headerBuff = new byte[500];
public byte[] dataBuff = new byte[500];
public byte[] tempBuff = new byte[500];
public int bytesRead;
public int dataPos;
public int dataMsgLen;
public int dataBytesRead = 0;
public bool READ_HEADER = true;
ConcurrentQueue<byte[]> serialQ = new ConcurrentQueue<byte[]>();
//private void si_DataReceived(byte[] data)
private void si_DataReceived(object s, EventArgs e)
{
//If we're supposed to be reading the header, read some bytes and look
// For the header identification sequence (0xF989)
if (READ_HEADER)
{
//Read some bytes, save how many we read
bytesRead = comport.Read(headerBuff, 0, comport.BytesToRead);
//Any time we call comport.REad, we automatically log those bytes to a file
using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
writer.Write(headerBuff.Skip(0).Take(bytesRead).ToArray());
//Loop through the bytes we just read and look for sequence
for (int i = 0; i < (bytesRead-1); i++)
{
if (headerBuff[i] == 0xF9 && headerBuff[i + 1] == 0x89)
{
//We have identified a header
// Lets copy it into a new array
dataPos = bytesRead-i;
Array.Copy(headerBuff, i, dataBuff, 0, dataPos);
dataMsgLen = dataBuff[4];
//Now we can switch to message logging
READ_HEADER = !READ_HEADER;
Array.Clear(headerBuff, 0, headerBuff.Length); //clear the buffer for next time
break; // don't need to look for headers anymore
}
}
}
//If we are done reading the header, let's wait until we get
// enough bytes to store the data message
else if (!READ_HEADER)
{
// Read some bytes into temp array
var tempNumBytes = comport.Read(tempBuff, 0, comport.BytesToRead);
//ADD this into data buffer
Array.Copy(tempBuff, 0, dataBuff, dataPos + dataBytesRead, tempNumBytes);
//Increment our counter
dataBytesRead += tempNumBytes;
//Save to stream
using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
writer.Write(tempBuff.Skip(0).Take(tempNumBytes).ToArray());
//Add to FIFO if we have read enough bytes
if (dataBytesRead >= (dataMsgLen * 2))
{
//Debug.Print(BitConverter.ToString(dataBuff));
serialQ.Enqueue(dataBuff.Select(x => x).ToArray()); // Add to queue for processing
READ_HEADER = !READ_HEADER; // Go back to looking for headers
dataBytesRead = 0;
}
}
}
I appreciate any help, let me know if you need any clarifications.
Thank you in advance.

All,
Thank you for your comments. Based on what I read, I re-wrote the serial data handler (see code below) and it seems to be working much better. I have had it running for about ten minutes now and I haven't seen this glitch at all.
//Declare some public variables for serial port reading
public byte[] headerBuff = new byte[500];
public byte[] dataBuff = new byte[500];
public byte[] tempBuff = new byte[500];
public int headerBytesRead = 0;
public int dataBytesRead = 0;
public const int HEADER_LENGTH = 10;
public int dataInd;
public int fullMsgLen;
public byte[] queuePop;
//Declare some states
public bool READ_HEADER = true;
//Where will we store the data log?
public string defDataDir;
//Declare a public queue as a FIFO for incoming serial data once the
// buffer is full
ConcurrentQueue<byte[]> serialQ = new ConcurrentQueue<byte[]>();
//private void si_DataReceived(byte[] data)
private void si_DataReceived(object s, EventArgs e)
{
//If we're supposed to read the headers, do that
if(READ_HEADER)
{
//Read some bytes
var numBytesRead = comport.Read(tempBuff, 0, comport.BytesToRead);
//Any time we call comport.Read, we automatically log those bytes to a file
using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
writer.Write(tempBuff.Skip(0).Take(numBytesRead).ToArray());
//Add these bytes to a header array
Array.Copy(tempBuff, 0, headerBuff, headerBytesRead, numBytesRead);
//Increment headerBytesRead counter
headerBytesRead += numBytesRead;
//Loop through header and see if we have a header
if(headerBytesRead>=HEADER_LENGTH)
{
//Loop through all the header bytes read so far
for(int i=0; i<headerBytesRead;i++)
{
//Look for the header start word. Note, 3rd bool statement
// here is to make sure we have enough bytes left to identify a header
// e.g. read 12 bytes, and bytes 11 and 12 are 0xF9 and 0x89, then we
// clearly don't have the rest of the header (since it is length 10)
if(headerBuff[i]==0xF9 && headerBuff[i+1]==0x89 && (headerBytesRead-i-1)>=9)
{
//We have identified a header, and have enough following characters to save it
//Copy the header into the data array
Array.Copy(headerBuff, i, dataBuff, 0, headerBytesRead - i);
dataInd = headerBytesRead - i;
//Save the message length
fullMsgLen = dataBuff[4]*2 + HEADER_LENGTH;
//Switch over to reading data
READ_HEADER = !READ_HEADER;
//Reset our header length counter
headerBytesRead = 0;
//Clear the header buffer for next time
Array.Clear(headerBuff, 0, headerBuff.Length);
break; // don't need to look for headers anymore
}
}
}
}
//Handle reading data into buffer here
else if (!READ_HEADER)
{
//We've just been told to start reading data bytes, and we know how many
var numBytesRead = comport.Read(tempBuff, 0, comport.BytesToRead);
//Any time we call comport.Read, we automatically log those bytes to a file
using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
writer.Write(tempBuff.Skip(0).Take(numBytesRead).ToArray());
//Add these bytes into the data array
Array.Copy(tempBuff, 0, dataBuff, dataInd+dataBytesRead, numBytesRead);
//Increment our data array counter
dataBytesRead += numBytesRead;
//Check to see if we have saved enough
if((dataInd+dataBytesRead) >= fullMsgLen)
{
//Copy the header+msg into the queue
serialQ.Enqueue(dataBuff.Skip(0).Take(fullMsgLen).ToArray());
//Copy the remaining bytes back into the header buffer
Array.Copy(dataBuff, fullMsgLen, headerBuff, 0, dataInd + dataBytesRead - fullMsgLen);
headerBytesRead = dataInd + dataBytesRead - fullMsgLen;
//Reset data bytes read countery
dataBytesRead = 0;
//Switch back to looking for headers
READ_HEADER = !READ_HEADER;
}
}
}

WaitFor() - How to wait for a specific buffer to arrive on Steam/SerialPort?

Requested Behaviour: I would like to hear proposed, generic solutions for suspending a calling thread until a specific buffer is received on a Stream/SerialPort. For the time being, I'm not concerned with timeouts etc, however I need something robust.
Attempted method:
Class myClass
{
private SerialPort _port; //Assume configured and connected.
public void WaitFor(byte[] buffer)
{
int bufferLength = buffer.Length;
byte[] comparisonBuffer = new byte[bufferLength];
while(true)
{
if(_port.BytesToRead >= bufferLength)
{
_port.Read(comparisonBuffer, 0, bufferLength);
if (comparisonBuffer.SequenceEqual(buffer)) { return; }
}
}
}
{
I've had a reasonable amount of success with this however it just has a "hacky" feel to it. It has quite often caused me trouble. I believe it's due to the fact that I cannot guarantee that other data isn't received either before or after the expected packet, so naturally this method can end up reading off the stream out of sync. In such a case I would not want to loose the leading/trailing data but the method should release the thread.
I need to implement in a procedural nature so event driven methods won't really work for me. In the generic sense I want to be able to implement as;
Do thing;
WaitFor(mybuffer);
Do other thing;

SerialPort.Read() already blocks until at least one byte has arrived. Therefore you don't need to (and shouldn't) use the BytesToRead the way you are - you've introduced a HORRIBLE busy-wait loop.
Instead, do something like this:
// Reads 'count' bytes from a serial port into the specified
// part of a buffer. This blocks until all the bytes have been read.
public void BlockingRead(SerialPort port, byte[] buffer, int offset, int count)
{
while (count > 0)
{
// SerialPort.Read() blocks until at least one byte has been read, or SerialPort.ReadTimeout milliseconds
// have elapsed. If a timeout occurs a TimeoutException will be thrown.
// Because SerialPort.Read() blocks until some data is available this is not a busy loop,
// and we do NOT need to issue any calls to Thread.Sleep().
int bytesRead = port.Read(buffer, offset, count);
offset += bytesRead;
count -= bytesRead;
}
}
Here's how you would implement your original code in terms of BlockingRead():
public void WaitFor(SerialPort port, byte[] buffer)
{
byte[] comparisonBuffer = new byte[buffer.Length];
while (true)
{
BlockingRead(port, comparisonBuffer, 0, comparisonBuffer.Length);
if (comparisonBuffer.SequenceEqual(buffer))
return;
}
}

Problem
Lets assume you wait for the byte pattern {1,1,1,2,2} and the serial port has buffered {1,1,1,1,2,2,5}.
Your code reads the first 5 bytes {1,1,1,1,2} which will not match the pattern. But after reading from the port the data you read has been removed from the buffer and contains only {2,5} and you will never get a match.
Solution
public void WaitFor( byte[ ] buffer )
{
if ( buffer.Length == 0 )
return;
var q = new List<byte>( buffer.Length );
while ( true )
{
var current = _reader.ReadByte();
q.Add( (byte)current );
// sequence match so far
if ( q.Last == buffer[ q.Count - 1 ] )
{
// check for total match
if ( q.Count == buffer.Length )
return;
}
else
{
// shift the data
while ( q.Any() && !q.SequenceEqual( buffer.Take( q.Count ) ) )
{
q.RemoveAt( 0 );
}
}
}
}

What do you think to this solution?
public override byte[] WaitFor(byte[] buffer, int timeout)
{
// List to stack stream into
List<byte> stack = new List<byte>();
// Index of first comparison byte
int index = 0;
// Index of last comparison byte
int upperBound = buffer.Length - 1;
// Timeout Manager
Stopwatch Sw = new Stopwatch();
Sw.Start();
while (Sw.Elapsed.Seconds <= timeout)
{
// Read off the last byte receievd and add to the stack
stack.Add((byte)_port.ReadByte());
// If my stack contains enough bytes to compare to the buffer
if (stack.Count > upperBound)
{
// If my first comparison byte matches my first buffer byte
if (stack[index] == buffer[0])
{
// Extract the comparison block to array
byte[] compBuffer = stack.GetRange(index,upperBound +1).ToArray();
// If the comparison matches, break and return the redundent bytes should I wish to handle them.
if ((compBuffer.SequenceEqual(buffer) && (index-1 > 0))) { return stack.GetRange(0, index - 1).ToArray(); }
// If there were no redundent bytes, just return zero.
else if (compBuffer.SequenceEqual(buffer)) { return new byte[] { 0}; }
}
// Increments
index += 1;
upperBound += 1;
}
}
throw new TimeoutException("Timeout: Expected buffer was not received prior to timeout");
}

C# NetworkStream data loss

I am currently working on a networking project where I worked out a binary protocol. My packets look like this:
[1 byte TYPE][2 bytes INDEX][2 bytes LENGTH][LENGTH bytes DATA]
And here's the code where I am receiving the packets:
NetworkStream clientStream= Client.GetStream();
while (Client.Connected)
{
Thread.Sleep(10);
try
{
if (clientStream.DataAvailable)
{
byte[] infobuffer = new byte[5];
int inforead = clientStream.Read(infobuffer, 0, 5);
if (inforead < 5) { continue; }
byte[] rawclient = new byte[2];
Array.Copy(infobuffer, 1, rawclient, 0, 2);
PacketType type = (PacketType)Convert.ToSByte(infobuffer[0]);
int clientIndex = BitConverter.ToInt16(rawclient, 0);
int readLength = BitConverter.ToInt16(infobuffer, 3);
byte[] readbuffer = new byte[readLength];
int count_read = clientStream.Read(readbuffer, 0, readLength);
byte[] read_data = new byte[count_read];
Array.Copy(readbuffer, read_data, count_read);
HandleData(read_data, type, clientIndex);
}
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("[E] " + ex.GetType().ToString());
Console.ResetColor();
break;
}
}
Well, and everything works fine... as long as I run it on 127.0.0.1. As soon as I try testing it over long distance, packets somehow get lost, and I am getting an overflow-exception on the line where I convert the first byte to PacketType. Also, if I try to convert the other values to int16, I get very strange values.
I assume the stream somehow looses some bytes on its way to the server, but can this be? Or is it just a little mistake of mine somewhere in the code?
edit:
I now edited the code, now it reads till it gets its 5 bytes. But I still get the same exception over long distance...
NetworkStream clientStream = Client.GetStream();
while (Client.Connected)
{
Thread.Sleep(10);
try
{
if (clientStream.DataAvailable)
{
int totalread = 0;
byte[] infobuffer = new byte[5];
while (totalread < 5)
{
int inforead = clientStream.Read(infobuffer, totalread, 5 - totalread);
if (inforead == 0)
{ break; }
totalread += inforead;
}
byte[] rawclient = new byte[2];
Array.Copy(infobuffer, 1, rawclient, 0, 2);
PacketType type = (PacketType)Convert.ToSByte(infobuffer[0]);
int clientIndex = BitConverter.ToInt16(rawclient, 0);
int readLength = BitConverter.ToInt16(infobuffer, 3);
byte[] readbuffer = new byte[readLength];
int count_read = clientStream.Read(readbuffer, 0, readLength);
byte[] read_data = new byte[count_read];
Array.Copy(readbuffer, read_data, count_read);
HandleData(read_data, type, clientIndex);
}
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("[E] " + ex.GetType().ToString());
Console.ResetColor();
break;
}
}
PacketType is an enum:
public enum PacketType
{
AddressSocks5 = 0,
Status = 1,
Data = 2,
Disconnect = 3,
AddressSocks4 = 4
}

So many things you're doing wrong here... so many bugs... where to even start...
First Network polling? Really? That's just a naïve way of doing network activity in this day and age.. but I won't harp on that.
Second, with this type of protocol, it's pretty easy to get "out of sync" and once you do, you have no way to get back in sync. This is typically accomplished with some kind of "framing protocol" which provides a unique sequence of bytes that you can use to indicate the start and end of a frame, so that if you ever find yourself out of sync you can read data until you get back in sync. Yes, you will lose data, but you've already lost it if you're out of sync.
Third, you're not really doing anything huge here, so I shamelessly stole the "ReadWholeArray" code from here, it's not the most efficient, but it works and there is other code there that might help:
http://www.yoda.arachsys.com/csharp/readbinary.html
Note: you don't mention how you are serializing the length, type and index values on the other side. So using the BitConverter may be the wrong thing depending on how that was done.
if (clientStream.DataAvailable)
{
byte[] data = new byte[5];
// if it can't read all 5 bytes, it throws an exception
ReadWholeArray(clientStream, data);
PacketType type = (PacketType)Convert.ToSByte(data[0]);
int clientIndex = BitConverter.ToInt16(data, 1);
int readLength = BitConverter.ToInt16(data, 3);
byte[] rawdata = new byte[readLength];
ReadWholeArray(clientStream, rawdata);
HandleData(rawdata, type, clientIndex);
}
/// <summary>
/// Reads data into a complete array, throwing an EndOfStreamException
/// if the stream runs out of data first, or if an IOException
/// naturally occurs.
/// </summary>
/// <param name="stream">The stream to read data from</param>
/// <param name="data">The array to read bytes into. The array
/// will be completely filled from the stream, so an appropriate
/// size must be given.</param>
public static void ReadWholeArray (Stream stream, byte[] data)
{
int offset=0;
int remaining = data.Length;
while (remaining > 0)
{
int read = stream.Read(data, offset, remaining);
if (read <= 0)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", remaining));
remaining -= read;
offset += read;
}
}

I think the problem is in these lines
int inforead = clientStream.Read(infobuffer, 0, 5);
if (inforead < 5) { continue; }
what happen to your previously read data if the length is under 5 byte? you should save the bytes you have read so far and append next bytes so you can have the header completely

You Read 5 - totalRead.
let totalRead equal 5 or more. When that happens you read nothing, and in cases of 1 - 4 you read that many arbitrary bytes. Not 5. You also then discard any result of less then 5.
You also copy at a offset 1 or another offset without really knowing the offset.
BitConverter.ToInt16(infobuffer, 3);
Is an example of this, what is at off 2?
So if it's not that (decoding error) and and not the structure of your data then unless you change the structure of your loop its you who's losing the bytes not the NetworkStream.
Calculate totalRead by increments of justRead when you recieve so you can handle any size of data as well as receiving it at the correct offset.

Using COM port in C# and not getting all sent data

Just to explain, I have a dev board that is, on command, shooting out 4 chars (Bytes). The terminal program (RealTerm) i'm using to debug this sees all 4 bytes. I've now moved on to writing the desktop software, and my program is only paying attention to 1 out of 4 bytes sent. TO clarify, i don't know which 1 of 4 bytes (first , last, middle two) but i can find out if its really necessary.
At first I thought the SerialPort.DataReceived event would fire for each byte that was received. This isn't true, I don't know what causes it to fire but its not the receiving of a singular Byte.
So I tried looping over SerialPort.BytesToRead, but this also only gets the first byte, even though it recognizes there are 3 bytes to read (why not 4??)
Its not essential for me to receive this data at the exact time it hits the port, but obviously I dot want to be loosing 3/4 of my data. However it wont always be 4 bytes, that's just what its doing now. I just want to get all bytes that are ready to be read.
Event Handler:
private void comPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
while (comPort.BytesToRead > 0)
{
RxString = comPort.ReadExisting();
RxByte = comPort.ReadByte();
byte[] myByte = new byte[6];
for (int i = 0; i < 6; i++)
{
myByte[i] = 0000000;
}
comPort.Read(myByte, 0, comPort.BytesToRead);
for (int i=0;i<6;i++)
{
if (myByte[i] != null)
{
thisBytes.Add(myByte[i]);
}
}
RxString = RxByte + "";
try
{
this.Invoke(new EventHandler(dealWithByte));
}
catch
{
}
}
}
private void dealWithByte(object sender, EventArgs e)
{
foreach (byte item in thisBytes)
{
RxByte = Convert.ToInt16(item);
string binary = Convert.ToString(RxByte, 2).PadLeft(8, '0');
//processTime(binary);
}
}

I am not a C# person but the code is pretty simple, pseudo code
numBytes As Int = SerialPort1.BytesToRead 'get # of bytes available
buf(numBytes - 1) As Byte 'allocate a buffer
br As Int = SerialPort1.Read(buf, 0, numBytes) 'read the bytes
If br <> numBytes {
Resize(buf, br) 'resize the buffer
}
at this point store the bytes into a list. This list can then be processed for messages.

TcpClient.GetStream().DataAvailable returns false, but stream has more data

So, it would seem that a blocking Read() can return before it is done receiving all of the data being sent to it. In turn we wrap the Read() with a loop that is controlled by the DataAvailable value from the stream in question. The problem is that you can receive more data while in this while loop, but there is no behind the scenes processing going on to let the system know this. Most of the solutions I have found to this on the net have not been applicable in one way or another to me.
What I have ended up doing is as the last step in my loop, I do a simple Thread.Sleep(1) after reading each block from the stream. This appears to give the system time to update and I am not getting accurate results but this seems a bit hacky and quite a bit 'circumstantial' for a solution.
Here is a list of the circumstances I am dealing with: Single TCP Connection between an IIS Application and a standalone application, both written in C# for send/receive communication. It sends a request and then waits for a response. This request is initiated by an HTTP request, but I am not having this issue reading data from the HTTP Request, it is after the fact.
Here is the basic code for handling an incoming connection
protected void OnClientCommunication(TcpClient oClient)
{
NetworkStream stream = oClient.GetStream();
MemoryStream msIn = new MemoryStream();
byte[] aMessage = new byte[4096];
int iBytesRead = 0;
while ( stream.DataAvailable )
{
int iRead = stream.Read(aMessage, 0, aMessage.Length);
iBytesRead += iRead;
msIn.Write(aMessage, 0, iRead);
Thread.Sleep(1);
}
MemoryStream msOut = new MemoryStream();
// .. Do some processing adding data to the msOut stream
msOut.WriteTo(stream);
stream.Flush();
oClient.Close();
}
All feedback welcome for a better solution or just a thumbs up on needing to give that Sleep(1) a go to allow things to update properly before we check the DataAvailable value.
Guess I am hoping after 2 years that the answer to this question isn't how things still are :)

You have to know how much data you need to read; you cannot simply loop reading data until there is no more data, because you can never be sure that no more is going to come.
This is why HTTP GET results have a byte count in the HTTP headers: so the client side will know when it has received all the data.
Here are two solutions for you depending on whether you have control over what the other side is sending:
Use "framing" characters: (SB)data(EB), where SB and EB are start-block and end-block characters (of your choosing) but which CANNOT occur inside the data. When you "see" EB, you know you are done.
Implement a length field in front of each message to indicate how much data follows: (len)data. Read (len), then read (len) bytes; repeat as necessary.
This isn't like reading from a file where a zero-length read means end-of-data (that DOES mean the other side has disconnected, but that's another story).
A third (not recommended) solution is that you can implement a timer. Once you start getting data, set the timer. If the receive loop is idle for some period of time (say a few seconds, if data doesn't come often), you can probably assume no more data is coming. This last method is a last resort... it's not very reliable, hard to tune, and it's fragile.

I'm seeing a problem with this.
You're expecting that the communication will be faster than the while() loop, which is very unlikely.
The while() loop will finish as soon as there is no more data, which may not be the case a few milliseconds just after it exits.
Are you expecting a certain amount of bytes?
How often is OnClientCommunication() fired? Who triggers it?
What do you do with the data after the while() loop? Do you keep appending to previous data?
DataAvailable WILL return false because you're reading faster than the communication, so that's fine only if you keep coming back to this code block to process more data coming in.

I was trying to check DataAvailable before reading data from a network stream and it would return false, although after reading a single byte it would return true. So I checked the MSDN documentation and they also read before checking. I would re-arrange the while loop to a do while loop to follow this pattern.
http://msdn.microsoft.com/en-us/library/system.net.sockets.networkstream.dataavailable.aspx
// Check to see if this NetworkStream is readable.
if(myNetworkStream.CanRead){
byte[] myReadBuffer = new byte[1024];
StringBuilder myCompleteMessage = new StringBuilder();
int numberOfBytesRead = 0;
// Incoming message may be larger than the buffer size.
do{
numberOfBytesRead = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
myCompleteMessage.AppendFormat("{0}", Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));
}
while(myNetworkStream.DataAvailable);
// Print out the received message to the console.
Console.WriteLine("You received the following message : " +
myCompleteMessage);
}
else{
Console.WriteLine("Sorry. You cannot read from this NetworkStream.");
}

When I have this code:
var readBuffer = new byte[1024];
using (var memoryStream = new MemoryStream())
{
do
{
int numberOfBytesRead = networkStream.Read(readBuffer, 0, readBuffer.Length);
memoryStream.Write(readBuffer, 0, numberOfBytesRead);
}
while (networkStream.DataAvailable);
}
From what I can observe:
When sender sends 1000 bytes and reader wants to read them. Then I suspect that NetworkStream somehow "knows" that it should receive 1000 bytes.
When I call .Read before any data arrives from NetworkStream then .Read should be blocking until it gets more than 0 bytes (or more if .NoDelay is false on networkStream)
Then when I read first batch of data I suspect that .Read is somehow updating from its result the counter of those 1000 bytes at NetworkStream and before this happens I suspect, that in this time the .DataAvailable is set to false and after the counter is updated then the .DataAvailable is then set to correct value if the counter data is less than 1000 bytes. It makes sense when you think about it. Because otherwise it would go to the next cycle before checking that 1000 bytes arrived and the .Read method would be blocking indefinitely, because reader could have already read 1000 bytes and no more data would arrive.
This I think is the point of failure here as already James said:
Yes, this is just the way these libraries work. They need to be given time to run to fully validate the data incoming. – James Apr 20 '16 at 5:24
I suspect that the update of internal counter between end of .Read and before accessing .DataAvailable is not as atomic operation (transaction) so the TcpClient needs more time to properly set the DataAvailable.
When I have this code:
var readBuffer = new byte[1024];
using (var memoryStream = new MemoryStream())
{
do
{
int numberOfBytesRead = networkStream.Read(readBuffer, 0, readBuffer.Length);
memoryStream.Write(readBuffer, 0, numberOfBytesRead);
if (!networkStream.DataAvailable)
System.Threading.Thread.Sleep(1); //Or 50 for non-believers ;)
}
while (networkStream.DataAvailable);
}
Then the NetworkStream have enough time to properly set .DataAvailable and this method should function correctly.
Fun fact... This seems to be somehow OS Version dependent. Because the first function without sleep worked for me on Win XP and Win 10, but was failing to receive whole 1000 bytes on Win 7. Don't ask me why, but I tested it quite thoroughly and it was easily reproducible.

Using TcpClient.Available will allow this code to read exactly what is available each time. TcpClient.Available is automatically set to TcpClient.ReceiveBufferSize when the amount of data remaining to be read is greater than or equal to TcpClient.ReceiveBufferSize. Otherwise it is set to the size of the remaining data.
Hence, you can indicate the maximum amount of data that is available for each read by setting TcpClient.ReceiveBufferSize (e.g., oClient.ReceiveBufferSize = 4096;).
protected void OnClientCommunication(TcpClient oClient)
{
NetworkStream stream = oClient.GetStream();
MemoryStream msIn = new MemoryStream();
byte[] aMessage;
oClient.ReceiveBufferSize = 4096;
int iBytesRead = 0;
while (stream.DataAvailable)
{
int myBufferSize = (oClient.Available < 1) ? 1 : oClient.Available;
aMessage = new byte[oClient.Available];
int iRead = stream.Read(aMessage, 0, aMessage.Length);
iBytesRead += iRead;
msIn.Write(aMessage, 0, iRead);
}
MemoryStream msOut = new MemoryStream();
// .. Do some processing adding data to the msOut stream
msOut.WriteTo(stream);
stream.Flush();
oClient.Close();
}

public class NetworkStream
{
private readonly Socket m_Socket;
public NetworkStream(Socket socket)
{
m_Socket = socket ?? throw new ArgumentNullException(nameof(socket));
}
public void Send(string message)
{
if (message is null)
{
throw new ArgumentNullException(nameof(message));
}
byte[] data = Encoding.UTF8.GetBytes(message);
SendInternal(data);
}
public string Receive()
{
byte[] buffer = ReceiveInternal();
string message = Encoding.UTF8.GetString(buffer);
return message;
}
private void SendInternal(byte[] message)
{
int size = message.Length;
if (size == 0)
{
m_Socket.Send(BitConverter.GetBytes(size), 0, sizeof(int), SocketFlags.None);
}
else
{
m_Socket.Send(BitConverter.GetBytes(size), 0, sizeof(int), SocketFlags.None);
m_Socket.Send(message, 0, size, SocketFlags.None);
}
}
private byte[] ReceiveInternal()
{
byte[] sizeData = CommonReceiveMessage(sizeof(int));
int size = BitConverter.ToInt32(sizeData);
if (size == 0)
{
return Array.Empty<byte>();
}
return CommonReceiveMessage(size);
}
private byte[] CommonReceiveMessage(int messageLength)
{
if (messageLength < 0)
{
throw new ArgumentOutOfRangeException(nameof(messageLength), messageLength, "Размер сообщения не может быть меньше нуля.");
}
if (messageLength == 0)
{
return Array.Empty<byte>();
}
byte[] buffer = new byte[m_Socket.ReceiveBufferSize];
int currentLength = 0;
int receivedDataLength;
using (MemoryStream memoryStream = new())
{
do
{
receivedDataLength = m_Socket.Receive(buffer, 0, m_Socket.ReceiveBufferSize, SocketFlags.None);
currentLength += receivedDataLength;
memoryStream.Write(buffer, 0, receivedDataLength);
}
while (currentLength < messageLength);
return memoryStream.ToArray();
}
}
}

This example presents an algorithm for sending and receiving data, namely text messages. You can also send files.
using System;
using System.IO;
using System.Net.Sockets;
using System.Text;
namespace Network
{
/// <summary>
/// Represents a network stream for transferring data.
/// </summary>
public class NetworkStream
{
#region Fields
private static readonly byte[] EmptyArray = Array.Empty<byte>();
private readonly Socket m_Socket;
#endregion
#region Constructors
/// <summary>
/// Initializes a new instance of the class <seealso cref="NetworkStream"/>.
/// </summary>
/// <param name="socket">
/// Berkeley socket interface.
/// </param>
public NetworkStream(Socket socket)
{
m_Socket = socket ?? throw new ArgumentNullException(nameof(socket));
}
#endregion
#region Properties
#endregion
#region Methods
/// <summary>
/// Sends a message.
/// </summary>
/// <param name="message">
/// Message text.
/// </param>
/// <exception cref="ArgumentNullException"/>
public void Send(string message)
{
if (message is null)
{
throw new ArgumentNullException(nameof(message));
}
byte[] data = Encoding.UTF8.GetBytes(message);
Write(data);
}
/// <summary>
/// Receives the sent message.
/// </summary>
/// <returns>
/// Sent message.
/// </returns>
public string Receive()
{
byte[] data = Read();
return Encoding.UTF8.GetString(data);
}
/// <summary>
/// Receives the specified number of bytes from a bound <seealso cref="Socket"/>.
/// </summary>
/// <param name="socket">
/// <seealso cref="Socket"/> for receiving data.
/// </param>
/// <param name="size">
/// The size of the received data.
/// </param>
/// <returns>
/// Returns an array of received data.
/// </returns>
private byte[] Read(int size)
{
if (size < 0)
{
// You can throw an exception.
return null;
}
if (size == 0)
{
// Don't throw an exception here, just return an empty data array.
return EmptyArray;
}
// There are many examples on the Internet where the
// Socket.Available property is used, this is WRONG!
// Important! The Socket.Available property is not working as expected.
// Data packages may be in transit, but the Socket.Available property may indicate otherwise.
// Therefore, we use a counter that will allow us to receive all data packets, no more and no less.
// The cycle will continue until we receive all the data packets or the timeout is triggered.
// Note. This algorithm is not designed to work with big data.
SimpleCounter counter = new(size, m_Socket.ReceiveBufferSize);
byte[] buffer = new byte[counter.BufferSize];
int received;
using MemoryStream storage = new();
// The cycle will run until we get all the data.
while (counter.IsExpected)
{
received = m_Socket.Receive(buffer, 0, counter.Available, SocketFlags.None);
// Pass the size of the received data to the counter.
counter.Count(received);
// Write data to memory.
storage.Write(buffer, 0, received);
}
return storage.ToArray();
}
/// <summary>
/// Receives the specified number of bytes from a bound <seealso cref="Socket"/>.
/// </summary>
/// <returns>
/// Returns an array of received data.
/// </returns>
private byte[] Read()
{
byte[] sizeData;
// First, we get the size of the master data.
sizeData = Read(sizeof(int));
// We convert the received data into a number.
int size = BitConverter.ToInt32(sizeData);
// If the data size is less than 0 then throws an exception.
// We inform the recipient that an error occurred while reading the data.
if (size < 0)
{
// Or return the value null.
throw new SocketException();
}
// If the data size is 0, then we will return an empty array.
// Do not allow an exception here.
if (size == 0)
{
return EmptyArray;
}
// Here we read the master data.
byte[] data = Read(size);
return data;
}
/// <summary>
/// Writes data to the stream.
/// </summary>
/// <param name="data"></param>
private void Write(byte[] data)
{
if (data is null)
{
// Throw an exception.
// Or send a negative number that will represent the value null.
throw new ArgumentNullException(nameof(data));
}
byte[] sizeData = BitConverter.GetBytes(data.Length);
// In any case, we inform the recipient about the size of the data.
m_Socket.Send(sizeData, 0, sizeof(int), SocketFlags.None);
if (data.Length != 0)
{
// We send data whose size is greater than zero.
m_Socket.Send(data, 0, data.Length, SocketFlags.None);
}
}
#endregion
#region Classes
/// <summary>
/// Represents a simple counter of received data over the network.
/// </summary>
private class SimpleCounter
{
#region Fields
private int m_Received;
private int m_Available;
private bool m_IsExpected;
#endregion
#region Constructors
/// <summary>
/// Initializes a new instance of the class <seealso cref="SimpleCounter"/>.
/// </summary>
/// <param name="dataSize">
/// Data size.
/// </param>
/// <param name="bufferSize">
/// Buffer size.
/// </param>
/// <exception cref="ArgumentOutOfRangeException"/>
public SimpleCounter(int dataSize, int bufferSize)
{
if (dataSize < 0)
{
throw new ArgumentOutOfRangeException(nameof(dataSize), dataSize, "Data size cannot be less than 0");
}
if (bufferSize < 0)
{
throw new ArgumentOutOfRangeException(nameof(dataSize), bufferSize, "Buffer size cannot be less than 0");
}
DataSize = dataSize;
BufferSize = bufferSize;
// Update the counter data.
UpdateCounter();
}
#endregion
#region Properties
/// <summary>
/// Returns the size of the expected data.
/// </summary>
/// <value>
/// Size of expected data.
/// </value>
public int DataSize { get; }
/// <summary>
/// Returns the size of the buffer.
/// </summary>
/// <value>
/// Buffer size.
/// </value>
public int BufferSize { get; }
/// <summary>
/// Returns the available buffer size for receiving data.
/// </summary>
/// <value>
/// Available buffer size.
/// </value>
public int Available
{
get
{
return m_Available;
}
}
/// <summary>
/// Returns a value indicating whether the thread should wait for data.
/// </summary>
/// <value>
/// <see langword="true"/> if the stream is waiting for data; otherwise, <see langword="false"/>.
/// </value>
public bool IsExpected
{
get
{
return m_IsExpected;
}
}
#endregion
#region Methods
// Updates the counter.
private void UpdateCounter()
{
int unreadDataSize = DataSize - m_Received;
m_Available = unreadDataSize < BufferSize ? unreadDataSize : BufferSize;
m_IsExpected = m_Available > 0;
}
/// <summary>
/// Specifies the size of the received data.
/// </summary>
/// <param name="bytes">
/// The size of the received data.
/// </param>
public void Count(int bytes)
{
// NOTE: Counter cannot decrease.
if (bytes > 0)
{
int received = m_Received += bytes;
// NOTE: The value of the received data cannot exceed the size of the expected data.
m_Received = (received < DataSize) ? received : DataSize;
// Update the counter data.
UpdateCounter();
}
}
/// <summary>
/// Resets counter data.
/// </summary>
public void Reset()
{
m_Received = 0;
UpdateCounter();
}
#endregion
}
#endregion
}
}

Use a do-while loop. This will make sure the memory stream pointers have moved. The first Read or ReadAsync will cause the memorystream pointer to move and then onwards the ".DataAvailable" property will continue to return true until we hit the end of the stream.
An example from microsoft docs:
// Check to see if this NetworkStream is readable.
if(myNetworkStream.CanRead){
byte[] myReadBuffer = new byte[1024];
StringBuilder myCompleteMessage = new StringBuilder();
int numberOfBytesRead = 0;
// Incoming message may be larger than the buffer size.
do{
numberOfBytesRead = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
myCompleteMessage.AppendFormat("{0}", Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));
}
while(myNetworkStream.DataAvailable);
// Print out the received message to the console.
Console.WriteLine("You received the following message : " +
myCompleteMessage);
}
else{
Console.WriteLine("Sorry. You cannot read from this NetworkStream.");
}
Original Micorosoft Doc