I am currently working on a networking project where I worked out a binary protocol. My packets look like this:
[1 byte TYPE][2 bytes INDEX][2 bytes LENGTH][LENGTH bytes DATA]
And here's the code where I am receiving the packets:
NetworkStream clientStream= Client.GetStream();
while (Client.Connected)
{
Thread.Sleep(10);
try
{
if (clientStream.DataAvailable)
{
byte[] infobuffer = new byte[5];
int inforead = clientStream.Read(infobuffer, 0, 5);
if (inforead < 5) { continue; }
byte[] rawclient = new byte[2];
Array.Copy(infobuffer, 1, rawclient, 0, 2);
PacketType type = (PacketType)Convert.ToSByte(infobuffer[0]);
int clientIndex = BitConverter.ToInt16(rawclient, 0);
int readLength = BitConverter.ToInt16(infobuffer, 3);
byte[] readbuffer = new byte[readLength];
int count_read = clientStream.Read(readbuffer, 0, readLength);
byte[] read_data = new byte[count_read];
Array.Copy(readbuffer, read_data, count_read);
HandleData(read_data, type, clientIndex);
}
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("[E] " + ex.GetType().ToString());
Console.ResetColor();
break;
}
}
Well, and everything works fine... as long as I run it on 127.0.0.1. As soon as I try testing it over long distance, packets somehow get lost, and I am getting an overflow-exception on the line where I convert the first byte to PacketType. Also, if I try to convert the other values to int16, I get very strange values.
I assume the stream somehow looses some bytes on its way to the server, but can this be? Or is it just a little mistake of mine somewhere in the code?
edit:
I now edited the code, now it reads till it gets its 5 bytes. But I still get the same exception over long distance...
NetworkStream clientStream = Client.GetStream();
while (Client.Connected)
{
Thread.Sleep(10);
try
{
if (clientStream.DataAvailable)
{
int totalread = 0;
byte[] infobuffer = new byte[5];
while (totalread < 5)
{
int inforead = clientStream.Read(infobuffer, totalread, 5 - totalread);
if (inforead == 0)
{ break; }
totalread += inforead;
}
byte[] rawclient = new byte[2];
Array.Copy(infobuffer, 1, rawclient, 0, 2);
PacketType type = (PacketType)Convert.ToSByte(infobuffer[0]);
int clientIndex = BitConverter.ToInt16(rawclient, 0);
int readLength = BitConverter.ToInt16(infobuffer, 3);
byte[] readbuffer = new byte[readLength];
int count_read = clientStream.Read(readbuffer, 0, readLength);
byte[] read_data = new byte[count_read];
Array.Copy(readbuffer, read_data, count_read);
HandleData(read_data, type, clientIndex);
}
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("[E] " + ex.GetType().ToString());
Console.ResetColor();
break;
}
}
PacketType is an enum:
public enum PacketType
{
AddressSocks5 = 0,
Status = 1,
Data = 2,
Disconnect = 3,
AddressSocks4 = 4
}
So many things you're doing wrong here... so many bugs... where to even start...
First Network polling? Really? That's just a naïve way of doing network activity in this day and age.. but I won't harp on that.
Second, with this type of protocol, it's pretty easy to get "out of sync" and once you do, you have no way to get back in sync. This is typically accomplished with some kind of "framing protocol" which provides a unique sequence of bytes that you can use to indicate the start and end of a frame, so that if you ever find yourself out of sync you can read data until you get back in sync. Yes, you will lose data, but you've already lost it if you're out of sync.
Third, you're not really doing anything huge here, so I shamelessly stole the "ReadWholeArray" code from here, it's not the most efficient, but it works and there is other code there that might help:
http://www.yoda.arachsys.com/csharp/readbinary.html
Note: you don't mention how you are serializing the length, type and index values on the other side. So using the BitConverter may be the wrong thing depending on how that was done.
if (clientStream.DataAvailable)
{
byte[] data = new byte[5];
// if it can't read all 5 bytes, it throws an exception
ReadWholeArray(clientStream, data);
PacketType type = (PacketType)Convert.ToSByte(data[0]);
int clientIndex = BitConverter.ToInt16(data, 1);
int readLength = BitConverter.ToInt16(data, 3);
byte[] rawdata = new byte[readLength];
ReadWholeArray(clientStream, rawdata);
HandleData(rawdata, type, clientIndex);
}
/// <summary>
/// Reads data into a complete array, throwing an EndOfStreamException
/// if the stream runs out of data first, or if an IOException
/// naturally occurs.
/// </summary>
/// <param name="stream">The stream to read data from</param>
/// <param name="data">The array to read bytes into. The array
/// will be completely filled from the stream, so an appropriate
/// size must be given.</param>
public static void ReadWholeArray (Stream stream, byte[] data)
{
int offset=0;
int remaining = data.Length;
while (remaining > 0)
{
int read = stream.Read(data, offset, remaining);
if (read <= 0)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", remaining));
remaining -= read;
offset += read;
}
}
I think the problem is in these lines
int inforead = clientStream.Read(infobuffer, 0, 5);
if (inforead < 5) { continue; }
what happen to your previously read data if the length is under 5 byte? you should save the bytes you have read so far and append next bytes so you can have the header completely
You Read 5 - totalRead.
let totalRead equal 5 or more. When that happens you read nothing, and in cases of 1 - 4 you read that many arbitrary bytes. Not 5. You also then discard any result of less then 5.
You also copy at a offset 1 or another offset without really knowing the offset.
BitConverter.ToInt16(infobuffer, 3);
Is an example of this, what is at off 2?
So if it's not that (decoding error) and and not the structure of your data then unless you change the structure of your loop its you who's losing the bytes not the NetworkStream.
Calculate totalRead by increments of justRead when you recieve so you can handle any size of data as well as receiving it at the correct offset.
Related
Sorry for such a vague title, I really dont know what to title this issue.
Basically when I get a stream thats chunked as told by Transfer-Encoding, I then do the following code:
private IEnumerable<byte[]> ReceiveMessageBodyChunked() {
readChunk:
#region Read a line from the Stream which should be a Block Length (Chunk Body Length)
string blockLength = _receiverHelper.ReadLine();
#endregion
#region If the end of the block is reached, re-read from the stream
if (blockLength == Http.NewLine) {
goto readChunk;
}
#endregion
#region Trim it so it should end up with JUST the number
blockLength = blockLength.Trim(' ', '\r', '\n');
#endregion
#region If the end of the message body is reached
if (blockLength == string.Empty) {
yield break;
}
#endregion
int blockLengthInt = 0;
#region Convert the Block Length String to an Int32 base16 (hex)
try {
blockLengthInt = Convert.ToInt32(blockLength, 16);
} catch (Exception ex) {
if (ex is FormatException || ex is OverflowException) {
throw new Exception(string.Format(ExceptionValues.HttpException_WrongChunkedBlockLength, blockLength), ex);
}
throw;
}
#endregion
// If the end of the message body is reached.
if (blockLengthInt == 0) {
yield break;
}
byte[] buffer = new byte[blockLengthInt];
int totalBytesRead = 0;
while (totalBytesRead != blockLengthInt) {
int length = blockLengthInt - totalBytesRead;
int bytesRead = _receiverHelper.HasData ? _receiverHelper.Read(buffer, 0, length) : _request.ClientStream.Read(buffer, 0, length);
if (bytesRead == 0) {
WaitData();
continue;
}
totalBytesRead += bytesRead;
System.Windows.Forms.MessageBox.Show("Chunk Length: " + blockLengthInt + "\nBytes Read/Total:" + bytesRead + "/" + totalBytesRead + "\n\n" + Encoding.ASCII.GetString(buffer));
yield return buffer;
}
goto readChunk;
}
What this is doing is reading 1 line of data from the stream which should be the Chunk's Length, does some checks here and there but eventually converts that to a Int32 Radix16 integer.
From there it essentially creates a byte buffer of that int32 as its length size.
It then just keeps reading from the stream until its read the same amount as the Int32 we converted.
This works splendid, however, for whatever reason, its responding incorrectly on the last read.
It will read the exact amount of bytes as the chunk length perfectly fine, and all data I expect is read. BUT it's ALSO reading again another small chunk of data that was ALREADY read at the very end, resulting in lets say all data from <!DOCTYPE html> down to </html> ASWELL as some data from inside somewhere like <form> e.t.c
Here's an example of what occured:
As you can see, the highlighted red text should NOT have been returned from the read! It should have ended at </html>.
Why is the Chunk's Length lying to me and how can I find the proper size to read at?
I'm not familiar with C# but if I understand your code and the semantics of Read in C# correctly (which seem to be similar to read in C) then the problem is that you are using the same buffer again and again without resetting it first:
byte[] buffer = new byte[blockLengthInt];
int totalBytesRead = 0;
while (totalBytesRead != blockLengthInt) {
int length = blockLengthInt - totalBytesRead;
int bytesRead = _receiverHelper.HasData ? _receiverHelper.Read(buffer, 0, length) : _request.ClientStream.Read(buffer, 0, length);
...
totalBytesRead += bytesRead;
...
yield return buffer;
}
To make some example of what goes wrong here: assume that the chunk size is 10, the content you read is 0123456789 and the first read will return 6 bytes and the second read the remaining 4 bytes. In this case your buffer will be 012345 after the first read and 567845 after the second read. These 45 at the end of the buffer remain from the previous read since you only replaced the first 4 bytes in the buffer but kept the rest.
Whats odd AF is that if I hand the request to another TCPStream proxied (127.0.0.1:8888 as a proxy which is fiddler) it works perfectly fine...
Fiddler is a proxy and might change how the response gets transferred. For example it might use Content-length instead of chunked encoding or it might use smaller chunks so that you always get the full chunk with the first read.
I have spent a lot of team researching the proper ways to use the serial port in C# such that you don't have problems reading in data. I think I have a solution which is pretty close to functional, but I have some glitches every once in a while which I cannot seem to figure out.
My goal: Read formatted binary messages from the serial port, and pass them along to a processor.
The message format looks something like this:
(MSG-HEADER)(MSG-ID)(MSG-LENGTH)(DATA0)(DATA1)(DATA2)...(DATA-N)
Each "word" in the data is 16 bits (2-bytes). My basic approach is to start in a "read message header" state, where each time the serial data received event occurs, I read from the serial port, store the data in a buffer, and then check to see if I detect the message header. If I detect the message header, I move into a "read data" state, where I keep reading data into a data buffer until I have read bytes.
This seems to work pretty well, except occasionally I see "data glitches". Where I end up storing a message that looks something like this:
(MSG1-HEADER)(MSG1-ID)(MSG1-LENGTH)(DATA0)(DATA1)(DATA2)(MSG2-HEADER)(MSG2-ID)..etc
Basically, every so often I get a proper message header, message ID, message length, then the data starts (typically around 200 bytes), and right in the middle of that data I see another message header, message id, and message length, and presumably the start of another message data section. And I can't seem to figure out why.
Here is the code in the serial port data received event I am using:
public byte[] headerBuff = new byte[500];
public byte[] dataBuff = new byte[500];
public byte[] tempBuff = new byte[500];
public int bytesRead;
public int dataPos;
public int dataMsgLen;
public int dataBytesRead = 0;
public bool READ_HEADER = true;
ConcurrentQueue<byte[]> serialQ = new ConcurrentQueue<byte[]>();
//private void si_DataReceived(byte[] data)
private void si_DataReceived(object s, EventArgs e)
{
//If we're supposed to be reading the header, read some bytes and look
// For the header identification sequence (0xF989)
if (READ_HEADER)
{
//Read some bytes, save how many we read
bytesRead = comport.Read(headerBuff, 0, comport.BytesToRead);
//Any time we call comport.REad, we automatically log those bytes to a file
using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
writer.Write(headerBuff.Skip(0).Take(bytesRead).ToArray());
//Loop through the bytes we just read and look for sequence
for (int i = 0; i < (bytesRead-1); i++)
{
if (headerBuff[i] == 0xF9 && headerBuff[i + 1] == 0x89)
{
//We have identified a header
// Lets copy it into a new array
dataPos = bytesRead-i;
Array.Copy(headerBuff, i, dataBuff, 0, dataPos);
dataMsgLen = dataBuff[4];
//Now we can switch to message logging
READ_HEADER = !READ_HEADER;
Array.Clear(headerBuff, 0, headerBuff.Length); //clear the buffer for next time
break; // don't need to look for headers anymore
}
}
}
//If we are done reading the header, let's wait until we get
// enough bytes to store the data message
else if (!READ_HEADER)
{
// Read some bytes into temp array
var tempNumBytes = comport.Read(tempBuff, 0, comport.BytesToRead);
//ADD this into data buffer
Array.Copy(tempBuff, 0, dataBuff, dataPos + dataBytesRead, tempNumBytes);
//Increment our counter
dataBytesRead += tempNumBytes;
//Save to stream
using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
writer.Write(tempBuff.Skip(0).Take(tempNumBytes).ToArray());
//Add to FIFO if we have read enough bytes
if (dataBytesRead >= (dataMsgLen * 2))
{
//Debug.Print(BitConverter.ToString(dataBuff));
serialQ.Enqueue(dataBuff.Select(x => x).ToArray()); // Add to queue for processing
READ_HEADER = !READ_HEADER; // Go back to looking for headers
dataBytesRead = 0;
}
}
}
I appreciate any help, let me know if you need any clarifications.
Thank you in advance.
All,
Thank you for your comments. Based on what I read, I re-wrote the serial data handler (see code below) and it seems to be working much better. I have had it running for about ten minutes now and I haven't seen this glitch at all.
//Declare some public variables for serial port reading
public byte[] headerBuff = new byte[500];
public byte[] dataBuff = new byte[500];
public byte[] tempBuff = new byte[500];
public int headerBytesRead = 0;
public int dataBytesRead = 0;
public const int HEADER_LENGTH = 10;
public int dataInd;
public int fullMsgLen;
public byte[] queuePop;
//Declare some states
public bool READ_HEADER = true;
//Where will we store the data log?
public string defDataDir;
//Declare a public queue as a FIFO for incoming serial data once the
// buffer is full
ConcurrentQueue<byte[]> serialQ = new ConcurrentQueue<byte[]>();
//private void si_DataReceived(byte[] data)
private void si_DataReceived(object s, EventArgs e)
{
//If we're supposed to read the headers, do that
if(READ_HEADER)
{
//Read some bytes
var numBytesRead = comport.Read(tempBuff, 0, comport.BytesToRead);
//Any time we call comport.Read, we automatically log those bytes to a file
using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
writer.Write(tempBuff.Skip(0).Take(numBytesRead).ToArray());
//Add these bytes to a header array
Array.Copy(tempBuff, 0, headerBuff, headerBytesRead, numBytesRead);
//Increment headerBytesRead counter
headerBytesRead += numBytesRead;
//Loop through header and see if we have a header
if(headerBytesRead>=HEADER_LENGTH)
{
//Loop through all the header bytes read so far
for(int i=0; i<headerBytesRead;i++)
{
//Look for the header start word. Note, 3rd bool statement
// here is to make sure we have enough bytes left to identify a header
// e.g. read 12 bytes, and bytes 11 and 12 are 0xF9 and 0x89, then we
// clearly don't have the rest of the header (since it is length 10)
if(headerBuff[i]==0xF9 && headerBuff[i+1]==0x89 && (headerBytesRead-i-1)>=9)
{
//We have identified a header, and have enough following characters to save it
//Copy the header into the data array
Array.Copy(headerBuff, i, dataBuff, 0, headerBytesRead - i);
dataInd = headerBytesRead - i;
//Save the message length
fullMsgLen = dataBuff[4]*2 + HEADER_LENGTH;
//Switch over to reading data
READ_HEADER = !READ_HEADER;
//Reset our header length counter
headerBytesRead = 0;
//Clear the header buffer for next time
Array.Clear(headerBuff, 0, headerBuff.Length);
break; // don't need to look for headers anymore
}
}
}
}
//Handle reading data into buffer here
else if (!READ_HEADER)
{
//We've just been told to start reading data bytes, and we know how many
var numBytesRead = comport.Read(tempBuff, 0, comport.BytesToRead);
//Any time we call comport.Read, we automatically log those bytes to a file
using (BinaryWriter writer = new BinaryWriter(File.Open(defDataDir, FileMode.Append)))
writer.Write(tempBuff.Skip(0).Take(numBytesRead).ToArray());
//Add these bytes into the data array
Array.Copy(tempBuff, 0, dataBuff, dataInd+dataBytesRead, numBytesRead);
//Increment our data array counter
dataBytesRead += numBytesRead;
//Check to see if we have saved enough
if((dataInd+dataBytesRead) >= fullMsgLen)
{
//Copy the header+msg into the queue
serialQ.Enqueue(dataBuff.Skip(0).Take(fullMsgLen).ToArray());
//Copy the remaining bytes back into the header buffer
Array.Copy(dataBuff, fullMsgLen, headerBuff, 0, dataInd + dataBytesRead - fullMsgLen);
headerBytesRead = dataInd + dataBytesRead - fullMsgLen;
//Reset data bytes read countery
dataBytesRead = 0;
//Switch back to looking for headers
READ_HEADER = !READ_HEADER;
}
}
}
TCP is stream-based protocol. To convert that stream into my messages, I send the size of each message with the message itself. At server side, I first read the first two bytes of message, which have the size. Then I create a byte array, of size equal to the size which was just read. Then I read the bytes into that array. But for some reason, more bytes are being read than specified. How can I read exactly the same number of bytes as I specify?
Here is my code:
while (true)
{
data = null;
length = null;
size = new byte[2];
handler.Receive(size);
length += Encoding.ASCII.GetString(size, 0, 2);
System.Console.WriteLine("Size: " + Int32.Parse(length));
bufferSize = Int32.Parse(length) + 2;
bytes = new byte[bufferSize];
handler.Receive(bytes);
data += Encoding.ASCII.GetString(bytes, 0, bufferSize);
System.Console.WriteLine("Data: " + data);
}
This is my server running in Windows PC, written in C#. My client is running in android phone, written in Java.
It's unclear why you're adding two to the size that's been transmitted - you've already accounted for the two additional bytes for storing the length during your previous receive. So I'd get rid of the +2.
You also need to respect the fact already stated in your question - TCP is a sequence of bytes, not messages. As such, you're never guaranteed whether a call to Receive is going to retrieve an entire "message" or just part of one (or, possible, parts of multiple messages). As such, you need to make sure that you respect the return value from Receive.
We can probably re-write your code as:
while (true)
{
data = null;
length = null;
size = ReceiveExactly(handler,2);
length = Encoding.ASCII.GetString(size, 0, 2); //Why +=?
bufferSize = Int32.Parse(length); //Why + 2?
System.Console.WriteLine("Size: " + bufferSize);
bytes = ReceiveExactly(handler,bufferSize);
data += Encoding.ASCII.GetString(bytes, 0, bufferSize);
System.Console.WriteLine("Data: " + data);
}
Where ReceiveExactly is defined something like this:
private byte[] ReceiveExactly(Socket handler, int length)
{
var buffer = new byte[length];
var receivedLength = 0;
while(receivedLength < length)
{
var nextLength = handler.Receive(buffer,receivedLength,length-receivedLength);
if(nextLength==0)
{
//Throw an exception? Something else?
//The socket's never going to receive more data
}
receivedLength += nextLength;
}
return buffer;
}
to receive a specific amount of bytes use the method
Socket.Receive(Byte[], Int32, Int32, SocketFlags)
rather than Socket.Receive(Byte[]). see spec here
I suspect you want something like
int len = Socket.Receive(bytes, 0, bufferSize, SocketFlags.None);
data += Encoding.ASCII.GetString(bytes, 0, len);
System.Console.WriteLine("Data: " + data);
This is C# related. We have a case where we need to copy the entire source stream into a destination stream except for the last 16 bytes.
EDIT: The streams can range upto 40GB, so can't do some static byte[] allocation (eg: .ToArray())
Looking at the MSDN documentation, it seems that we can reliably determine the end of stream only when the return value is 0. Return values between 0 and the requested size can imply bytes are "not currently available" (what does that really mean?)
Currently it copies every single byte as follows. inStream and outStream are generic - can be memory, disk or network streams (actually some more too).
public static void StreamCopy(Stream inStream, Stream outStream)
{
var buffer = new byte[8*1024];
var last16Bytes = new byte[16];
int bytesRead;
while ((bytesRead = inStream.Read(buffer, 0, buffer.Length)) > 0)
{
outStream.Write(buffer, 0, bytesRead);
}
// Issues:
// 1. We already wrote the last 16 bytes into
// outStream (possibly over the n/w)
// 2. last16Bytes = ? (inStream may not necessarily support rewinding)
}
What is a reliable way to ensure all but the last 16 are copied? I can think of using Position and Length on the inStream but there is a gotcha on MSDN that says
If a class derived from Stream does not support seeking, calls to Length, SetLength, Position, and Seek throw a NotSupportedException. .
Read between 1 and n bytes from the input stream.1
Append the bytes to a circular buffer.2
Write the first max(0, b - 16) bytes from the circular buffer to the output stream, where b is the number of bytes in the circular buffer.
Remove the bytes that you just have written from the circular buffer.
Go to step 1.
1This is what the Read method does – if you call int n = Read(buffer, 0, 500); it will read between 1 and 500 bytes into buffer and return the number of bytes read. If Read returns 0, you have reached the end of the stream.
2For maximum performance, you can read the bytes directly from the input stream into the circular buffer. This is a bit tricky, because you have to deal with the wraparound within the array underlying the buffer.
The following solution is fast and tested. Hope it's useful. It uses the double buffering idea you already had in mind. EDIT: simplified loop removing the conditional that separated the first iteration from the rest.
public static void StreamCopy(Stream inStream, Stream outStream) {
// Define the size of the chunk to copy during each iteration (1 KiB)
const int blockSize = 1024;
const int bytesToOmit = 16;
const int buffSize = blockSize + bytesToOmit;
// Generate working buffers
byte[] buffer1 = new byte[buffSize];
byte[] buffer2 = new byte[buffSize];
// Initialize first iteration
byte[] curBuffer = buffer1;
byte[] prevBuffer = null;
int bytesRead;
// Attempt to fully fill the buffer
bytesRead = inStream.Read(curBuffer, 0, buffSize);
if( bytesRead == buffSize ) {
// We succesfully retrieved a whole buffer, we will output
// only [blockSize] bytes, to avoid writing to the last
// bytes in the buffer in case the remaining 16 bytes happen to
// be the last ones
outStream.Write(curBuffer, 0, blockSize);
} else {
// We couldn't retrieve the whole buffer
int bytesToWrite = bytesRead - bytesToOmit;
if( bytesToWrite > 0 ) {
outStream.Write(curBuffer, 0, bytesToWrite);
}
// There's no more data to process
return;
}
curBuffer = buffer2;
prevBuffer = buffer1;
while( true ) {
// Attempt again to fully fill the buffer
bytesRead = inStream.Read(curBuffer, 0, buffSize);
if( bytesRead == buffSize ) {
// We retrieved the whole buffer, output first the last 16
// bytes of the previous buffer, and output just [blockSize]
// bytes from the current buffer
outStream.Write(prevBuffer, blockSize, bytesToOmit);
outStream.Write(curBuffer, 0, blockSize);
} else {
// We could not retrieve a complete buffer
if( bytesRead <= bytesToOmit ) {
// The bytes to output come solely from the previous buffer
outStream.Write(prevBuffer, blockSize, bytesRead);
} else {
// The bytes to output come from the previous buffer and
// the current buffer
outStream.Write(prevBuffer, blockSize, bytesToOmit);
outStream.Write(curBuffer, 0, bytesRead - bytesToOmit);
}
break;
}
// swap buffers for next iteration
byte[] swap = prevBuffer;
prevBuffer = curBuffer;
curBuffer = swap;
}
}
static void Assert(Stream inStream, Stream outStream) {
// Routine that tests the copy worked as expected
inStream.Seek(0, SeekOrigin.Begin);
outStream.Seek(0, SeekOrigin.Begin);
Debug.Assert(outStream.Length == Math.Max(inStream.Length - bytesToOmit, 0));
for( int i = 0; i < outStream.Length; i++ ) {
int byte1 = inStream.ReadByte();
int byte2 = outStream.ReadByte();
Debug.Assert(byte1 == byte2);
}
}
A much easier solution to code, yet slower since it would work at a byte level, would be to use an intermediate queue between the input stream and the output stream. The process would first read and enqueue 16 bytes from the input stream. Then it would iterate over the remaining input bytes, reading a single byte from the input stream, enqueuing it and then dequeuing a byte. The dequeued byte would be written to the output stream, until all bytes from the input stream are processed. The unwanted 16 bytes should linger in the intermediate queue.
Hope this helps!
=)
Use a circular buffer sounds great but there is no circular buffer class in .NET which means additional code anyways. I ended up with the following algorithm, a sort of map and copy - I think it's simple. The variable names are longer than usual for the sake of being self descriptive here.
This flows thru the buffers as
[outStream] <== [tailBuf] <== [mainBuf] <== [inStream]
public byte[] CopyStreamExtractLastBytes(Stream inStream, Stream outStream,
int extractByteCount)
{
//var mainBuf = new byte[1024*4]; // 4K buffer ok for network too
var mainBuf = new byte[4651]; // nearby prime for testing
int mainBufValidCount;
var tailBuf = new byte[extractByteCount];
int tailBufValidCount = 0;
while ((mainBufValidCount = inStream.Read(mainBuf, 0, mainBuf.Length)) > 0)
{
// Map: how much of what (passthru/tail) lives where (MainBuf/tailBuf)
// more than tail is passthru
int totalPassthruCount = Math.Max(0, tailBufValidCount +
mainBufValidCount - extractByteCount);
int tailBufPassthruCount = Math.Min(tailBufValidCount, totalPassthruCount);
int tailBufTailCount = tailBufValidCount - tailBufPassthruCount;
int mainBufPassthruCount = totalPassthruCount - tailBufPassthruCount;
int mainBufResidualCount = mainBufValidCount - mainBufPassthruCount;
// Copy: Passthru must be flushed per FIFO order (tailBuf then mainBuf)
outStream.Write(tailBuf, 0, tailBufPassthruCount);
outStream.Write(mainBuf, 0, mainBufPassthruCount);
// Copy: Now reassemble/compact tail into tailBuf
var tempResidualBuf = new byte[extractByteCount];
Array.Copy(tailBuf, tailBufPassthruCount, tempResidualBuf, 0,
tailBufTailCount);
Array.Copy(mainBuf, mainBufPassthruCount, tempResidualBuf,
tailBufTailCount, mainBufResidualCount);
tailBufValidCount = tailBufTailCount + mainBufResidualCount;
tailBuf = tempResidualBuf;
}
return tailBuf;
}
I have a TCP socket application where I have to read several types of replies being the max size of a buffer 8192 some replies are splitted in more packets.
Currently I receive a list of members at reply 44, so the first idea I had to be able to deal with splitted packets was to define a stream out side of it to stored the incoming data until it is complete with a bool and current size variable.
Once it hits the reply 44 it will check if extraList is true or false, if false it means it is an initial request to incoming members list.
If the 4 initial bytes of the packet is bigger then bytes.Legth which is 8192 it will trigger the extraList to true and fill the initial data to the buffer I had previously set with the with the total packet size as it is size.
Since extraList has been trigged and turned into true, the packet reading will fall into it until the data is complete, which will then set it back to false and trigger the MemberList function with the complete list.
Would like some advices, suggestions, etc to improve this code.
int storedCurSize = 0;
MemoryStream stored = null;
bool extraList = false;
while (roomSocket.Connected)
{
byte[] bytes = new byte[roomSocket.ReceiveBufferSize];
roomSocket.Receive(bytes);
MemoryStream bufferReceived = new MemoryStream(bytes, 0, bytes.Length);
using (var reader = new BinaryReader(bufferReceived))
{
int packetSize = (int)reader.ReadInt32() + 9;
int reply = (int)reader.ReadByte();
if (reply == 44 || extraList)
{
if (!extraList && packetSize <= bytes.Length)
{
MemberList(bytes);
}
else
{
if (!extraList)
{
stored = new MemoryStream(new byte[packetSize], 0, packetSize);
stored.Write(bytes, 0, bytes.Length);
storedCurSize = bytes.Length;
extraList = true;
}
else
{
if (storedCurSize < stored.Length)
{
int storedLeftSize = (int)stored.Length - storedCurSize;
stored.Write(bytes, 0, (storedLeftSzie < bytes.Length) ? storedLeftSize : bytes.Length);
storedCurSize += (storedLeftSize < bytes.Length) ? storedLeftSize : bytes.Length;
if (storedCurSize >= stored.Length)
{
extraList = false;
MemberList(stored.ToArray());
stored.Close();
}
}
}
}
}
}
}
While reading code briefly what is flaring is magic numbers (9, 44) and very deep nesting.
Replace numbers with good-named constants and move out some parts of code as methods.
In case they are tightly twisted by used local variables - probably all this method worth moving out to worker class with single responsibility - to read the message. Thus local variables becomes class fields and methods wouldn't be so inflexible to refactoring.
Also MemberList(...) is poor name for a method as for me. Make it a verb that will describe what method is doing.
To merge bytes that aren't together, you can use Buffer.BlockCopy().
byte[] buf1;
byte[] buf2;
byte[] concatenated = new byte[buf1.Length + buf2.Length];
Buffer.BlockCopy(buf1, 0, concatenated, 0, buf1.Length);
Buffer.BlockCopy(buf2, 0, concatenated, buf1.ength, buf2.Length);