We Keep Coding

Fastest way to split a Stream according to a pattern

What would be the most optimal/fastest way to split a Steam into chunks delimited by a byte pattern (eg. new byte[] { 0, 0 })?
My current, naieve and slow, implementation reads the stream byte per byte, decrements a counter each time it encounters the delimiter. If the counter is zero, it yields a memory chunk.
const int NUMBER_CONSECUTIVE_DELIMITER = 2;
const int DELIMITER = 0;
public IEnumerable<ReadOnlyMemory<byte>> Chunk(Stream stream)
{
var chunk = new MemoryStream();
try
{
int b; //the byte being read
int c = NUMBER_CONSECUTIVE_DELIMITER;
while ((b = stream.ReadByte()) != -1) //Read the stream byte by byte, -1 = end of the stream
{
chunk.WriteByte((byte)b); //Write this byte to the next chunk
if (b == DELIMITER)
c--; //if we hit the delimiter (ie '0') decrement the counter
else
c = NUMBER_CONSECUTIVE_DELIMITER; //else, reset the couter
if ((c <= 0 || stream.Position == stream.Length) //we hit two subsequent '0's
{
var r = chunk.ToArray().AsMemory(); //parse it to a Memory<T>
chunk.Dispose();
chunk = new();
yield return r;
}
}
}
finally
{
chunk.Dispose();
}
}

Such an implementation is extremely difficult to implement because a stream has to be read out in fixed buffer sizes. The buffer can be too big or too small for the content to be interpreted. To solve this problem, the ReadOnlySequence<T> struct was added. More information about this topic can be seen here.
By using System.IO.Pipelines (package must be obtained) this problem can be solved as follows:
public static async Task FillPipeAsync(Stream stream, PipeWriter writer, CancellationToken cancellationToken = default)
{
// The minimum buffer size that is used for the current buffer segment.
const int bufferSize = 65536;
while (true)
{
// Request 65536 bytes from the PipeWriter.
Memory<byte> memory = writer.GetMemory(bufferSize);
// Read the content from the stream.
int bytesRead = await stream.ReadAsync(memory, cancellationToken).ConfigureAwait(false);
if (bytesRead == 0) break;
// Tell the writer how many bytes are read.
writer.Advance(bytesRead);
// Flush the data to the PipeWriter.
FlushResult result = await writer.FlushAsync(cancellationToken).ConfigureAwait(false);
if (result.IsCompleted) break;
}
// This enables our reading process to be notified that no more new data is coming.
await writer.CompleteAsync().ConfigureAwait(false);
}
This will read your stream asynchronously and write a buffer segment to the pipe. Next you have to implement a read logic to slice/merge the concatenated buffer segments into chunks:
public static async IAsyncEnumerable<ReadOnlySequence<byte>> ReadPipeAsync(PipeReader reader, ReadOnlyMemory<byte> delimiter,
[EnumeratorCancellation] CancellationToken cancellationToken = default)
{
while (true)
{
// Read from the PipeReader.
ReadResult result = await reader.ReadAsync(cancellationToken).ConfigureAwait(false);
ReadOnlySequence<byte> buffer = result.Buffer;
while (TryReadChunk(ref buffer, delimiter.Span, out ReadOnlySequence<byte> chunk))
yield return chunk;
// Tell the PipeReader how many bytes are read.
// This is essential because the Pipe will release last used buffer segments that are not longer in use.
reader.AdvanceTo(buffer.Start, buffer.End);
// Take care of the complete notification and return the last buffer. UPDATE: Corrected issue 2/.
if (result.IsCompleted)
{
yield return buffer;
break;
}
}
await reader.CompleteAsync().ConfigureAwait(false);
}
private static bool TryReadChunk(ref ReadOnlySequence<byte> buffer, ReadOnlySpan<byte> delimiter,
out ReadOnlySequence<byte> chunk)
{
// Search the buffer for the first byte of the delimiter.
SequencePosition? position = buffer.PositionOf(delimiter[0]);
// If no occurence was found or the next bytes of the data in the buffer does not match the delimiter, return false.
// UPDATE: Corrected issue 3/.
if (position is null || !buffer.Slice(position.Value, delimiter.Length).FirstSpan.StartsWith(delimiter))
{
chunk = default;
return false;
}
// Return the calculated chunk and update the buffer to cut the start.
chunk = buffer.Slice(0, position.Value);
buffer = buffer.Slice(buffer.GetPosition(delimiter.Length, position.Value));
return true;
}
For this to work in that form you have to use an IAsyncEnumerable so that the chunks can be streamed into a foreach loop. Merging and slicing is largely handled by the pipe, so that a reliable algorithm can be built here with relatively little code. This code will also handle this in a high-performance manner.
Usage:
// Create a Pipe that manages the buffer.
Pipe pipe = new Pipe();
ConfiguredTaskAwaitable writing = FillPipeAsync(stream, pipe.Writer).ConfigureAwait(false);
// The delimiter that should be used. This can be any data with length > 0.
ReadOnlyMemory<byte> delimiter = new ReadOnlyMemory<byte>(new byte[] { 0, 0 });
// 'await foreach' and 'await writing' are executed asynchronously (in parallel).
await foreach (ReadOnlySequence<byte> chunk in ReadPipeAsync(pipe.Reader, delimiter))
{
// Use "chunk" to retrieve your chunked content.
};
await writing;
Note that reading and chunking is done asynchronously and independently.

I eventually ended up with the below code, strongly inspired by Philipp's answer above and https://keestalkstech.com/2010/11/seek-position-of-a-string-in-a-file-or-filestream/.
public override IEnumerable<byte[]> Chunk(Stream stream)
{
var buffer = new byte[bufferSize];
var size = bufferSize;
var offset = 0;
var position = stream.Position;
var nextChunk = Array.Empty<byte>();
while (true)
{
var bytesRead = stream.Read(buffer, offset, size);
// when no bytes are read -- the string could not be found
if (bytesRead <= 0)
break;
// when less then size bytes are read, we need to slice the buffer to prevent reading of "previous" bytes
ReadOnlySpan<byte> ro = buffer;
if (bytesRead < size)
ro = ro.Slice(0, offset + bytesRead);
// check if we can find our search bytes in the buffer
var i = ro.IndexOf(Delimiter);
if (i > -1 && // we found something
i <= bytesRead && //i <= r -- we found something in the area that was read (at the end of the buffer, the last values are not overwritten). i = r if the delimiter is at the end of the buffer
nextChunk.Length + (i + Delimiter.Length - offset) >= MinChunkSize) //the size of the chunk that will be made is large enough
{
var chunk = buffer[offset..(i + Delimiter.Length)];
yield return new byte[](Concat(nextChunk, chunk));
nextChunk = Array.Empty<byte>();
offset = 0;
size = bufferSize;
position += i + Delimiter.Length;
stream.Position = position;
continue;
}
else if (stream.Position == stream.Length)
{
// we re at the end of the stream
var chunk = buffer[offset..(bytesRead + offset)]; //return the bytes read
yield return new byte[](Concat(nextChunk, chunk));
break;
}
// the stream is not finished. Copy the last 2 bytes to the beginning of the buffer and set the offset to fill the buffer as of byte 3
nextChunk = Concat(nextChunk, buffer[offset..buffer.Length]);
offset = Delimiter.Length;
size = bufferSize - offset;
Array.Copy(buffer, buffer.Length - offset, buffer, 0, offset);
position += bufferSize - offset;
}
}

Sending multiple files from C# to C

I have a wpf gui that I want to upload files from and send to a C client .
I want to send 3 files and for some reason 1 of them is being sent and written (but it adds 8 nulls in the end and removes 4 of the first letters in the file)
and in the other two when I try to receive the size it says their size is 0
I've been stuck on this problem for a while now and i'm becoming depserate as i'm probably missing a small thing , if any of u could give a hand that'll mean a lot ! I really wanna know whats the problem in my code.
I have the files paths in an array and sending it in main like so
C# Main
IPEndPoint ipPoint = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 5555);//switch the port
Socket listenSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
listenSocket.Bind(ipPoint);
listenSocket.Listen(1);
Socket clientSocket = listenSocket.Accept();
for (int i = 0; i < 1; i++)
{
SendFile(clientSocket, filePaths[i]);
}
clientSocket.Shutdown(SocketShutdown.Both);
clientSocket.Close();
SendFile(C# side)
public static void SendFile(Socket clientSocket, string filePath)
{
if (File.Exists(filePath))
{
FileInfo fi = new FileInfo(filePath);
long file_size = fi.Length;
byte[] preBuffer;
using (var memoryStream = new MemoryStream())
{
using (BinaryWriter writer = new BinaryWriter(memoryStream))
{
writer.Write(file_size);
}
preBuffer = memoryStream.ToArray();
byte[] fixedBuffer = new byte[4];
Array.Copy(preBuffer, 0, fixedBuffer, 0, 4);
Console.WriteLine(BitConverter.ToString(preBuffer));
Console.WriteLine(BitConverter.ToString(fixedBuffer)); //fixing the problem i had with the converting to array that it added 4 useless zeros.
clientSocket.Send(fixedBuffer); // sending size
}
byte[] data = new Byte[4096];
using (FileStream fs = new FileStream(filePath, FileMode.Open))
{
int actualRead;
do
{
actualRead = fs.Read(data, 0, data.Length);
clientSocket.Send(data);
file_size -= actualRead;
} while (file_size - filePath.Length > 0);
}
}
else
{
MessageBox.Show("File for the program is missing! lua/pcap/csv");
}
}
C Receive(built from 3 functions)
/*
============================================
General : function is responsible for receiving a length of data from the client
Parameters : sock - client socket to receive the data from
*buf - holds a pointer to the buffer that needs to update
bufsize - the length of the buffer
Return Value : returns TRUE when the data is read correctly
else, FALSE when there was a socket error or no bytes are received.
============================================
*/
bool recv_raw(SOCKET sock, void* buf, int bufsize)
{
unsigned char* pbuf = (unsigned char*)buf;
while (bufsize > 0) {
int num = recv(sock, pbuf, bufsize, 0);
if (num <= 0) { return false; }
pbuf += num;
bufsize -= num;
}
return true;
}
/*
===================================================
General : receives the length of the file and updates it
Parameters : sock - client socket to receive the data from
*filesize - holds a pointer to the size of the buffer that needs to update
filesize_len - the length of the file size pointer
Return Value : returns TRUE when the size is read correctly
else, FALSE when there was a socket error or no bytes are received.
===================================================
*/
bool recv_file_len(SOCKET sock, long* filesize)
{
if (!recv_raw(sock, filesize, sizeof(*filesize))) { return false; }
return true;
}
/*
================================================== =
General : writes to the lua file the data from the file
that was received in the socket
Parameters : sock - the socket between the client and server
*f - the file to write the data received to
Return Value : returns TRUE when everything was written to the file.
returns FALSE if there's no data received or detected a socket problem.
================================================== =
*/
bool write_data(SOCKET sock, FILE *f)
{
long filesize;//size of address
char buffer[BUFFER_SIZE];
if (!recv_file_len(sock, &filesize)) { return false; }
printf("file size (From C#) : %ld\n", filesize);
int n = recv_raw(sock, buffer, 8); // need to get the size of the name
if (filesize > 0)
{
do {
int num = min(filesize, BUFFER_SIZE);
if (!recv_raw(sock, buffer, num)) {
return false;
}
int offset = 0;
do
{
size_t written = fwrite(&buffer[offset], 1, num - offset, f);
if (written < 1) { return false; }
offset += written;
} while (offset < num);
filesize -= num;
} while (filesize > 0);
}
return true;
}
C Main
FILE* luafhandler = fopen("test.lua", "wb");//the new lua file
if (luafhandler == NULL)
{
fclose(luafhandler);
printf("GUI CONNECT lua file failed to open!\n");
}
FILE* pcapfhandler = fopen("test.pcap", "wb");//the new lua file
if (pcapfhandler == NULL)
{
fclose(pcapfhandler);
printf("GUI CONNECT pcap file failed to open!\n");
}
FILE* csvfhandler = fopen("AlgoTest.csv", "wb");//the new lua file
if (csvfhandler == NULL)
{
fclose(csvfhandler);
printf("GUI CONNECT csv file failed to open!\n");
}
else {
SOCKET sock1 = open_socket(5555, SERVER_IP);
bool check = write_data(sock1, luafhandler);
bool check1 = write_data(sock1, pcapfhandler);
bool check2 = write_data(sock1, csvfhandler);
fclose(luafhandler);
fclose(pcapfhandler);
fclose(csvfhandler);
}

Sending multiple files on different threads Sockets c#

I am working on a software that transfers multiple files between different computers but i am faced with the problem of concurrency. When both threads start sending using Task.Run() the bytes mumble up on the receiving side i.e the data clashes together. I have tried locking every possible part of my code that could cause the error all to no avail. This is part of my code.
//code to transfer asynchronously
public void StartTransferAsync(Socket socket)
{
Task.Factory.StartNew(() =>
{
lock (this)
{
uploader = new Uploader(this, socket);
uploader.Upload();
}
});
}
//code to receive the asynchronous files coming in
public byte[] DownloadData()
{
int total = 0;
int recv = 0;
int size = 0;
int dataleft = 0;
byte[] data;
byte[] dataSize = new byte[4];
recv = socket.Receive(dataSize, 0, 4, 0);
size = BitConverter.ToInt32(dataSize, 0);
Console.WriteLine($"Size received: {size}");
if (size > 100038)
{
Console.WriteLine("Shii here");
}
dataleft = size;
data = new byte[size];
while (total < size)
{
recv = socket.Receive(data, total, dataleft, 0);
total += recv;
dataleft -= recv;
}
return data;
}
Please help i really need this.

WaitFor() - How to wait for a specific buffer to arrive on Steam/SerialPort?

Requested Behaviour: I would like to hear proposed, generic solutions for suspending a calling thread until a specific buffer is received on a Stream/SerialPort. For the time being, I'm not concerned with timeouts etc, however I need something robust.
Attempted method:
Class myClass
{
private SerialPort _port; //Assume configured and connected.
public void WaitFor(byte[] buffer)
{
int bufferLength = buffer.Length;
byte[] comparisonBuffer = new byte[bufferLength];
while(true)
{
if(_port.BytesToRead >= bufferLength)
{
_port.Read(comparisonBuffer, 0, bufferLength);
if (comparisonBuffer.SequenceEqual(buffer)) { return; }
}
}
}
{
I've had a reasonable amount of success with this however it just has a "hacky" feel to it. It has quite often caused me trouble. I believe it's due to the fact that I cannot guarantee that other data isn't received either before or after the expected packet, so naturally this method can end up reading off the stream out of sync. In such a case I would not want to loose the leading/trailing data but the method should release the thread.
I need to implement in a procedural nature so event driven methods won't really work for me. In the generic sense I want to be able to implement as;
Do thing;
WaitFor(mybuffer);
Do other thing;

SerialPort.Read() already blocks until at least one byte has arrived. Therefore you don't need to (and shouldn't) use the BytesToRead the way you are - you've introduced a HORRIBLE busy-wait loop.
Instead, do something like this:
// Reads 'count' bytes from a serial port into the specified
// part of a buffer. This blocks until all the bytes have been read.
public void BlockingRead(SerialPort port, byte[] buffer, int offset, int count)
{
while (count > 0)
{
// SerialPort.Read() blocks until at least one byte has been read, or SerialPort.ReadTimeout milliseconds
// have elapsed. If a timeout occurs a TimeoutException will be thrown.
// Because SerialPort.Read() blocks until some data is available this is not a busy loop,
// and we do NOT need to issue any calls to Thread.Sleep().
int bytesRead = port.Read(buffer, offset, count);
offset += bytesRead;
count -= bytesRead;
}
}
Here's how you would implement your original code in terms of BlockingRead():
public void WaitFor(SerialPort port, byte[] buffer)
{
byte[] comparisonBuffer = new byte[buffer.Length];
while (true)
{
BlockingRead(port, comparisonBuffer, 0, comparisonBuffer.Length);
if (comparisonBuffer.SequenceEqual(buffer))
return;
}
}

Problem
Lets assume you wait for the byte pattern {1,1,1,2,2} and the serial port has buffered {1,1,1,1,2,2,5}.
Your code reads the first 5 bytes {1,1,1,1,2} which will not match the pattern. But after reading from the port the data you read has been removed from the buffer and contains only {2,5} and you will never get a match.
Solution
public void WaitFor( byte[ ] buffer )
{
if ( buffer.Length == 0 )
return;
var q = new List<byte>( buffer.Length );
while ( true )
{
var current = _reader.ReadByte();
q.Add( (byte)current );
// sequence match so far
if ( q.Last == buffer[ q.Count - 1 ] )
{
// check for total match
if ( q.Count == buffer.Length )
return;
}
else
{
// shift the data
while ( q.Any() && !q.SequenceEqual( buffer.Take( q.Count ) ) )
{
q.RemoveAt( 0 );
}
}
}
}

What do you think to this solution?
public override byte[] WaitFor(byte[] buffer, int timeout)
{
// List to stack stream into
List<byte> stack = new List<byte>();
// Index of first comparison byte
int index = 0;
// Index of last comparison byte
int upperBound = buffer.Length - 1;
// Timeout Manager
Stopwatch Sw = new Stopwatch();
Sw.Start();
while (Sw.Elapsed.Seconds <= timeout)
{
// Read off the last byte receievd and add to the stack
stack.Add((byte)_port.ReadByte());
// If my stack contains enough bytes to compare to the buffer
if (stack.Count > upperBound)
{
// If my first comparison byte matches my first buffer byte
if (stack[index] == buffer[0])
{
// Extract the comparison block to array
byte[] compBuffer = stack.GetRange(index,upperBound +1).ToArray();
// If the comparison matches, break and return the redundent bytes should I wish to handle them.
if ((compBuffer.SequenceEqual(buffer) && (index-1 > 0))) { return stack.GetRange(0, index - 1).ToArray(); }
// If there were no redundent bytes, just return zero.
else if (compBuffer.SequenceEqual(buffer)) { return new byte[] { 0}; }
}
// Increments
index += 1;
upperBound += 1;
}
}
throw new TimeoutException("Timeout: Expected buffer was not received prior to timeout");
}

C# NetworkStream data loss

I am currently working on a networking project where I worked out a binary protocol. My packets look like this:
[1 byte TYPE][2 bytes INDEX][2 bytes LENGTH][LENGTH bytes DATA]
And here's the code where I am receiving the packets:
NetworkStream clientStream= Client.GetStream();
while (Client.Connected)
{
Thread.Sleep(10);
try
{
if (clientStream.DataAvailable)
{
byte[] infobuffer = new byte[5];
int inforead = clientStream.Read(infobuffer, 0, 5);
if (inforead < 5) { continue; }
byte[] rawclient = new byte[2];
Array.Copy(infobuffer, 1, rawclient, 0, 2);
PacketType type = (PacketType)Convert.ToSByte(infobuffer[0]);
int clientIndex = BitConverter.ToInt16(rawclient, 0);
int readLength = BitConverter.ToInt16(infobuffer, 3);
byte[] readbuffer = new byte[readLength];
int count_read = clientStream.Read(readbuffer, 0, readLength);
byte[] read_data = new byte[count_read];
Array.Copy(readbuffer, read_data, count_read);
HandleData(read_data, type, clientIndex);
}
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("[E] " + ex.GetType().ToString());
Console.ResetColor();
break;
}
}
Well, and everything works fine... as long as I run it on 127.0.0.1. As soon as I try testing it over long distance, packets somehow get lost, and I am getting an overflow-exception on the line where I convert the first byte to PacketType. Also, if I try to convert the other values to int16, I get very strange values.
I assume the stream somehow looses some bytes on its way to the server, but can this be? Or is it just a little mistake of mine somewhere in the code?
edit:
I now edited the code, now it reads till it gets its 5 bytes. But I still get the same exception over long distance...
NetworkStream clientStream = Client.GetStream();
while (Client.Connected)
{
Thread.Sleep(10);
try
{
if (clientStream.DataAvailable)
{
int totalread = 0;
byte[] infobuffer = new byte[5];
while (totalread < 5)
{
int inforead = clientStream.Read(infobuffer, totalread, 5 - totalread);
if (inforead == 0)
{ break; }
totalread += inforead;
}
byte[] rawclient = new byte[2];
Array.Copy(infobuffer, 1, rawclient, 0, 2);
PacketType type = (PacketType)Convert.ToSByte(infobuffer[0]);
int clientIndex = BitConverter.ToInt16(rawclient, 0);
int readLength = BitConverter.ToInt16(infobuffer, 3);
byte[] readbuffer = new byte[readLength];
int count_read = clientStream.Read(readbuffer, 0, readLength);
byte[] read_data = new byte[count_read];
Array.Copy(readbuffer, read_data, count_read);
HandleData(read_data, type, clientIndex);
}
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("[E] " + ex.GetType().ToString());
Console.ResetColor();
break;
}
}
PacketType is an enum:
public enum PacketType
{
AddressSocks5 = 0,
Status = 1,
Data = 2,
Disconnect = 3,
AddressSocks4 = 4
}

So many things you're doing wrong here... so many bugs... where to even start...
First Network polling? Really? That's just a naïve way of doing network activity in this day and age.. but I won't harp on that.
Second, with this type of protocol, it's pretty easy to get "out of sync" and once you do, you have no way to get back in sync. This is typically accomplished with some kind of "framing protocol" which provides a unique sequence of bytes that you can use to indicate the start and end of a frame, so that if you ever find yourself out of sync you can read data until you get back in sync. Yes, you will lose data, but you've already lost it if you're out of sync.
Third, you're not really doing anything huge here, so I shamelessly stole the "ReadWholeArray" code from here, it's not the most efficient, but it works and there is other code there that might help:
http://www.yoda.arachsys.com/csharp/readbinary.html
Note: you don't mention how you are serializing the length, type and index values on the other side. So using the BitConverter may be the wrong thing depending on how that was done.
if (clientStream.DataAvailable)
{
byte[] data = new byte[5];
// if it can't read all 5 bytes, it throws an exception
ReadWholeArray(clientStream, data);
PacketType type = (PacketType)Convert.ToSByte(data[0]);
int clientIndex = BitConverter.ToInt16(data, 1);
int readLength = BitConverter.ToInt16(data, 3);
byte[] rawdata = new byte[readLength];
ReadWholeArray(clientStream, rawdata);
HandleData(rawdata, type, clientIndex);
}
/// <summary>
/// Reads data into a complete array, throwing an EndOfStreamException
/// if the stream runs out of data first, or if an IOException
/// naturally occurs.
/// </summary>
/// <param name="stream">The stream to read data from</param>
/// <param name="data">The array to read bytes into. The array
/// will be completely filled from the stream, so an appropriate
/// size must be given.</param>
public static void ReadWholeArray (Stream stream, byte[] data)
{
int offset=0;
int remaining = data.Length;
while (remaining > 0)
{
int read = stream.Read(data, offset, remaining);
if (read <= 0)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", remaining));
remaining -= read;
offset += read;
}
}

I think the problem is in these lines
int inforead = clientStream.Read(infobuffer, 0, 5);
if (inforead < 5) { continue; }
what happen to your previously read data if the length is under 5 byte? you should save the bytes you have read so far and append next bytes so you can have the header completely

You Read 5 - totalRead.
let totalRead equal 5 or more. When that happens you read nothing, and in cases of 1 - 4 you read that many arbitrary bytes. Not 5. You also then discard any result of less then 5.
You also copy at a offset 1 or another offset without really knowing the offset.
BitConverter.ToInt16(infobuffer, 3);
Is an example of this, what is at off 2?
So if it's not that (decoding error) and and not the structure of your data then unless you change the structure of your loop its you who's losing the bytes not the NetworkStream.
Calculate totalRead by increments of justRead when you recieve so you can handle any size of data as well as receiving it at the correct offset.