We Keep Coding

How do I keep reading data from my Socket to append my buffer

So I'm trying to send an image from my client to my server and I'm sending the entire image as a whole, meaning that I'm not splitting it up into chunks, I'm just sending the entire byte array as is.
CLIENT
private void SendImage(byte[] opcode, byte[] length, byte[] payload)
{
var packet = new byte[payload.Length + length.Length + 1];
Array.Copy(opcode, 0, packet, 0, 1);
//Set the length
Array.Copy(length, 0, packet, 1, length.Length);
Array.Copy(payload, 0, packet, 5, payload.Length);
_clientSocket.Send(packet);
}
This sends just fine, I'm using the OpCode 0x15 which will be interpreted by the server as "there is an image incoming". The length is the payload.Length which I've done this with
var length = BitConverter.GetBytes(myImage.Length);
So it occupies 4 bytes.
So my packet structure looks like this OpCode(1 byte), Length(4 bytes), Payload(imageBytes)
This method works just fine, it sends without any issues what so ever, I only included the code for it so that the next part will make sense.
SERVER
private byte[] _buffer = new byte[1024];
private void ReceiveCallback(IAsyncResult ar)
{
var client = (Socket)ar.AsyncState;
int received = client.EndReceive(ar);
//Temporary buffer
var dataBuf = new byte[received];
Array.Copy(_buffer, dataBuf, received);
switch (dataBuf[0])
{
//Received image
case 0x15:
//Read the packet header and check the length of the payload
var length = BitConverter.ToInt32(dataBuf.Skip(1).Take(4).ToArray(), 0);
//This will hold the bytes for the image
var imageBuffer = new byte[length];
//First incoming packet payload (image bytes)
var imgData = dataBuf.Skip(5).ToArray();
//Copy that into the buffer
Array.Copy(imgData, 0, imageBuffer, 0, imgData.Length);
var pos = imgData.Length;
//Keep reading bytes from the incoming stream
while (dataBuf.Length > 0)
{
imgData = new byte[1024];
client.Receive(imgData);
Array.Copy(imgData, 0, imageBuffer, pos, dataBuf.Length);
pos += 1024;
}
//This takes the bytes and creates a bitmap from it
AnotherViewModel.SetImage(imageBuffer);
break;
default:
Debug.WriteLine("Wat");
break;
}
client.BeginReceive(_buffer, 0, _buffer.Length, SocketFlags.None, new AsyncCallback(ReceiveCallback), client);
}
The issues I'm facing is that I don't know how to keep appending the incoming data to the imageBuffer because I want to receive the full image and not just the first 4 bytes and with my while loop I'm currently getting this exception
Destination array was not long enough. Check destIndex and length, and
the array's lower bounds.
Over at this line Array.Copy(imgData, 0, imageBuffer, pos, dataBuf.Length);
How do I properly read the entire image that the client sends to the server?

Unable to create a dynamic byte[] to process data received through Async TCP socket response

In my C# code I am receiving a byte[1024] from a DLL through TCP link.
Code I did for this I am attaching below.
The workflow is
An asynchronous packet gets received at tcp port into a byte array,
which then gets copied to a different array for processing.
This copied array then gets processed according to received packet
size(each message could be of different size one behind another).
While in the meantime async method received another data which
should get appended to array used for processing packet.
Problem coming is being method as asynchronous data arriving at TCP port is getting appended anytime to the processing buffer array which many times throws exceptions while its size is getting referred during Copy methods.
I am unable to block this async call as after processing one packet.
// Method to process recieved packet
_message.OnRecieve(message);
UI is getting updated and on user's action again the send and receive methods getting called where receive buffer array gets overwritten with new data.
Sharing piece of code:
public class Client
{
// State object for receiving data from remote device.
public class StateObject
{
// Client socket.
public Socket workSocket = null;
// Size of receive buffer.
public const int BufferSize = 1024;
// Receive buffer.
public byte[] buffer = new byte[1024];
}
public static T ByteArrayToStructure<T>(byte[] bytes) where T : struct
{
var handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
try
{
return (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
}
finally
{
handle.Free();
}
}
[StructLayout(LayoutKind.Sequential,Pack = 2)]
public struct MessageHeader
{
public int size;
}
private static byte[] data = new byte[1024 * 10];
private static void ReceiveCallback(IAsyncResult ar)
{
try
{
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 3)
{
//if data does not contain any data to process
if (data.Length == 0)
Array.Copy(state.buffer, 0, data, 0, bytesRead);
else
{
//resize data array according to data it contains+data arrived
Array.Resize(ref data, data.Length + bytesRead);
//if data array contains data and state.buffer arrives with new data
Array.Copy(state.buffer, 0, data, data.Length, bytesRead);
}
// process all data that exist in data array
while (data.Length > 2)
{
byte[] headerbyte = new byte[2];
//read two byes of data contains message length in format IPAddress.NetworkToHostOrder
Array.Copy(data, 0, headerbyte, 0, 2);
//reverse bytes in headerbyte
Array.Reverse(headerbyte);
Array.Copy(headerbyte, 0, data, 1, 1);
Array.Copy(headerbyte, 1, data, 0, 1);
//getting recieved message size from structure
MessageHeader header = ByteArrayToStructure<MessageHeader>(data);
int packetSize = header.size;
//if data contains within data array is partial packet
if (data.Length < packetSize)
{
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
else
{
byte[] message = new byte[packetSize];
byte[] remainingData = new byte[data.Length - packetSize];
//copy message data to process into message array
Array.Copy(data, 0, message, 0, packetSize);
//copy remainng data into a temp array
Array.Copy(data, packetSize, remainingData, 0, data.Length - packetSize);
//Method to process recieved packet
_message.OnRecieve(message);
//Removing processed Message from data array by resizing it to of size remaingin data and copying content into it
Array.Resize(ref data, remainingData.Length);
Array.Copy(remainingData, 0, data, 0, remainingData.Length);
}
}
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
else
{
// Create call back for next incoming packet
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
}

Client server socket C#

I am working on socket C#. I've implemented a client server application using socket, but the problem is that the client doesn't receive all data sent by the server.
Here is the client application code. What should I do so that it would receive all data sent by the server?
strRecieved = "";
Socket soc = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
IPEndPoint endPoint = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 9001);
soc.Connect(endPoint);
byte[] msgBuffer = Encoding.Default.GetBytes(path);
soc.Send(msgBuffer, 0, msgBuffer.Length, 0);
byte[] buffer = new byte[2000];
int rec = soc.Receive(buffer);
strRecieved = String.Format(Encoding.Default.GetString(buffer));

First of all. If you're implementing some kind of streaming feature ( tcp/udp/file ) you should consider using some kind of protocol.
What is a protocol? It's just a scheme to use when streaming data. Example:
[4Bytes - length][lengthBytes - message][1Byte - termination indicator]
Knowing the protocol you can read all of the incoming bytes simply as such :
byte[] buffer = new byte[4];
stream.ReadBytes(buffer, 0, 4); // cast that to int and read the rest
int packetLen = BitConverter.ToInt32(buffer, 0);
buffer = new byte[packetLen];
stream.ReadBytes(buffer, 0, buffer.Length); // all bytes that was sent
Remember that you have to subtract thease 4 bytes in the length before sending the message.
EDIT:
Simple example on how to send and receive data using shared protocol.
// sender.cs
string _stringToSend = "some fancy string";
byte[] encodedString = Encoding.UTF8.GetBytes(_stringToSend);
List<byte> buffer = new List<byte>();
buffer.AddRange(BitConverter.GetBytes(encodedString.Length));
buffer.AddRange(encodedString);
netStream.WriteBytes(buffer.ToArray(), 0, buffer.Count);
// netStream sent message in protocol [#LEN - 4Bytes][#MSG - #LENBytes]
// simply speaking something like: 5ABCDE
// receiver.cs
byte[] buffer = new byte[sizeof(int)];
netStream.ReadBytes(buffer, 0, buffer.Length);
// receiver got the length of the message eg. 5
int dataLen = BitConverter.ToInt32(buffer, 0);
buffer = new byte[dataLen];
// now we can read an actual message because we know it's length
netStream.ReadBytes(buffer, 0, buffer.Length);
string receivedString = Encoding.UTF8.GetString(buffer);
// received string is equal to "some fancy string"
Making it simpler
This technique forces you to use desired protocol which in this example will be :
First 4 bytes sizeof(int) are indicating the length of the incoming packet
Every byte further is your packet until the end.
So right now you should make ProtocolHelper object:
public static class ProtocolHelper
{
public byte[] PackIntoProtocol(string message)
{
List<byte> result = new List<byte>();
byte[] messageBuffer = Encoding.UTF8.GetBytes(message);
result.AddRange(BitConverter.GetBytes(messageBuffer.Length), 0); // this is the first part of the protocol ( length of the message )
result.AddRange(messageBuffer); // this is actual message
return result.ToArray();
}
public string UnpackProtocol(byte[] buffer)
{
return Encoding.UTF8.GetString(buffer, 0, buffer.Length);
}
}
Now ( depending on method you've chosen to read from network ) you have to send and receive your message.
// sender.cs
string meMessage = "network message 1";
byte[] buffer = ProtocolHelper.PackIntoProtocol(meMessage);
socket.Send(buffer, 0, buffer.Length, 0);
// receiver.cs
string message = string.Empty;
byte[] buffer = new byte[sizeof(int)]; // or simply new byte[4];
int received = socket.Receive(buffer);
if(received == sizeof(int))
{
int packetLen = BitConverter.ToInt32(buffer);// size of our message
buffer = new byte[packetLen];
received = socket.Receive(buffer);
if( packetLen == received ) // we have full buffer
{
message = PacketHelper.UnpackProtocol(buffer);
}
}
Console.WriteLine(message); // output: "network message 1"

You're limiting the size of received messages by 2KB as you're using new byte[2000].
I think you could either:
Size up you buffer to meet you message's size needs; and/or
Split you message into more than one socket messages.
Given that 4-8K is a good size for buffering socket messages and assuming RAM size is not a issue I would start with that, say, new byte[8000].
Also, you can send socket messages splitted in chunks. Maybe this is a good idea for the case. For example, if you have msg as the message (or object) you want to send:
private static async Task SendAnswer(Message msg, WebSocket socket)
{
var answer = System.Text.Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(msg).ToCharArray());
var bufferSize = 8000;
var endOfMessage = false;
for (var offset = 0; offset < answer.Length; offset += bufferSize)
{
if (offset + bufferSize >= answer.Length)
{
bufferSize = answer.Length - offset;
endOfMessage = true;
}
await socket.SendAsync(new ArraySegment<byte>(answer, offset, bufferSize),
WebSocketMessageType.Text, endOfMessage, CancellationToken.None);
}
}
And when receiving, you can also split the reception in chunks, so you can control you buffer (and therefore you memory consumption). After hanlding the whole message, you should wait for another message from the client to do more stuff. Source
private async Task ReceiveMessage(WebSocket webSocket)
{
var buffer = new byte[8000];
var result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
string msg = Encoding.UTF8.GetString(new ArraySegment<byte>(buffer, 0, result.Count).ToArray());
while (!result.EndOfMessage)
{
result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
msg += Encoding.UTF8.GetString(new ArraySegment<byte>(buffer, 0, result.Count).ToArray());
}
//At this point, `msg` has the whole message you sent, you can do whatever you want with it.
// [...]
//After you handle the message, wait for another message from the client
result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
}
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}

Receive correct amount of data TCP

I'm bringing up a TCP async server using C# and I'm struggling on the right way to receive the correct amount of data from the server.
My question is this one: due to the nature of TCP being a STREAM protocol, there is no delimiter at the end of each message received so the only thing I can do is add at the beginning of the message the upcoming message size and react consequently; the thing is, how can I recv and be sure that I'm not reading the "next" message in the stream?
My pseudo code looks like this:
// client accepted, begin receiving
client.BeginReceive(state.buffer, 0, StateObject.bufferSize, 0, new AsyncCallback(_cbck_Read), state);
private void _cbck_Read(IAsyncResult ar)
{
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.clientSocket;
int bytesRead = client.EndReceive(ar);
// if data have been received
if (bytesRead > 0)
{
state.bytesReceived += bytesRead;
// no message header received so far go on reading
if (state.bytesReceived < Marshal.SizeOf(header))
{
client.BeginReceive(state.buffer, 0, StateObject.bufferSize, 0, new AsyncCallback(_cbck_Read), state);
}
// ... go ahead reading
If the first recv does not get the whole message, could the next recv go far beyond the boundaries of the first and possibly add some unwanted bytes to the message I'm actually wanting to read?

Here is how it can be done using "async/await" with some helper extension methods.
Socket s = new Socket(SocketType.Stream, ProtocolType.Tcp);
await s.ConnectTaskAsync("stackoverflow.com", 80);
await s.SendTaskAsync(Encoding.UTF8.GetBytes("GET /\r\n\r\n"));
var buf1 = await s.ReceiveExactTaskAsync(100); //read exactly 100 bytes
Console.Write(Encoding.UTF8.GetString(buf1));
var buf2 = await s.ReceiveExactTaskAsync(100); //read exactly 100 bytes
Console.Write(Encoding.UTF8.GetString(buf2));
public static class SocketExtensions
{
public static Task<int> ReceiveTaskAsync(this Socket socket, byte[] buffer, int offset, int count)
{
return Task.Factory.FromAsync<int>(
socket.BeginReceive(buffer, offset, count, SocketFlags.None, null, socket),
socket.EndReceive);
}
public static async Task<byte[]> ReceiveExactTaskAsync(this Socket socket, int len)
{
byte[] buf = new byte[len];
int totalRead = 0;
do{
int read = await ReceiveTaskAsync(socket, buf, totalRead, buf.Length - totalRead);
if (read <= 0) throw new SocketException();
totalRead += read;
}while (totalRead != buf.Length);
return buf;
}
public static Task ConnectTaskAsync(this Socket socket, string host, int port)
{
return Task.Factory.FromAsync(
socket.BeginConnect(host, port, null, null),
socket.EndConnect);
}
public static Task SendTaskAsync(this Socket socket, byte[] buffer)
{
return Task.Factory.FromAsync<int>(
socket.BeginSend(buffer, 0, buffer.Length, SocketFlags.None, null, socket),
socket.EndSend);
}
}

As you observe, TCP provides no native framing. Worse, the a sync I/O events are reported for each TCP segment received from the network stack, and may not even match the send calls.
While building up the contents of the received stream, you will need to use one of:
fixed length messages, possibly with some leading type discrimination,
explicit length prefix indication, or
an unambiguous message delimeter.
The choice will generally depend on non-technical factors.

if you know the pending length (which you say you do based on protocol header of some sort) then only read the known pending length. IN your case Marshal.Sizeof(header) - state.bytesReceived

socket auto buffer size

I'm trying to make a C# socket client but when i create a buffer size with 256
public class StateObject
{
// Size of receive buffer.
public const int BufferSize = 256; // the buffer size :/
// Receive buffer.
public byte[] Buffer = new byte[BufferSize];
// Received data string.
public StringBuilder Sb = new StringBuilder();
public Socket WorkSocket = null;
}
i miss some received buffers and begin received as a separated buffer array,
if i create it for example a 1000 buffer size i'm wasting too many memory if i'm receiving a buffer with low size is there any way to make the buffer size auto ?
this is the code i'm using for connect call back and receive call back
private void OnConnectCallBack(IAsyncResult ar)
{
Socket.EndConnect(ar);
Connection = Connection.Connected;
if (OnConnect != null) OnConnect(this, CONNECTION_FROM.CLIENT);
// Begin receiving the data from the remote device.
Socket.BeginReceive(State.Buffer, 0, StateObject.BufferSize, 0, OnReceiveCallBack, State);
}
private void OnReceiveCallBack(IAsyncResult ar)
{
int bytesRead = Socket.EndReceive(ar);
// There might be more data, so store the data received so far.
if (OnReceive != null) OnReceive(this, State.Buffer, CONNECTION_FROM.CLIENT);
// Get the rest of the data.
Socket.BeginReceive(State.Buffer, 0, StateObject.BufferSize, 0,
OnReceiveCallBack, State);
}