I'm bringing up a TCP async server using C# and I'm struggling on the right way to receive the correct amount of data from the server.
My question is this one: due to the nature of TCP being a STREAM protocol, there is no delimiter at the end of each message received so the only thing I can do is add at the beginning of the message the upcoming message size and react consequently; the thing is, how can I recv and be sure that I'm not reading the "next" message in the stream?
My pseudo code looks like this:
// client accepted, begin receiving
client.BeginReceive(state.buffer, 0, StateObject.bufferSize, 0, new AsyncCallback(_cbck_Read), state);
private void _cbck_Read(IAsyncResult ar)
{
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.clientSocket;
int bytesRead = client.EndReceive(ar);
// if data have been received
if (bytesRead > 0)
{
state.bytesReceived += bytesRead;
// no message header received so far go on reading
if (state.bytesReceived < Marshal.SizeOf(header))
{
client.BeginReceive(state.buffer, 0, StateObject.bufferSize, 0, new AsyncCallback(_cbck_Read), state);
}
// ... go ahead reading
If the first recv does not get the whole message, could the next recv go far beyond the boundaries of the first and possibly add some unwanted bytes to the message I'm actually wanting to read?
Here is how it can be done using "async/await" with some helper extension methods.
Socket s = new Socket(SocketType.Stream, ProtocolType.Tcp);
await s.ConnectTaskAsync("stackoverflow.com", 80);
await s.SendTaskAsync(Encoding.UTF8.GetBytes("GET /\r\n\r\n"));
var buf1 = await s.ReceiveExactTaskAsync(100); //read exactly 100 bytes
Console.Write(Encoding.UTF8.GetString(buf1));
var buf2 = await s.ReceiveExactTaskAsync(100); //read exactly 100 bytes
Console.Write(Encoding.UTF8.GetString(buf2));
public static class SocketExtensions
{
public static Task<int> ReceiveTaskAsync(this Socket socket, byte[] buffer, int offset, int count)
{
return Task.Factory.FromAsync<int>(
socket.BeginReceive(buffer, offset, count, SocketFlags.None, null, socket),
socket.EndReceive);
}
public static async Task<byte[]> ReceiveExactTaskAsync(this Socket socket, int len)
{
byte[] buf = new byte[len];
int totalRead = 0;
do{
int read = await ReceiveTaskAsync(socket, buf, totalRead, buf.Length - totalRead);
if (read <= 0) throw new SocketException();
totalRead += read;
}while (totalRead != buf.Length);
return buf;
}
public static Task ConnectTaskAsync(this Socket socket, string host, int port)
{
return Task.Factory.FromAsync(
socket.BeginConnect(host, port, null, null),
socket.EndConnect);
}
public static Task SendTaskAsync(this Socket socket, byte[] buffer)
{
return Task.Factory.FromAsync<int>(
socket.BeginSend(buffer, 0, buffer.Length, SocketFlags.None, null, socket),
socket.EndSend);
}
}
As you observe, TCP provides no native framing. Worse, the a sync I/O events are reported for each TCP segment received from the network stack, and may not even match the send calls.
While building up the contents of the received stream, you will need to use one of:
fixed length messages, possibly with some leading type discrimination,
explicit length prefix indication, or
an unambiguous message delimeter.
The choice will generally depend on non-technical factors.
if you know the pending length (which you say you do based on protocol header of some sort) then only read the known pending length. IN your case Marshal.Sizeof(header) - state.bytesReceived
Related
In my C# code I am receiving a byte[1024] from a DLL through TCP link.
Code I did for this I am attaching below.
The workflow is
An asynchronous packet gets received at tcp port into a byte array,
which then gets copied to a different array for processing.
This copied array then gets processed according to received packet
size(each message could be of different size one behind another).
While in the meantime async method received another data which
should get appended to array used for processing packet.
Problem coming is being method as asynchronous data arriving at TCP port is getting appended anytime to the processing buffer array which many times throws exceptions while its size is getting referred during Copy methods.
I am unable to block this async call as after processing one packet.
// Method to process recieved packet
_message.OnRecieve(message);
UI is getting updated and on user's action again the send and receive methods getting called where receive buffer array gets overwritten with new data.
Sharing piece of code:
public class Client
{
// State object for receiving data from remote device.
public class StateObject
{
// Client socket.
public Socket workSocket = null;
// Size of receive buffer.
public const int BufferSize = 1024;
// Receive buffer.
public byte[] buffer = new byte[1024];
}
public static T ByteArrayToStructure<T>(byte[] bytes) where T : struct
{
var handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
try
{
return (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
}
finally
{
handle.Free();
}
}
[StructLayout(LayoutKind.Sequential,Pack = 2)]
public struct MessageHeader
{
public int size;
}
private static byte[] data = new byte[1024 * 10];
private static void ReceiveCallback(IAsyncResult ar)
{
try
{
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 3)
{
//if data does not contain any data to process
if (data.Length == 0)
Array.Copy(state.buffer, 0, data, 0, bytesRead);
else
{
//resize data array according to data it contains+data arrived
Array.Resize(ref data, data.Length + bytesRead);
//if data array contains data and state.buffer arrives with new data
Array.Copy(state.buffer, 0, data, data.Length, bytesRead);
}
// process all data that exist in data array
while (data.Length > 2)
{
byte[] headerbyte = new byte[2];
//read two byes of data contains message length in format IPAddress.NetworkToHostOrder
Array.Copy(data, 0, headerbyte, 0, 2);
//reverse bytes in headerbyte
Array.Reverse(headerbyte);
Array.Copy(headerbyte, 0, data, 1, 1);
Array.Copy(headerbyte, 1, data, 0, 1);
//getting recieved message size from structure
MessageHeader header = ByteArrayToStructure<MessageHeader>(data);
int packetSize = header.size;
//if data contains within data array is partial packet
if (data.Length < packetSize)
{
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
else
{
byte[] message = new byte[packetSize];
byte[] remainingData = new byte[data.Length - packetSize];
//copy message data to process into message array
Array.Copy(data, 0, message, 0, packetSize);
//copy remainng data into a temp array
Array.Copy(data, packetSize, remainingData, 0, data.Length - packetSize);
//Method to process recieved packet
_message.OnRecieve(message);
//Removing processed Message from data array by resizing it to of size remaingin data and copying content into it
Array.Resize(ref data, remainingData.Length);
Array.Copy(remainingData, 0, data, 0, remainingData.Length);
}
}
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
else
{
// Create call back for next incoming packet
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
}
I am working on socket C#. I've implemented a client server application using socket, but the problem is that the client doesn't receive all data sent by the server.
Here is the client application code. What should I do so that it would receive all data sent by the server?
strRecieved = "";
Socket soc = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
IPEndPoint endPoint = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 9001);
soc.Connect(endPoint);
byte[] msgBuffer = Encoding.Default.GetBytes(path);
soc.Send(msgBuffer, 0, msgBuffer.Length, 0);
byte[] buffer = new byte[2000];
int rec = soc.Receive(buffer);
strRecieved = String.Format(Encoding.Default.GetString(buffer));
First of all. If you're implementing some kind of streaming feature ( tcp/udp/file ) you should consider using some kind of protocol.
What is a protocol? It's just a scheme to use when streaming data. Example:
[4Bytes - length][lengthBytes - message][1Byte - termination indicator]
Knowing the protocol you can read all of the incoming bytes simply as such :
byte[] buffer = new byte[4];
stream.ReadBytes(buffer, 0, 4); // cast that to int and read the rest
int packetLen = BitConverter.ToInt32(buffer, 0);
buffer = new byte[packetLen];
stream.ReadBytes(buffer, 0, buffer.Length); // all bytes that was sent
Remember that you have to subtract thease 4 bytes in the length before sending the message.
EDIT:
Simple example on how to send and receive data using shared protocol.
// sender.cs
string _stringToSend = "some fancy string";
byte[] encodedString = Encoding.UTF8.GetBytes(_stringToSend);
List<byte> buffer = new List<byte>();
buffer.AddRange(BitConverter.GetBytes(encodedString.Length));
buffer.AddRange(encodedString);
netStream.WriteBytes(buffer.ToArray(), 0, buffer.Count);
// netStream sent message in protocol [#LEN - 4Bytes][#MSG - #LENBytes]
// simply speaking something like: 5ABCDE
// receiver.cs
byte[] buffer = new byte[sizeof(int)];
netStream.ReadBytes(buffer, 0, buffer.Length);
// receiver got the length of the message eg. 5
int dataLen = BitConverter.ToInt32(buffer, 0);
buffer = new byte[dataLen];
// now we can read an actual message because we know it's length
netStream.ReadBytes(buffer, 0, buffer.Length);
string receivedString = Encoding.UTF8.GetString(buffer);
// received string is equal to "some fancy string"
Making it simpler
This technique forces you to use desired protocol which in this example will be :
First 4 bytes sizeof(int) are indicating the length of the incoming packet
Every byte further is your packet until the end.
So right now you should make ProtocolHelper object:
public static class ProtocolHelper
{
public byte[] PackIntoProtocol(string message)
{
List<byte> result = new List<byte>();
byte[] messageBuffer = Encoding.UTF8.GetBytes(message);
result.AddRange(BitConverter.GetBytes(messageBuffer.Length), 0); // this is the first part of the protocol ( length of the message )
result.AddRange(messageBuffer); // this is actual message
return result.ToArray();
}
public string UnpackProtocol(byte[] buffer)
{
return Encoding.UTF8.GetString(buffer, 0, buffer.Length);
}
}
Now ( depending on method you've chosen to read from network ) you have to send and receive your message.
// sender.cs
string meMessage = "network message 1";
byte[] buffer = ProtocolHelper.PackIntoProtocol(meMessage);
socket.Send(buffer, 0, buffer.Length, 0);
// receiver.cs
string message = string.Empty;
byte[] buffer = new byte[sizeof(int)]; // or simply new byte[4];
int received = socket.Receive(buffer);
if(received == sizeof(int))
{
int packetLen = BitConverter.ToInt32(buffer);// size of our message
buffer = new byte[packetLen];
received = socket.Receive(buffer);
if( packetLen == received ) // we have full buffer
{
message = PacketHelper.UnpackProtocol(buffer);
}
}
Console.WriteLine(message); // output: "network message 1"
You're limiting the size of received messages by 2KB as you're using new byte[2000].
I think you could either:
Size up you buffer to meet you message's size needs; and/or
Split you message into more than one socket messages.
Given that 4-8K is a good size for buffering socket messages and assuming RAM size is not a issue I would start with that, say, new byte[8000].
Also, you can send socket messages splitted in chunks. Maybe this is a good idea for the case. For example, if you have msg as the message (or object) you want to send:
private static async Task SendAnswer(Message msg, WebSocket socket)
{
var answer = System.Text.Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(msg).ToCharArray());
var bufferSize = 8000;
var endOfMessage = false;
for (var offset = 0; offset < answer.Length; offset += bufferSize)
{
if (offset + bufferSize >= answer.Length)
{
bufferSize = answer.Length - offset;
endOfMessage = true;
}
await socket.SendAsync(new ArraySegment<byte>(answer, offset, bufferSize),
WebSocketMessageType.Text, endOfMessage, CancellationToken.None);
}
}
And when receiving, you can also split the reception in chunks, so you can control you buffer (and therefore you memory consumption). After hanlding the whole message, you should wait for another message from the client to do more stuff. Source
private async Task ReceiveMessage(WebSocket webSocket)
{
var buffer = new byte[8000];
var result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
while (!result.CloseStatus.HasValue)
{
string msg = Encoding.UTF8.GetString(new ArraySegment<byte>(buffer, 0, result.Count).ToArray());
while (!result.EndOfMessage)
{
result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
msg += Encoding.UTF8.GetString(new ArraySegment<byte>(buffer, 0, result.Count).ToArray());
}
//At this point, `msg` has the whole message you sent, you can do whatever you want with it.
// [...]
//After you handle the message, wait for another message from the client
result = await webSocket.ReceiveAsync(buffer, CancellationToken.None);
}
await webSocket.CloseAsync(result.CloseStatus.Value, result.CloseStatusDescription, CancellationToken.None);
}
I'm working with an Access Control Device, that allows or denies access to rooms, verifying rights through biometric data. I need to listen indefinitely for data on a connected TcpClient(Socket). But how to do this without the following approach:
byte[] bb = new byte[1024]
while(true)
{
if (tcpClient.Client.Available > 0)
{
tcpClient.Client.Receive(bb, bb.Length, SocketFlags.None);
int k = tcpClient.Client.Receive(bb);
string result = Encoding.Default.GetString(bb.Take(k).ToArray());
// do sth here, rise an event, etc...
}
Thread.Sleep(500);
}
I'm following this example, did some adaptation (see below), but only the first chunk of data is read (if the device sends something again, the sent data are not read). I think that the EndReceive method is closing the socket or else, but I want to read the socket continuously, to raise an event every time socket reads data.
...
TcpClient tcpClient // Connected with BeginConnect
byte[] BufferData = new byte[1024]
public IAsyncResult StartReceivingData()
{
//some code here
return tcpClient.Client.BeginReceive(DataBuffer, 0,
DataBuffer.Length, SocketFlags.None,
new AsyncCallback(ReceivedData), tcpClient);
}
public void ReceivedData(IAsyncResult callerResult)
{
TcpClient remote = (TcpClient)callerResult.AsyncState;
int recv = remote.Client.EndReceive(callerResult);
string stringData = Encoding.ASCII.GetString(BufferDados, 0, recv);
//rise an event here containing stringData
}
EDIT
I'm working with .NET 3.5 and I can't upgrade to .NET 4/4.5, because this is part of a legacy system.
TcpClient has a very useful method GetStream. I would use it.
var buf = new byte[0x10000];
var stream = tcpClient.GetStream();
int len = await stream.ReadAsync(buf, 0, buf.Length);
while (len > 0)
{
//your work ....
len = await stream.ReadAsync(buf, 0, buf.Length);
}
If you are getting strings separated by newline chars, then you can also use this
var stream = new StreamReader(tcpClient.GetStream());
string line = await stream.ReadLineAsync();
while (line!=null)
{
//...
line = await stream.ReadLineAsync();
}
EDIT
but only the first chunk of data is read It is because BeginReceive doesn't mean you'll get a callback for every data you receive. You should call it everytime you receive your data (for ex, in ReceivedData method)
So I'm writing an application in where I need to steam data from a client to my server and I've noticed some very weird behavior with how the NetworkStream.Read function behaves. I'm sending packets of about 18 bytes in length and if I make the while loop in the server Read and then Write then I get all of the packets properly. However, if I only Read and never Write then the Read does not return after a single packet. Instead, it gathers several packets before returning, saying that it received some 300+ bytes at each iteration. Does anybody know why I'm getting this behavior? I could simply write back 'a' every time but this seems a bit ghetto.
Here is the section of code:
private static void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
byte[] message = new byte[4096];
int bytesRead;
int count = 0;
while (true)
{
bytesRead = 0;
try
{
// Block until a client sends a message
bytesRead = clientStream.Read(message, 0, 4096);
}
catch
{
// A socket error has occured
break;
}
if (bytesRead == 0)
{
// The client has disconnected from the server
break;
}
Debug.print("Number of bytes: {0:D}", bytesRead);
count++;
// Message has successfully been received. Echo back.
clientStream.Write(new byte[2], 0, 1);
}
Debug.print("Count: {0:F}", count);
tcpClient.Close();
}
I have two programs both written in c# and one sends the following by using networkStream.beginWrite command:
1,2,3,4,5......200,201,202...(some termination byte array)
Now, I have the other program get the bytes and the first thing it reads for some reason is:
197, 198, 199....(termination array)
My question is, why is my TCP beginWrite sending it (the buffer) out of order?
Also, for some background information, I am reading it on the other side with beginReceive. Also, the byte array I am sending is 30000 bytes long and I am reading it into a 1024 byte buffer in the other computer. Would it be an issue if I do this and split it using the termination array?
Here is my send command on computer 1:
public bool sendToServer(SocketAsyncEventArgs e, params byte[][] buffer)
{
int length = 0;
foreach (byte[] bytes in buffer)
{
length += bytes.Length;
}
byte[] buffer2 = new byte[length + 5];
int index = 0;
foreach (byte[] bytes in buffer)
{
Buffer.BlockCopy(bytes, 0, buffer2, index, bytes.Length);
index += bytes.Length;
}
byte[] eof = { 60, 69, 79, 70, 62 };
Buffer.BlockCopy(eof, 0, buffer2, index, 5);
// netStream.Write(buffer2, 0, buffer2.Length);
netStream.BeginWrite(buffer2, 0, buffer2.Length, new AsyncCallback(SendCallback), clientSocket);
//socketEventArg.SetBuffer(buffer2, 0, buffer2.Length);
//Socket sock = socketEventArg.UserToken as Socket;
//bool willRaiseEvent = sock.SendAsync(socketEventArg);
Console.WriteLine("Sending: " + buffer2.Length + " bytes of data");
//foreach (byte bi in buffer2)
{
// Console.WriteLine(bi);
}
// clientSocket.BeginSend(buffer2, 0, buffer2.Length, 0,
// new AsyncCallback(SendCallback), clientSocket);
return true;
}
Here is my code to receive that data:
public void AcceptCallback(IAsyncResult ar)
{
// Get the socket that handles the client request.
Socket listener = (Socket)ar.AsyncState;
Socket handler = listener.EndAccept(ar);
Console.WriteLine("Connected!");
// Create the state object.
MonkeyObject state = new MonkeyObject();
state.workSocket = handler;
MonkeyObjects.Add(state);
listener.BeginAccept(
new AsyncCallback(AcceptCallback),
listener);
byte[] buffer = new byte[1024];
state.currentBuffer = buffer;
handler.BeginReceive(buffer, 0, MonkeyObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
You may want to check the order of the packets on the network using a sniffer like Wireshark, just to be sure the code is doing what you think it's doing.
But TCP assigns a sequence number to each packet as it's sent, and the receiver uses the sequence numbers to reassemble the message in the proper order. The packets don't need to arrive in the proper order, and in fact they sometimes don't.
One reason (as you show no code) is you are using same buffer for multiple send commands and they overwrite data that still should be send. In this case you may send semi-random data (depending when you update buffer and when actual send starts).