We Keep Coding

Socket.EndReceive of IAsyncResult is collecting more TCP messages than one

I have a situation here.
I'm using System.Net.Sockets.Socket to read and send TCP messages.
Using the recursion for receiving the data and reading again new data.
void rcvTCP(IAsyncResult ar)
{
var socket = (Socket)ar.AsyncState;
try
{
var bytesRead = socket.EndReceive(ar);
if (bytesRead > 0)
{
var data = new byte[bytesRead];
Array.Copy(this.mBuffer, data, data.Length);
dataReceived(this, mMaster.SlaveAddress, data);
}
socket.BeginReceive(mBuffer, 0, mBuffer.Length, SocketFlags.None, new AsyncCallback(rcvTCP), socket);
}
catch (Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
I'm sending messages from a device to my app, each 10ms, 8 bytes each message. I'm receiving the messages ok, so the "bytesRead" is 8, until a certain point when the app freezes randomly, and "bytesRead" is 768. When I look inside the data that came, I see that I have 96 messages in one.
I've read in the internet, like the messages are coming Sync instead of Async, so checking the "CompletedSinchronously" gives true at previous messages and this message also...
if (ar.CompletedSynchronously)
{
Debugger.Break();
}
I tried with TcpClient + NetworkStream instead of Socket(I know that TcpClient is pretty the same as Socket), and I have the same result.
Please help me. I want all the messages separated instead of collected, but I can't find any solution to this...
Any idea how to do this?
PS: I am already putting the flag - NoDelay to true.
FIX: For me the fix was splitting the collected messages after the receive, and running the splitting in a separate thread, so the UI does not freeze. Keep in mind, if you have a property used in different Threads, don't forget to use "lock".

Async Socket reading multiple messages as one

In my "client" i have the following piece of code:
socket.Send(ASCIIEncoding.UTF8.GetBytes("Test0"), SocketFlags.None);
socket.Send(ASCIIEncoding.UTF8.GetBytes("Test1"), SocketFlags.None);
socket.Send(ASCIIEncoding.UTF8.GetBytes("Test2"), SocketFlags.None);
In my "server" i have the following:
public void ReadCallback(IAsyncResult ar)
{
StateObject state = (StateObject)ar.AsyncState;
Socket handler = state.WorkSocket;
int read = handler.EndReceive(ar);
if (read > 0)
{
string aux = Encoding.ASCII.GetString(state.buffer, 0, read);
//Rest of the code...
My problem is: aux is "Test0Test1Test2"... I was expecting that my ReadCallback would be called 3 times, one for each send... but its being called only once... what do i have to do to make the readcallback behave as i expected?

You seem to be using a stream-type socket (e.g. TCP as opposed to UDP).
Stream sockets are streams - the receiver can't necessarily tell whether the message was sent with multiple sends or in a single send.
If you need messages to be separated you need to either use a packet-oriented or message-oriented protocol or implement your own method of showing where messages begin and end.

I'll just echo what Ben said, and try to add a tiny bit of additional info. I ran into the same situation 5-6 years ago, and did as much reasearch as I could, and concluded that "that's the way it is". Here's the only "reference" I have that more-or-less confirms that:
http://www.xtremevbtalk.com/archive/index.php/t-204348.html
So what I did was prefix each packet with the length of the packet, and "parse" the received packets at the other end.

I was using a TCPclient socket type and had this problem. It has to do with the Nagle algorithm. I was able to solve the problem by setting the NoDelay property to true, which allows the data to be sent immediately even if the buffer is not yet full.
Edit: After reading the question again, I am not sure you have the same problem that I did. I was sending commands to a device then reading response data. Is this what you are trying to do, or are you trying to see what was sent?

.NET Async Server receiving data for no apparent reason

I am totally confused right now.
Edit: Okay, nevermind. The Python socket as well is starting to do it now.
Edit 2: Well, not quite sure if this is causing high CPU usage, but something randomly is. Is there an efficient way to figure out what is causing spikes in the usage? This project is a bit large and has various threads.
I have an asynchronous server that listens and waits for incoming connections, then keeps them alive and waits for the socket to flush and give the server data. It is only closed when the user wants the socket to be closed.
However, whenever I let a socket & stream stay connected, it starts to go haywire and starts sending empty data on an endless loop... it may take anywhere from 15 seconds to over a minute before it starts going wack. If I let it go for a really long time, it starts to cause really high CPU usage.
Aside from the high CPU usage, oddly enough, everything works as it should; messages are sent & received fine.
This is my read callback function:
protected void ReadCallback(IAsyncResult ar)
{
StateObject state = (StateObject)ar.AsyncState;
Socket handler = state.SocketHandle;
try
{
int bytesRead = (state.BytesRead += handler.EndReceive(ar)), offset = 0;
string line = m_Encoder.GetString(state.Buffer, 0, bytesRead);
if ( state.Buddy != null )
Console.WriteLine(state.Buddy.Address);
if (bytesRead > 0)
{
Console.WriteLine("!!!");
/* A complete request? */
if (line.EndsWith("\n") || line.EndsWith("\x00"))
{
string[] lines = line.Split('\n'); // ... *facepalm*
foreach (string ln in lines)
this.MessageReceieved(ln, state);
state.Buffer = new byte[StateObject.BUFFER_SIZE];
state.BytesRead = 0; // reset
}
/* Incomplete; resize the array to accommodate more data... */
else
{
offset = bytesRead;
Array.Resize<byte>(ref state.Buffer, bytesRead + StateObject.BUFFER_SIZE);
}
}
if (handler != null && handler.Connected )
handler.BeginReceive(state.Buffer, offset, state.Buffer.Length - offset, SocketFlags.None, new AsyncCallback(ReadCallback), state);
}
catch (SocketException)
{
if (state.Buddy != null)
state.Buddy.Kill();
else
handler.Close();
}
}
I know this is somehow caused by calling BeginReceive, but I don't know how else to keep the connection alive.

There is nothing in that code that can make it go haywire.
I do see some problems though.
Connection detection
No need to check if the socket is connected. You can detect disconnections in two ways in the receive callback:
Zero bytes is returned by EndReceive
An exception is being thrown.
I would recommend that the first thing you do after EndReceive is to check the return value and handle disconnect accordingly. It makes the code clearer.
Your code will currently do nothing if 0 bytes are received. The handler will just stop receiving and still think that the connection is open.
Buffer handling
Your buffer handling is very inneffecient. Do not resize the buffer every time. It will slow your server down a lot. Allocate a large buffer from start.
String handling
Don't build a string every time you receive something. Check inside the byte buffer after new line or null instead. THEN build a string, and only make it as large as needed. You might receive more bytes that just a message (for instance one and a half message)

Send Message to Net (Similar to net send)

Im not sure whether this topic have been disscussed or not, but i have no idea what to do. In c#, i need to send message to other people in my networks. I made some researches, and all pointing to socket methods. With this method, logically it will work like this:
open up socket
Encode the message(to ascii for example)
Put the encoded bytes in an array
Send over the array
On the other side there's also a socket opened, and then it's the same thing in reverse order.
So i found this code, Can anyone help me understanding the code, which are similar on what i want:
IPEndPoint ip = new IPEndPoint(IPAddress.Any,9999);
//no.1
Socket socket = new Socket(AddressFamily.InterNetwork,SocketType.Stream, ProtocolType.Tcp);
socket.Bind(ip);
socket.Listen(10);
Console.WriteLine("Waiting for a client...");
Socket client = socket.Accept();
IPEndPoint clientep =(IPEndPoint)client.RemoteEndPoint;
Console.WriteLine("Connected with {0} at port {1}",clientep.Address, clientep.Port);
//no.2 & 3
string welcome = "Welcome";
byte[] data = new byte[1024];
data = Encoding.ASCII.GetBytes(welcome);
//no.4
client.Send(data, data.Length,SocketFlags.None);
while(true)
{
data = new byte[1024];
int receivedDataLength = client.Receive(data);
Console.WriteLine(Encoding.ASCII.GetString(data, 0, receivedDataLength));
client.Send(data, receivedDataLength, SocketFlags.None);
}
Console.WriteLine("Disconnected from {0}",clientep.Address);
client.Close();
socket.Close();
I manage to understand what the code do until point no 4. but this code:
while(true)
{
data = new byte[1024];
int receivedDataLength = client.Receive(data);
Console.WriteLine(Encoding.ASCII.GetString(data, 0, receivedDataLength));
client.Send(data, receivedDataLength, SocketFlags.None);
}
I'm not sure what the above codes do?
and for the point no 5, can someone explain in to me how to do that? is that mean, that i need another program as a receiver?
~Thank you~

The code you're looking at is for an echo server, which reads some data sent to a socket by another client, prints it to the console, and then sends it back to the client that it came from. This is what the code in point 5 does. Here's what is going on:
data = new byte[1024];
this is just allocating a buffer to hold the data that we're going to read.
int receivedDataLength = client.Receive(data);
this receives data from the socket. Receive is a blocking call, so it won't return until there is data to be read OR until the socket is disconnected or has an error, at which point it throws. The recievedDataLength variable is EXTREMELY important because this tells us how many bytes we actually read. The number of bytes read may easily be LESS than the size of the buffer, because we don't know how many bytes were sent to the socket by the other client. By detecting how many bytes were read, we can be sure that we know how much data is actually in our buffer. If this returns 0, it means that the connection was closed by the other side.
Console.WriteLine(Encoding.ASCII.GetString(data, 0, receivedDataLength));
This just converts the buffer from arbitrary byte values to an ASCII string. Note that receivedDataLength allows us to only write out a string for the number of bytes that were read from the socket. If we just wrote the contents of the entire buffer, you'd get random crap on the console since any data present in the buffer that occurs AFTER receivedDataLenth is unknown.
client.Send(data, receivedDataLength, SocketFlags.None);
Finally, we're sending the data back to the other end of the connection by writing it to the socket. We use receivedDatalength to indicate that we only want to send the bytes that we actually read instead of the whole buffer. If you were sending a different message here, you'd want to use the length in bytes of the message that you're sending. You'll most likely never need to set any SocketFlags so don't worry about these.
Now this code here is for a server that wants to receive connections. To make a connection, the code is very similar, except after you call Socket.Bind(ip) you'll call Socket.Connect(remotehost) instead of Listen() and Accept(). Then the code to write data to the Socket is the same as in step 5 above.
One last thing: the Socket.Accept() method is non-intuitive at first so I'll explain what it's doing. Calling Listen() tells the socket to allow incoming connections, but in order to actually use those connections for something, you have to accept them. Accept() blocks until a new connection is made, then it returns a NEW socket object. This socket is similar to the socket that you're listening on, except that this socket represents a socket-pair, meaning that on the other end of the connection there is another similar socket that is going to talk to this one. You should use this new socket to talk to the client that has connected.
Once you've gotten the new Socket from Accept(), you still have the old socket, which is the one you originally called Accept() on. This socket is still listening for new connections, so if you want to allow more machines to connect to this socket, you need to keep calling Accept() until you are no longer willing to accept connections (like if you don't want there to be more than some arbitrary number of connections) or until you want to shut down the application. Now I said before that Accept() is a blocking call, so if you call it again it will not allow the thread you're on to do anything else until a new connection comes in. This means that you need to do one of two things: Either call BeginAccept to accept new clients asynchronously, OR use BeginRead or BeginSend to handle your socket I/O asynchronously. You CANNOT do both of these things on the same thread. It's a bit harder to do this, but check out the IPv6Sockets sample on MSDN (http://msdn.microsoft.com/en-us/library/8sd71sfs(VS.80).aspx) and you can see how this is done. Remember, it is absolutely necessary that you do NOT try to accept new connections and perform I/O on existing connections on the same thread. I actually recommend that you do everything with your sockets asynchronously.
If you don't want to have to use all this socket stuff, you can also check out the TcpClient class. It's a thin wrapper around Socket that allows you to grab a Stream object from the TcpClient and just read/write from that stream. It's way easier to use but it's not as powerful as a Socket, although for your application I doubt you'll need to use anything fancy with Sockets anyway so TcpClient is what I would use.

It looks like it's sitting in a loop receiving data and writing it to the console, and echoing back to the sender.

C# Begin/EndReceive - how do I read large data?

When reading data in chunks of say, 1024, how do I continue to read from a socket that receives a message bigger than 1024 bytes until there is no data left? Should I just use BeginReceive to read a packet's length prefix only, and then once that is retrieved, use Receive() (in the async thread) to read the rest of the packet? Or is there another way?
edit:
I thought Jon Skeet's link had the solution, but there is a bit of a speedbump with that code. The code I used is:
public class StateObject
{
public Socket workSocket = null;
public const int BUFFER_SIZE = 1024;
public byte[] buffer = new byte[BUFFER_SIZE];
public StringBuilder sb = new StringBuilder();
}
public static void Read_Callback(IAsyncResult ar)
{
StateObject so = (StateObject) ar.AsyncState;
Socket s = so.workSocket;
int read = s.EndReceive(ar);
if (read > 0)
{
so.sb.Append(Encoding.ASCII.GetString(so.buffer, 0, read));
if (read == StateObject.BUFFER_SIZE)
{
s.BeginReceive(so.buffer, 0, StateObject.BUFFER_SIZE, 0,
new AyncCallback(Async_Send_Receive.Read_Callback), so);
return;
}
}
if (so.sb.Length > 0)
{
//All of the data has been read, so displays it to the console
string strContent;
strContent = so.sb.ToString();
Console.WriteLine(String.Format("Read {0} byte from socket" +
"data = {1} ", strContent.Length, strContent));
}
s.Close();
}
Now this corrected works fine most of the time, but it fails when the packet's size is a multiple of the buffer. The reason for this is if the buffer gets filled on a read it is assumed there is more data; but the same problem happens as before. A 2 byte buffer, for exmaple, gets filled twice on a 4 byte packet, and assumes there is more data. It then blocks because there is nothing left to read. The problem is that the receive function doesn't know when the end of the packet is.
This got me thinking to two possible solutions: I could either have an end-of-packet delimiter or I could read the packet header to find the length and then receive exactly that amount (as I originally suggested).
There's problems with each of these, though. I don't like the idea of using a delimiter, as a user could somehow work that into a packet in an input string from the app and screw it up. It also just seems kinda sloppy to me.
The length header sounds ok, but I'm planning on using protocol buffers - I don't know the format of the data. Is there a length header? How many bytes is it? Would this be something I implement myself? Etc..
What should I do?

No - call BeginReceive again from the callback handler, until EndReceive returns 0. Basically, you should keep on receiving asynchronously, assuming you want the fullest benefit of asynchronous IO.
If you look at the MSDN page for Socket.BeginReceive you'll see an example of this. (Admittedly it's not as easy to follow as it might be.)

Dang. I'm hesitant to even reply to this given the dignitaries that have already weighed in, but here goes. Be gentle, O Great Ones!
Without having the benefit of reading Marc's blog (it's blocked here due the corporate internet policy), I'm going to offer "another way."
The trick, in my mind, is to separate the receipt of the data from the processing of that data.
I use a StateObject class defined like this. It differs from the MSDN StateObject implementation in that it does not include the StringBuilder object, the BUFFER_SIZE constant is private, and it includes a constructor for convenience.
public class StateObject
{
private const int BUFFER_SIZE = 65535;
public byte[] Buffer = new byte[BUFFER_SIZE];
public readonly Socket WorkSocket = null;
public StateObject(Socket workSocket)
{
WorkSocket = workSocket;
}
}
I also have a Packet class that is simply a wrapper around a buffer and a timestamp.
public class Packet
{
public readonly byte[] Buffer;
public readonly DateTime Timestamp;
public Packet(DateTime timestamp, byte[] buffer, int size)
{
Timestamp = timestamp;
Buffer = new byte[size];
System.Buffer.BlockCopy(buffer, 0, Buffer, 0, size);
}
}
My ReceiveCallback() function looks like this.
public static ManualResetEvent PacketReceived = new ManualResetEvent(false);
public static List<Packet> PacketList = new List<Packet>();
public static object SyncRoot = new object();
public static void ReceiveCallback(IAsyncResult ar)
{
try {
StateObject so = (StateObject)ar.AsyncState;
int read = so.WorkSocket.EndReceive(ar);
if (read > 0) {
Packet packet = new Packet(DateTime.Now, so.Buffer, read);
lock (SyncRoot) {
PacketList.Add(packet);
}
PacketReceived.Set();
}
so.WorkSocket.BeginReceive(so.Buffer, 0, so.Buffer.Length, 0, ReceiveCallback, so);
} catch (ObjectDisposedException) {
// Handle the socket being closed with an async receive pending
} catch (Exception e) {
// Handle all other exceptions
}
}
Notice that this implementation does absolutely no processing of the received data, nor does it have any expections as to how many bytes are supposed to have been received. It simply receives whatever data happens to be on the socket (up to 65535 bytes) and stores that data in the packet list, and then it immediately queues up another asynchronous receive.
Since processing no longer occurs in the thread that handles each asynchronous receive, the data will obviously be processed by a different thread, which is why the Add() operation is synchronized via the lock statement. In addition, the processing thread (whether it is the main thread or some other dedicated thread) needs to know when there is data to process. To do this, I usually use a ManualResetEvent, which is what I've shown above.
Here is how the processing works.
static void Main(string[] args)
{
Thread t = new Thread(
delegate() {
List<Packet> packets;
while (true) {
PacketReceived.WaitOne();
PacketReceived.Reset();
lock (SyncRoot) {
packets = PacketList;
PacketList = new List<Packet>();
}
foreach (Packet packet in packets) {
// Process the packet
}
}
}
);
t.IsBackground = true;
t.Name = "Data Processing Thread";
t.Start();
}
That's the basic infrastructure I use for all of my socket communication. It provides a nice separation between the receipt of the data and the processing of that data.
As to the other question you had, it is important to remember with this approach that each Packet instance does not necessarily represent a complete message within the context of your application. A Packet instance might contain a partial message, a single message, or multiple messages, and your messages might span multiple Packet instances. I've addressed how to know when you've received a full message in the related question you posted here.

You would read the length prefix first. Once you have that, you would just keep reading the bytes in blocks (and you can do this async, as you surmised) until you have exhausted the number of bytes you know are coming in off the wire.
Note that at some point, when reading the last block you won't want to read the full 1024 bytes, depending on what the length-prefix says the total is, and how many bytes you have read.

Also I troubled same problem.
When I tested several times, I found that sometimes multiple BeginReceive - EndReceive makes packet loss. (This loop was ended improperly)
In my case, I used two solution.
First, I defined the enough packet size to make only 1 time BeginReceive() ~ EndReceive();
Second, When I receive large size of data, I used NetworkStream.Read() instead of BeginReceive() - EndReceive().
Asynchronous socket is not easy to use, and it need a lot of understanding about socket.

For info (general Begin/End usage), you might want to see this blog post; this approach is working OK for me, and saving much pain...

There seems to be a lot of confusion surrounding this. The examples on MSDN's site for async socket communication using TCP are misleading and not well explained. The EndReceive call will indeed block if the message size is an exact multiple of the receive buffer. This will cause you to never get your message and the application to hang.
Just to clear things up - You MUST provide your own delimiter for data if you are using TCP. Read the following (this is from a VERY reliable source).
The Need For Application Data
Delimiting
The other impact of TCP treating
incoming data as a stream is that data
received by an application using TCP
is unstructured. For transmission, a
stream of data goes into TCP on one
device, and on reception, a stream of
data goes back to the application on
the receiving device. Even though the
stream is broken into segments for
transmission by TCP, these segments
are TCP-level details that are hidden
from the application. So, when a
device wants to send multiple pieces
of data, TCP provides no mechanism for
indicating where the “dividing line”
is between the pieces, since TCP
doesn't examine the meaning of the
data at all. The application must
provide a means for doing this.
Consider for example an application
that is sending database records. It
needs to transmit record #579 from the
Employees database table, followed by
record #581 and record #611. It sends
these records to TCP, which treats
them all collectively as a stream of
bytes. TCP will package these bytes
into segments, but in a manner the
application cannot predict. It is
possible that each will end up in a
different segment, but more likely
they will all be in one segment, or
part of each will end up in different
segments, depending on their length.
The records themselves must have some
sort of explicit markers so the
receiving device can tell where one
record ends and the next starts.
Source: http://www.tcpipguide.com/free/t_TCPDataHandlingandProcessingStreamsSegmentsandSequ-3.htm
Most examples I see online for using EndReceive are wrong or misleading. It usually causes no problems in the examples because only one predefined message is sent and then the connection is closed.

This a very old topic, but I got here looking for something else and found this:
Now this corrected works fine most of the time, but it fails when the packet's size is a multiple of the buffer. The reason for this is if the buffer gets filled on a read it is assumed there is more data; but the same problem happens as before. A 2 byte buffer, for exmaple, gets filled twice on a 4 byte packet, and assumes there is more data. It then blocks because there is nothing left to read. The problem is that the receive function doesn't know when the end of the packet is.
I had this same problem, and since none of the replies seems to solve this, the way I did it was using Socket.Available
public static void Read_Callback(IAsyncResult ar)
{
StateObject so = (StateObject) ar.AsyncState;
Socket s = so.workSocket;
int read = s.EndReceive(ar);
if (read > 0)
{
so.sb.Append(Encoding.ASCII.GetString(so.buffer, 0, read));
if (s.Available == 0)
{
// All data received, process it as you wish
}
}
// Listen for more data
s.BeginReceive(so.buffer, 0, StateObject.BUFFER_SIZE, 0,
new AyncCallback(Async_Send_Receive.Read_Callback), so);
}
Hope this helps others, SO have helped me many times, thank you all!