We Keep Coding

UDP Sockets aren't receiving all the packets

At the moment I'm working on a project where I have to build an RTP client and server with the .NET via C#. The current question concern the the client side.
I created a UDP Socket like the following for listening to the server :
Socket RTPSocket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
I also created an array of bytes to store received bytes :
byte[] RTPBuffer = new byte[1500];
The size is at 1500 because after packet analysis with wireshark i saw that the max packet size sent by VLC was 1428. After that i created a thread for the listening process, this thread contains the following instructions :
while(!Disposing)
{
RTPSocket.Receive(RTPBuffer);
OnRTPDataReceived(RTPBuffer);
}
The OnRTPDataReceived instruction raise an event wich is handled by the following method :
public void RTPPacketReceived(object sender, byte[] packet)
{
PacketQueue.Enqueue(packet);
}
Finally the bytes array contained in the PacketQueue Queue are processed by another thread but this is off topic.
The problem is that my socket is sometimes receiving one packet out of two and sometines one out of three and some other times every packets. In fact it's random.
I looked to this kind of problem in different forums but I didn't find any solutions to resolve my problem.
I also tried to set a better priority to the listening thread and a lower one to the others, didn't worked. Last thing I tried was reduce the number of instruction after receiving data to see if the process duration was the cause of the packet loss but this too didn't worked.
Thanks in advance for your time !

Async Socket reading multiple messages as one

In my "client" i have the following piece of code:
socket.Send(ASCIIEncoding.UTF8.GetBytes("Test0"), SocketFlags.None);
socket.Send(ASCIIEncoding.UTF8.GetBytes("Test1"), SocketFlags.None);
socket.Send(ASCIIEncoding.UTF8.GetBytes("Test2"), SocketFlags.None);
In my "server" i have the following:
public void ReadCallback(IAsyncResult ar)
{
StateObject state = (StateObject)ar.AsyncState;
Socket handler = state.WorkSocket;
int read = handler.EndReceive(ar);
if (read > 0)
{
string aux = Encoding.ASCII.GetString(state.buffer, 0, read);
//Rest of the code...
My problem is: aux is "Test0Test1Test2"... I was expecting that my ReadCallback would be called 3 times, one for each send... but its being called only once... what do i have to do to make the readcallback behave as i expected?

You seem to be using a stream-type socket (e.g. TCP as opposed to UDP).
Stream sockets are streams - the receiver can't necessarily tell whether the message was sent with multiple sends or in a single send.
If you need messages to be separated you need to either use a packet-oriented or message-oriented protocol or implement your own method of showing where messages begin and end.

I'll just echo what Ben said, and try to add a tiny bit of additional info. I ran into the same situation 5-6 years ago, and did as much reasearch as I could, and concluded that "that's the way it is". Here's the only "reference" I have that more-or-less confirms that:
http://www.xtremevbtalk.com/archive/index.php/t-204348.html
So what I did was prefix each packet with the length of the packet, and "parse" the received packets at the other end.

I was using a TCPclient socket type and had this problem. It has to do with the Nagle algorithm. I was able to solve the problem by setting the NoDelay property to true, which allows the data to be sent immediately even if the buffer is not yet full.
Edit: After reading the question again, I am not sure you have the same problem that I did. I was sending commands to a device then reading response data. Is this what you are trying to do, or are you trying to see what was sent?

C# Socket.Receive message length

I'm currently in the process of developing a C# Socket server that can accept multiple connections from multiple client computers. The objective of the server is to allow clients to "subscribe" and "un-subscribe" from server events.
So far I've taken a jolly good look over here: http://msdn.microsoft.com/en-us/library/5w7b7x5f(v=VS.100).aspx and http://msdn.microsoft.com/en-us/library/fx6588te.aspx for ideas.
All the messages I send are encrypted, so I take the string message that I wish to send, convert it into a byte[] array and then encrypt the data before pre-pending the message length to the data and sending it out over the connection.
One thing that strikes me as an issue is this: on the receiving end it seems possible that Socket.EndReceive() (or the associated callback) could return when only half of the message has been received. Is there an easy way to ensure each message is received "complete" and only one message at a time?
EDIT: For example, I take it .NET / Windows sockets does not "wrap" the messages to ensure that a single message sent with Socket.Send() is received in one Socket.Receive() call? Or does it?
My implementation so far:
private void StartListening()
{
IPHostEntry ipHostInfo = Dns.GetHostEntry(Dns.GetHostName());
IPEndPoint localEP = new IPEndPoint(ipHostInfo.AddressList[0], Constants.PortNumber);
Socket listener = new Socket(localEP.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
listener.Bind(localEP);
listener.Listen(10);
while (true)
{
// Reset the event.
this.listenAllDone.Reset();
// Begin waiting for a connection
listener.BeginAccept(new AsyncCallback(this.AcceptCallback), listener);
// Wait for the event.
this.listenAllDone.WaitOne();
}
}
private void AcceptCallback(IAsyncResult ar)
{
// Get the socket that handles the client request.
Socket listener = (Socket) ar.AsyncState;
Socket handler = listener.EndAccept(ar);
// Signal the main thread to continue.
this.listenAllDone.Set();
// Accept the incoming connection and save a reference to the new Socket in the client data.
CClient client = new CClient();
client.Socket = handler;
lock (this.clientList)
{
this.clientList.Add(client);
}
while (true)
{
this.readAllDone.Reset();
// Begin waiting on data from the client.
handler.BeginReceive(client.DataBuffer, 0, client.DataBuffer.Length, 0, new AsyncCallback(this.ReadCallback), client);
this.readAllDone.WaitOne();
}
}
private void ReadCallback(IAsyncResult asyn)
{
CClient theClient = (CClient)asyn.AsyncState;
// End the receive and get the number of bytes read.
int iRx = theClient.Socket.EndReceive(asyn);
if (iRx != 0)
{
// Data was read from the socket.
// So save the data
byte[] recievedMsg = new byte[iRx];
Array.Copy(theClient.DataBuffer, recievedMsg, iRx);
this.readAllDone.Set();
// Decode the message recieved and act accordingly.
theClient.DecodeAndProcessMessage(recievedMsg);
// Go back to waiting for data.
this.WaitForData(theClient);
}
}

Yes, it is possible you'll have only part of message per one receiving, also it can be even worse during transfer only part of message will be sent. Usually you can see that during bad network conditions or under heavy network load.
To be clear on network level TCP guaranteed to transfer your data in specified order but it not guaranteed that portions of data will be same as you sent. There are many reasons for that software (take a look to Nagle's algorithm for example), hardware (different routers in trace), OS implementation, so in general you should never assume what part of data already transferred or received.
Sorry for long introduction, below some advices:
Try to use relatevely "new" API for high-performance socket server, here samples Networking Samples for .NET v4.0
Do not assume you always send full packet. Socket.EndSend() returns number of bytes actually scheduled to send, it can be even 1-2 bytes under heavy network load. So you have to implement resend rest part of buffer when it required.
There is warning on MSDN:
There is no guarantee that the data
you send will appear on the network
immediately. To increase network
efficiency, the underlying system may
delay transmission until a significant
amount of outgoing data is collected.
A successful completion of the
BeginSend method means that the
underlying system has had room to
buffer your data for a network send.
Do not assume you always receive full packet. Join received data in some kind of buffer and analyze it when it have enough data.
Usually, for binary protocols, I add field to indicate how much data incoming, field with message type (or you can use fixed length per message type (generally not good, e.g. versioning problem)), version field (where applicable) and add CRC-field to end of message.
It not really required to read, a bit old and applies directly to Winsock but maybe worth to study: Winsock Programmer's FAQ
Take a look to ProtocolBuffers, it worth to learn: http://code.google.com/p/protobuf-csharp-port/, http://code.google.com/p/protobuf-net/
Hope it helps.
P.S. Sadly sample on MSDN you refer in question effectively ruin async paradigm as stated in other answers.

Your code is very wrong. Doing loops like that defeats the purpose of asynchronous programming. Async IO is used to not block the thread but let them continue doing other work. By looping like that, you are blocking the thread.
void StartListening()
{
_listener.BeginAccept(OnAccept, null);
}
void OnAccept(IAsyncResult res)
{
var clientSocket = listener.EndAccept(res);
//begin accepting again
_listener.BeginAccept(OnAccept, null);
clientSocket.BeginReceive(xxxxxx, OnRead, clientSocket);
}
void OnReceive(IAsyncResult res)
{
var socket = (Socket)res.Asyncstate;
var bytesRead = socket.EndReceive(res);
socket.BeginReceive(xxxxx, OnReceive, socket);
//handle buffer here.
}
Note that I've removed all error handling to make the code cleaner. That code do not block any thread and is therefore much more effecient. I would break the code up in two classes: the server handling code and the client handling code. It makes it easier to maintain and extend.
Next thing to understand is that TCP is a stream protocol. It do not guarentee that a message arrives in one Receive. Therefore you must know either how large a message is or when it ends.
The first solution is to prefix each message with an header which you parse first and then continue reading until you get the complete body/message.
The second solution is to put some control character at the end of each message and continue reading until the control character is read. Keep in mind that you should encode that character if it can exist in the actual message.

You need to send fixed length messages or include in the header the length of the message. Try to have something that allows you to clearly identify the start of a packet.

I found 2 very useful links:
http://vadmyst.blogspot.com/2008/03/part-2-how-to-transfer-fixed-sized-data.html
C# Async TCP sockets: Handling buffer size and huge transfers

C# Begin Send within a foreach loop issue

I have a group of "Packets" which are custom classed that are coverted to byte[] and then sent to the client. When a client joins, they are updated with the previous "Catch Up Packets" that were sent previous to the user joining. Think of it as a chat room where you are updated with the previous conversations.
My issue is on the client end, we do not receive all the information; Sometimes not at all..
Below is pseudo c# code for what I see
code looks like this.
lock(CatchUpQueue.SyncRoot)
{
foreach(Packet packet in CatchUpQueue)
{
// If I put Console.WriteLine("I am Sending Packets"); It will work fine up to (2) client sockets else if fails again.
clientSocket.BeginSend(data, 0, data.length, SocketFlags.None, new AsyncCallback(EndSend), data);
}
}
Is this some sort of throttle issue or an issue with sending to many times: ie: if there are 4 packets in the queue then it calls begin send 4 times.
I have searched for a topic similiar and I cannot find one. Thank you for your help.
Edit: I would also like to point out that the sending between clients continues normally for any sends after the client connects. But for some reason the packets within this for loop are not all sent.

I would suspect that you are flooding the TCP port with packets, and probably overflowing its send buffer, at which point it will probably return errors rather than sending the data.
The idea of Async I/O is not to allow you to send an infinite amount of data packets simultaneously, but to allow your foreground thread to continue processing while a linear sequence of one or more I/O operations occurs in the background.
As the TCP stream is a serial stream, try respecting that and send each packet in turn. That is, after BeginSend, use the Async callback to detect when the Send has completed before you send again. You are effectively doing this by adding a Sleep, but this is not a very good solution (you will either be sending packets more slowly than possible, or you may not sleep for long enough and packets will be lost again)
Or, if you don't need the I/O to run in the background, use your simple foreach loop, but use a synchronous rather than Async send.

Okay,
Apparently a fix, so far still has me confused, is to Thread.Sleep for the number of ms for each packet I am sending.
So...
for(int i = 0; i < PacketQueue.Count; i++)
{
Packet packet = PacketQueue[i];
clientSocket.BeginSend(data, 0, data.length, SocketFlags.None, new AsyncCallback(EndSend), data);
Thread.Sleep(PacketQueue.Count);
}
I assume that for some reason the loop stops some of the calls from happening... Well I will continue to work with this and try to find the real answer.

Send Message to Net (Similar to net send)

Im not sure whether this topic have been disscussed or not, but i have no idea what to do. In c#, i need to send message to other people in my networks. I made some researches, and all pointing to socket methods. With this method, logically it will work like this:
open up socket
Encode the message(to ascii for example)
Put the encoded bytes in an array
Send over the array
On the other side there's also a socket opened, and then it's the same thing in reverse order.
So i found this code, Can anyone help me understanding the code, which are similar on what i want:
IPEndPoint ip = new IPEndPoint(IPAddress.Any,9999);
//no.1
Socket socket = new Socket(AddressFamily.InterNetwork,SocketType.Stream, ProtocolType.Tcp);
socket.Bind(ip);
socket.Listen(10);
Console.WriteLine("Waiting for a client...");
Socket client = socket.Accept();
IPEndPoint clientep =(IPEndPoint)client.RemoteEndPoint;
Console.WriteLine("Connected with {0} at port {1}",clientep.Address, clientep.Port);
//no.2 & 3
string welcome = "Welcome";
byte[] data = new byte[1024];
data = Encoding.ASCII.GetBytes(welcome);
//no.4
client.Send(data, data.Length,SocketFlags.None);
while(true)
{
data = new byte[1024];
int receivedDataLength = client.Receive(data);
Console.WriteLine(Encoding.ASCII.GetString(data, 0, receivedDataLength));
client.Send(data, receivedDataLength, SocketFlags.None);
}
Console.WriteLine("Disconnected from {0}",clientep.Address);
client.Close();
socket.Close();
I manage to understand what the code do until point no 4. but this code:
while(true)
{
data = new byte[1024];
int receivedDataLength = client.Receive(data);
Console.WriteLine(Encoding.ASCII.GetString(data, 0, receivedDataLength));
client.Send(data, receivedDataLength, SocketFlags.None);
}
I'm not sure what the above codes do?
and for the point no 5, can someone explain in to me how to do that? is that mean, that i need another program as a receiver?
~Thank you~

The code you're looking at is for an echo server, which reads some data sent to a socket by another client, prints it to the console, and then sends it back to the client that it came from. This is what the code in point 5 does. Here's what is going on:
data = new byte[1024];
this is just allocating a buffer to hold the data that we're going to read.
int receivedDataLength = client.Receive(data);
this receives data from the socket. Receive is a blocking call, so it won't return until there is data to be read OR until the socket is disconnected or has an error, at which point it throws. The recievedDataLength variable is EXTREMELY important because this tells us how many bytes we actually read. The number of bytes read may easily be LESS than the size of the buffer, because we don't know how many bytes were sent to the socket by the other client. By detecting how many bytes were read, we can be sure that we know how much data is actually in our buffer. If this returns 0, it means that the connection was closed by the other side.
Console.WriteLine(Encoding.ASCII.GetString(data, 0, receivedDataLength));
This just converts the buffer from arbitrary byte values to an ASCII string. Note that receivedDataLength allows us to only write out a string for the number of bytes that were read from the socket. If we just wrote the contents of the entire buffer, you'd get random crap on the console since any data present in the buffer that occurs AFTER receivedDataLenth is unknown.
client.Send(data, receivedDataLength, SocketFlags.None);
Finally, we're sending the data back to the other end of the connection by writing it to the socket. We use receivedDatalength to indicate that we only want to send the bytes that we actually read instead of the whole buffer. If you were sending a different message here, you'd want to use the length in bytes of the message that you're sending. You'll most likely never need to set any SocketFlags so don't worry about these.
Now this code here is for a server that wants to receive connections. To make a connection, the code is very similar, except after you call Socket.Bind(ip) you'll call Socket.Connect(remotehost) instead of Listen() and Accept(). Then the code to write data to the Socket is the same as in step 5 above.
One last thing: the Socket.Accept() method is non-intuitive at first so I'll explain what it's doing. Calling Listen() tells the socket to allow incoming connections, but in order to actually use those connections for something, you have to accept them. Accept() blocks until a new connection is made, then it returns a NEW socket object. This socket is similar to the socket that you're listening on, except that this socket represents a socket-pair, meaning that on the other end of the connection there is another similar socket that is going to talk to this one. You should use this new socket to talk to the client that has connected.
Once you've gotten the new Socket from Accept(), you still have the old socket, which is the one you originally called Accept() on. This socket is still listening for new connections, so if you want to allow more machines to connect to this socket, you need to keep calling Accept() until you are no longer willing to accept connections (like if you don't want there to be more than some arbitrary number of connections) or until you want to shut down the application. Now I said before that Accept() is a blocking call, so if you call it again it will not allow the thread you're on to do anything else until a new connection comes in. This means that you need to do one of two things: Either call BeginAccept to accept new clients asynchronously, OR use BeginRead or BeginSend to handle your socket I/O asynchronously. You CANNOT do both of these things on the same thread. It's a bit harder to do this, but check out the IPv6Sockets sample on MSDN (http://msdn.microsoft.com/en-us/library/8sd71sfs(VS.80).aspx) and you can see how this is done. Remember, it is absolutely necessary that you do NOT try to accept new connections and perform I/O on existing connections on the same thread. I actually recommend that you do everything with your sockets asynchronously.
If you don't want to have to use all this socket stuff, you can also check out the TcpClient class. It's a thin wrapper around Socket that allows you to grab a Stream object from the TcpClient and just read/write from that stream. It's way easier to use but it's not as powerful as a Socket, although for your application I doubt you'll need to use anything fancy with Sockets anyway so TcpClient is what I would use.

It looks like it's sitting in a loop receiving data and writing it to the console, and echoing back to the sender.