We Keep Coding

TCP packets out of order in Unity TCP client connection on Mac only

I have a Unity script which connects to a server using TCP. The server sends messages of arbitrary length. The length of the message is described in the first 4 bytes of the message, which I use to determine when the full message has arrived. For some reason, the message isn't being "stitched" together in the correct order. This problem, however, only occurs on Mac. The same script works perfectly fine on Windows (same version of Unity too). Any idea?
Unity version: 2018.4.19f
The connection code is as follows (I edited out things for simplicity of the question):
using System;
using System.Collections.Concurrent;
using System.Net.Sockets;
using System.Threading.Tasks;
using UnityEngine;
using UnityEngine.Events;
public class ServerConnection : MonoBehaviour {
// Example connection parameters
public string IP = "192.168.0.10";
public int Port = 8085;
/// The task object where TCP connection runs.
Task ServerTask { get; set; }
TcpClient Client { get; set; }
Socket Socket { get; set; }
string Message { get; set; }
/// Gets called on the push of a button
public void Connect() {
ServerTask = Task.Factory.StartNew(listenForDataAsync, TaskCreationOptions.LongRunning);
}
async private void listenForDataAsync() {
Debug.Log(string.Format("Connecting to server: IP={0}, Port={1}", IP, Port));
try {
Client = new TcpClient(IP, Port);
Debug.Log("Connection established");
byte[] message = new byte[0];
byte[] msgSize = new byte[4];
int messageSize = 0;
using (NetworkStream stream = Client.GetStream()) {
while (true) {
// Wait for 4 bytes to get message size
await stream.ReadAsync(msgSize, 0, 4);
// Convert message size byte array to an int
int totalMessageSize = BitConverter.ToInt32(msgSize, 0);
// subtract 4 from total message size (4 bytes previously read)
messageSize = totalMessageSize - 4;
// Wait for the rest of the message
message = new byte[messageSize];
int readBytes = 0;
while (messageSize != 0) {
readBytes = await stream.ReadAsync(message, readBytes, messageSize);
messageSize -= readBytes;
}
// Decode byte array to string
string response = System.Text.ASCIIEncoding.ASCII.GetString(message);
// On Mac, response has the message out of order.
// On Windows, it is always perfectly fine.
}
}
}
}
}
Edit: The server is implemented in C++ using Qt and is running on a Windows machine. The server sends responses that may contain large chunks of data. The entire message is composed of 4 bytes indicating the length of the whole message followed by the response itself. The response is a Json string. The error I'm getting is that the final string obtained from the code above is not ordered the same way as it was when it was sent. The message ends in } (closing the outer Json object). However, in the received response, after the }, there are tons of other values that were supposed to be part of an an array inside the Json object. To give you an example, take the following response sent by the server:
{data: [0, 1, 2, 3, 4, 5, 6, 7]}
The code above gets something like:
{data: [0, 1, 2]}, 3, 4, 5, 7
This only happens when the responses are starting to hit the kilobytes range (and higher), and I emphasize, the exact same code works perfectly fine on Windows.
The server sends the message in the following way:
// The response in Json
QJsonObject responseJson;
// Convert to byte array
QJsonDocument doc(responseJson);
QByteArray message = doc.toJson(QJsonDocument::JsonFormat::Compact);
// Insert 4 bytes indicating length of message
qint32 messageSize = sizeof(qint32) + message.size();
message.insert(0, (const char*)&messageSize, sizeof(qint32));
// Send message
// In here, socket is a QTcpSocket
socket->write(message);
Edit #2 Here is an image of the output. The values after the } should have been part of the big array right before Message.

any time you see Read/ReadAsync without capturing and using the result: the code is simply wrong.
Fortunately, it isn't a huge fix:
int toRead = 4, offset = 0, read;
while (toRead != 0 && (read = stream.ReadAsync(msgSize, offset, toRead)) > 0)
{
offset += read;
toRead -= read;
}
if (toRead != 0) throw new EndOfStreamException();
As a side note: BitConverter.ToInt32 is dangerous - it assumes CPU endianness, which is not fixed. It would be better to use BinaryPrimitives.ReadInt32BigEndian or BinaryPrimitives.ReadInt32LittleEndian.

EndReceive in Async Socket without strings

I'm following this example about the creation of an async tcp listener in C#.
MSDN Example
I see that all data is encoded as string to check for message completeness. More precisely, every message sent is already a string, which we append the 'EOF' char to for string termination.
The server side part i'm talking about is in this snippet:
public static void ReadCallback(IAsyncResult ar) {
String content = String.Empty;
// Retrieve the state object and the handler socket
// from the asynchronous state object.
StateObject state = (StateObject) ar.AsyncState;
Socket handler = state.workSocket;
// Read data from the client socket.
int bytesRead = handler.EndReceive(ar);
if (bytesRead > 0) {
// There might be more data, so store the data received so far.
state.sb.Append(Encoding.ASCII.GetString(
state.buffer, 0, bytesRead));
// Check for end-of-file tag. If it is not there, read
// more data.
content = state.sb.ToString();
if (content.IndexOf("<EOF>") > -1) {
// All the data has been read from the
// client. Display it on the console.
Console.WriteLine("Read {0} bytes from socket. \n Data : {1}",
content.Length, content );
// Echo the data back to the client.
Send(handler, content);
} else {
// Not all data received. Get more.
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
}
}
Is there a way, as i usually do with TcpListener/TcpClient classes, to check if received bytes are available on the socket?
I mean something like this:
private void HandleClientConnection(TcpClient client)
{
NetworkStream clientStream = client.GetStream();
MemoryStream memoryStream = new MemoryStream();
while (true)
{
int read = clientStream.ReadByte();
if (read != -1)
{
memoryStream.WriteByte((byte)read);
}
else
{
break;
}
}
}
I'm aware that i probably misunderstood this example, or at least the Begin/End part and the "legacy" async pattern. But this is my goal, do you know some way to get it working without involving strings?

You said : "Is there a way to check if received bytes are available on the socket?"
In general 'EndReceive' will block the thread until data is available. So you don't need to do anything because 'EndReceive' is doing all the job for you.
'bytesRead' is an int that shows you how much data you have received.
a quote from docs.microsoft
The EndReceive method will block until data is available.1
But if you are using a SYNC socket (which you are not) then it's another topic.

Messages aren't properly displayed on the client side

I have written a simple server and client code in C#. As soon as the client is connected, the server will send a welcome message, a file size and a string to the client one by one and the client will display it. The client will convert the string into a string array and display it. After that the client will send an id to server and the server will display it. But the problem is, the client is not properly displaying. When I run the client program after running the server, it's displaying the following thing whereas it's supposed to display each message in a single line.
welcome1.cpp,.jpg,.png
Moreover, on the client side, The line which I have written to display the converted string array, is not working at all neither the lines after this are executing. Seems like,the code hangs. I have marked it in my code. My sample code is given below:
Server:
using System;
using System.Collections.Generic;
using System.Net;
using System.Net.Sockets;
using System.IO;
using System.Text;
namespace server
{
class Program
{
static void Main(string[] args)
{
// Listen on port 1234.
try
{
TcpListener tcpListener = new TcpListener(IPAddress.Any, 1234);
tcpListener.Start();
byte[] data = new byte[1024];
// Infinite loop to connect to new clients.
while (true)
{
// Accept a TcpClient
TcpClient tcpClient = tcpListener.AcceptTcpClient();
NetworkStream ns = tcpClient.GetStream();
//sending welcome message
string welcome = "welcome";
ns.Write(Encoding.ASCII.GetBytes(welcome), 0, welcome.Length);
ns.Flush();
//sending file size
string fsize = "1";
ns.Write(Encoding.ASCII.GetBytes(fsize), 0, fsize.Length);
ns.Flush();
//sending extensions
string[] extensions = { ".cpp", ".jpg", ".png" };
string str = string.Join(",", extensions);
Console.WriteLine(str);
ns.Write(Encoding.ASCII.GetBytes(str), 0, str.Length);
ns.Flush();
//receiving id
int recv = ns.Read(data, 0, data.Length);
string id = Encoding.ASCII.GetString(data, 0, recv);
Console.WriteLine(id);
}
}
catch (Exception e)
{
Console.Write(e.Message);
}
Console.Read();
}
}
}
Client:
using System;
using System.Collections.Generic;
using System.Net;
using System.Net.Sockets;
using System.IO;
using System.Text;
namespace client
{
class Program
{
static void Main(string[] args)
{
try
{
TcpClient tcpClient = new TcpClient("127.0.0.1", 1234);
NetworkStream ns = tcpClient.GetStream();
byte[] data = new byte[1024];
StreamWriter sWriter = new StreamWriter(tcpClient.GetStream());
//receiving welcome message
int recv = ns.Read(data, 0, data.Length);
string message = Encoding.ASCII.GetString(data, 0, recv);
Console.WriteLine(message);
//receive filesize
int recv2 = ns.Read(data, 0, data.Length);
string message2 = Encoding.ASCII.GetString(data, 0, recv2);
Console.WriteLine(message2);
//receiving extensions
int recv1 = ns.Read(data, 0, data.Length);
string message1 = Encoding.ASCII.GetString(data, 0, recv1);
Console.WriteLine(message1);
var array2 = message1.Split(',');
foreach (string s in array2) //from this line the program isn't working
{
Console.WriteLine(s);
}
string input = Console.ReadLine();
ns.Write(Encoding.ASCII.GetBytes(input), 0, input.Length);
ns.Flush();
}
catch (Exception e)
{
Console.Write(e.Message);
}
Console.Read();
}
}
}
What's wrong in the code?

After looking at your post again, I am sure that my comment is the actual answer:
The code already hangs at int recv2 because all data will be received by the 1st ns.read and the 2nd call in the mentioned line is blocking as no more data is present.
You need to add a 'high level protocol' that lets you identify the data of each packet.
Sample:
000007welcome
6 bytes (4 could be enough though) at the Start of each message specify the length of the user data. So you can easily separate your data which will look like
000007welcome0000011000014.cpp,.jpg,.png
You have to create the functions for creating/adding and separating/interpreting your 6 bytes header (which actually is just a length info, but could be enhanced to contain multiple info) by yourself of course, but that's quite easy.
In general you should always consider timing issues on EVERY operation that is not just adding 2 integers: file access, data transfer over network or other media, ... really any type of hardware access. Therefore sending data over network means that the packets could
arrive in correct order, as desired, with some delay so that you could read them separately. This is quite unlikely though.
arrive in correct order, but at the same time or with major delay (which is a couple of 100ms, so not really large for a human, but indeed in terms of networking). This is very common, see your case.
arrive partially. Packet fragmentation usually occurs only on large packets (greater than MTU), but you should have a routine that stores and concats incoming data, while another routine checks the stored data for complete messages, removes complete messages from the stored data and processes them. For this, use a loop (in a separate thread) and call ns.read when ns.DataAvailable is true.
arrive in different order. This may happen on long transmission paths (internet; rarely on LAN). You would have to enhance your protocol by an incrementing sequence number.
get lost! On UDP, the packets will NOT be resent. On TCP they will, but actually I cannot tell what happens on the sender side if the resend fails, too...
This means, depending on the importance of your data you may need to take safety measures for these cases.
And finally you might want to handle (unexpected) connection losses...
I also suggest you take a look on my answers on other networking questions for advices on debugging, testing, common issues, ...

Analysis
The code:
// Receive the welcome message.
int recv = ns.Read(data, 0, data.Length);
string message = Encoding.ASCII.GetString(data, 0, recv);
Console.WriteLine(message);
// Receive the file size.
int recv2 = ns.Read(data, 0, data.Length);
string message2 = Encoding.ASCII.GetString(data, 0, recv2);
Console.WriteLine(message2);
// Receive the extensions.
int recv1 = ns.Read(data, 0, data.Length);
string message1 = Encoding.ASCII.GetString(data, 0, recv1);
Console.WriteLine(message1);
does not work properly because the Stream.Read(Byte[], Int32, Int32) method has the following return value:
Return Value
Type: System.Int32
The total number of bytes read into the buffer. This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.
— Stream.Read Method (Byte[], Int32, Int32), MSDN.
The Stream class does not guarantee the "data correspondence" between the Stream.Write() and the Stream.Read() method calls. For example, if the Stream.Write() method call writes 6 bytes, the first Stream.Read() method call could return 1 byte (first byte).
Conceptual solution
Because of "streaming" it is necessary to define the logical messages to "extract" ("detect") them from the stream.
It is required to "join" the messages using the "separator" when sending and "split" the messages using the "separator" when receiving. One of the following alternatives could be considered to implement the "separator" concept:
Introduce the "end-of-the-message" marker.
Introduce the message header which contains the length of the message.
The small article, TCP/IP client-server application: exchange with string messages, may be useful to understand the mentioned alternatives.

How to get the Socket Buffer size in C#

I have a Socket code which is communicating through TCP/IP.The machine to which i am communicating has buffer data in its buffer.At present i am trying to get the buffer data using this code.
byte data = new byte[1024];
int recv = sock.Receive(data);
stringData = Encoding.ASCII.GetString(data, 0, recv);
But this code retrieves only 11 lines of data whereas more data is there in the machines buffer.Is this because i have used int recv = sock.Receive(data); and data is 1024 ?
If yes ,How to get the total buffer size and retrieve it into string.

If you think you are missing some data, then you need to check recv and almost certainly: loop. Fortunately, ASCII is always single byte - in most other encodings you would also have to worry about receiving partial characters.
A common approach is basically:
int recv;
while((recv = sock.Receive(data)) > 0)
{
// process recv-many bytes
// ... stringData = Encoding.ASCII.GetString(data, 0, recv);
}
Keep in mind that there is no guarantee that stringData will be any particular entire unit of work; what you send is not always what you receive, and that could be a single character, 14 lines, or the second half of one word and the first half of another. You generally need to maintain your own back-buffer of received data until you have a complete logical frame to process.
Note, however, Receive always tries to return something (at least one byte), unless the inbound stream has closed - and will block to do so. If this is a problem, you may need to check the available buffer (sock.Available) to decide whether to do synchronous versus asynchronous receive (i.e. read synchronously while data is available, otherwise request an asynchronous read).

Try something along these lines:
StringBuilder sbContent=new StringBuilder();
byte data = new byte[1024];
int numBytes;
while ((numBytes = sock.Receive(data))>0)
{
sbContent.Append(Encoding.UTF8.GetString(data));
}
// use sbContent.ToString()

Socket tcpSocket = new Socket(ipe.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
Console.WriteLine(" ReceiveBufferSize {0}", tcpSocket.ReceiveBufferSize);
For actual data you can put below condition:-
int receiveBytes;
while((receiveBytes = tcpSocket.Receive.Data(receiveBytes)) > 0)
{
}

TCP client\server - client doesn't always read

Client Code:
TcpClient client = new TcpClient();
NetworkStream ns;
private void Form1_Load(object sender, EventArgs e)
{
try
{
client.Connect("127.0.0.1", 560);
ns = client.GetStream();
byte[] buffer = ReadFully(ns, client.Available);
//working with the buffer...
}
catch
{
//displaying error...
}
}
public static byte[] ReadFully(NetworkStream stream , int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[initialLength];
long read = 0;
int chunk;
while ((chunk = stream.Read(buffer, (int)read, buffer.Length - (int)read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
Server Code:
private static TcpListener tcpListener;
private static Thread listenThread;
private static int clients;
static void Main(string[] args)
{
tcpListener = new TcpListener(IPAddress.Any, 560);
listenThread = new Thread(new ThreadStart(ListenForClients));
listenThread.Start();
}
private static void ListenForClients()
{
tcpListener.Start();
Console.WriteLine("Server started.");
while (true)
{
//blocks until a client has connected to the server
TcpClient client = tcpListener.AcceptTcpClient();
//create a thread to handle communication
//with connected client
Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
clientThread.Start(client);
}
}
private static void HandleClientComm(object client)
{
clients++;
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
ASCIIEncoding encoder = new ASCIIEncoding();
Console.WriteLine("Client connected. ({0} connected)", clients.ToString());
#region sendingHandler
byte[] buffer = encoder.GetBytes(AddressBookServer.Properties.Settings.Default.contacts);
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
#endregion
}
As you can see from the code, i'm trying to send AddressBookServer.Properties.Settings.Default.contacts (a string, not empty) to the connected client.
The problam is that sometimes(that's the wierd part) the client recieves the string and sometimes its keep being blocked on the ns.Read line waiting to recieve something.
I tryed debuging by putting a breakpoint on the line after ns.Read and i saw that when it doesn't work it never gets to that line, so it doesn't recieve the message that was sent by the server.
My question: How can I fix it?
My assumption: The server is sending the message before the client can recieve it therefor it's never get recieved by the client.

As Mark Gravell pointed out, this is a framing problem. Here is a simple client and server to show you how to frame your messages with a length prefix on the message. Keep in mind that this is just a sample to get you started. I would not consider it production ready code:
Client Code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Sockets;
using System.Text;
namespace SimpleClient
{
internal class Client
{
private static void Main(string[] args)
{
try
{
TcpClient client = new TcpClient();
NetworkStream ns;
client.Connect("127.0.0.1", 560);
ns = client.GetStream();
byte[] buffer = ReadNBytes(ns, 4);
// read out the length field we know is there, because the server always sends it.
int msgLenth = BitConverter.ToInt32(buffer, 0);
buffer = ReadNBytes(ns, msgLenth);
//working with the buffer...
ASCIIEncoding encoder = new ASCIIEncoding();
string msg = encoder.GetString(buffer);
Console.WriteLine(msg);
client.Close();
}
catch
{
//displaying error...
}
}
public static byte[] ReadNBytes(NetworkStream stream, int n)
{
byte[] buffer = new byte[n];
int bytesRead = 0;
int chunk;
while (bytesRead < n)
{
chunk = stream.Read(buffer, (int) bytesRead, buffer.Length - (int) bytesRead);
if (chunk == 0)
{
// error out
throw new Exception("Unexpected disconnect");
}
bytesRead += chunk;
}
return buffer;
}
}
}
Server Code:
using System;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
namespace SimpleServer
{
class Server
{
private static TcpListener tcpListener;
private static int clients;
static void Main(string[] args)
{
tcpListener = new TcpListener(IPAddress.Any, 560);
tcpListener.Start();
Console.WriteLine("Server started.");
while (true)
{
//blocks until a client has connected to the server
TcpClient client = tcpListener.AcceptTcpClient();
//create a thread to handle communication
//with connected client
Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
clientThread.Start(client);
}
}
private static void HandleClientComm(object client)
{
int clientCount = Interlocked.Increment(ref clients);
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
ASCIIEncoding encoder = new ASCIIEncoding();
Console.WriteLine("Client connected. ({0} connected)", clientCount);
#region sendingHandler
byte[] buffer = encoder.GetBytes("Some Contacts as a string!");
byte[] lengthBuffer = BitConverter.GetBytes(buffer.Length);
clientStream.Write(lengthBuffer, 0, lengthBuffer.Length);
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
tcpClient.Close();
#endregion
}
}
}
The reason your code sometimes worked, and sometimes failed is that client.Available can return 0. When it did you were setting the bytes to read to 32k, so the read call was waiting for those bytes to come in. They never did, and since the server never closed the socket, read would not error out either.
Hope this gets you moving in the right direction.
Edit:
I forgot to mention endianess in my original post. You can see the documentation here about endianess and using BitConverter: http://msdn.microsoft.com/en-us/library/system.bitconverter(v=vs.100).aspx
Basically you need to make sure both server and client are running on architectures with the same endianess, or handle the conversion from one endianess to another as needed.
Edit 2 (to answer question in the comments):
1) Why can client.available return 0?
This is a timing issue. The client is connecting to the server, then immediately asking which bytes are available. Depending on what other processes are running, time slices for the available processor etc, the client may be asking what is available before the server has had a chance to send anything at all. In this case it will return 0.
2) Why did I use Interlocked to increment the clients?
The code as you had originally written it was incrementing the clients in the newly created thread running HandleClientComm(...). If two or more clients connected simultaneously it is possible a race condition could occur as multiple threads were trying to increment clients. The end result would be clients would be less than it should be.
3) Why did I change ReadFully method?
You version of ReadFully, which I changed to ReadNBytes, was close to be correct, but had a few flaws:
Setting initialLenth to 32768 if the original initialLength was zero or less. You should never guess how many bytes you need to read from a socket. As Mark Gravell mentioned you need to frame your messages with either a length prefix or some sort of delimiter.
NetworkStream.Read blocks until some bytes are read or returns a 0 if the socket is closed out from underneath it. There was no need to call stream.ReadByte, since chunk would already be 0 if the socket was disconnected. Making that change simplified the method, especially since we know exactly how many bytes we need to read based on our simple header.
Now that we know how much we are going to read, we can allocate exactly what we need up front. This removes the necessity of re-sizing the buffer upon return. We can just return what we allocated.
4) How do I know the length buffer size is 4?
The length I serialized was 32 bits. You can see that in the documentation for BitConverter.GetBytes(int value). We know a byte is 8 bits, so simply divide 32 by 8 giving us 4.
5) Why is this not production ready/How can I improve the code?
Basically there is no real error handling. NetworkStream.Read() can throw several exceptions, none of which I am handling. Adding the proper error handling would go a long way to making it production ready.
I have not really tested the code beyond a cursory run. It would need to be tested under a variety of conditions.
There is no provision for the client to reconnect or retry, though you may not need this for your purposes.
This was written as a simple example, and may not actually meet the requirements you are trying to fulfill. Not knowing those requirements I cannot claim this is ready for your production environment (whatever that is).
Conceptually ReadNBytes is fine, but if someone sends you malicious message which claim the length of the message is 2 gigabytes or something, you are going to try and allocate 2 gigabytes blindly. Most byte level protocols (the description of what goes over the wire) specify the maximum size of messages. That would need to be checked, and if the messages can actually be large you would need to handle it differently than just allocating a buffer, maybe writing to a file or other output stream as you read it in. Again, not knowing your full requirements, I cannot be sure what is needed there.
Hope this helps.