We Keep Coding

How can I efficiently receive & send data simultaneously using c# sockets

I have a C# .net server & client (TCP). The server listens for multiple clients in an asynchronous manner. The clients can send data ranging from screenshots, to messages, to a videostream (really just multiple screenshots sent extremely fast which never works...), to system information. The problem that I'm facing is that when all of these are possibilites, the server can only receive so little. I can't seem to download a file from a client & receive a screenshot at the same time. Or when receiving multiple screenshots really fast to emulate screensharing, I can't download a file (this usually crashes the data receiving process.)
I don't even think my server is correctly receiving data for clients, as data often is corrupt and throws exceptions. I'll start with my server listening code itself:
public void Listen()
{
_socket.Listen(100);
_socket.BeginAccept(new AsyncCallback(AcceptCallback), null);
CSCLog("Server started || Listening...");
void AcceptCallback(IAsyncResult AR)
{
Socket socket = _socket.EndAccept(AR);
clienthandler client = new clienthandler(socket, _dash);
_socketClientList.Add(client);
// Repeat the listening process...
_socket.BeginAccept(new AsyncCallback(AcceptCallback), null);
}
}
The server holds a custom class, clienthandler, that requires a socket and provides individual socket "control." Here is the basic run down of that class:
public class clienthandler
{
public Socket _clientSocket;
public clienthandler(Socket paramSocket)
{
_clientSocket = paramSocket;
receiveAll();
}
public void receiveAll()
{
int BUFF_SIZE = 4024 * 2;
byte[] _lengthBuffer = new byte[BUFF_SIZE];
_clientSocket.BeginReceive(_lengthBuffer, 0, _lengthBuffer.Length, 0, new AsyncCallback(LengthCallBack), _clientSocket);
void LengthCallBack(IAsyncResult lengthAR)
{
try
{
Socket buffSocket = (Socket)lengthAR.AsyncState;
int lengthreceived = _clientSocket.EndReceive(lengthAR);
byte[] lengthdata = new byte[lengthreceived];
Array.Copy(_lengthBuffer, lengthdata, lengthreceived);
// Handle the received incoming data length
DataInformation datai = (DataInformation)ObjectHandler.Deserialize(_lengthBuffer);
int datalength = datai.datalength;
Array.Clear(_lengthBuffer, 0, _lengthBuffer.Length);
PrepCol(datai, buffSocket);
// Repeat the data length listening process...
_clientSocket.BeginReceive(_lengthBuffer, 0, _lengthBuffer.Length, 0, new AsyncCallback(LengthCallBack), _clientSocket);
}
catch (Exception ex) { // handle exception... }
}
}
}
To further explain, the receiveAll() function tries to asynchronously listen for data. I've created a custom class (these are all stored in a .DLL that the client and server share) called DataInformation that acts as a header for data being sent through the stream. It only includes an enum of an objectType & int datalength that signifies the amount of incoming data my client sends right afterwards (as any data sent from the client has its own DataInformation sent first, then immediately after is the actual data. Both are serialized.)
The PropCol() methods takes in the deserialized DataInformation class and socket object. It then collects data using the datainformation variables provided (the Enum & int datalength.):
public void PrepCol(DataInformation paramDatainformation, Socket paramBuffSocket)
{
int datalength = paramDatainformation.datalength;
object streamobj;
MemoryStream ms = new MemoryStream();
if(paramDatainformation.objectType == ObjectType.Universal)
{
// Prepare & collect the incoming data
while (datalength > 0)
{
byte[] buffer;
if (datalength < paramBuffSocket.ReceiveBufferSize)
{
buffer = new byte[datalength];
}
else
buffer = new byte[paramBuffSocket.ReceiveBufferSize];
int rec = paramBuffSocket.Receive(buffer, 0, buffer.Length, 0);
datalength -= rec;
ms.Write(buffer, 0, rec);
}
// Handle the collected data
ms.Close();
byte[] data = ms.ToArray(); ms.Dispose();
streamobj = ObjectHandler.Deserialize(data);
Array.Clear(data, 0, data.Length);
// Check the data that is received & determine type
CheckData(streamobj);
}
if(paramDatainformation.objectType == ObjectType.Screenshot)
{ // Receive data certain way }
if(paramDatainformation.objectType == ObjectType.TransferFile)
{ // Receive data certain way }
}
It utilizes a while loop to synchronously receive data until it has received the amount that the DataInformation says it needs. It takes the data object and passes it into the CheckData() method that requires an object. It takes the object and checks if it is any object it can handle (such as a screenshot, or message, ect...), if so then it does.
The problem I find the most is that when receiving large data, or data really fast, my DataInformation deseralization in the receiveAll() method returns corrupt/ invalid data which isn't good at all because I'm losing something that I could be needing.
My question really boils down to what am I doing wrong? Or how should I take the approach to receive multiple data simultaneously?
Any more information I could provide is that the Serialization & Desserialization methods are in the .DLL. That's really all. I apologize for dumping large amounts of code out, but felt like those were the most relevent things. Thank you for your time.

First of all, as mentioned in comments, you should consider your TCP socket as continous stream of data. Eventually, you'll end up in situation where you read uncomplete set of bytes, and being unable to deserialize it.
Said all the above, when you receive next portion of data from the socket, you need to know two things:
how much data you received so far;
how much data you need to begin the deserialization process.
Another point is that you are using old-fashioned callback-style asynchronous operations.
You can rewrite your code to use async/await-friendly methods, like Socket.ReceiveAsync().
And finally, don't use Socket in your logic. Use Stream instead.
Here is the approach I'd choose to receive and parse binary data according to some protocol. I assume that your protocol consists of some fixed-size header followed by the data (file/screenshot/video etc.). The size and type of data is stored in the header.
using System;
using System.IO;
using System.Net.Sockets;
using System.Threading;
using System.Threading.Tasks;
namespace Test1
{
public interface IData
{
}
public interface IHeader
{
int DataSize { get; }
}
public interface IHeaderParser
{
int HeaderSize { get; }
IHeader Parse(byte[] buffer, int offset);
}
public interface IDataParser
{
IData Parse(byte[] buffer, int offset);
}
public interface IDataParserSelector
{
IDataParser GetParser(IHeader header);
}
public class StreamReceiver : IDisposable
{
public StreamReceiver(
Stream stream,
IHeaderParser headerParser,
IDataParserSelector dataParserSelector)
{
_stream = stream;
_headerParser = headerParser;
_dataParserSelector = dataParserSelector;
}
public virtual void Dispose()
{
_stream.Dispose();
}
public async Task<IData> ReceiveAsync(CancellationToken token)
{
const int headerOffset = 0;
await ReadAsync(headerOffset, _headerParser.HeaderSize, token).ConfigureAwait(false);
var header = _headerParser.Parse(_buffer, headerOffset);
var dataOffset = headerOffset + _headerParser.HeaderSize;
await ReadAsync(dataOffset, header.DataSize, token).ConfigureAwait(false);
var dataParser = _dataParserSelector.GetParser(header);
var data = dataParser.Parse(_buffer, dataOffset);
return data;
}
private async Task ReadAsync(int offset, int count, CancellationToken token)
{
if (_buffer.Length < offset + count)
{
var oldBuffer = _buffer;
_buffer = new byte[offset + count];
Array.Copy(oldBuffer, _buffer, oldBuffer.Length);
}
var nread = 0;
while (nread < count)
{
nread += await _stream.ReadAsync(
_buffer, offset + nread, count - nread, token)
.ConfigureAwait(false);
}
}
private readonly Stream _stream;
private readonly IHeaderParser _headerParser;
private readonly IDataParserSelector _dataParserSelector;
private byte[] _buffer = new byte[0];
}
public class TcpReceiver : StreamReceiver
{
public TcpReceiver(
TcpClient tcpClient,
IHeaderParser headerParser,
IDataParserSelector dataParserSelector)
: base(tcpClient.GetStream(), headerParser, dataParserSelector)
{
_tcpClient = tcpClient;
}
public override void Dispose()
{
base.Dispose();
_tcpClient.Dispose();
}
private readonly TcpClient _tcpClient;
}
}
This is just a stub, I leave interface implementations up to you, if you ever consider using this approach.
Also, I didn't cover the connection process here, so it is up to you too :).

EndReceive in Async Socket without strings

I'm following this example about the creation of an async tcp listener in C#.
MSDN Example
I see that all data is encoded as string to check for message completeness. More precisely, every message sent is already a string, which we append the 'EOF' char to for string termination.
The server side part i'm talking about is in this snippet:
public static void ReadCallback(IAsyncResult ar) {
String content = String.Empty;
// Retrieve the state object and the handler socket
// from the asynchronous state object.
StateObject state = (StateObject) ar.AsyncState;
Socket handler = state.workSocket;
// Read data from the client socket.
int bytesRead = handler.EndReceive(ar);
if (bytesRead > 0) {
// There might be more data, so store the data received so far.
state.sb.Append(Encoding.ASCII.GetString(
state.buffer, 0, bytesRead));
// Check for end-of-file tag. If it is not there, read
// more data.
content = state.sb.ToString();
if (content.IndexOf("<EOF>") > -1) {
// All the data has been read from the
// client. Display it on the console.
Console.WriteLine("Read {0} bytes from socket. \n Data : {1}",
content.Length, content );
// Echo the data back to the client.
Send(handler, content);
} else {
// Not all data received. Get more.
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
}
}
Is there a way, as i usually do with TcpListener/TcpClient classes, to check if received bytes are available on the socket?
I mean something like this:
private void HandleClientConnection(TcpClient client)
{
NetworkStream clientStream = client.GetStream();
MemoryStream memoryStream = new MemoryStream();
while (true)
{
int read = clientStream.ReadByte();
if (read != -1)
{
memoryStream.WriteByte((byte)read);
}
else
{
break;
}
}
}
I'm aware that i probably misunderstood this example, or at least the Begin/End part and the "legacy" async pattern. But this is my goal, do you know some way to get it working without involving strings?

You said : "Is there a way to check if received bytes are available on the socket?"
In general 'EndReceive' will block the thread until data is available. So you don't need to do anything because 'EndReceive' is doing all the job for you.
'bytesRead' is an int that shows you how much data you have received.
a quote from docs.microsoft
The EndReceive method will block until data is available.1
But if you are using a SYNC socket (which you are not) then it's another topic.

C# Multithreading socket server receiving data

I have developed this small multi-threaded socket server, that creates one socket to not block the main program, and another thread for each new accepted connection.
The question comes when reading the data, I have a method called receiveData that reads all the data input, so right now the code only reads one time for each connection.
I solved this by adding inside the method receiveData a while loop with the flag socket.CanRead, what does this flag exactly do? What difference is there between CanRead and DataAvailable?
class Server
{
static void receiveData(TcpClient client, NetworkStream stream)
{
byte[] bytes = new byte[client.ReceiveBufferSize];
stream.Read (bytes, 0, (int) client.ReceiveBufferSize);
string returndata = Encoding.UTF8.GetString (bytes);
Console.WriteLine ("SERVER RECEIVED: " + returndata);
}
static void startServer()
{
new Thread(() =>
{
TcpListener listener = new TcpListener(IPAddress.Any, 5000);
listener.Start();
while (true)
{
if (listener.Pending()) {
new Thread(() =>
{
TcpClient client = listener.AcceptTcpClient();
NetworkStream stream = client.GetStream();
Server.receiveData(client, stream);
}).Start();
}
}
}).Start();
}
static void Main(string[] args)
{
Server.startServer ();
}
}

Refer to MSDN NetworkStream.CanRead Property,
CanRead:
Gets a value that indicates whether the NetworkStream supports reading.
while DataAvailable:
Gets a value that indicates whether data is available on the NetworkStream to be read.
It's like incoming data will be indicated as available only if the NetworkStream supports reading, thus can be read.
Take a look at sample code, I think it will answer all of your questions:
// Examples for CanRead, Read, and DataAvailable.
// Check to see if this NetworkStream is readable.
if(myNetworkStream.CanRead){
byte[] myReadBuffer = new byte[1024];
StringBuilder myCompleteMessage = new StringBuilder();
int numberOfBytesRead = 0;
// Incoming message may be larger than the buffer size.
do{
numberOfBytesRead = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
myCompleteMessage.AppendFormat("{0}", Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));
}
while(myNetworkStream.DataAvailable);
// Print out the received message to the console.
Console.WriteLine("You received the following message : " +
myCompleteMessage);
}
else{
Console.WriteLine("Sorry. You cannot read from this NetworkStream.");
}

According to MSDN documentation NetworkStream.CanRead Can Read MSDN is used to determine if network stream is support reading. NetworkStream.DataAvailable Data Acalible MSDN shows you if there is data to read right now. So you should use NetworkStream.CanRead to determine if it is readable, and use NetworkStream.DataAvailable to determine if there is something to be read.

C# Tcp Async socket data read logic

I'm working on a C# Tcp Async socket server program which will simply read socket data and put it into a file. Now the issue is before sending actual file content I'm sending client name and file name. Please note that I'm sending these data from C++ MFC application with simple send() API in below manner.
CString csInitData = "Client_Name | File_Name <EOF>";
send(hSocket, (LPCTSTR)csInitData, csInitData.GetLength(), 0);
Now how do I receive this data inside below async callback function? Cannot figure out the byte break up logic.
public void OnDataReceived(IAsyncResult ar)
{
StateObject state = (StateObject)ar.AsyncState;
Socket clientSocket = state.m_currentSocket;
int bytesRead = clientSocket.EndReceive(ar);
if (bytesRead > 0)
{
//HERE I WANT TO RECEIVE THOSE DATA AND CREATE FOLDER AND FILE WITH THOSE NAMES RECEIVED FROM CLIENT
//Process 'state.dataBuffer'//DATA INSIDE THIS BYTE BUFFER
string fileName = ""; //FILE NAME SHOULD GO HERE
string folderName = ""; //CLIENT NAME SHOULD GO HERE
BinaryWriter writer;
if (!File.Exists(folderName+fileName))
{
Directory.CreateDirectory(folderName);
writer = new BinaryWriter(File.Open(folderName+fileName, FileMode.Create));
}
else
{
writer = new BinaryWriter(File.Open(folderName+fileName, FileMode.Append));
}
writer.Write(state.dataBuffer, 0, bytesRead);
writer.Flush();
writer.Close();
// Recursively receive the rest file.
try
{
clientSocket.BeginReceive(state.dataBuffer, 0, StateObject.BufferSize, 0, new AsyncCallback(OnDataReceived), state);
}
catch
{
if (!clientSocket.Connected)
{
MessageBox.Show("Catched from OnDataReceived!");
}
}
}
else
{
// Signal if all the file received.
}
}

For the buffering, it sounds like you want MemoryStream:
// Somewhere to put the data:
// for simplicity, I'll assume an instance field is fine; could
// also be tracked via async-state object, if preferred
private MemoryStream backlog = new MemoryStream();
if (bytesRead > 0)
{
// we want to append, so move to the end
backlog.Position = backlog.Length;
// copy new data into the buffer
backlog.Write(state.dataBuffer, 0, bytesRead);
...
}
General feedback:
don't use BinaryWriter here; just Stream is fine
use using on the file-stream
have a cached delegate instance in a field so you don't need to new a delegate each time (this can even be static if you pass the target instance as state)
use Path.Combine, preferably once ahead of time, to avoid lots of string concatenations
check for sync-completion; in the callback, check CompletedSynchronously and exit immediately if true; when starting work, capture the IAsyncResult, check CompletedSynchronously, and process right away on the same thread if true
consider moving to the newer async API - ReceiveAsync (this is quite a significant change, though)
don't MessageBox from an IO method

TCP client\server - client doesn't always read

Client Code:
TcpClient client = new TcpClient();
NetworkStream ns;
private void Form1_Load(object sender, EventArgs e)
{
try
{
client.Connect("127.0.0.1", 560);
ns = client.GetStream();
byte[] buffer = ReadFully(ns, client.Available);
//working with the buffer...
}
catch
{
//displaying error...
}
}
public static byte[] ReadFully(NetworkStream stream , int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[initialLength];
long read = 0;
int chunk;
while ((chunk = stream.Read(buffer, (int)read, buffer.Length - (int)read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
Server Code:
private static TcpListener tcpListener;
private static Thread listenThread;
private static int clients;
static void Main(string[] args)
{
tcpListener = new TcpListener(IPAddress.Any, 560);
listenThread = new Thread(new ThreadStart(ListenForClients));
listenThread.Start();
}
private static void ListenForClients()
{
tcpListener.Start();
Console.WriteLine("Server started.");
while (true)
{
//blocks until a client has connected to the server
TcpClient client = tcpListener.AcceptTcpClient();
//create a thread to handle communication
//with connected client
Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
clientThread.Start(client);
}
}
private static void HandleClientComm(object client)
{
clients++;
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
ASCIIEncoding encoder = new ASCIIEncoding();
Console.WriteLine("Client connected. ({0} connected)", clients.ToString());
#region sendingHandler
byte[] buffer = encoder.GetBytes(AddressBookServer.Properties.Settings.Default.contacts);
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
#endregion
}
As you can see from the code, i'm trying to send AddressBookServer.Properties.Settings.Default.contacts (a string, not empty) to the connected client.
The problam is that sometimes(that's the wierd part) the client recieves the string and sometimes its keep being blocked on the ns.Read line waiting to recieve something.
I tryed debuging by putting a breakpoint on the line after ns.Read and i saw that when it doesn't work it never gets to that line, so it doesn't recieve the message that was sent by the server.
My question: How can I fix it?
My assumption: The server is sending the message before the client can recieve it therefor it's never get recieved by the client.

As Mark Gravell pointed out, this is a framing problem. Here is a simple client and server to show you how to frame your messages with a length prefix on the message. Keep in mind that this is just a sample to get you started. I would not consider it production ready code:
Client Code:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Sockets;
using System.Text;
namespace SimpleClient
{
internal class Client
{
private static void Main(string[] args)
{
try
{
TcpClient client = new TcpClient();
NetworkStream ns;
client.Connect("127.0.0.1", 560);
ns = client.GetStream();
byte[] buffer = ReadNBytes(ns, 4);
// read out the length field we know is there, because the server always sends it.
int msgLenth = BitConverter.ToInt32(buffer, 0);
buffer = ReadNBytes(ns, msgLenth);
//working with the buffer...
ASCIIEncoding encoder = new ASCIIEncoding();
string msg = encoder.GetString(buffer);
Console.WriteLine(msg);
client.Close();
}
catch
{
//displaying error...
}
}
public static byte[] ReadNBytes(NetworkStream stream, int n)
{
byte[] buffer = new byte[n];
int bytesRead = 0;
int chunk;
while (bytesRead < n)
{
chunk = stream.Read(buffer, (int) bytesRead, buffer.Length - (int) bytesRead);
if (chunk == 0)
{
// error out
throw new Exception("Unexpected disconnect");
}
bytesRead += chunk;
}
return buffer;
}
}
}
Server Code:
using System;
using System.Net;
using System.Net.Sockets;
using System.Text;
using System.Threading;
namespace SimpleServer
{
class Server
{
private static TcpListener tcpListener;
private static int clients;
static void Main(string[] args)
{
tcpListener = new TcpListener(IPAddress.Any, 560);
tcpListener.Start();
Console.WriteLine("Server started.");
while (true)
{
//blocks until a client has connected to the server
TcpClient client = tcpListener.AcceptTcpClient();
//create a thread to handle communication
//with connected client
Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
clientThread.Start(client);
}
}
private static void HandleClientComm(object client)
{
int clientCount = Interlocked.Increment(ref clients);
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
ASCIIEncoding encoder = new ASCIIEncoding();
Console.WriteLine("Client connected. ({0} connected)", clientCount);
#region sendingHandler
byte[] buffer = encoder.GetBytes("Some Contacts as a string!");
byte[] lengthBuffer = BitConverter.GetBytes(buffer.Length);
clientStream.Write(lengthBuffer, 0, lengthBuffer.Length);
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
tcpClient.Close();
#endregion
}
}
}
The reason your code sometimes worked, and sometimes failed is that client.Available can return 0. When it did you were setting the bytes to read to 32k, so the read call was waiting for those bytes to come in. They never did, and since the server never closed the socket, read would not error out either.
Hope this gets you moving in the right direction.
Edit:
I forgot to mention endianess in my original post. You can see the documentation here about endianess and using BitConverter: http://msdn.microsoft.com/en-us/library/system.bitconverter(v=vs.100).aspx
Basically you need to make sure both server and client are running on architectures with the same endianess, or handle the conversion from one endianess to another as needed.
Edit 2 (to answer question in the comments):
1) Why can client.available return 0?
This is a timing issue. The client is connecting to the server, then immediately asking which bytes are available. Depending on what other processes are running, time slices for the available processor etc, the client may be asking what is available before the server has had a chance to send anything at all. In this case it will return 0.
2) Why did I use Interlocked to increment the clients?
The code as you had originally written it was incrementing the clients in the newly created thread running HandleClientComm(...). If two or more clients connected simultaneously it is possible a race condition could occur as multiple threads were trying to increment clients. The end result would be clients would be less than it should be.
3) Why did I change ReadFully method?
You version of ReadFully, which I changed to ReadNBytes, was close to be correct, but had a few flaws:
Setting initialLenth to 32768 if the original initialLength was zero or less. You should never guess how many bytes you need to read from a socket. As Mark Gravell mentioned you need to frame your messages with either a length prefix or some sort of delimiter.
NetworkStream.Read blocks until some bytes are read or returns a 0 if the socket is closed out from underneath it. There was no need to call stream.ReadByte, since chunk would already be 0 if the socket was disconnected. Making that change simplified the method, especially since we know exactly how many bytes we need to read based on our simple header.
Now that we know how much we are going to read, we can allocate exactly what we need up front. This removes the necessity of re-sizing the buffer upon return. We can just return what we allocated.
4) How do I know the length buffer size is 4?
The length I serialized was 32 bits. You can see that in the documentation for BitConverter.GetBytes(int value). We know a byte is 8 bits, so simply divide 32 by 8 giving us 4.
5) Why is this not production ready/How can I improve the code?
Basically there is no real error handling. NetworkStream.Read() can throw several exceptions, none of which I am handling. Adding the proper error handling would go a long way to making it production ready.
I have not really tested the code beyond a cursory run. It would need to be tested under a variety of conditions.
There is no provision for the client to reconnect or retry, though you may not need this for your purposes.
This was written as a simple example, and may not actually meet the requirements you are trying to fulfill. Not knowing those requirements I cannot claim this is ready for your production environment (whatever that is).
Conceptually ReadNBytes is fine, but if someone sends you malicious message which claim the length of the message is 2 gigabytes or something, you are going to try and allocate 2 gigabytes blindly. Most byte level protocols (the description of what goes over the wire) specify the maximum size of messages. That would need to be checked, and if the messages can actually be large you would need to handle it differently than just allocating a buffer, maybe writing to a file or other output stream as you read it in. Again, not knowing your full requirements, I cannot be sure what is needed there.
Hope this helps.