We Keep Coding

C# TCP Socket receive part of data on slow cellular network connection while connection still not broken

I am using Console application as a server on windows 10 Pro.
Android application made with Unity Engine as Client.
For both application I use TCP socket on the .Net framework.
Note: the server is always running on wire connection.
Everything work pretty well on both wire and Wi-Fi connections with no problem no meter how packet length is.
The Problem
Each evening the cellular network became too bad and the client app receive only part of the data when the data is little bigger about 50 kb
The code on the server side:
_socket.Send(Encoding.ASCII.GetBytes(s));
_socket: is an instance System.Net.Sockets.Socket.
s: is string.
The code on the client side:
public static void ReceiveData()
{
message = string.Empty;
_socket.BeginReceive(globalBuffer, 0, globalBuffer.Length, SocketFlags.None, ReceiveCallBack, null);
}
public static void ReceiveCallBack(IAsyncResult ar)
{
int internalBuffer = _socket.EndReceive(ar);
byte[] subtractedBuffer = new byte[internalBuffer];
Array.Copy(globalBuffer, subtractedBuffer, internalBuffer);
string stmp = Encoding.ASCII.GetString(subtractedBuffer);
message += stmp;
while (message.Contains("</Cargo>"))
{
string stringOne = message.Substring(0, message.IndexOf("</Cargo>") + "</Cargo>".Length);
string stringLeft = message.Substring(message.IndexOf("</Cargo>") + "</Cargo>".Length);
message = stringLeft;
Thread thread = new Thread(TreatOrder);
thread.Start(stringOne);
}
if (!(_socket.Poll(1000, SelectMode.SelectRead) && _socket.Available == 0))
{
NetWorkScript.LastConnectedTime = DateTime.Now;
_socket.BeginReceive(globalBuffer, 0, globalBuffer.Length, SocketFlags.None, ReceiveCallBack, null);
}
else
{
Debug.log("Connection lost");
}
}
What I have notice:
The next line of code never executed
else
{
Debug.log("Connection lost");
}
The server keep receiving messages from the client on the same Socket instance by another thread and that is how came to know the connection is not broken unless TCP connection could be broken on one way only.
What I understand so far
The TCP will make sure the packets will arrive at the exact order.
The next line of code in my server application end its responsibility and the Operation System will carry on the mission to send the data.
_socket.Send(Encoding.ASCII.GetBytes(s));
What I am confused about
How the Operation System will deliver the data to the client?, why it is not keep trying until send all the data if the connection still exist?
Should I created my own protocol and send small packet then retransmit if client does not confirm receiving the packet?

ZeroMQ performance issue

I'm having an issue with ZeroMQ, which I believe is because I'm not very familiar with it.
I'm trying to build a very simple service where multiple clients connect to a server and sends a query. The server responds to this query.
When I use REQ-REP socket combination (client using REQ, server binding to a REP socket) I'm able to get close to 60,000 messages per second at server side (when client and server are on the same machine). When distributed across machines, each new instance of client on a different machine linearly increases the messages per second at the server and easily reaches 40,000+ with enough client instances.
Now REP socket is blocking, so I followed ZeroMQ guide and used the rrbroker pattern (http://zguide.zeromq.org/cs:rrbroker):
REQ (client) <----> [server ROUTER -- DEALER --- REP (workers running on different threads)]
However, this completely screws up the performance. I'm getting only around 4000 messages per second at the server when running across machines. Not only that, each new client started on a different machine reduces the throughput of every other client.
I'm pretty sure I'm doing something stupid. I'm wondering if ZeroMQ experts here can point out any obvious mistakes. Thanks!
Edit: Adding code as per advice. I'm using the clrzmq nuget package (https://www.nuget.org/packages/clrzmq-x64/)
Here's the client code. A timer counts how many responses are received every second.
for (int i = 0; i < numTasks; i++) { Task.Factory.StartNew(() => Client(), TaskCreationOptions.LongRunning); }
void Client()
{
using (var ctx = new Context())
{
Socket socket = ctx.Socket(SocketType.REQ);
socket.Connect("tcp://192.168.1.10:1234");
while (true)
{
socket.Send("ping", Encoding.Unicode);
string res = socket.Recv(Encoding.Unicode);
}
}
}
Server - case 1: The server keeps track of how many requests are received per second
using (var zmqContext = new Context())
{
Socket socket = zmqContext.Socket(SocketType.REP);
socket.Bind("tcp://*:1234");
while (true)
{
string q = socket.Recv(Encoding.Unicode);
if (q.CompareTo("ping") == 0) {
socket.Send("pong", Encoding.Unicode);
}
}
}
With this setup, at server side, I can see around 60,000 requests received per second (when client is on the same machine). When on different machines, each new client increases number of requests received at server as expected.
Server Case 2: This is essentially rrbroker from ZMQ guide.
void ReceiveMessages(Context zmqContext, string zmqConnectionString, int numWorkers)
{
List<PollItem> pollItemsList = new List<PollItem>();
routerSocket = zmqContext.Socket(SocketType.ROUTER);
try
{
routerSocket.Bind(zmqConnectionString);
PollItem pollItem = routerSocket.CreatePollItem(IOMultiPlex.POLLIN);
pollItem.PollInHandler += RouterSocket_PollInHandler;
pollItemsList.Add(pollItem);
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("{0}", ze.Message);
return;
}
dealerSocket = zmqContext.Socket(SocketType.DEALER);
try
{
dealerSocket.Bind("inproc://workers");
PollItem pollItem = dealerSocket.CreatePollItem(IOMultiPlex.POLLIN);
pollItem.PollInHandler += DealerSocket_PollInHandler;
pollItemsList.Add(pollItem);
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("{0}", ze.Message);
return;
}
// Start the worker pool; cant connect
// to inproc socket before binding.
workerPool.Start(numWorkers);
while (true)
{
zmqContext.Poll(pollItemsList.ToArray());
}
}
void RouterSocket_PollInHandler(Socket socket, IOMultiPlex revents)
{
RelayMessage(routerSocket, dealerSocket);
}
void DealerSocket_PollInHandler(Socket socket, IOMultiPlex revents)
{
RelayMessage(dealerSocket, routerSocket);
}
void RelayMessage(Socket source, Socket destination)
{
bool hasMore = true;
while (hasMore)
{
byte[] message = source.Recv();
hasMore = source.RcvMore;
destination.Send(message, message.Length, hasMore ? SendRecvOpt.SNDMORE : SendRecvOpt.NONE);
}
}
Where the worker pool's start method is:
public void Start(int numWorkerTasks=8)
{
for (int i = 0; i < numWorkerTasks; i++)
{
QueryWorker worker = new QueryWorker(this.zmqContext);
Task task = Task.Factory.StartNew(() =>
worker.Start(),
TaskCreationOptions.LongRunning);
}
Console.WriteLine("Started {0} with {1} workers.", this.GetType().Name, numWorkerTasks);
}
public class QueryWorker
{
Context zmqContext;
public QueryWorker(Context zmqContext)
{
this.zmqContext = zmqContext;
}
public void Start()
{
Socket socket = this.zmqContext.Socket(SocketType.REP);
try
{
socket.Connect("inproc://workers");
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("Could not create worker, error: {0}", ze.Message);
return;
}
while (true)
{
try
{
string message = socket.Recv(Encoding.Unicode);
if (message.CompareTo("ping") == 0)
{
socket.Send("pong", Encoding.Unicode);
}
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("Could not receive message, error: " + ze.ToString());
}
}
}
}

Could you post some source code or at least a more detailed explanation of your test case? In general the way to build out your design is to make one change at a time, and measure at each change. You can always move stepwise from a known working design to more complex ones.

Most probably the 'ROUTER' is the bottleneck.
Check out these related questions on this:
Client maintenance in ZMQ ROUTER
Load testing ZeroMQ (ZMQ_STREAM) for finding the maximum simultaneous users it can handle
ROUTER (and ZMQ_STREAM, which is just a variant of ROUTER) internally has to maintain the client mapping, hence IMO it can accept limited connections from a particular client. It looks like ROUTER can multiplex multiple clients, only as long as, each client has only one active connection.
I could be wrong here - but I am not seeing much proof to the contrary (simple working code that scales to multi-clients with multi-connections with ROUTER or STREAM).
There certainly is a very severe restriction on concurrent connections with ZeroMQ, though it looks like no one know what is causing it.

I have done done performance testing on calling a native unmanaged DLL function with various methods from C#:
1. C++/CLI wrapper
2. PInvoke
3. ZeroMQ/clrzmq
The last might be interesting for you.
My finding at the end of my performance test was that using the ZMQ binding clrzmq was not useful and produced a factor of 100 performance overhead after I tried to optimize the PInvoke calls within the source code of the binding. Therefore I have used the ZMQ without a binding but with PInvoke calls.these calls must be done with the cdecl convention and with the option "SuppressUnmanagedCodeSecurity" to get most speed.
I had to import just 5 functions which was fairly easy.
At the end the speed was a bit slower than a PInvoke call but with the ZMQ-in my case over "inproc".
This may give you the hint to try it without the binding, if speed is interesting for you.
This is not a direct answer for your question but may help you to increase performance in general.

How to create a TcpClient in C# that can recover from a lost network connection?

I am trying to write a TCP client thread that will connect to a server and continue to process data as it receives it. Sometimes the client will lose connection to the server. When the connection is re-established, the thread should resume/recover and automatically start processing data again. I can't seem to get this to work. The basic algorithm is below. My problem is that I just don't quite understand how the TcpClient behaves when the network connection is lost. How can I tell that the connection has been lost? Do I then need to close the connection? How do I reestablish the connection and continue on?
TcpClient _tcpClient;
IPEndPoint _ipEndPoint;
bool _cancelled = false;
bool _error = true;
while (!_cancelled)
{
try
{
if(_error)
{
_ipEndPoint = new IPEndPoint(_myAddress, _myPort);
_tcpClient.Connect(_ipEndPoint);
_networkStream = _tcpClient.GetStream();
_error = false;
}
else
{
_data = new byte[10025];
if(_networkStream.CanRead)
{
_bytesRead = _networkStream.Read(_data, 0, (int)_tcpClient.ReceiveBufferSize);
if(_bytesRead > 0)
{
...process the data...
}
else
{
_error = true;
}
}
else
{
_error = true;
}
}
}
catch(Exception ex)
{
...log error...
_error = true;
}
}

there is a Connected property on the TcpClient, but it is only updated when a Write or Read is executed on the NetworkStream.
I think you can just execute a Read(null,0,0) or Write(null,0,0) if you want to force a connectivity check. But in the example you have you can check the Connected or CanRead properties after your _networkStream.Read is completed.
As for reestablishing the link what you have will work. I would suggest Disposing of the old network stream before getting a new one. Something like this:
if(_networkStream != null)
{
_networkStream.Dispose();
}
_networkStream = _tcpClient.GetStream();
What I do is start a reconnect timer that will attempt to reconnect on a configured interval. Depending on the TcpHost your connecting to you may even want to start trying at a small interval 500-1000ms and increment is after a given number of failed retries so your not wasting a lot of time trying to connect to a host that is gone. Then after a max number of tries just give up unless the user explicitly requests to try again. But that also depends on what else your app is doing, if the connection is its sole purpose or if its just one piece.
The wrapper class I use to interact with TcpClient is around 700 lines of code and it handles reconnects, sending data as well as reading it. I work in a closed shop so i cant post it but if you have any other specific question I'd be happy to help.
Good luck.

C# TCP Server stop receiving client messages, resumes when service is restarted

I working in a managed Windows Service written with C#. It keeps receiving messages from several clients connected over TCP/IP. The Client is basically a router that receive and resend messages from thermometers to the Server. The Server parse the messages and store them in a SQL Server database.
The problem I am facing is that some clients, suddenly, stops sending messages. But, as soon the service is restarted, they connect again and resume sending. I don't have the code of the Client since it is a third party device and I pretty sure the problem is with the Server.
I manage to reduce the problem by implementing a timer that keeps checking if each client is still connected (see code below). Also, I added a Keep Alive mode to the Socket, using the socket.IOControl(IOControlCode.KeepAliveValues, ...) method, but the problem still happening.
I'm posting some code from specific parts I consider relevant. But, if more snippets are needed to understand the problem, please ask me and I'll edit the post. All the try/catch blocks was removed to reduce the ammount of code.
I don't want a perfect solution, just any guidance will be appreciated.
private Socket _listener;
private ConcurrentDictionary<int, ConnectionState> _connections;
public TcpServer(TcpServiceProvider provider, int port)
{
this._provider = provider;
this._port = port;
this._listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
this._connections = new ConcurrentDictionary<int, ConnectionState>();
ConnectionReady = new AsyncCallback(ConnectionReady_Handler);
AcceptConnection = new WaitCallback(AcceptConnection_Handler);
ReceivedDataReady = new AsyncCallback(ReceivedDataReady_Handler);
}
public bool Start()
{
this._listener.Bind(new IPEndPoint(IPAddress.Any, this._port));
this._listener.Listen(10000);
this._listener.BeginAccept(ConnectionReady, null);
}
// Check every 5 minutes for clients that have not send any message in the past 30 minutes
// MSG_RESTART is a command that the devices accepts to restart
private void CheckForBrokenConnections()
{
foreach (var entry in this._connections)
{
ConnectionState conn = entry.Value;
if (conn.ReconnectAttemptCount > 3)
{
DropConnection(conn);
continue;
}
if (!conn.Connected || (DateTime.Now - conn.LastResponse).TotalMinutes > 30)
{
byte[] message = HexStringToByteArray(MSG_RESTART);
if (!conn.WaitingToRestart && conn.Write(message, 0, message.Length))
{
conn.WaitingToRestart = true;
}
else
{
DropConnection(conn);
}
}
}
}
private void ConnectionReady_Handler(IAsyncResult ar)
{
lock (thisLock)
{
if (this._listener == null)
return;
ConnectionState connectionState = new ConnectionState();
connectionState.Connection = this._listener.EndAccept(ar);
connectionState.Server = this;
connectionState.Provider = (TcpServiceProvider)this._provider.Clone();
connectionState.Buffer = new byte[4];
Util.SetKeepAlive(connectionState.Connection, KEEP_ALIVE_TIME, KEEP_ALIVE_TIME);
int newID = (this._connections.Count == 0 ? 0 : this._connections.Max(x => x.Key)) + 1;
connectionState.ID = newID;
this._connections.TryAdd(newID, connectionState);
ThreadPool.QueueUserWorkItem(AcceptConnection, connectionState);
this._listener.BeginAccept(ConnectionReady, null);
}
}
private void AcceptConnection_Handler(object state)
{
ConnectionState st = state as ConnectionState;
st.Provider.OnAcceptConnection(st);
if (st.Connection.Connected)
st.Connection.BeginReceive(st.Buffer, 0, 0, SocketFlags.None, ReceivedDataReady, st);
}
private void ReceivedDataReady_Handler(IAsyncResult result)
{
ConnectionState connectionState = null;
lock (thisLock)
{
connectionState = result.AsyncState as ConnectionState;
connectionState.Connection.EndReceive(result);
if (connectionState.Connection.Available == 0)
return;
// Here the message is parsed
connectionState.Provider.OnReceiveData(connectionState);
if (connectionState.Connection.Connected)
connectionState.Connection.BeginReceive(connectionState.Buffer, 0, 0, SocketFlags.None, ReceivedDataReady, connectionState);
}
}
internal void DropConnection(ConnectionState connectionState)
{
lock (thisLock)
{
if (this._connections.Values.Contains(connectionState))
{
ConnectionState conn;
this._connections.TryRemove(connectionState.ID, out conn);
}
if (connectionState.Connection != null && connectionState.Connection.Connected)
{
connectionState.Connection.Shutdown(SocketShutdown.Both);
connectionState.Connection.Close();
}
}
}

2 things I think I see...
If this is a connection you keep for multiple messages, you probably should not return from ReceivedDataReady_Handler when connectionState.Connection.Available == 0 IIRC a 0 length data paket can be received. So if the connection is still open, you should call connectionState.Connection.BeginReceive( ... ) before leaving the handler.
(I hesitate to put this here because I do not remember specifics) There is an event you can handle that tells you when things happen to your underlying connection including errors and failures connecting or closing a connection. For the life of me I cannot remember the name(s)... This would likely be more efficient than a timer every few seconds. It also gives you a way to break out of connections stuck in the connecting or closing states.

Add try/catch blocks around all the IO calls, and write the errors to a log file. As it is, it can't recover on error.
Also, be careful with any lock that doesn't have a timeout. These operations should be given a reasonable TTL.

I have experienced these kind of situation many times. The problem is probably not with your code at all but with the network and the way Windows (on boths ends) or the routers handle the network. What happens quite often is that a temporary network outage "breaks" the socket, but Windows isn't aware of it, so it doesn't close the socket.
The only way to overcome this is exactly what you did - sending keep-alives and monitoring connection health. Once you recognize the the connection is down, you need to restart it. However, in your code you don't restart the listener socket which is also broken and can't accept new connections. That's why restarting the service helps, it restarts the listener.

TcpClient communication with server to keep alive connection in c#?

I've this TcpClient code which works fine. It connects to perl server on linux system and receives anything that server sents to it. Works nicely.
public static void Main() {
foreach (ProtocolConnection tcpConnection in TcpConnectionsList) {
ProtocolConnection connection = tcpConnection;
ThreadPool.QueueUserWorkItem(_ => {
ThreadTcpClient(connection);
ManualResetEventTcp.Set();
});
}
... Some code...
}
public static void TcpConnect(ProtocolConnection varConnection) {
int retryCountSeconds = varConnection.RetryEverySeconds*Program.MilisecondsMultiplier;
int count = 0;
while (true) {
try {
using (var client = new TcpClient(varConnection.IpAddress.ToString(), varConnection.Port) { NoDelay = true })
using (var stream = client.GetStream()) {
var data = new Byte[256];
while (!Program.PrepareExit) {
Int32 bytes = stream.Read(data, 0, data.Length);
string varReadData = Encoding.ASCII.GetString(data, 0, bytes).Trim();
if (varReadData != "" && varReadData != "PONG") {
VerificationQueue.EnqueueData(varReadData);
Logging.AddToLog("[TCP][" + varConnection.Name + "][DATA ARRIVED]" + varReadData);
} else {
Logging.AddToLog("[TCP]" + varReadData);
}
}
}
} catch (Exception e) {
if (e.ToString().Contains("No connection could be made because the target machine actively refused it")) {
Logging.AddToLog("[TCP][ERROR] Can't connect to server (" + varConnection.Name + ") " + varConnection.IpAddress + ":" + varConnection.Port );
} else {
Logging.AddToLog(e.ToString());
}
}
DateTime startTimeFunction = DateTime.Now;
do {
Thread.Sleep(1000);
} while (((DateTime.Now - startTimeFunction).TotalSeconds < retryCountSeconds));
}
}
However in certain conditions I'm having some problems with it:
My work connection often drops connection after some idle time so I've implemented in server so when it receives PING it responds with PONG. I can send PING with UDP to server and it will respond with PONG on tcp but i would prefer built-in way into tcp client so it does send PING every 60 seconds or so. Even if UDP solution would be acceptable I have no timeout on string varReadData = Encoding.ASCII.GetString(data, 0, bytes).Trim(); so when PONG doesn't arrive my client doesn't even notice it anyway. It just keeps waiting ... which brings me to..
My other problem is that at some point string varReadData = Encoding.ASCII.GetString(data, 0, bytes).Trim(); this is waiting for data all the time. When server crashes or disconnects my client i don't even notice that. I would like server to have some kind of timeout or check if connection is active. If it's not active it should try to reconnect.
What would be simplest way to fix this TcpClient ? How do i implement both way communication making sure that if server drops my connections or my net gets disconnected client will notice it and reestablish connection ?

It's not Encoding.ASCII.GetString(data, 0, bytes).Trim(); that blocks forever, it's the stream.Read()
If you're reading, you can't easily distinguish between the server(or any NAT gateway inbetween) dropping your connection , and the case where the server simply doesn't have anything to send you. Atleast in the case where the TCP FIN/RST packets doesn't reach your client in case of failure, or a NAT gateway silently dropping your connection.
What you can do;
Set a Send/ReceiveTimeout , and ping the server if a timeout occurs, or implement your own heartbeat messages over your TCP connection. Reestablish or take other actions if you don't receive a heartbeat within a reasonable time.
Set the TCP keepalive option, and rely on that to tell you if the server is gone. See code here.
The last point will tell you if the tcp connection fails, it won't tell you if the server has somewhat failed - e.g. if you CTRL+Z your perl server, it'll just sit there not doing anything as the tcp window closes , so you might need to implement your own heatbeat messges to cover such a case too if you need to.

You should get rid of the UDP heartbeat attempt and put in a real TCP heartbeat. "Pinging" the server using UDP is almost meaningless.
Your protocol is also missing message framing.
Read both of those linked articles carefully (especially message framing). The protocol you're currently using does need serious revision.