We Keep Coding

TCP server connection causing processor overload

I have a TCP/IP server that is supposed to allow a connection to remain open as messages are sent across it. However, it seems that some clients open a new connection for each message, which causes the CPU usage to max out. I tried to fix this by adding a time-out but still seem to have the problem occasionally. I suspect that my solution was not the best choice, but I'm not sure what would be.
Below is my basic code with logging, error handling and processing removed.
private void StartListening()
{
try
{
_tcpListener = new TcpListener( IPAddress.Any, _settings.Port );
_tcpListener.Start();
while (DeviceState == State.Running)
{
var incomingConnection = _tcpListener.AcceptTcpClient();
var processThread = new Thread( ReceiveMessage );
processThread.Start( incomingConnection );
}
}
catch (Exception e)
{
// Unfortunately, a SocketException is expected when stopping AcceptTcpClient
if (DeviceState == State.Running) { throw; }
}
finally { _tcpListener?.Stop(); }
}
I believe the actual issue is that multiple process threads are being created, but are not being closed. Below is the code for ReceiveMessage.
private void ReceiveMessage( object IncomingConnection )
{
var buffer = new byte[_settings.BufferSize];
int bytesReceived = 0;
var messageData = String.Empty;
bool isConnected = true;
using (TcpClient connection = (TcpClient)IncomingConnection)
using (NetworkStream netStream = connection.GetStream())
{
netStream.ReadTimeout = 1000;
try
{
while (DeviceState == State.Running && isConnected)
{
// An IOException will be thrown and captured if no message comes in each second. This is the
// only way to send a signal to close the connection when shutting down. The exception is caught,
// and the connection is checked to confirm that it is still open. If it is, and the Router has
// not been shut down, the server will continue listening.
try { bytesReceived = netStream.Read( buffer, 0, buffer.Length ); }
catch (IOException e)
{
if (e.InnerException is SocketException se && se.SocketErrorCode == SocketError.TimedOut)
{
bytesReceived = 0;
if(GlobalSettings.IsLeaveConnectionOpen)
isConnected = GetConnectionState(connection);
else
isConnected = false;
}
else
throw;
}
if (bytesReceived > 0)
{
messageData += Encoding.UTF8.GetString( buffer, 0, bytesReceived );
string ack = ProcessMessage( messageData );
var writeBuffer = Encoding.UTF8.GetBytes( ack );
if (netStream.CanWrite) { netStream.Write( writeBuffer, 0, writeBuffer.Length ); }
messageData = String.Empty;
}
}
}
catch (Exception e) { ... }
finally { FileLogger.Log( "Closing the message stream.", Verbose.Debug, DeviceName ); }
}
}
For most clients the code is running correctly, but there are a few that seem to create a new connection for each message. I suspect that the issue lies around how I handle the IOException. For the systems that fail, the code does not seem to reach the finally statement until 30 seconds after the first message comes in, and each message creates a new ReceiveMessage thread. So the logs will show messages coming in, and 30 seconds in it will start to show multiple messages about the message stream being closed.
Below is how I check the connection, in case this is important.
public static bool GetConnectionState( TcpClient tcpClient )
{
var state = IPGlobalProperties.GetIPGlobalProperties()
.GetActiveTcpConnections()
.FirstOrDefault( x => x.LocalEndPoint.Equals( tcpClient.Client.LocalEndPoint )
&& x.RemoteEndPoint.Equals( tcpClient.Client.RemoteEndPoint ) );
return state != null ? state.State == TcpState.Established : false;
}

You're reinventing the wheel (in a worse way) at quite a few levels:
You're doing pseudo-blocking sockets. That combined with creating a whole new thread for every connection in an OS like Linux which doesn't have real threads can get expensive fast. Instead you should create a pure blocking socket with no read timeout (-1) and just listen on it. Unlike UDP, TCP will catch the connection being terminated by the client without you needing to poll for it.
And the reason why you seem to be doing the above is that you reinvent the standard Keep-Alive TCP mechanism. It's already written and works efficiently, simply use it. And as a bonus, the standard Keep-Alive mechanism is on the client side, not the server side, so even less processing for you.
Edit: And 3. You really need to cache the threads you so painstakingly created. The system thread pool won't suffice if you have that many long-term connections with a single socket communication per thread, but you can build your own expandable thread pool. You can also share multiple sockets on one thread using select, but that's going to change your logic quite a bit.

Delay in processing command from socket listener

I have a special socket listener, that works in it's thread. It's job to get commands from external program that updates database. When command comes on socket i am calling special method, that updates my application cache from database.
I have a problem, that is delay between sending command from external program and processing that command in my app (ASP .NET Application). Every day my app restarting at 4 a.m. and by the end of the day i have delay about 1-2 hours.
How i can reduce this delay?
You can find code of my listener below.
Thanks.
public delegate void OnECIGetCommand( string command );
public class ECIMain
{
protected Socket socket;
protected string ip;
protected int port;
private static ECIMain INSTANCE = null;
const int receivedDataSize = 250;
protected static byte[] buffer = new byte[ receivedDataSize ];
protected static StringBuilder sb;
protected static DoWorkEventHandler onCommand;
private ECIMain()
{
socket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);
sb = new StringBuilder();
}
private void StartSocket()
{
sb.Clear();
socket.Listen(1);
socket.BeginAccept(null, receivedDataSize,
new AsyncCallback(AcceptReceiveDataCallback), socket);
}
private static void AcceptReceiveDataCallback(IAsyncResult ar)
{
// Get the socket that handles the client request.
Socket listener = (Socket)ar.AsyncState;
// End the operation and display the received data on the console.
byte[] Buffer;
int bytesTransferred;
Socket handler = listener.EndAccept(out Buffer,
out bytesTransferred, ar);
HandleBuff(bytesTransferred, Buffer);
// Create the state object for the asynchronous receive.
handler.BeginReceive(buffer, 0, receivedDataSize,
SocketFlags.None, new AsyncCallback(ReadCallback), handler);
}
private static void HandleBuff(int size, byte[] buff )
{
if (size > 0)
{
// There might be more data, so store the data received so far.
sb.Append(Encoding.ASCII.GetString(buff, 0, size));
// Check for end-of-file tag. If it is not there, read more data.
var content = sb.ToString();
int pos = -1;
if ((pos = content.IndexOf("</cmd>")) > -1)
{
// All the data has been read from the
// client.
pos += 6;
if( pos < content.Length )
content = content.Remove(pos);
var startPos = content.LastIndexOf("<cmd>");
if( startPos > -1 )
{
if (startPos > 0)
content = content.Remove(0, startPos);
BackgroundWorker worker = new BackgroundWorker();
worker.DoWork += onCommand;
worker.RunWorkerAsync(content);
}
sb.Remove(0, pos);
}
}
}
private static void ReadCallback(IAsyncResult ar)
{
// Retrieve the state object and the handler socket
// from the asynchronous state object.
Socket handler = (Socket)ar.AsyncState;
SocketError error;
// Read data from the client socket.
int bytesRead = handler.EndReceive(ar, out error );
if (error == SocketError.Success)
{
if (bytesRead > 0)
{
HandleBuff(bytesRead, buffer);
handler.BeginReceive(buffer, 0, receivedDataSize,
SocketFlags.None, new AsyncCallback(ReadCallback), handler);
}
else
{
handler.Disconnect(true);
INSTANCE.StartSocket();
}
}
else if (error == SocketError.Shutdown || error == SocketError.ConnectionReset)
{
INSTANCE.StartSocket();
}
}
public static string InitECI(int port, DoWorkEventHandler commandHandler)
{
if (INSTANCE == null)
{
INSTANCE = new ECIMain();
INSTANCE.port = port;
onCommand += commandHandler;
IPHostEntry ipHostInfo = Dns.GetHostEntry(Dns.GetHostName());
IPAddress ipAddress = ipHostInfo.AddressList
.FirstOrDefault(a => a.AddressFamily == AddressFamily.InterNetwork);
IPEndPoint localEndPoint = new IPEndPoint(ipAddress, port);
INSTANCE.ip = ipAddress.ToString();
try
{
INSTANCE.socket.Bind(localEndPoint);
}
catch (System.Net.Sockets.SocketException e)
{
if (e.SocketErrorCode == System.Net.Sockets
.SocketError.AddressAlreadyInUse)
{
//INSTANCE.socket.Bind(localEndPoint);
}
throw e;
}
INSTANCE.StartSocket();
}
return INSTANCE.ip;
}
public static void ShutDownECI()
{
if( INSTANCE.socket.Connected )
INSTANCE.socket.Disconnect(false);
INSTANCE.socket.Close();
}
}

When using the TCP Stack (either sending or receiving) one must look at the stack as it's own system, which is bullet proof and works very well... Most of these types of issues deal with the fact that the application layer is easily able to fire off as many async actions as it wants, but that doesn't mean the TCP Stack is going to be faster, especially if it's overwhelmed. What it means instead is that it will queue up and process tasks as it's able.
One of the symptoms of the stack being overwhelmed is the presence of many half session states visualized via the Netstat command. Anything that has the word "Wait" is an indicator of a half state. This occurs when one side posts a packet but the other doesn't respond right away. From there it's all downhill because TCP kicks in and attempts to keep the sessions alive by re-sending the same packet. Multiply this by the number of active sessions and the fact that TCP retries up to 10 times for each packet before timing out, you can see this is the last thing an overwhelmed stack needs.
Your situation could be that the network traffic is exceeding the capacity of the single Network adapter. This is usually solved by adding more network cards and using other means to do load balancing. One way is what they call DNS round robin and it's the least expensive. The other way is an F5 device.
Bottom line, it sounds like your network adapter is being overwhelmed.
Here's few things to check at both side of the wire.
Are all of the sockets being fully closed when the gig is up for each session?
Are you able to run a network monitor to review the overall loads... You typically want 60% or less utilization on any single network adapter.
If your loads are too high then you can talk to DNS folks about using round robin, and put in another card into the same server (it will have different address)
Sometimes it's due to lack of compression of the data being sent on the wire. You can investigate compression techniques too.
Sometimes switches go bad which give MAC-To-MAC connect-ability.
Improper router configuration can allow for re-tranmissions on the wire from different access points.
Incorrectly configured servers can also broadcast too much noise (total junk)
Wireless Access Points are notorious for going flaky they could be the source of noise too.
There's a lot to getting to the root of this type of issue, but hope that some of these ideas will help you.

C# sockets missing some data unless I set a breakpoint

I am troubleshooting an issue on a relatively simple socket application which is listening for status updates from a third party machine. I have set up a TcpListener object to wait for a connection request and then establish the socket to read the data coming in. I get the periodic heartbeat as expected without issue, but whenever there is a sudden change in status the server machine sends out an immediate update which I don't get. The bizarre thing here is that I get the update no problem if I set a breakpoint in the code.
The server itself handles these connections a little strangely and doesn't maintain an open socket connection. when it tries to send data, it opens the connection, sends data, and then closes the connection, which is why I've built this to similarly wait for a connection and close it when the data transfer is done before beginning to listen for another connection request.
private void ListeningThread()
{
bool keep_going = CreateConnection();
CreateTimer();
while (keep_going)
{
try
{
if (m_ThreadShutdownEvent.IsSet)
{
// event was set, so shut down
keep_going = false;
m_Listener.Stop();
bool appshuttingdown = false;
DestroyTimer();
lock (m_Lock)
{
appshuttingdown = m_ApplicationShutDown;
}
if (!appshuttingdown)
{
RunStatusNotification();
}
Connected = false;
}
else
{
if (m_Listener.Pending())
{
Socket socket = m_Listener.AcceptSocket();
if (socket != null)
{
StateObject state = new StateObject();
state.Socket = socket;
try
{
int bytes_read = socket.Receive(state.Buffer, 0, StateObject.BUFFER_SIZE, SocketFlags.None);
DateTime now = DateTime.UtcNow;
if (bytes_read == 14)
{
if (state.Buffer.Count() > 13)
{
int packet = state.Buffer[13];
InterpretRelevantByte(packet, now);
}
}
}
catch (Exception ex)
{
FireUnknownException(ex);
}
finally
{
socket.Close();
}
}
}
}
}
catch (Exception ex)
{
m_Logger.Error(ex);
}
}
}

It's possible that your call to receive gets you some value greater than or less than 14, you should probably add some logic to inspect the data you receive when bytes read is not equal to 14 since in these cases you are discarding what you've read.
int bytes_read = socket.Receive(state.Buffer, 0, StateObject.BUFFER_SIZE, SocketFlags.None);
DateTime now = DateTime.UtcNow;
if (bytes_read == 14)
{
if (state.Buffer.Count() > 13)
{
int packet = state.Buffer[13];
InterpretRelevantByte(packet, now);
}
}
else if (bytes_read > 14)
{
// maybe you received multiple messages in one packet
}
else
{
// maybe there is more data on the way
}

Ok, I've resolved this. Turns out I was closing the socket too soon which led to some weird behavior that, honestly, I don't fully understand, but I do know how I fixed it.
After opening the socket I needed to continue listening for data until receiving a 0 length message which signaled that the server had closed the connection. At that point I could start listening for a new socket connection request. I'm still not sure why I would get the heartbeats only, but everything has been working perfectly since I made the change.

How to create a TcpClient in C# that can recover from a lost network connection?

I am trying to write a TCP client thread that will connect to a server and continue to process data as it receives it. Sometimes the client will lose connection to the server. When the connection is re-established, the thread should resume/recover and automatically start processing data again. I can't seem to get this to work. The basic algorithm is below. My problem is that I just don't quite understand how the TcpClient behaves when the network connection is lost. How can I tell that the connection has been lost? Do I then need to close the connection? How do I reestablish the connection and continue on?
TcpClient _tcpClient;
IPEndPoint _ipEndPoint;
bool _cancelled = false;
bool _error = true;
while (!_cancelled)
{
try
{
if(_error)
{
_ipEndPoint = new IPEndPoint(_myAddress, _myPort);
_tcpClient.Connect(_ipEndPoint);
_networkStream = _tcpClient.GetStream();
_error = false;
}
else
{
_data = new byte[10025];
if(_networkStream.CanRead)
{
_bytesRead = _networkStream.Read(_data, 0, (int)_tcpClient.ReceiveBufferSize);
if(_bytesRead > 0)
{
...process the data...
}
else
{
_error = true;
}
}
else
{
_error = true;
}
}
}
catch(Exception ex)
{
...log error...
_error = true;
}
}

there is a Connected property on the TcpClient, but it is only updated when a Write or Read is executed on the NetworkStream.
I think you can just execute a Read(null,0,0) or Write(null,0,0) if you want to force a connectivity check. But in the example you have you can check the Connected or CanRead properties after your _networkStream.Read is completed.
As for reestablishing the link what you have will work. I would suggest Disposing of the old network stream before getting a new one. Something like this:
if(_networkStream != null)
{
_networkStream.Dispose();
}
_networkStream = _tcpClient.GetStream();
What I do is start a reconnect timer that will attempt to reconnect on a configured interval. Depending on the TcpHost your connecting to you may even want to start trying at a small interval 500-1000ms and increment is after a given number of failed retries so your not wasting a lot of time trying to connect to a host that is gone. Then after a max number of tries just give up unless the user explicitly requests to try again. But that also depends on what else your app is doing, if the connection is its sole purpose or if its just one piece.
The wrapper class I use to interact with TcpClient is around 700 lines of code and it handles reconnects, sending data as well as reading it. I work in a closed shop so i cant post it but if you have any other specific question I'd be happy to help.
Good luck.

Non blocking UDP socket .NET "A non-blocking socket operation could not be completed immediately" exception

I have written a small UDP client server class in C# that is used to provide comms between a Linux and a Windows machines.
The implementation of the UDP client and server in C# in Windows is a direct rewrite from C++ code I wrote for Linux originally.
I have no problems during run time between Linux machines but there is an intermittent problem that occasionally appears between Linux and Windows link.
Due to the application I need fast, non blocking operation of the UDP socket.
Since one client is Linux the code under C# I had to use some magic of marshalling.
Here is the code:
public bool Connect(string sIPAddr, int portNumber)
{
try
{
if (portNumber > 65535 && portNumber < 0)
{
this._isReady = false;
return this._isReady;
}
this._ipPort = portNumber;
this._ipAddress = IPAddress.Parse(sIPAddr);
IPEndPoint ipep = new IPEndPoint(this._ipAddress, this._ipPort);
this._myUDPClient = new Socket(ipep.Address.AddressFamily, SocketType.Dgram, ProtocolType.Udp);
this._myUDPClient.Blocking = false;
this._myUDPClient.Connect(this._ipAddress, this._ipPort);
this._isReady = true;
return this._isReady;
}
catch (Exception)
{
this._isReady = false;
return this._isReady;
}
}
I use connect on UDP to simplify send and receive calls.
The problem happens when I try and read from the socket.
More code:
public bool NewMessageReceived()
{
try
{
if (this._newMessaageReceived)
{
return this._newMessaageReceived;
}
else
{
_messageBuffer = new byte[65507];
int numBytesRcvd = _myUDPClient.Receive(this._messageBuffer, 65507, SocketFlags.None);
Marshal.Copy(_messageBuffer, 0, _pmessageBuffer, 65507);
if (numBytesRcvd < 0)
{
this._newMessaageReceived = false;
// TODO: Add Socket Error Checking
}
else
{
this._newMessaageReceived = true;
}
Array.Clear(_messageBuffer, 0, _messageBuffer.GetLength(0));
return this._newMessaageReceived;
}
}
catch (Exception e)
{
System.Windows.Forms.MessageBox.Show(e.Message);
return false;
}
}
I have Wireshark running on both machines and I can see that the datagram sent from Linux machine arrives on Windows machine unharmed. However the UDP client Receive call throws and exception saying: "A non-blocking socket operation could not be completed
immediately" which from what I understand is a WSAEWOULDBLOCK error. However I explicitly set blocking option to false.
The sequence of events is the following:
Windows machine sends a datagram on port 2 and listens for acknowledge on port 1. I have a while loop which implements timeout
Code:
DateTime TimeAtStart = new DateTime();
TimeAtStart = DateTime.Now;
TimeSpan TimeOut = new TimeSpan(0,0,0,0,800);
IntPtr RecievedTelPtr = new IntPtr();
bool UnpackingResult;
while (TimeOut > (DateTime.Now - TimeAtStart))
{
if (!NackAckRecieveConnection.GetIsReady())
{
ErrorEventArguements.SetAllHmiNetworkEventArgs(ID, -3, 2);
return (false);
}
if (NackAckRecieveConnection.NewMessageReceived())
{
RecievedTelPtr = NackAckRecieveConnection.GetMessage();
UnpackingResult = UnpackHmiTelegram(RecievedTelPtr, AckNackType);
NackAckRecieveConnection.MessageRetrieved();
return (UnpackingResult);
}
}
//if escape loop return timeout err msg
ErrorEventArguements.SetAllHmiNetworkEventArgs(ID, -4, (AckNackType == 0) ? (1) : (3));
return (false);
I would like to be able to understand the issue and why the problem occurs and how can I fix it as I have fun out of ideas.
Thank you

I'm not answering the question, but I do need to point out something very important:
catch (Exception)
{
this._isReady = false;
return this._isReady;
}
Do NOT hide exceptions like that. When something fails you will have no chance what so ever to try to fix it, since you will never know why something failed. Do use proper exception handling.
Due to the application I need fast, non blocking operation of the UDP socket
That statement is not correct. Non-blocking sockets are not faster, they simply return before the operation has completed.
I do recommend that you switch back to blocking sockets, since you seem to be new to socket programming. Get the application running first, then try to optimize it.

You are setting the socket that you're reading messages from to non-blocking. This instructs the socket to NOT BLOCK if the operation cannot be completed immediately. What this means in practical terms is that if you attempt to read from the socket and there is nothing waiting to be read, the call will not return successfully.
I don't know how MessageReceived is being called, however I would assume that whatever is calling it is not checking that information is actually ready to be read from the socket, prior to the call.
As you're experiencing an intermittent problem, it would suggest that most of the time, the when MessageReceived is being called, there is data to be read from the socket.
If you want to continue to use non-blocking IO, you need to either change your logic, so that it catches the IO exception and retrys after a short delay (if you're sure there's going to be data there), or check to see if there is actually data available to be read from the socket, prior to attempting to perform the read.
One way to check if there is information available on the socket (prior to attempting to read from it) would be to use Socket.Poll. Something like:
if (_myUDPClient.Poll(myTimeoutInMicroSeconds, SelectMode.SelectRead)){
// Try to read the new messsage
} else {
continue; // go back to top of loop and try again
}
You may also need to check for SelectError state, to determine if the socket has a failure. Most of my socket programming has been from C++, so I'm not sure about the .Net specifics.