We Keep Coding

TCP server connection causing processor overload

I have a TCP/IP server that is supposed to allow a connection to remain open as messages are sent across it. However, it seems that some clients open a new connection for each message, which causes the CPU usage to max out. I tried to fix this by adding a time-out but still seem to have the problem occasionally. I suspect that my solution was not the best choice, but I'm not sure what would be.
Below is my basic code with logging, error handling and processing removed.
private void StartListening()
{
try
{
_tcpListener = new TcpListener( IPAddress.Any, _settings.Port );
_tcpListener.Start();
while (DeviceState == State.Running)
{
var incomingConnection = _tcpListener.AcceptTcpClient();
var processThread = new Thread( ReceiveMessage );
processThread.Start( incomingConnection );
}
}
catch (Exception e)
{
// Unfortunately, a SocketException is expected when stopping AcceptTcpClient
if (DeviceState == State.Running) { throw; }
}
finally { _tcpListener?.Stop(); }
}
I believe the actual issue is that multiple process threads are being created, but are not being closed. Below is the code for ReceiveMessage.
private void ReceiveMessage( object IncomingConnection )
{
var buffer = new byte[_settings.BufferSize];
int bytesReceived = 0;
var messageData = String.Empty;
bool isConnected = true;
using (TcpClient connection = (TcpClient)IncomingConnection)
using (NetworkStream netStream = connection.GetStream())
{
netStream.ReadTimeout = 1000;
try
{
while (DeviceState == State.Running && isConnected)
{
// An IOException will be thrown and captured if no message comes in each second. This is the
// only way to send a signal to close the connection when shutting down. The exception is caught,
// and the connection is checked to confirm that it is still open. If it is, and the Router has
// not been shut down, the server will continue listening.
try { bytesReceived = netStream.Read( buffer, 0, buffer.Length ); }
catch (IOException e)
{
if (e.InnerException is SocketException se && se.SocketErrorCode == SocketError.TimedOut)
{
bytesReceived = 0;
if(GlobalSettings.IsLeaveConnectionOpen)
isConnected = GetConnectionState(connection);
else
isConnected = false;
}
else
throw;
}
if (bytesReceived > 0)
{
messageData += Encoding.UTF8.GetString( buffer, 0, bytesReceived );
string ack = ProcessMessage( messageData );
var writeBuffer = Encoding.UTF8.GetBytes( ack );
if (netStream.CanWrite) { netStream.Write( writeBuffer, 0, writeBuffer.Length ); }
messageData = String.Empty;
}
}
}
catch (Exception e) { ... }
finally { FileLogger.Log( "Closing the message stream.", Verbose.Debug, DeviceName ); }
}
}
For most clients the code is running correctly, but there are a few that seem to create a new connection for each message. I suspect that the issue lies around how I handle the IOException. For the systems that fail, the code does not seem to reach the finally statement until 30 seconds after the first message comes in, and each message creates a new ReceiveMessage thread. So the logs will show messages coming in, and 30 seconds in it will start to show multiple messages about the message stream being closed.
Below is how I check the connection, in case this is important.
public static bool GetConnectionState( TcpClient tcpClient )
{
var state = IPGlobalProperties.GetIPGlobalProperties()
.GetActiveTcpConnections()
.FirstOrDefault( x => x.LocalEndPoint.Equals( tcpClient.Client.LocalEndPoint )
&& x.RemoteEndPoint.Equals( tcpClient.Client.RemoteEndPoint ) );
return state != null ? state.State == TcpState.Established : false;
}

You're reinventing the wheel (in a worse way) at quite a few levels:
You're doing pseudo-blocking sockets. That combined with creating a whole new thread for every connection in an OS like Linux which doesn't have real threads can get expensive fast. Instead you should create a pure blocking socket with no read timeout (-1) and just listen on it. Unlike UDP, TCP will catch the connection being terminated by the client without you needing to poll for it.
And the reason why you seem to be doing the above is that you reinvent the standard Keep-Alive TCP mechanism. It's already written and works efficiently, simply use it. And as a bonus, the standard Keep-Alive mechanism is on the client side, not the server side, so even less processing for you.
Edit: And 3. You really need to cache the threads you so painstakingly created. The system thread pool won't suffice if you have that many long-term connections with a single socket communication per thread, but you can build your own expandable thread pool. You can also share multiple sockets on one thread using select, but that's going to change your logic quite a bit.

Reset TCP connection if server closes/crashes mid connection

I have a tcp connection like follows:
public void ConnectToServer()
{
string mac = GetUID();
while(true)
{
try
{
tcpClient = new TcpClient("xx.x.xx.xxx", xxxx);
networkstream = new SslStream(tcpClient.GetStream());
networkstream.AuthenticateAsClient("xx.x.xx.xxx");
networkstream.Write(Encoding.UTF8.GetBytes("0002:" + mac + "\r\n"));
networkstream.Flush();
string serverMessage = ReadMessage(networkstream);
Console.WriteLine("MESSAGE FROM SERVER: " + serverMessage);
}
catch (Exception e)
{
tcpClient.GetStream().Close();
tcpClient.Close();
}
}
}
This works fine and can send a receive data to/from the server.
What I need help with, if the server isn't running when the client starts, it'll wait and then connect once the server is up. But, if both the client and server are running and everything is working, if I close the server, the client will not reconnect(because I don't have anything to handle the event yet).
I have seen some answers on here that suggest polling and such. Is that the only way? The ReadMessage method that I call get into an infinite loop as well. I can post that code if need be.
I would really like to detect when the server closes/crashes and close the stream and the tcpclient and reconnect ASAP.
Here is my readmessage:
static string ReadMessage(SslStream sslStream)
{
if (sslStream.CanRead)
{
byte[] buffer = new byte[2048];
StringBuilder messageData = new StringBuilder();
int bytes = -1;
string message_type = null;
string actual_message = null;
do
{
try
{
Console.WriteLine("LENGTH: " + buffer.Length);
bytes = sslStream.Read(buffer, 0, buffer.Length);
Decoder decoder = Encoding.UTF8.GetDecoder();
char[] chars = new char[decoder.GetCharCount(buffer, 0, bytes)];
decoder.GetChars(buffer, 0, bytes, chars, 0);
messageData.Append(chars);
message_type = messageData.ToString().Substring(0, 5);
actual_message = messageData.ToString().Substring(5);
if (message_type.Equals("0001:"))
{
m_Window pop = new m_Window();
pop.callHttpPost(null, new EventArgs());
}
if (messageData.ToString().IndexOf("\r\n") != -1)
{
break;
}
}
catch (Exception e)
{
Console.WriteLine("ERROR: " + e.Message);
}
} while (bytes != 0);
return messageData.ToString();
}
return("CONNECTION HAS BEEN LOST");
}

With TCP you have 2 kinds of a server disconnect:
the server is closed
the server crashes
When the server is closed, you are going to receive 0 bytes on your client socket, this is the way you know that the peer has closed its end of the socket, which is called a half close.
But thing get more ugly if the server crashes.
When that happens again you have several possibilities.
If you don't send anything from the client to the server, the you have not way to find out that the server has indeed crashed.
The only way to find out that the server crashed is by letting the client send something or by activating keep alive. If you send something to a server socket that does not exist, you will have to wait a rather long period, because TCP is going to try several times, with retransmits untill there is a server response. When TCP has retried several times, then it will finally bail out and if you have a blocking socket you will see that the send failed, which means you should close your socket.
Actually there is a third possible server disconnect, that is a reset, but this is exceptionally used. I assume here that if there is a gracefull server shutdown, a normal close on the socket on the server end is executed. Which will end up in a FIN being sent instead of a RST, which is the exceptional case.
Now back to your situation, if the server crashes, it is inherently in the design of TCP, because of all those retransmission timeouts and increasing delays, that you will have to wait some time to actually detect that there is a problem. If the server is gracefully closed and startup again, this is not the case, this you detect immediately by receiving 0 bytes.

C# TCPClient - Leave It Open or Close After Each Transmission?

I have an industrial application that uses the .NET TcpClient object in C#.
I am simply receiving a short barcode string from two different barcode readers at fairly long, inconsistent intervals. The application works great for awhile, but eventually I get a connection issue with one or the other barcode reader. The connection gets lost, and sometimes the scanner will stop accepting connections until it is restarted.
As the application is now, I create the connection just once. When the scanner is triggered (from another IO device) I get the stream and the barcode comes in fine.
I am wondering if I should instead be waiting for the trigger to do everything at once- i.e. create the connection, get the stream, close the connection and dispose of the TcpClient each trigger. I am hoping this or some other method will keep my devices from dropping their connections and hanging up.
The tough part is that I have no idea what the "servers" (scanners) are doing because I can't debug them.
Any help is appreciated!
Thanks.
Simple connection code:
private void Connect()
{
IPAddress IP = IPAddress.Parse(IPString);
try
{
TCPClient.Connect(IP, PortNumber);
}
catch
{
// Connection failed
Message = "Connection Failed #" + IP;
}
if (TCPClient.Connected == true)
{
// Connection Succeeded
Message = "Connection Established #" + IP;
TCPSocket = TCPClient.Client;
}
}
Simple Data Receive Code:
public bool DataAvailable()
{
String data = null;
Byte[] buffer = new Byte[256];
int bytesRead;
NetworkStream Stream = TCPClient.GetStream();
if (!Stream.DataAvailable) return false;
else
{
do
{
bytesRead = Stream.Read(buffer, 0, buffer.Length);
data = Encoding.ASCII.GetString(buffer, 0, bytesRead);
}
while (Stream.DataAvailable);
}
Message = data;
return true;
}
I also have this simple connection test that gets run on each scanner every minute or so, but I'm pretty sure it's not doing what I intended it to (and it also may be responsible for the drops?):
public bool TestConnection()
{
try
{
return !(TCPSocket.Poll(1000, SelectMode.SelectRead) && TCPSocket.Available == 0);
}
catch (SocketException) { return false; }
}

C# sockets missing some data unless I set a breakpoint

I am troubleshooting an issue on a relatively simple socket application which is listening for status updates from a third party machine. I have set up a TcpListener object to wait for a connection request and then establish the socket to read the data coming in. I get the periodic heartbeat as expected without issue, but whenever there is a sudden change in status the server machine sends out an immediate update which I don't get. The bizarre thing here is that I get the update no problem if I set a breakpoint in the code.
The server itself handles these connections a little strangely and doesn't maintain an open socket connection. when it tries to send data, it opens the connection, sends data, and then closes the connection, which is why I've built this to similarly wait for a connection and close it when the data transfer is done before beginning to listen for another connection request.
private void ListeningThread()
{
bool keep_going = CreateConnection();
CreateTimer();
while (keep_going)
{
try
{
if (m_ThreadShutdownEvent.IsSet)
{
// event was set, so shut down
keep_going = false;
m_Listener.Stop();
bool appshuttingdown = false;
DestroyTimer();
lock (m_Lock)
{
appshuttingdown = m_ApplicationShutDown;
}
if (!appshuttingdown)
{
RunStatusNotification();
}
Connected = false;
}
else
{
if (m_Listener.Pending())
{
Socket socket = m_Listener.AcceptSocket();
if (socket != null)
{
StateObject state = new StateObject();
state.Socket = socket;
try
{
int bytes_read = socket.Receive(state.Buffer, 0, StateObject.BUFFER_SIZE, SocketFlags.None);
DateTime now = DateTime.UtcNow;
if (bytes_read == 14)
{
if (state.Buffer.Count() > 13)
{
int packet = state.Buffer[13];
InterpretRelevantByte(packet, now);
}
}
}
catch (Exception ex)
{
FireUnknownException(ex);
}
finally
{
socket.Close();
}
}
}
}
}
catch (Exception ex)
{
m_Logger.Error(ex);
}
}
}

It's possible that your call to receive gets you some value greater than or less than 14, you should probably add some logic to inspect the data you receive when bytes read is not equal to 14 since in these cases you are discarding what you've read.
int bytes_read = socket.Receive(state.Buffer, 0, StateObject.BUFFER_SIZE, SocketFlags.None);
DateTime now = DateTime.UtcNow;
if (bytes_read == 14)
{
if (state.Buffer.Count() > 13)
{
int packet = state.Buffer[13];
InterpretRelevantByte(packet, now);
}
}
else if (bytes_read > 14)
{
// maybe you received multiple messages in one packet
}
else
{
// maybe there is more data on the way
}

Ok, I've resolved this. Turns out I was closing the socket too soon which led to some weird behavior that, honestly, I don't fully understand, but I do know how I fixed it.
After opening the socket I needed to continue listening for data until receiving a 0 length message which signaled that the server had closed the connection. At that point I could start listening for a new socket connection request. I'm still not sure why I would get the heartbeats only, but everything has been working perfectly since I made the change.

How to detect a disconnected socket C#?

I've been working on a socket client program in C# and am wondering how to detect when the other end of a socket has disconnected itself "ungracefully" as in a network cable being unplugged or a hard reset.
I have these functions below to access the socket and according to the SO question here and this MSDN article, the best way to check for a disconnected socket is to send a 1-byte message with a length of 0. If an exception is thrown and WSAEWOULDBLOCK is not the error code then the socket is disconnected. I have tried this but after hard reseting the server connection the client will call Send(new byte[1], 0, 0, SocketFlags.None) and return successfully and the Receive() command right afterwards returns the WSAEWOULDBLOCK error.
What gives??
Here's my code below. _socket is set to non-blocking mode:
private int nonBlockRecv(byte[] recvBytes, int offset, int size, SocketFlags sf)
{
int bytesRecv = 0;
while (true)
{
try
{
nonBlockSend(new byte[1], 0, 0, sf);
bytesRecv = _socket.Receive(recvBytes, offset, size, sf);
break;
}
catch (SocketException excp)
{
if (excp.ErrorCode != 10035) // WSAEWOULDBLOCK
throw excp;
}
}
return bytesRecv;
}
private int nonBlockSend(byte[] sendBytes, int offset, int size, SocketFlags sf)
{
int bytesSent = 0;
while (true)
{
try
{
_socket.Send(sendBytes, offset, size, sf);
break;
}
catch (SocketException excp)
{
if (excp.ErrorCode != 10035) // WSAEWOULDBLOCK
throw excp;
}
}
return bytesSent;
}
Edit: This may be beneficial but the server is Windows Mobile device. I read in another thread that different OSs maybe able to send socket close signals when they're dying. Perhaps the Windows Mobile OS does not do this??

If the remote computer gracefully disconnects the session, the
Socket.Receive() method will return with 0 bytes. You must detect that
to know that the remote end has disconnected:
int recv = sock.Receive(data);
if (recv == 0)
{
// Remote client has disconnected.
}
else
{
// Remote client has sent data.
}
Also, even if there SocketException arises you can identify the exception for socket disconnection.
Hope this helps solve your problem.

I know this is late but I came up with a cunning solution for this.
I had to communicate with 3rd party software which expected a carriage return on every command sent, otherwise it ignored it.
During the main phase my client socket was in a loop receiving responses from the 3rd party software. My solution isn't ideal but the basic premise is that I put a receive timeout on the socket so that the loop will try to read for 5 seconds then fall into the catch, then loop again. Before each receive I call my own isconnected method which performs a small write without a carriage return, so it's ignored by the 3rd party software yet will give me a reliable fallover if the network has dropped. All I do if the write fails is throw a LostConnectionException and handle that externally.
If you are writing both server and client, you can quite easily come up with some checkdigit that the other ignores.
This may not be perfect but it's reliable for me.
while (numberOfBytesRead == 0)
{
try
{
IsConnected();
_socket.ReceiveTimeout = 5000;
numberOfBytesRead = _socket.Receive(myReadBuffer, 0, myReadBuffer.Length, SocketFlags.None);
}
catch (Exception e)
{
if (e.GetType() == typeof (LostConnection))
{
Status = SocketStatus.offline;
throw;
}
}
}
and the isconnected method would look something like this
public bool IsConnected(Socket s)
{
try
{
ASCIIEncoding encoder = new ASCIIEncoding();
byte[] buffer = encoder.GetBytes("test");
s.Send(buffer, 0, buffer.Length, SocketFlags.None);
}
catch (Exception)
{
throw new LostConnection();
}
return s.Connected;
}