We Keep Coding

C# TcpListener and MySqlConnection stops accepting connections after a while

I have an async socket server written in C#, running on a Lightsail server running Amazon Linux. It consists of a TcpListener that accepts connections, starts up a new thread to listen when someone connects, initiates an SSL connection, and then acts as a server for an online game.
This server works fine for about a day, until suddenly all networking stops working on the server. The crash takes anywhere from 22 hours to one week to occur. The symptoms are as follows:
Anyone already connected to the server will suddenly stop receiving/sending data. I can see in the logs that my inactivity checking code will eventually kick them for not sending heartbeat packets.
The server will also be unable to connect to its MySQL database (which is running on the same system, so it's unable to connect to localhost? I can still access it through PHPMyAdmin during this time).
It is, however, still able to write both to files and to console, as my logger is still able to write to both.
The code looks like everyone else's (I did try the changes suggested for this question, but it still crashed after ~24 hours). None of the errors get logged, so it looks like it never encounters an exception. No exceptions precede the crash, which is why I've been having problems figuring this one out.
For completeness, here is my main loop:
public void ListenLoop()
{
TcpListener listener = new TcpListener(IPAddress.Any, 26000);
listener.Start();
while (true)
{
try
{
if (listener.Pending())
{
listener.BeginAcceptTcpClient(new AsyncCallback(AcceptConnection), listener);
Logger.Write(Logger.Level.INFO, "continuing the main loop");
}
// Yield so we're not stuck in a busy-loop
Thread.Sleep(5);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
}
and here are the accept parts:
/// <summary>
/// Finish an async callback but spawn a new thread to handle it if necessary
/// </summary>
/// <param name="ar"></param>
private void AcceptConnection(IAsyncResult ar)
{
if (ar.CompletedSynchronously)
{
// Force the accept logic to run async, to keep our listening
// thread free.
Action accept = () => AcceptCallback(ar);
accept.BeginInvoke(accept.EndInvoke, null);
} else
{
AcceptCallback(ar);
}
}
private void AcceptCallback(IAsyncResult ar)
{
try
{
TcpListener listener = (TcpListener) ar.AsyncState;
TcpClient client = listener.EndAcceptTcpClient(ar);
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = (IPEndPoint)client.Client.RemoteEndPoint;
string ip = ipep.Address.ToString();
Logger.Write(Logger.Level.INFO, $"Connection begun to {ip}");
// Authenticate and begin communicating with the client
SslStream stream = new SslStream(client.GetStream(), false);
try
{
stream.AuthenticateAsServer(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
stream.BeginRead(player.Connection.Buffer, 0, 1024, new AsyncCallback(ReadCallback), player);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}

I'm guessing that your primary issue is that you are not disposing the stream and therefore you are getting socket exhaustion.
Apart from that I would advise you to move to fully async code using Task.
public async Task ListenLoop(CancellationToken cancel) // use a cancellation token to shutdown the loop
{
using (var TcpListener listener = new TcpListener(IPAddress.Any, 26000))
{
listener.Start();
while (!cancel.IsCancellationRequested)
{
try
{
var client = await listener.AcceptTcpClientAsync(cancel);
Task.Run(async () => await AcceptConnection(client, cancel));
Logger.Write(Logger.Level.INFO, "continuing the main loop");
// no need to yield due to async
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
listener.Stop();
}
}
private async Task AcceptConnection(TcpClient client, CancellationToken cancel)
{
try
{
using (client)
{
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
await AcceptConnectionImpl(client, cancel);
}
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}
private async Task AcceptConnectionImpl(TcpClient client, CancellationToken cancel)
{
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = client.Client.RemoteEndPoint;
Logger.Write(Logger.Level.INFO, $"Connection begun to {ipep.Address}");
// Authenticate and begin communicating with the client
using (SslStream stream = new SslStream(client.GetStream(), false))
{
try
{
await stream.AuthenticateAsServerAsync(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
player.Cancellation = cancel;
await player.YourReadLoopAsync();
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
}
The function YourReadLoopAsync should read data from the stream using ReadAsync, or using classes like StreamReader which also has async functions.
You don't need to use CancellationToken, but it does make it easier to deal with shutting everything down cleanly. Make sure to catch OperationCanceledException on every try.
See also this link for further tips.

The solution I found after consulting some people more familiar with C# than me is that I was running into Thread Pool Exhaustion. Essentially, I had a bunch of other async tasks (not shown in the code in the question, as they didn't look like they could cause what I was seeing) that were stuck executing some extremely-long-IOs (talking to users that had either disconnected improperly or were behind very high latency), which prevented the async AcceptCallback in my post from being picked up by the Thread Pool. This had a myriad of other side-effects which I outlined in the question:
Creating a new connection to a MySQL database involves an async task behind-the-scenes, which was being starved out due to exhaustion.
Completing the EndAcceptTcpClient required my async task to run, which requires an available thread.
Tasks which did not involve the async keyword, such as Timer() bound tasks (like my logger I/O) were unaffected and could still run.
My solution involved reducing the number of synchronization steps elsewhere in my program, and restructuring any tasks that could take a long time to execute so that they didn't block threads. Thank you to everyone who looked/commented.

TCP Socket High CPU/Memory usage after 5mins

I'm building a server app that accepts incoming TCP connections. (roughly 300 unique clients). It's important to note that I do not have control over the clients.
I have found that some of the connecting clients remain idle for quite some time after making the initial connection and sending the first status update. When they remain idle for over 5 mins the application's CPU usage jumps to over 90% and remains there.
To address this issue I built in a cancellation token that is triggered after 4 mins. This allows me to kill the connection. The client then detects this and reconnects about a minute later. This solves the high CPU usage issue, but has the side effect of high memory usage, there seems to be a memory leak. I suspect the resources is being held by the previous socket object.
I have a client object that contains the socket connection and information about the connected client. It also manages the incoming messages. There is also a manager class which accepts the incoming connections. It then creates the client object, assigns the socket to it and adds the client object to a concurrent dictionary. Every 10 seconds it checks the dictionary for clients that have been set to _closeConnection = true and calls their dispose method.
Here is the some of client object code:
public void StartCommunication()
{
Task.Run(async () =>
{
ArraySegment<byte> buffer = new ArraySegment<byte>(new byte[75]);
while (IsConnected)
{
try
{
// This is where I suspect the memory leak is originating - this call I suspect is not properly cleaned up when the object is diposed
var result = await SocketTaskExtensions.ReceiveAsync(ClientConnection.Client, buffer, SocketFlags.None).WithCancellation(cts.Token);
if (result > 0)
{
var message = new ClientMessage(buffer.Array, true);
if(message.IsValid)
HandleClientMessage(message);
}
}
catch (OperationCanceledException)
{
_closeConnection = true;
DisconnectReason = "Client has not reported in 4 mins";
}
catch (Exception e)
{
_closeConnection = true;
DisconnectReason = "Error during receive opperation";
}
}
});
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (disposing)
{
_closeConnection = true;
cts.Cancel();
// Explicitly kill the underlying socket
if (UnitConnection.Client != null)
{
UnitConnection.Client.Close();
}
UnitConnection.Close();
cts.Dispose();
}
}
Task Extension Method:
public static async Task<T> WithCancellation<T>(this Task<T> task, CancellationToken cancellationToken)
{
var tcs = new TaskCompletionSource<bool>();
using (cancellationToken.Register(s => ((TaskCompletionSource<bool>)s).TrySetResult(true), tcs))
{
if (task != await Task.WhenAny(task, tcs.Task))
{
throw new OperationCanceledException(cancellationToken);
}
}
return task.Result;
}
Mananger Code:
public bool StartListener()
{
_listener = new TcpListenerEx(IPAddress.Any, Convert.ToInt32(_serverPort));
_listener.Start();
Task.Run(async () =>
{
while (_maintainConnection) // <--- boolean flag to exit loop
{
try
{
HandleClientConnection(await _listener.AcceptTcpClientAsync());
}
catch (Exception e)
{
//<snip>
}
}
});
return true;
}
private void HandleClientConnection(TcpClient client)
{
Task.Run(async () =>
{
try
{
// Create new Coms object
var client = new ClientComsAsync();
client.ClientConnection = client;
// Start client communication
client.StartCommunication();
//_clients is the ConcurrentDictionary
ClientComsAsync existingClient;
if (_clients.TryGetValue(client.ClientName, out existingClient) && existingClient != null)
{
if (existingClient.IsConnected)
existingClient.SendHeatbeat();
if (!existingClient.IsConnected)
{
// Call Dispose on existing client
CleanUpClient(existingClient, "Reconnected with new connection");
}
}
}
catch (Exception e)
{
//<snip>
}
finally
{
//<snip>
}
});
}
private void CleanUpClient(ClientComsAsync client, string reason)
{
ClientComsAsync _client;
_units.TryRemove(client.ClientName, out _client);
if (_client != null)
{
_client.Dispose();
}
}

When they remain idle for over 5 mins the application's CPU usage jumps to over 90% and remains there.
To address this issue I built in a cancellation token that is triggered after 4 mins.
The proper response is to solve the high CPU usage problem.
Looks to me like it's here:
while (IsConnected)
{
try
{
var result = await SocketTaskExtensions.ReceiveAsync(ClientConnection.Client, buffer, SocketFlags.None);
if (result > 0)
{
...
}
}
catch ...
{
...
}
}
Sockets are weird, and dealing with raw TCP/IP sockets is quite difficult to do correctly. On a side note, I always encourage devs to use something more standard like HTTP or WebSockets, but in this case you don't control the clients, so that's not an option.
Specifically, your code is not handling the case where result == 0. If the client devices gracefully closed their socket, you'd see a result of 0, immediately loop back and keep getting a result of 0 - a tight loop that uses up CPU.
This is, of course, assuming that IsConnected remains true. And that may be possible...
You don't show where IsConnected is set in your code, but I suspect it's in the error handling after sending the heartbeat message. So here's why that may not work as expected... I suspect that the client devices are closing their sending stream (your receiving stream) while keeping their receiving stream (your sending stream) open. This is one way to shut down a socket, sometimes considered "more polite" because it allows the other side to continue sending data even though this side is done sending. (This is from the client device perspective, so the "other side" is your code, and "this side" is the client device).
And this is perfectly legal socket-wise because each connected socket is two streams, not one, each of which can be independently closed. If this happens, your heartbeats will still be send and received without error (and likely just silently discarded by the client device), IsConnected will remain true, and the read loop will become synchronous and eat up your CPU.
To resolve, add a check for result == 0 in your read loop and clean up the client just the same as if a heartbeat failed to send.

Lock on dedicated lock-object causes deadlock

Yesterday I came accross a strange problem, which gave me quite some headaches. I have a server application with a Server class, which in turn is derived from a Connection class. The Connection class provides information about the connection state and the possibility to close the connection
public bool Connected
{
get
{
if (connection != null)
{
lock (lockObject)
{
bool blockingState = connection.Blocking;
try
{
connection.Blocking = false;
connection.Send(new byte[1], 1, 0);
}
catch (SocketException e)
{
if (!e.NativeErrorCode.Equals(10035))
{
return false;
}
//is connected, but would block
}
finally
{
connection.Blocking = blockingState;
}
return connection.Connected;
}
}
return false;
}
}
public virtual void CloseConnection()
{
if (Connected)
{
lock (lockObject)
{
connection.Close();
}
}
}
The Server class is resonsible for actually sending data
private void ConnectAndPollForData()
{
try
{
TcpListener listener = new TcpListener(Port);
listener.Start();
while (true)
{
connection = listener.AcceptSocket();
string currentBuffr = string.Empty;
const int READ_BUFFER_SIZE = 1024;
byte[] readBuffr = new byte[READ_BUFFER_SIZE];
while (Connected)
{
int bytesReceived;
lock (lockObject)
{
bytesReceived = connection.Receive(readBuffr, READ_BUFFER_SIZE, SocketFlags.None);
}
currentBuffr += ASCIIEncoding.ASCII.GetString(readBuffr, 0, bytesReceived);
//do stuff
}
}
catch(ThreadAbortException)
{
Thread.ResetAbort();
}
finally
{
}
}
public void SendString(string stringToSend)
{
stringToSend += "\r\n";
if(Connected)
{
lock(lockObject)
{
connection.Send(ASCIIEncoding.UTF7.GetBytes(stringToSend));
}
}
}
There is no other explicit access to the connection object. The ConnectAndPollForData function executes in a seperate thread. Whenever I ran the host in this version (I am currently using a non thread-safe version, which causes other problems) it hang after quite a few lines received via TCP. Pausing the debugger showed me, that one thread tried to execute the code withing the lock of Connected, while the other tried to receive data in the lock of ConnectAndPollForData. This behavior seems strange to me, for I would expect to execute the code within the first lock and then do the second. There do seem to be similar problems when using callbacks like in Deadlocking lock() method or 'Deadlock' with only one locked object? but the situation here is a bit different, for in my situation (I think) the code within the locks should not emit any events that themselves try to obtain a lock on the object.

Let's assume it gets the lock in the second method first. So it is holding the lock, and waiting for data. It is unclear whether this is directly receiving the data sent by the first method, or whether this is looking for a reply from an unrelated server - a reply to the message sent in the first method. But either way, I'm assuming that there won't be data incoming until the outbound message is sent.
Now consider: the outbound message can't be sent, because you are holding an exclusive lock.
So yes, you've deadlocked yourself. Basically, don't do that. There is no need to synchronize between inbound and outbound socket operations, even on the same socket. And since it makes very little sense to have concurrent readers on the same socket, or concurrent writers, I'm guessing you don't actually need those locks at all.

App freezes after Dispatcher.Invoke

I have this application that freezes when calling the dispatcher.invoke for any control.
When i Call the Dispatcher in radiobutton, Grid, Image..etc the App freezes but without giving errors. any help please !!! thank you
I call the thread Method RunClient
private void RunClient()
{
TcpClient client;
// instantiate TcpClient for sending data to server
try
{
// Step 1: create TcpClient and connect to server
client = new TcpClient();
client.Connect(ip, 5001);
// Step 2: get NetworkStream associated with TcpClient
output = client.GetStream();
// create objects for writing and reading across stream
writer = new BinaryWriter(output);
reader = new BinaryReader(output);
string theReply = "";
do
{
try
{
// read the string sent to the server
theReply = reader.ReadString();
int i = 0;
foreach (var x in theReply.Split('#'))
{
ReadString[i] = x;
i++;
}
CheckConnection(ReadString[0]);
}
catch (Exception)
{
//do nothing
}
} while (ReadString[6].Equals(" ") &&
connection.Connected);
updatelabel = () => GameResult(ReadString[6]);
Dispatcher.Invoke(new Action(updatelabel));
if (!connection.Connected)
{
MessageBox.Show("The connection was lost. The game will be closed automatically.");
writer.Close();
reader.Close();
output.Close();
connection.Close();
this.Close();
}
}
// handle exception if error in establishing connection
catch (Exception error)
{
MessageBox.Show("Check Internet Connectivity. Couldn't connect!");
}
}
when the code enters the method ( check connection ) and calls the dispatcher the app freezes.
void CheckConnection(string ii)
{
try
{
if (ii.Equals("Connected"))
{
MessageBox.Show("A Connection was established");
int x = Convert.ToInt32(ReadString[1]);
if (x == 1)
{
updatelabel = () => char1RadioButton2.IsEnabled = false;
char1RadioButton2.Dispatcher.Invoke(new Action(updatelabel));
}
else
{
updatelabel = () => char5RadioButton2.IsEnabled = false;
char5RadioButton2.Dispatcher.Invoke(new Action(updatelabel));
}
updatelabel = () => CreatingGameGrid.Visibility = System.Windows.Visibility.Visible;
CreatingGameGrid.Dispatcher.Invoke(new Action(updatelabel));
updatelabel = () => JoinGameGrid.Visibility = System.Windows.Visibility.Visible;
JoinGameGrid.Dispatcher.Invoke(new Action(updatelabel));
}
else
{
MessageBox.Show("No Such Game found");
this.Close();
}
}
catch (Exception x)
{
MessageBox.Show(x.ToString());
}
}

The Dispatcher.Invoke attempts to synchronously run the specified action on the Dispatcher Thread.
Assuming the RunClient is run on the Dispatcher Thread, and the while loop continues to run while you are trying to Invoke back onto the Dispatcher Thread, the call will freeze.
The simplest solution is to replace all the Dispatcher.Invoke with Dispatcher.BeginInvoke and give it a priority that will run once your RunClient is finished.
The other solution is to run RunClient on a BackgroundWorker.
Similar questions with answers are
Dispatcher.Invoke loop freeze UI
Dispatcher.Invoke hangs main window.
Response to comment on ReadString freeze
Calling Read on a NetworkStream is a blocking call. Well, in fact, it is the Stream obtained by calling TcpClient.GetStream() that blocks. The documentation on MSDN states 'After you have obtained the NetworkStream, call the Write method to send data to the remote host. Call the Read method to receive data arriving from the remote host. Both of these methods block until the specified operation is performed'.
I used dotPeek to see what ReadString was doing and the first thing it does is read the length of the incoming string off the stream using NetworkStream.ReadByte which will block until it has a value to read.
That means the ReadString will sit there until there is data available to read and the amount of data is the same as or more than is expected. You will need to check if you have anything to read before you do by calling stream.DataAvailable or reader.PeekChar.
Alternatively, you could run your socket code on a separate thread. If you are using .Net 4.5, I would take a good look at the Task Parallel Library. ntziolis says in an answer to this question that 'We have made good experiences with that (long being days rather than minutes or hours).'

fast parallel TcpClients with await BeginConnect new Thread Pulse

I am trying to establish a TCP connection with a number of IPs in parallel, and do that as fast as possible. I have converted some older code to use AsyncCTP for that purpose, introducing the parallelism.
Changes to Design and Speed, and Accessing Successful Connections?
My question is three-fold:
How bad is the following flow / what should I change?
i.e. the await starts a bunch of parallel TcpRequest threads,
but within each TcpRequest there is a tcpClient.BeginConnect
as well as another thread being spawn for reading (if connection is successful)
and the writing to the stream is done with a Wait / Pulse mechanism in a while loop.
Secondly, how could i make the process of connecting to a number of targets faster?
Currently, if the ip:port targets are not actually running any servers, then i get the "All Done" printed after about 18 seconds from the start, when trying to connect to about 500 local targets (that are not listening, and thus fail, on those ports).
How could i access the WriteToQueue method of successful connections, from the mothership?
Async Mothership Trying to Connect to All Targets in Parallel
// First get a bunch of IPAddress:Port targets
var endpoints = EndPointer.Get();
// Try connect to all those targets
var tasks = from t in topList select TcpRequester.ConnectAsync(t);
await TaskEx.WhenAll(tasks);
Debug.WriteLine("All Done");
Static Accessor for Individual TcpRequest Tasks
public static Task<TcpRequester> ConnectAsync(IPEndPoint endPoint)
{
var tcpRequester = Task<TcpRequester>.Factory.StartNew(() =>
{
var request = new TcpRequester();
request.Connect(endPoint);
return request;
}
);
return tcpRequester;
}
TcpRequester with BeginConnect TimeOut and new Thread for Reading
public void Connect(IPEndPoint endPoint)
{
TcpClient tcpClient = null;
Stream stream = null;
using (tcpClient = new TcpClient())
{
tcpClient.ReceiveTimeout = 1000;
tcpClient.SendTimeout = 1000;
IAsyncResult ar = tcpClient.BeginConnect(endPoint.Address, endPoint.Port, null, null);
WaitHandle wh;
wh = ar.AsyncWaitHandle;
try
{
if (!ar.AsyncWaitHandle.WaitOne(TimeSpan.FromMilliseconds(1000), false))
{
throw new TimeoutException();
}
if (tcpClient.Client != null)
{
// Success
tcpClient.EndConnect(ar);
}
if (tcpClient.Connected)
{
stream = tcpClient.GetStream();
}
// Start to read stream until told to close or remote close
ThreadStart reader = () => Read(stream);
// Reading is done in a separate thread
var thread = new Thread(reader);
thread.Start();
// See Writer method below
Writer(stream);
} finally
{
wh.Close();
}
}
} catch (Exception ex)
{
if (tcpClient != null)
tcpClient.Close();
}
}
}
Writing to Stream with Wait and Pulse
readonly Object _writeLock = new Object();
public void WriteToQueue(String message)
{
_bytesToBeWritten.Add(Convert(message));
lock (_writeLock)
{
Monitor.Pulse(_writeLock);
}
}
void Writer(Stream stream)
{
while (!_halt)
{
while (_bytesToBeWritten.Count > 0 && !_halt)
{
// Write method does the actual writing to the stream:
if (Write(stream, _bytesToBeWritten.ElementAt(0)))
{
_bytesToBeWritten.RemoveAt(0);
} else
{
Discontinue();
}
}
if (!(_bytesToBeWritten.Count > 0) && !_halt)
{
lock (_writeLock)
{
Monitor.Wait(_writeLock);
}
}
}
Debug.WriteLine("Discontinuing Writer and TcpRequester");
}

There are a few red flags that pop out at a cursory glance.
You have this Stream that is accepting reads and writes, but there is no clear indication that the operations have been synchronized appropriately. The documentation does state that a Stream's instance methods are not safe for multithreaded operations.
There does not appear to be synchronization around operations involving _bytesToBeWritten.
Acquiring a lock solely to execute Monitor.Wait and Monitor.Pulse is a little weird, if not downright incorrect. It is basically equivalent to using a ManualResetEvent.
It is almost never correct to use Monitor.Wait without a while loop. To understand why you have to understand the purpose of pulsing and waiting on a lock. That is really outside the scope of this answer.
It appears like the Writer and WriteToQueue methods are an attempt to generate a producer-consumer queue. The .NET BCL already contains the innards for this via the BlockingCollection class.
For what it is worth I see nothing flagrantly wrong with the general approach and usage of the await keyword.