We Keep Coding

C# TcpListener and MySqlConnection stops accepting connections after a while

I have an async socket server written in C#, running on a Lightsail server running Amazon Linux. It consists of a TcpListener that accepts connections, starts up a new thread to listen when someone connects, initiates an SSL connection, and then acts as a server for an online game.
This server works fine for about a day, until suddenly all networking stops working on the server. The crash takes anywhere from 22 hours to one week to occur. The symptoms are as follows:
Anyone already connected to the server will suddenly stop receiving/sending data. I can see in the logs that my inactivity checking code will eventually kick them for not sending heartbeat packets.
The server will also be unable to connect to its MySQL database (which is running on the same system, so it's unable to connect to localhost? I can still access it through PHPMyAdmin during this time).
It is, however, still able to write both to files and to console, as my logger is still able to write to both.
The code looks like everyone else's (I did try the changes suggested for this question, but it still crashed after ~24 hours). None of the errors get logged, so it looks like it never encounters an exception. No exceptions precede the crash, which is why I've been having problems figuring this one out.
For completeness, here is my main loop:
public void ListenLoop()
{
TcpListener listener = new TcpListener(IPAddress.Any, 26000);
listener.Start();
while (true)
{
try
{
if (listener.Pending())
{
listener.BeginAcceptTcpClient(new AsyncCallback(AcceptConnection), listener);
Logger.Write(Logger.Level.INFO, "continuing the main loop");
}
// Yield so we're not stuck in a busy-loop
Thread.Sleep(5);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
}
and here are the accept parts:
/// <summary>
/// Finish an async callback but spawn a new thread to handle it if necessary
/// </summary>
/// <param name="ar"></param>
private void AcceptConnection(IAsyncResult ar)
{
if (ar.CompletedSynchronously)
{
// Force the accept logic to run async, to keep our listening
// thread free.
Action accept = () => AcceptCallback(ar);
accept.BeginInvoke(accept.EndInvoke, null);
} else
{
AcceptCallback(ar);
}
}
private void AcceptCallback(IAsyncResult ar)
{
try
{
TcpListener listener = (TcpListener) ar.AsyncState;
TcpClient client = listener.EndAcceptTcpClient(ar);
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = (IPEndPoint)client.Client.RemoteEndPoint;
string ip = ipep.Address.ToString();
Logger.Write(Logger.Level.INFO, $"Connection begun to {ip}");
// Authenticate and begin communicating with the client
SslStream stream = new SslStream(client.GetStream(), false);
try
{
stream.AuthenticateAsServer(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
stream.BeginRead(player.Connection.Buffer, 0, 1024, new AsyncCallback(ReadCallback), player);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}

I'm guessing that your primary issue is that you are not disposing the stream and therefore you are getting socket exhaustion.
Apart from that I would advise you to move to fully async code using Task.
public async Task ListenLoop(CancellationToken cancel) // use a cancellation token to shutdown the loop
{
using (var TcpListener listener = new TcpListener(IPAddress.Any, 26000))
{
listener.Start();
while (!cancel.IsCancellationRequested)
{
try
{
var client = await listener.AcceptTcpClientAsync(cancel);
Task.Run(async () => await AcceptConnection(client, cancel));
Logger.Write(Logger.Level.INFO, "continuing the main loop");
// no need to yield due to async
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
listener.Stop();
}
}
private async Task AcceptConnection(TcpClient client, CancellationToken cancel)
{
try
{
using (client)
{
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
await AcceptConnectionImpl(client, cancel);
}
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}
private async Task AcceptConnectionImpl(TcpClient client, CancellationToken cancel)
{
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = client.Client.RemoteEndPoint;
Logger.Write(Logger.Level.INFO, $"Connection begun to {ipep.Address}");
// Authenticate and begin communicating with the client
using (SslStream stream = new SslStream(client.GetStream(), false))
{
try
{
await stream.AuthenticateAsServerAsync(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
player.Cancellation = cancel;
await player.YourReadLoopAsync();
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
}
The function YourReadLoopAsync should read data from the stream using ReadAsync, or using classes like StreamReader which also has async functions.
You don't need to use CancellationToken, but it does make it easier to deal with shutting everything down cleanly. Make sure to catch OperationCanceledException on every try.
See also this link for further tips.

The solution I found after consulting some people more familiar with C# than me is that I was running into Thread Pool Exhaustion. Essentially, I had a bunch of other async tasks (not shown in the code in the question, as they didn't look like they could cause what I was seeing) that were stuck executing some extremely-long-IOs (talking to users that had either disconnected improperly or were behind very high latency), which prevented the async AcceptCallback in my post from being picked up by the Thread Pool. This had a myriad of other side-effects which I outlined in the question:
Creating a new connection to a MySQL database involves an async task behind-the-scenes, which was being starved out due to exhaustion.
Completing the EndAcceptTcpClient required my async task to run, which requires an available thread.
Tasks which did not involve the async keyword, such as Timer() bound tasks (like my logger I/O) were unaffected and could still run.
My solution involved reducing the number of synchronization steps elsewhere in my program, and restructuring any tasks that could take a long time to execute so that they didn't block threads. Thank you to everyone who looked/commented.

Loop won't stop with Thread and CancellationToken

I'm working with a Windows socket application using async callbacks. If I use Thread to start _StartListening, when I call StopListening, the loop still stops at allDone.WaitOne(). But the Task version will be OK.
What's the difference?
My code is a modified version of this
The original version with ManualResetEvent has a race condition mentioned by felix-b. I changed it to SemaphoreSlim but the problem is still there.
I tried in debug mode and it seems that the break point never be hit at if (cancelToken.IsCancellationRequested) after I call StopListening even I don't start the client.
Sorry. I found that I accidentally started two socket servers. That's the problem.
class WinSocketServer:IDisposable
{
public SemaphoreSlim semaphore = new SemaphoreSlim(0);
private CancellationTokenSource cancelSource = new CancellationTokenSource();
public void AcceptCallback(IAsyncResult ar)
{
semaphore.Release();
//Do something
}
private void _StartListening(CancellationToken cancelToken)
{
try
{
while (true)
{
if (cancelToken.IsCancellationRequested)
break;
Console.WriteLine("Waiting for a connection...");
listener.BeginAccept(new AsyncCallback(AcceptCallback),listener);
semaphore.Wait();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
Console.WriteLine("Complete");
}
public void StartListening()
{
Task.Run(() => _StartListening(cancelSource.Token));//OK
var t = new Thread(() => _StartListening(cancelSource.Token));
t.Start();//Can't be stopped by calling StopListening
}
public void StopListening()
{
listener.Close();
cancelSource.Cancel();
semaphore.Release();
}
public void Dispose()
{
StopListening();
cancelSource.Dispose();
semaphore.Dispose();
}
}

Your code has a race condition that can lead to deadlock (sometimes). Let's name the threads "listener" (one that runs _StartListening) and "control" (one that runs StopListening):
Listener thread: if (cancelToken.IsCancellationRequested) -> false
Control thread: cancelSource.Cancel()
Control thread: allDone.Set()
Listener thread: allDone.Reset() -> accidentally resets the stop request!
Listener thread: listener.BeginAccept(...)
Control thread: stopListening() exits, while the listener continues to work!
Listener thread: allDone.WaitOne() -> deadlock! no one will do allDone.Set().
The problem is in how you use the allDone event, it should be the other way around: _StartListening should do allDone.Set() just before it exits for whatever reason, whereas StopListening should do allDone.WaitOne():
class WinSocketServer:IDisposable
{
// I guess this was in your code, necessary to show proper stopping
private Socket listener = new Socket(......);
public ManualResetEvent allDone = new ManualResetEvent(false);
private CancellationTokenSource cancelSource = new CancellationTokenSource();
private void _StartListening(CancellationToken cancelToken)
{
try
{
listener.Listen(...); // I guess
allDone.Reset(); // reset once before starting the loop
while (!cancelToken.IsCancellationRequested)
{
Console.WriteLine("Waiting for a connection...");
listener.BeginAccept(new AsyncCallback(AcceptCallback),listener);
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
allDone.Set(); // notify that the listener is exiting
Console.WriteLine("Complete");
}
public void StartListening()
{
Task.Run(() => _StartListening(cancelSource.Token));
}
public void StopListening()
{
// notify the listener it should exit
cancelSource.Cancel();
// cancel possibly pending BeginAccept
listener.Close();
// wait until the listener notifies that it's actually exiting
allDone.WaitOne();
}
public void Dispose()
{
StopListening();
cancelSource.Dispose();
allDone.Dispose();
}
}
UPDATE
It worth noting that listener.BeginAccept won't return until there is a new client connection. When stopping the listener, it is necessary to close the socket (listener.Close()) to force BeginAccept to exit.
The difference in Thread/Task behavior is really weird, it probably can originate from the Task thread being a background thread, while the regular thread being a foreground one.

.NET - Socket server with async/await

I'm writing simple socket server with C#. The idea is to do it as simple as possible since the communication won't be heavy. I've used some of the TAP/APM patterns on socket async calls so the code now looks like this:
public async Task StartListening()
{
try
{
var endpoint = new IPEndPoint(IPAddress.Loopback, Port);
using (Socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
{
Socket.Bind(endpoint);
Socket.Listen(Limit);
while (!Shutdown)
{
// await till we get the connection - let the main thread/caller continue (I expect this loop to continue in separate thread after await)
var socket = await Socket.AcceptAsync().ConfigureAwait(false);
var state= new ClientStateObject(socket, id);
// do not await for receive - continue to listen for connections
Receive(socket, state);
}
}
}
catch (Exception ex)
{
// handle errors via callbacks
}
}
private async void Receive(Socket socket, ClientStateObject state)
{
try
{
while (!Shutdown)
{
var bytes = await socket.ReceiveAsync(state).ConfigureAwait(false);
var readResult = state.Read(bytes);
if (readResult == CloseConn)
{
// close connection
return;
}
if (readResult == Completed)
{
// process the message
state.Reset();
}
}
}
catch (Exception ex)
{
// handle errors via callbacks
}
}
This code seems to run fine in development builds, but sometimes behaves strange in release mode. I assume this might be related to race conditions or something that is related to threading and async/await. Does anyone see what might be the problem with the above code?
Just to be clear AcceptAsync/ReceiveAsync are wrappers around socket methods to return tasks.

Lock on dedicated lock-object causes deadlock

Yesterday I came accross a strange problem, which gave me quite some headaches. I have a server application with a Server class, which in turn is derived from a Connection class. The Connection class provides information about the connection state and the possibility to close the connection
public bool Connected
{
get
{
if (connection != null)
{
lock (lockObject)
{
bool blockingState = connection.Blocking;
try
{
connection.Blocking = false;
connection.Send(new byte[1], 1, 0);
}
catch (SocketException e)
{
if (!e.NativeErrorCode.Equals(10035))
{
return false;
}
//is connected, but would block
}
finally
{
connection.Blocking = blockingState;
}
return connection.Connected;
}
}
return false;
}
}
public virtual void CloseConnection()
{
if (Connected)
{
lock (lockObject)
{
connection.Close();
}
}
}
The Server class is resonsible for actually sending data
private void ConnectAndPollForData()
{
try
{
TcpListener listener = new TcpListener(Port);
listener.Start();
while (true)
{
connection = listener.AcceptSocket();
string currentBuffr = string.Empty;
const int READ_BUFFER_SIZE = 1024;
byte[] readBuffr = new byte[READ_BUFFER_SIZE];
while (Connected)
{
int bytesReceived;
lock (lockObject)
{
bytesReceived = connection.Receive(readBuffr, READ_BUFFER_SIZE, SocketFlags.None);
}
currentBuffr += ASCIIEncoding.ASCII.GetString(readBuffr, 0, bytesReceived);
//do stuff
}
}
catch(ThreadAbortException)
{
Thread.ResetAbort();
}
finally
{
}
}
public void SendString(string stringToSend)
{
stringToSend += "\r\n";
if(Connected)
{
lock(lockObject)
{
connection.Send(ASCIIEncoding.UTF7.GetBytes(stringToSend));
}
}
}
There is no other explicit access to the connection object. The ConnectAndPollForData function executes in a seperate thread. Whenever I ran the host in this version (I am currently using a non thread-safe version, which causes other problems) it hang after quite a few lines received via TCP. Pausing the debugger showed me, that one thread tried to execute the code withing the lock of Connected, while the other tried to receive data in the lock of ConnectAndPollForData. This behavior seems strange to me, for I would expect to execute the code within the first lock and then do the second. There do seem to be similar problems when using callbacks like in Deadlocking lock() method or 'Deadlock' with only one locked object? but the situation here is a bit different, for in my situation (I think) the code within the locks should not emit any events that themselves try to obtain a lock on the object.

Let's assume it gets the lock in the second method first. So it is holding the lock, and waiting for data. It is unclear whether this is directly receiving the data sent by the first method, or whether this is looking for a reply from an unrelated server - a reply to the message sent in the first method. But either way, I'm assuming that there won't be data incoming until the outbound message is sent.
Now consider: the outbound message can't be sent, because you are holding an exclusive lock.
So yes, you've deadlocked yourself. Basically, don't do that. There is no need to synchronize between inbound and outbound socket operations, even on the same socket. And since it makes very little sense to have concurrent readers on the same socket, or concurrent writers, I'm guessing you don't actually need those locks at all.

WCF + Thread, cant .Open() out of the thread

So I have a service interface (IServiceContract) and its implementation (ServiceContract)
In my client I create an instance of this on the constructor of the client.
Service.ServiceContractClient objProxy; //This is the server instance
myCallbacks = new ClientBase(new InstanceContext(this));
objProxy = new Service.ServiceContractClient(new InstanceContext(myCallbacks), "1");
So I should be able to do a objProxy.Open() at every moment.
But to start the communication, it is done through a thread that does a handshake.
Handshake is a wcf method in the server, called from the client.
So if I do the Open out of the thread, it will say that it is faulted and not let me do it
If I put it inside, it will work, but that is not correct, it should be out.
The thread is just a mechanism to sleep my client in a connected mode.
public void Connect()
{
objClientThread = new Thread(start) { IsBackground = true };
objClientThread.Start();
}
private void start()
{
if (performHandshake())
{
IsConnected = true;
while (IsConnected)
{
System.Threading.Thread.Sleep(500);
}
}
}
private Boolean performHandshake()
{
objProxy.Open();
Debug.WriteLine(String.Format("{0} is Performing handshake...", Name));
string xmlConfig = objProxy.Handshake(Name, ModuleType, RecordTypes, Description, Version);
Debug.WriteLine(String.Format("{0} is loading configuration...", Name));
objConfiguration.LoadXml(xmlConfig as String);
try
{
if (OnHandshakeCompleted != null)
OnHandshakeCompleted(this, new EventArgs());
}
catch
{
return false;
}
//Handshake successfull
return true;
}
How can I put the Open out of the thread and make it work inside the thread (I can call it but it will say it is in a faulted state.

I finally putted on the start() method and it worked.