We Keep Coding

C# TcpListener and MySqlConnection stops accepting connections after a while

I have an async socket server written in C#, running on a Lightsail server running Amazon Linux. It consists of a TcpListener that accepts connections, starts up a new thread to listen when someone connects, initiates an SSL connection, and then acts as a server for an online game.
This server works fine for about a day, until suddenly all networking stops working on the server. The crash takes anywhere from 22 hours to one week to occur. The symptoms are as follows:
Anyone already connected to the server will suddenly stop receiving/sending data. I can see in the logs that my inactivity checking code will eventually kick them for not sending heartbeat packets.
The server will also be unable to connect to its MySQL database (which is running on the same system, so it's unable to connect to localhost? I can still access it through PHPMyAdmin during this time).
It is, however, still able to write both to files and to console, as my logger is still able to write to both.
The code looks like everyone else's (I did try the changes suggested for this question, but it still crashed after ~24 hours). None of the errors get logged, so it looks like it never encounters an exception. No exceptions precede the crash, which is why I've been having problems figuring this one out.
For completeness, here is my main loop:
public void ListenLoop()
{
TcpListener listener = new TcpListener(IPAddress.Any, 26000);
listener.Start();
while (true)
{
try
{
if (listener.Pending())
{
listener.BeginAcceptTcpClient(new AsyncCallback(AcceptConnection), listener);
Logger.Write(Logger.Level.INFO, "continuing the main loop");
}
// Yield so we're not stuck in a busy-loop
Thread.Sleep(5);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
}
and here are the accept parts:
/// <summary>
/// Finish an async callback but spawn a new thread to handle it if necessary
/// </summary>
/// <param name="ar"></param>
private void AcceptConnection(IAsyncResult ar)
{
if (ar.CompletedSynchronously)
{
// Force the accept logic to run async, to keep our listening
// thread free.
Action accept = () => AcceptCallback(ar);
accept.BeginInvoke(accept.EndInvoke, null);
} else
{
AcceptCallback(ar);
}
}
private void AcceptCallback(IAsyncResult ar)
{
try
{
TcpListener listener = (TcpListener) ar.AsyncState;
TcpClient client = listener.EndAcceptTcpClient(ar);
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = (IPEndPoint)client.Client.RemoteEndPoint;
string ip = ipep.Address.ToString();
Logger.Write(Logger.Level.INFO, $"Connection begun to {ip}");
// Authenticate and begin communicating with the client
SslStream stream = new SslStream(client.GetStream(), false);
try
{
stream.AuthenticateAsServer(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
stream.BeginRead(player.Connection.Buffer, 0, 1024, new AsyncCallback(ReadCallback), player);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}

I'm guessing that your primary issue is that you are not disposing the stream and therefore you are getting socket exhaustion.
Apart from that I would advise you to move to fully async code using Task.
public async Task ListenLoop(CancellationToken cancel) // use a cancellation token to shutdown the loop
{
using (var TcpListener listener = new TcpListener(IPAddress.Any, 26000))
{
listener.Start();
while (!cancel.IsCancellationRequested)
{
try
{
var client = await listener.AcceptTcpClientAsync(cancel);
Task.Run(async () => await AcceptConnection(client, cancel));
Logger.Write(Logger.Level.INFO, "continuing the main loop");
// no need to yield due to async
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
listener.Stop();
}
}
private async Task AcceptConnection(TcpClient client, CancellationToken cancel)
{
try
{
using (client)
{
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
await AcceptConnectionImpl(client, cancel);
}
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}
private async Task AcceptConnectionImpl(TcpClient client, CancellationToken cancel)
{
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = client.Client.RemoteEndPoint;
Logger.Write(Logger.Level.INFO, $"Connection begun to {ipep.Address}");
// Authenticate and begin communicating with the client
using (SslStream stream = new SslStream(client.GetStream(), false))
{
try
{
await stream.AuthenticateAsServerAsync(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
player.Cancellation = cancel;
await player.YourReadLoopAsync();
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
}
The function YourReadLoopAsync should read data from the stream using ReadAsync, or using classes like StreamReader which also has async functions.
You don't need to use CancellationToken, but it does make it easier to deal with shutting everything down cleanly. Make sure to catch OperationCanceledException on every try.
See also this link for further tips.

The solution I found after consulting some people more familiar with C# than me is that I was running into Thread Pool Exhaustion. Essentially, I had a bunch of other async tasks (not shown in the code in the question, as they didn't look like they could cause what I was seeing) that were stuck executing some extremely-long-IOs (talking to users that had either disconnected improperly or were behind very high latency), which prevented the async AcceptCallback in my post from being picked up by the Thread Pool. This had a myriad of other side-effects which I outlined in the question:
Creating a new connection to a MySQL database involves an async task behind-the-scenes, which was being starved out due to exhaustion.
Completing the EndAcceptTcpClient required my async task to run, which requires an available thread.
Tasks which did not involve the async keyword, such as Timer() bound tasks (like my logger I/O) were unaffected and could still run.
My solution involved reducing the number of synchronization steps elsewhere in my program, and restructuring any tasks that could take a long time to execute so that they didn't block threads. Thank you to everyone who looked/commented.

c# tcp socket receive loop break logic

I'm writng a c# tcp socket server which will receive data from c++ socket application. Now everything works good except the loop break logic inside my c# receiving thread. I'm showing a code snippet below to explain better.
//creating socket object
private Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
//accepting connections
TcpListener tcpListener = new TcpListener(ipAdd, iPort);
tcpListener.Start();
this.socket = tcpListener.AcceptSocket();
tcpListener.Stop();
//code for
//public byte[] ReceiveMessage(ref int intError)
while (this.socket.Available > 0)
{
try
{
intBytesReceived = this.socket.Receive(byteReceivedBuffer, byteReceivedBuffer.Length, SocketFlags.None);
Thread.Sleep(100);
}
catch (SocketException ex)
{
string str = ex.Message;
intError = -1;
}
}
Below is the thread function to receive data continuously
//Thread to continuous poll data from client
//Now this has been done as I need to receive multiple messages from client after server ack received at client side.
//Thread quit logic based on manual stop button press
//I want to have one more logic by which thread will exit if the connection closed by client socket. i.e. a call of 'CloseSocket()' function from c++ application.
if (TCPServerListener.serverStatus == TCPServerListener.ServerStatus.Stopped)
{
tcpCommunication.Disconnect();
break;
}
while(true)
{
int sockError = 0;
byte[] byteData = tcpCommunication.ReceiveMessage(ref sockError);
if (byteData.Length > 0)
{
ParseBuffer(byteData, ref tcpCommunication);
}
}
For better picture I'm wrting below my communication protocol.
Client initiate a connection. Send a data set
Server receives, process, send ack
Client is in halt mode until it receive the ack from server, but didn't close the socket
This time, server will not receive any message but the receiving thread should active as I'm creating a single thread for each client.
Once the client receives the ack it will continue to send next data
Server receiving thread will get data now and process accordingly
Once client closes down its socket, server receiver thread should close also
I'm able to getting my work done but the server receiver thread is not getting closed. That's my problem. Any suggestions would be a off great help.

It has been my experience that when you use a TCP server/client socket relationship, the TCP server socket side will be in a thread that is controlled by a flag to continue listening until told to stop (Your comments mention a stop button).
The client side runs until all data is done, or the same flag controlling the server side is tripped.
When a client connects, I would store the AcceptSocket into a list. When I did this, I had a custom class that the takes the AcceptSocket and the custom class had counters for like how many messages, or bytes was sent/received to/from this client.
// This should be a threaded method if you're using a button to stop
private void StartTcpListener()
{
List<ClientClass> clientClassList = new List<ClientClass>();
TcpListener tcpListener = new TcpListener(ipAdd, iPort);
tcpListener.Start();
while (keepListening)
{
Socket socket = tcpListener.AcceptSocket();
// Create a client object and store it in a list
ClientClass clientClass = new ClientClass(socket);
clientClassList.Add(clientClass);
// This method would start a thread to read data from the accept socket while it was connected.
clientClass.Start();
}
foreach (ClientClass client in clientClassList)
{
// This method would stop the thread that reads data from the accept socket and close the accept socket
client.Stop();
}
tcpListener.Stop();
}
For the client side, when reading from the client accept socket this would be thread controlled and either the thread is aborted or the client closes its end causing the read to return a -1 for number of bytes read.
// Threaded method
private void Receive()
{
// Don't know how much of a buffer you need
byte[] dataIn = byte[1000];
while (acceptSocket.Connected)
{
// Exception handling so you either end the thread or keep processing data
try
{
int bytesRead = acceptSocket.Read(dataIn);
if (bytesRead != -1)
{
// Process your data
}
else
{
// -1 Bytes read should indicate the client shutdown on their end
break;
}
}
catch(SocketException se)
{
// You could exit this loop depending on the SocketException
}
catch(ThreadAbortException tae)
{
// Exit the loop
}
catch (Exception e)
{
// Handle exception, but keep reading for data
}
}
// You have to check in case the socket was disposed or was never successfully created
if (acceptSocket != null)
{
acceptSocket.Close();
}
}
// This is the stop method if you press your stop button
private void Stop()
{
// Aborts your read thread and the socket should be closed in the read thread. The read thread should have a ThreadState.Stopped if the while loop was gracefully ended and the socket has already been closed
if (readThread != null && readThread.ThreadState != ThreadState.Stopped)
{
readThread.Abort();
}
}
I think this is the basics of what I've done before. I'm sure there are "cleaner" ways of doing this now. This was years ago when I use to implement TCP client/server socket objects.
Hope this helps...

Lock on dedicated lock-object causes deadlock

Yesterday I came accross a strange problem, which gave me quite some headaches. I have a server application with a Server class, which in turn is derived from a Connection class. The Connection class provides information about the connection state and the possibility to close the connection
public bool Connected
{
get
{
if (connection != null)
{
lock (lockObject)
{
bool blockingState = connection.Blocking;
try
{
connection.Blocking = false;
connection.Send(new byte[1], 1, 0);
}
catch (SocketException e)
{
if (!e.NativeErrorCode.Equals(10035))
{
return false;
}
//is connected, but would block
}
finally
{
connection.Blocking = blockingState;
}
return connection.Connected;
}
}
return false;
}
}
public virtual void CloseConnection()
{
if (Connected)
{
lock (lockObject)
{
connection.Close();
}
}
}
The Server class is resonsible for actually sending data
private void ConnectAndPollForData()
{
try
{
TcpListener listener = new TcpListener(Port);
listener.Start();
while (true)
{
connection = listener.AcceptSocket();
string currentBuffr = string.Empty;
const int READ_BUFFER_SIZE = 1024;
byte[] readBuffr = new byte[READ_BUFFER_SIZE];
while (Connected)
{
int bytesReceived;
lock (lockObject)
{
bytesReceived = connection.Receive(readBuffr, READ_BUFFER_SIZE, SocketFlags.None);
}
currentBuffr += ASCIIEncoding.ASCII.GetString(readBuffr, 0, bytesReceived);
//do stuff
}
}
catch(ThreadAbortException)
{
Thread.ResetAbort();
}
finally
{
}
}
public void SendString(string stringToSend)
{
stringToSend += "\r\n";
if(Connected)
{
lock(lockObject)
{
connection.Send(ASCIIEncoding.UTF7.GetBytes(stringToSend));
}
}
}
There is no other explicit access to the connection object. The ConnectAndPollForData function executes in a seperate thread. Whenever I ran the host in this version (I am currently using a non thread-safe version, which causes other problems) it hang after quite a few lines received via TCP. Pausing the debugger showed me, that one thread tried to execute the code withing the lock of Connected, while the other tried to receive data in the lock of ConnectAndPollForData. This behavior seems strange to me, for I would expect to execute the code within the first lock and then do the second. There do seem to be similar problems when using callbacks like in Deadlocking lock() method or 'Deadlock' with only one locked object? but the situation here is a bit different, for in my situation (I think) the code within the locks should not emit any events that themselves try to obtain a lock on the object.

Let's assume it gets the lock in the second method first. So it is holding the lock, and waiting for data. It is unclear whether this is directly receiving the data sent by the first method, or whether this is looking for a reply from an unrelated server - a reply to the message sent in the first method. But either way, I'm assuming that there won't be data incoming until the outbound message is sent.
Now consider: the outbound message can't be sent, because you are holding an exclusive lock.
So yes, you've deadlocked yourself. Basically, don't do that. There is no need to synchronize between inbound and outbound socket operations, even on the same socket. And since it makes very little sense to have concurrent readers on the same socket, or concurrent writers, I'm guessing you don't actually need those locks at all.

App freezes after Dispatcher.Invoke

I have this application that freezes when calling the dispatcher.invoke for any control.
When i Call the Dispatcher in radiobutton, Grid, Image..etc the App freezes but without giving errors. any help please !!! thank you
I call the thread Method RunClient
private void RunClient()
{
TcpClient client;
// instantiate TcpClient for sending data to server
try
{
// Step 1: create TcpClient and connect to server
client = new TcpClient();
client.Connect(ip, 5001);
// Step 2: get NetworkStream associated with TcpClient
output = client.GetStream();
// create objects for writing and reading across stream
writer = new BinaryWriter(output);
reader = new BinaryReader(output);
string theReply = "";
do
{
try
{
// read the string sent to the server
theReply = reader.ReadString();
int i = 0;
foreach (var x in theReply.Split('#'))
{
ReadString[i] = x;
i++;
}
CheckConnection(ReadString[0]);
}
catch (Exception)
{
//do nothing
}
} while (ReadString[6].Equals(" ") &&
connection.Connected);
updatelabel = () => GameResult(ReadString[6]);
Dispatcher.Invoke(new Action(updatelabel));
if (!connection.Connected)
{
MessageBox.Show("The connection was lost. The game will be closed automatically.");
writer.Close();
reader.Close();
output.Close();
connection.Close();
this.Close();
}
}
// handle exception if error in establishing connection
catch (Exception error)
{
MessageBox.Show("Check Internet Connectivity. Couldn't connect!");
}
}
when the code enters the method ( check connection ) and calls the dispatcher the app freezes.
void CheckConnection(string ii)
{
try
{
if (ii.Equals("Connected"))
{
MessageBox.Show("A Connection was established");
int x = Convert.ToInt32(ReadString[1]);
if (x == 1)
{
updatelabel = () => char1RadioButton2.IsEnabled = false;
char1RadioButton2.Dispatcher.Invoke(new Action(updatelabel));
}
else
{
updatelabel = () => char5RadioButton2.IsEnabled = false;
char5RadioButton2.Dispatcher.Invoke(new Action(updatelabel));
}
updatelabel = () => CreatingGameGrid.Visibility = System.Windows.Visibility.Visible;
CreatingGameGrid.Dispatcher.Invoke(new Action(updatelabel));
updatelabel = () => JoinGameGrid.Visibility = System.Windows.Visibility.Visible;
JoinGameGrid.Dispatcher.Invoke(new Action(updatelabel));
}
else
{
MessageBox.Show("No Such Game found");
this.Close();
}
}
catch (Exception x)
{
MessageBox.Show(x.ToString());
}
}

The Dispatcher.Invoke attempts to synchronously run the specified action on the Dispatcher Thread.
Assuming the RunClient is run on the Dispatcher Thread, and the while loop continues to run while you are trying to Invoke back onto the Dispatcher Thread, the call will freeze.
The simplest solution is to replace all the Dispatcher.Invoke with Dispatcher.BeginInvoke and give it a priority that will run once your RunClient is finished.
The other solution is to run RunClient on a BackgroundWorker.
Similar questions with answers are
Dispatcher.Invoke loop freeze UI
Dispatcher.Invoke hangs main window.
Response to comment on ReadString freeze
Calling Read on a NetworkStream is a blocking call. Well, in fact, it is the Stream obtained by calling TcpClient.GetStream() that blocks. The documentation on MSDN states 'After you have obtained the NetworkStream, call the Write method to send data to the remote host. Call the Read method to receive data arriving from the remote host. Both of these methods block until the specified operation is performed'.
I used dotPeek to see what ReadString was doing and the first thing it does is read the length of the incoming string off the stream using NetworkStream.ReadByte which will block until it has a value to read.
That means the ReadString will sit there until there is data available to read and the amount of data is the same as or more than is expected. You will need to check if you have anything to read before you do by calling stream.DataAvailable or reader.PeekChar.
Alternatively, you could run your socket code on a separate thread. If you are using .Net 4.5, I would take a good look at the Task Parallel Library. ntziolis says in an answer to this question that 'We have made good experiences with that (long being days rather than minutes or hours).'

proper way to abort a blocked thread

I'm creating a server that a TCP connection. The TCP Connection is run in its own thread for an indefinite amount of time. Is there a good pattern to allow safe shutdown of the TcpListener and Client as well as the thread? Below is what I have so far.
private volatile bool Shudown;
void ThreadStart1()
{
TcpListener listener = null;
TcpClient client = null;
Stream s = null;
try
{
listener = new TcpListener(60000);
client = listener.AcceptTcpClient();
Stream s = client.GetStrea();
while(!Shutdown) // use shutdown to gracefully shutdown thread.
{
try
{
string msg = s.ReadLine(); // This blocks the thread so setting shutdown = true will never occur unless a client sends a message.
DoSomething(msg);
}
catch(IOException ex){ } // I would like to avoid using Exceptions for flow control
catch(Exception ex) { throw; }
}
}
catch(Exception ex)
{
LogException(ex);
throw ex;
}
finally
{
if(listener != null) listener.Close();
if(s != null) s.Close();
if(client != null) client.Close();
}
}

Set a timeout on the NetworkStream (client.ReadTimeout=...). Once the read operation times out, check to see if the main thread signalled you to stop (by setting a variable or an AutoResetEvent). If it's been signalled to stop, exit gracefully. If not, try reading again until the next timeout.
Setting a 0.5 or 1 second timeout should suffice - you will be able to exit the thread in a timely manner, and yet be very easy on the CPU.

Is there a good pattern to allow safe shutdown of the thread?
Change the while loop to the following:
while (!interrupted){
// Do something
}
// declare interrupted as volatile boolean
volatile bool interrupted;
Check this MSDN example for details.
Setting interrupted boolean to true will make the thread come out of the loop when it checks for the while condition.
Is there a good pattern to allow safe shutdown of the TcpListener and
Client?
To avoid duplication, please check this SO question
As for your question on how to terminate a blocking thread on ReadLine(); the following listener.Server.Close(); should do the job and return from the blocking call.

Perhaps instead of calling Read on a NetworkStream object synchronously, you should use BeginRead and EndRead to do it asynchronously, and call Close() on the NetworkStream when you are done with it.