We Keep Coding

C# TcpListener and MySqlConnection stops accepting connections after a while

I have an async socket server written in C#, running on a Lightsail server running Amazon Linux. It consists of a TcpListener that accepts connections, starts up a new thread to listen when someone connects, initiates an SSL connection, and then acts as a server for an online game.
This server works fine for about a day, until suddenly all networking stops working on the server. The crash takes anywhere from 22 hours to one week to occur. The symptoms are as follows:
Anyone already connected to the server will suddenly stop receiving/sending data. I can see in the logs that my inactivity checking code will eventually kick them for not sending heartbeat packets.
The server will also be unable to connect to its MySQL database (which is running on the same system, so it's unable to connect to localhost? I can still access it through PHPMyAdmin during this time).
It is, however, still able to write both to files and to console, as my logger is still able to write to both.
The code looks like everyone else's (I did try the changes suggested for this question, but it still crashed after ~24 hours). None of the errors get logged, so it looks like it never encounters an exception. No exceptions precede the crash, which is why I've been having problems figuring this one out.
For completeness, here is my main loop:
public void ListenLoop()
{
TcpListener listener = new TcpListener(IPAddress.Any, 26000);
listener.Start();
while (true)
{
try
{
if (listener.Pending())
{
listener.BeginAcceptTcpClient(new AsyncCallback(AcceptConnection), listener);
Logger.Write(Logger.Level.INFO, "continuing the main loop");
}
// Yield so we're not stuck in a busy-loop
Thread.Sleep(5);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
}
and here are the accept parts:
/// <summary>
/// Finish an async callback but spawn a new thread to handle it if necessary
/// </summary>
/// <param name="ar"></param>
private void AcceptConnection(IAsyncResult ar)
{
if (ar.CompletedSynchronously)
{
// Force the accept logic to run async, to keep our listening
// thread free.
Action accept = () => AcceptCallback(ar);
accept.BeginInvoke(accept.EndInvoke, null);
} else
{
AcceptCallback(ar);
}
}
private void AcceptCallback(IAsyncResult ar)
{
try
{
TcpListener listener = (TcpListener) ar.AsyncState;
TcpClient client = listener.EndAcceptTcpClient(ar);
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = (IPEndPoint)client.Client.RemoteEndPoint;
string ip = ipep.Address.ToString();
Logger.Write(Logger.Level.INFO, $"Connection begun to {ip}");
// Authenticate and begin communicating with the client
SslStream stream = new SslStream(client.GetStream(), false);
try
{
stream.AuthenticateAsServer(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
stream.BeginRead(player.Connection.Buffer, 0, 1024, new AsyncCallback(ReadCallback), player);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}

I'm guessing that your primary issue is that you are not disposing the stream and therefore you are getting socket exhaustion.
Apart from that I would advise you to move to fully async code using Task.
public async Task ListenLoop(CancellationToken cancel) // use a cancellation token to shutdown the loop
{
using (var TcpListener listener = new TcpListener(IPAddress.Any, 26000))
{
listener.Start();
while (!cancel.IsCancellationRequested)
{
try
{
var client = await listener.AcceptTcpClientAsync(cancel);
Task.Run(async () => await AcceptConnection(client, cancel));
Logger.Write(Logger.Level.INFO, "continuing the main loop");
// no need to yield due to async
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
listener.Stop();
}
}
private async Task AcceptConnection(TcpClient client, CancellationToken cancel)
{
try
{
using (client)
{
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
await AcceptConnectionImpl(client, cancel);
}
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}
private async Task AcceptConnectionImpl(TcpClient client, CancellationToken cancel)
{
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = client.Client.RemoteEndPoint;
Logger.Write(Logger.Level.INFO, $"Connection begun to {ipep.Address}");
// Authenticate and begin communicating with the client
using (SslStream stream = new SslStream(client.GetStream(), false))
{
try
{
await stream.AuthenticateAsServerAsync(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
player.Cancellation = cancel;
await player.YourReadLoopAsync();
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
}
The function YourReadLoopAsync should read data from the stream using ReadAsync, or using classes like StreamReader which also has async functions.
You don't need to use CancellationToken, but it does make it easier to deal with shutting everything down cleanly. Make sure to catch OperationCanceledException on every try.
See also this link for further tips.

The solution I found after consulting some people more familiar with C# than me is that I was running into Thread Pool Exhaustion. Essentially, I had a bunch of other async tasks (not shown in the code in the question, as they didn't look like they could cause what I was seeing) that were stuck executing some extremely-long-IOs (talking to users that had either disconnected improperly or were behind very high latency), which prevented the async AcceptCallback in my post from being picked up by the Thread Pool. This had a myriad of other side-effects which I outlined in the question:
Creating a new connection to a MySQL database involves an async task behind-the-scenes, which was being starved out due to exhaustion.
Completing the EndAcceptTcpClient required my async task to run, which requires an available thread.
Tasks which did not involve the async keyword, such as Timer() bound tasks (like my logger I/O) were unaffected and could still run.
My solution involved reducing the number of synchronization steps elsewhere in my program, and restructuring any tasks that could take a long time to execute so that they didn't block threads. Thank you to everyone who looked/commented.

c# tcp socket receive loop break logic

I'm writng a c# tcp socket server which will receive data from c++ socket application. Now everything works good except the loop break logic inside my c# receiving thread. I'm showing a code snippet below to explain better.
//creating socket object
private Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
//accepting connections
TcpListener tcpListener = new TcpListener(ipAdd, iPort);
tcpListener.Start();
this.socket = tcpListener.AcceptSocket();
tcpListener.Stop();
//code for
//public byte[] ReceiveMessage(ref int intError)
while (this.socket.Available > 0)
{
try
{
intBytesReceived = this.socket.Receive(byteReceivedBuffer, byteReceivedBuffer.Length, SocketFlags.None);
Thread.Sleep(100);
}
catch (SocketException ex)
{
string str = ex.Message;
intError = -1;
}
}
Below is the thread function to receive data continuously
//Thread to continuous poll data from client
//Now this has been done as I need to receive multiple messages from client after server ack received at client side.
//Thread quit logic based on manual stop button press
//I want to have one more logic by which thread will exit if the connection closed by client socket. i.e. a call of 'CloseSocket()' function from c++ application.
if (TCPServerListener.serverStatus == TCPServerListener.ServerStatus.Stopped)
{
tcpCommunication.Disconnect();
break;
}
while(true)
{
int sockError = 0;
byte[] byteData = tcpCommunication.ReceiveMessage(ref sockError);
if (byteData.Length > 0)
{
ParseBuffer(byteData, ref tcpCommunication);
}
}
For better picture I'm wrting below my communication protocol.
Client initiate a connection. Send a data set
Server receives, process, send ack
Client is in halt mode until it receive the ack from server, but didn't close the socket
This time, server will not receive any message but the receiving thread should active as I'm creating a single thread for each client.
Once the client receives the ack it will continue to send next data
Server receiving thread will get data now and process accordingly
Once client closes down its socket, server receiver thread should close also
I'm able to getting my work done but the server receiver thread is not getting closed. That's my problem. Any suggestions would be a off great help.

It has been my experience that when you use a TCP server/client socket relationship, the TCP server socket side will be in a thread that is controlled by a flag to continue listening until told to stop (Your comments mention a stop button).
The client side runs until all data is done, or the same flag controlling the server side is tripped.
When a client connects, I would store the AcceptSocket into a list. When I did this, I had a custom class that the takes the AcceptSocket and the custom class had counters for like how many messages, or bytes was sent/received to/from this client.
// This should be a threaded method if you're using a button to stop
private void StartTcpListener()
{
List<ClientClass> clientClassList = new List<ClientClass>();
TcpListener tcpListener = new TcpListener(ipAdd, iPort);
tcpListener.Start();
while (keepListening)
{
Socket socket = tcpListener.AcceptSocket();
// Create a client object and store it in a list
ClientClass clientClass = new ClientClass(socket);
clientClassList.Add(clientClass);
// This method would start a thread to read data from the accept socket while it was connected.
clientClass.Start();
}
foreach (ClientClass client in clientClassList)
{
// This method would stop the thread that reads data from the accept socket and close the accept socket
client.Stop();
}
tcpListener.Stop();
}
For the client side, when reading from the client accept socket this would be thread controlled and either the thread is aborted or the client closes its end causing the read to return a -1 for number of bytes read.
// Threaded method
private void Receive()
{
// Don't know how much of a buffer you need
byte[] dataIn = byte[1000];
while (acceptSocket.Connected)
{
// Exception handling so you either end the thread or keep processing data
try
{
int bytesRead = acceptSocket.Read(dataIn);
if (bytesRead != -1)
{
// Process your data
}
else
{
// -1 Bytes read should indicate the client shutdown on their end
break;
}
}
catch(SocketException se)
{
// You could exit this loop depending on the SocketException
}
catch(ThreadAbortException tae)
{
// Exit the loop
}
catch (Exception e)
{
// Handle exception, but keep reading for data
}
}
// You have to check in case the socket was disposed or was never successfully created
if (acceptSocket != null)
{
acceptSocket.Close();
}
}
// This is the stop method if you press your stop button
private void Stop()
{
// Aborts your read thread and the socket should be closed in the read thread. The read thread should have a ThreadState.Stopped if the while loop was gracefully ended and the socket has already been closed
if (readThread != null && readThread.ThreadState != ThreadState.Stopped)
{
readThread.Abort();
}
}
I think this is the basics of what I've done before. I'm sure there are "cleaner" ways of doing this now. This was years ago when I use to implement TCP client/server socket objects.
Hope this helps...

Windows Service thread is not stopping in timely fashion in c#

I have a windows service which is getting started using Thread.After installation as windows service i am able to start the service properly but as soon as i try to stop the service it is taking too much time and and not getting stopped.I am using ManualResetEvent to stop the windows service Here is my code.
protected override void OnStart(string[] args)
{
_thread = new Thread(DoWork);
_thread.Start();
}
private void DoWork()
{
while (!_shutdownEvent.WaitOne(0))
{
data = new byte[1024];
int recv = sock.Receive(data);
stringData = Encoding.ASCII.GetString(data, 0, recv);
}
sock.Shutdown(SocketShutdown.Both);
sock.Close();
}
catch (Exception DFGFD)
{
}
}
protected override void OnStop()
{
_shutdownEvent.Set();
_thread.Join(); // wait for thread to stop
}
}
}
Please help me to resolve this.

You socket is blocking on the receive code. I would suggest issuing:
sock.Shutdown(SocketShutdown.Both);
sock.Close();
in a method called from the OnStop() handler (so it is called from another thread to the blocking Receive). This will cause the blocking sock.Receive to fail with an exception that you can handle by quitting the loop.

Maybe the code in the while is blocking ( ie waiting data from the conenction synchronously )
Typically you shoul use asynchronous I/O operation with the socket, that generally allow you to avoid starting a new thread.

The problem is in the call to Receive in the loop. From MSDN:
If no data is available for reading, the Receive method will block
until data is available, unless a time-out value was set by using
Socket.ReceiveTimeout. If the time-out value was exceeded, the Receive
call will throw a SocketException. If you are in non-blocking mode,
and there is no data available in the in the protocol stack buffer,
the Receive method will complete immediately and throw a
SocketException. You can use the Available property to determine if
data is available for reading. When Available is non-zero, retry the
receive operation.
Therefore, I'd suggest to change the code in the loop to this:
if (sock.Available > 0)
{
data = new byte[1024];
int recv = sock.Receive(data);
stringData = Encoding.ASCII.GetString(data, 0, recv);
}
Thread.Sleep(200);
This code checks whether data is available for reading and only calls Receive if there is data to read. In order to avoid busy waiting, it issues a call the Thread.Sleep.
If you want to avoid the call to Thread.Sleep, you can specify a ReceiveTimeout on the socket:
sock.ReceiveTimeout = 200;
The code in the while loop would then look like this:
try
{
data = new byte[1024];
int recv = sock.Receive(data);
stringData = Encoding.ASCII.GetString(data, 0, recv);
}
catch(SocketException ex)
{
if (ex.SocketErrorCode != SocketError.TimedOut)
throw;
// In case of Timeout, do nothing, continue loop
}

Problem: Receive() is waiting for additional input.
The Thread.Join() function is waiting for the thread to finish.
Possible Solution:
When I've been in similar situations, I've used a timeout on the Join function thread.Join(3000); This gives the thread an opportunity to do a clean shutdown, then continue.

Send data from a background thread to the main thread

I am running a C# server from the main application and I would like to pass the message received from the server thread to the main thread. The server should be running on the background for new connections. Every time there is a new connection the server should pass the message received to the main app. How can let the main app know when the message is received? and how can pass the message from the server thread to the main when there is a new connection?
Main application
public partial class MainWindow : Window
{
TCPServer Server = new TCPServer(); //start running the server
//get the message (Server.message) when a client sent it to the server
//TODO process the message
}
TCP server
class TCPServer
{
private TcpListener tcpListener;
private Thread listenThread;
private String message;
public TCPServer()
{
this.tcpListener = new TcpListener(IPAddress.Any, 3200);
this.listenThread = new Thread(new ThreadStart(ListenForClients));
this.listenThread.Start();
}
//starts the tcp listener and accept connections
private void ListenForClients()
{
this.tcpListener.Start();
while (true)
{
//blocks until a client has connected to the server
System.Diagnostics.Debug.WriteLine("Listening...");
TcpClient client = this.tcpListener.AcceptTcpClient();
System.Diagnostics.Debug.WriteLine("Client connected");
//create a thread to handle communication
//with connected client
Thread clientThread = new Thread(new ParameterizedThreadStart(HandleClientComm));
clientThread.Start(client);
}
}
//Read the data from the client
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client; //start the client
NetworkStream clientStream = tcpClient.GetStream(); //get the stream of data for network access
byte[] message = new byte[4096];
int bytesRead;
while (true)
{
bytesRead = 0;
try
{
//blocks until a client sends a message
bytesRead = clientStream.Read(message, 0, 4096);
}
catch
{
//a socket error has occured
break;
}
if (bytesRead == 0) //if we receive 0 bytes
{
//the client has disconnected from the server
break;
}
//message has successfully been received
ASCIIEncoding encoder = new ASCIIEncoding();
message = encoder.GetString(message, 0, bytesRead);
//Reply
byte[] buffer = encoder.GetBytes("ACK");
clientStream.Write(buffer, 0, buffer.Length);
System.Diagnostics.Debug.WriteLine("ACK");
clientStream.Flush();
}
tcpClient.Close();
System.Diagnostics.Debug.WriteLine("Client disconnected");
}

This is already well supported by TcpListener, use the BeginAcceptTcpClient() method instead. When you call it from the main thread of a WPF or Winforms app then the callback will run on the same main thread automatically. The same applies to its BeginReceive() method. Internally it uses the dispatcher loop to get the callback method activated, pretty similar to the way a class like BackgroundWorker and the C# v5 async/await keywords work.
This saves you from the hassle of starting end terminating your own thread and ensuring you marshal back properly. And significantly cutting down on the resource usage of your program. Highly recommended.

A Queue is the answer. Specifically in this case a Concurrent Queue.
Your socket thread puts messages into the queue. Your worker thread(s) poll the queue and pulls work items out.
For socket based applications, this pattern is very, very common.
Alternatively, you can QueueUserWorkItem against the system thread pool, and let it manage the work load.
Note: You're in multi-threaded land now. You'll need to read about synchronization and other issues that are going to arise. Failure to do this means your app will have very weird bugs.

c# gui hangs but still can listen and process

why is that my c# server gui hangs? any idea where did i go wrong? thank you
its like, the moment i click the button1, the gui hangs, but it can still process requests and listen and accept for incoming client connections.
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
}
TcpListener listener = null;
TcpClient client = null;
NetworkStream stream = null;
BinaryWriter writer = null;
BinaryReader reader = null;
string vouchercode;
string username;
string password;
string reseller;
string fresh;
string result;
private void button1_Click(object sender, EventArgs e)
{
try
{
listener = new TcpListener(new IPAddress(new byte[] {127,0,0,1}), 6666);
listener.Start();
while (true)
{
label1.Text = "waiting....";
using (client = listener.AcceptTcpClient())
{
label1.Text = "Connection request accepted!";
using (stream = client.GetStream())
{
//some codes here ..
}
}
}
}
catch (WebException ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (listener != null) listener.Stop();
if (writer != null) writer.Close();
if (reader != null) reader.Close();
}
}
}
}

It hangs because AcceptTcpClient() is a blocking method. You can look into and try incorporating BeginAcceptTcpClient() for it to be non-blocking. There is an example in the msdn page.

When you are doing processing on the UI thread (like you are in the button click handler), it's important not to block. As Bala pointed out, you have a blocking call which is sitting in a (possibly infinite) loop, and this is a problem because you never return out of the function, allowing window messages to be processed (window messages do things like repainting windows, respond to UI controls like button clicks, etc...).
The answer is to either make button1_Click non-blocking, or move the sockets code to a different thread.
Check out this SO thread:
How to spread tcplistener incoming connections over threads in .NET?

You are also entering a while loop with no logic that I can see to exit it. So you are going to hang. It is also always good practice to get your heavy lifting out of your events and in this case into a different thread.