We Keep Coding

C# TcpListener and MySqlConnection stops accepting connections after a while

I have an async socket server written in C#, running on a Lightsail server running Amazon Linux. It consists of a TcpListener that accepts connections, starts up a new thread to listen when someone connects, initiates an SSL connection, and then acts as a server for an online game.
This server works fine for about a day, until suddenly all networking stops working on the server. The crash takes anywhere from 22 hours to one week to occur. The symptoms are as follows:
Anyone already connected to the server will suddenly stop receiving/sending data. I can see in the logs that my inactivity checking code will eventually kick them for not sending heartbeat packets.
The server will also be unable to connect to its MySQL database (which is running on the same system, so it's unable to connect to localhost? I can still access it through PHPMyAdmin during this time).
It is, however, still able to write both to files and to console, as my logger is still able to write to both.
The code looks like everyone else's (I did try the changes suggested for this question, but it still crashed after ~24 hours). None of the errors get logged, so it looks like it never encounters an exception. No exceptions precede the crash, which is why I've been having problems figuring this one out.
For completeness, here is my main loop:
public void ListenLoop()
{
TcpListener listener = new TcpListener(IPAddress.Any, 26000);
listener.Start();
while (true)
{
try
{
if (listener.Pending())
{
listener.BeginAcceptTcpClient(new AsyncCallback(AcceptConnection), listener);
Logger.Write(Logger.Level.INFO, "continuing the main loop");
}
// Yield so we're not stuck in a busy-loop
Thread.Sleep(5);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
}
and here are the accept parts:
/// <summary>
/// Finish an async callback but spawn a new thread to handle it if necessary
/// </summary>
/// <param name="ar"></param>
private void AcceptConnection(IAsyncResult ar)
{
if (ar.CompletedSynchronously)
{
// Force the accept logic to run async, to keep our listening
// thread free.
Action accept = () => AcceptCallback(ar);
accept.BeginInvoke(accept.EndInvoke, null);
} else
{
AcceptCallback(ar);
}
}
private void AcceptCallback(IAsyncResult ar)
{
try
{
TcpListener listener = (TcpListener) ar.AsyncState;
TcpClient client = listener.EndAcceptTcpClient(ar);
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = (IPEndPoint)client.Client.RemoteEndPoint;
string ip = ipep.Address.ToString();
Logger.Write(Logger.Level.INFO, $"Connection begun to {ip}");
// Authenticate and begin communicating with the client
SslStream stream = new SslStream(client.GetStream(), false);
try
{
stream.AuthenticateAsServer(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
stream.BeginRead(player.Connection.Buffer, 0, 1024, new AsyncCallback(ReadCallback), player);
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}

I'm guessing that your primary issue is that you are not disposing the stream and therefore you are getting socket exhaustion.
Apart from that I would advise you to move to fully async code using Task.
public async Task ListenLoop(CancellationToken cancel) // use a cancellation token to shutdown the loop
{
using (var TcpListener listener = new TcpListener(IPAddress.Any, 26000))
{
listener.Start();
while (!cancel.IsCancellationRequested)
{
try
{
var client = await listener.AcceptTcpClientAsync(cancel);
Task.Run(async () => await AcceptConnection(client, cancel));
Logger.Write(Logger.Level.INFO, "continuing the main loop");
// no need to yield due to async
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while waiting for listeners: {e.Message}\n{e.StackTrace}");
}
}
listener.Stop();
}
}
private async Task AcceptConnection(TcpClient client, CancellationToken cancel)
{
try
{
using (client)
{
// If the SSL connection takes longer than 5s we have a problem, and should stop
client.Client.ReceiveTimeout = 5000;
await AcceptConnectionImpl(client, cancel);
}
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting a connection to an unknown user: {e.Message}");
}
}
private async Task AcceptConnectionImpl(TcpClient client, CancellationToken cancel)
{
// Attempt to get the IP address of the client we're connecting to
IPEndPoint ipep = client.Client.RemoteEndPoint;
Logger.Write(Logger.Level.INFO, $"Connection begun to {ipep.Address}");
// Authenticate and begin communicating with the client
using (SslStream stream = new SslStream(client.GetStream(), false))
{
try
{
await stream.AuthenticateAsServerAsync(
serverCertificate,
enabledSslProtocols: System.Security.Authentication.SslProtocols.Tls12,
clientCertificateRequired: false,
checkCertificateRevocation: true
);
stream.ReadTimeout = 3600000;
stream.WriteTimeout = 3600000;
NetworkPlayer player = new NetworkPlayer();
player.Name = ip;
player.Connection.Stream = stream;
player.Connection.Connected = true;
player.Connection.Client = client;
player.Cancellation = cancel;
await player.YourReadLoopAsync();
}
catch (OperationCanceledException) { }
catch (Exception e)
{
Logger.Write(Logger.Level.ERROR, $"Error while starting the connection to {ip}: {e.Message}");
// The following code just calls stream.Close(); and client.Close(); but sends exceptions to my logger.
CloseConnectionSafely(client, stream);
}
}
}
The function YourReadLoopAsync should read data from the stream using ReadAsync, or using classes like StreamReader which also has async functions.
You don't need to use CancellationToken, but it does make it easier to deal with shutting everything down cleanly. Make sure to catch OperationCanceledException on every try.
See also this link for further tips.

The solution I found after consulting some people more familiar with C# than me is that I was running into Thread Pool Exhaustion. Essentially, I had a bunch of other async tasks (not shown in the code in the question, as they didn't look like they could cause what I was seeing) that were stuck executing some extremely-long-IOs (talking to users that had either disconnected improperly or were behind very high latency), which prevented the async AcceptCallback in my post from being picked up by the Thread Pool. This had a myriad of other side-effects which I outlined in the question:
Creating a new connection to a MySQL database involves an async task behind-the-scenes, which was being starved out due to exhaustion.
Completing the EndAcceptTcpClient required my async task to run, which requires an available thread.
Tasks which did not involve the async keyword, such as Timer() bound tasks (like my logger I/O) were unaffected and could still run.
My solution involved reducing the number of synchronization steps elsewhere in my program, and restructuring any tasks that could take a long time to execute so that they didn't block threads. Thank you to everyone who looked/commented.

Working on client server using sockets and tcp\ip in C#

I am working on client-server data receiving. I am able to receive data from the server which is my meter. The code is below
listner = new TcpListener(new IPEndPoint(IPAddress.Any, port));
Console.WriteLine("Listening...");
listner.Start();
while (true)
{
try
{
//---incoming client connected---
TcpClient client = listner.AcceptTcpClient();
//---get the incoming data through a network stream---
NetworkStream nwStream = client.GetStream();
byte[] buffer = new byte[client.ReceiveBufferSize];
//---read incoming stream---
int bytesRead = nwStream.Read(buffer, 0, client.ReceiveBufferSize);
//---convert the data received into a string---
string dataReceived = BitConverter.ToString(buffer, 0, bytesRead);
//Encoding.ASCII.GetString(buffer, 0, bytesRead);
//Console.WriteLine("Received : " + dataReceived);
//MessageBox.Show("Data Received", dataReceived);
//---write back the text to the client---
//Console.WriteLine("Sending back : " + dataReceived);
//nwStream.Write(buffer, 0, bytesRead);
client.Close();
}
catch (Exception ex)
{
}
Using the above code I am able to receive the data. But I want to perform the following
Keeping listening on the port
While listening I want to send some bytes to the server (meter)
Get a response from the server after sending some data.
I know there are lots of client-server tutorials but they have a separate implementation of server and client. I just want to handle the client(my software). I have also tried with Asynchronous Server Socket Example.
Any help would be highly appreciated.

Your server needs to continually listen for connections from clients. Each time it gets a connection you get from the OS a new TcpClient to handle that connection. the TcpClient that you get from the OS is sometimes called a Child TcpClient because it's created by the listening port. But it's functionally the same as any other socket.
This is very easy to achieve with the async/await pattern.
First you need a listener that waits for connections, everytime it gets a connection it passes this off to another task that processes the child TcpClient. e.g.,
static async Task StartTcpServerAsync()
{
var tcpListener = new TcpListener(new IPEndPoint(IPAddress.Any, 9999));
tcpListener.Start();
while (true)
{
var tcpClient = await tcpListener.AcceptTcpClientAsync();
// Fire and forget the Child connection
_ = StartChildTcpClientAsync(tcpClient);
}
}
In the above boiler plate code we're completely forgetting about the ChildTcpClient task. You might or might not want this. If you make the task completely self sufficient (as this demo is) then the main task may never need to know if it finished or not. But if you need to be able to provide feedback between the ChildTcpClient Task and the main task then you'll need to provide some additional code to manage this and you'd start by storing the Task returned from StartChildTcpClientAsync(tcpClient); somewhere so you can observe it. You could use Task.Run() here, but it is not necessary in a Console application.
Now that you've got your listener Task, you need a ChildTcpClient Task, eg:
static async Task StartChildTcpClientAsync(TcpClient tcpClient)
{
Console.WriteLine($"Connection received from: {tcpClient.Client.RemoteEndPoint.ToString()}");
using var stream = tcpClient.GetStream();
var buffer = new byte[1024];
while (true)
{
var bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length);
await stream.WriteAsync(buffer, 0, bytesRead);
}
}
That's a super simple Tcp echo client. It's got no flow control and no error handling the task will just sit there forever waiting for the client to send some data even after the client has disconnected. This Task just "awaits" for infinity (or until an exception occurs) so you need to add some code in here to manage the client, check that the client is still connected, etc etc
To pull it all together you simply need a main like this:
static async Task Main(string[] args)
{
await StartTcpServerAsync();
// Will never exit unless tcpListener.AcceptTcpClientAsync();
// throws an exception
}
And that's it, you have a Console Application that waits for an infinite number of clients to connect, and each time one does it has it's own Task to deal with the IO.

.NET - Socket server with async/await

I'm writing simple socket server with C#. The idea is to do it as simple as possible since the communication won't be heavy. I've used some of the TAP/APM patterns on socket async calls so the code now looks like this:
public async Task StartListening()
{
try
{
var endpoint = new IPEndPoint(IPAddress.Loopback, Port);
using (Socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp))
{
Socket.Bind(endpoint);
Socket.Listen(Limit);
while (!Shutdown)
{
// await till we get the connection - let the main thread/caller continue (I expect this loop to continue in separate thread after await)
var socket = await Socket.AcceptAsync().ConfigureAwait(false);
var state= new ClientStateObject(socket, id);
// do not await for receive - continue to listen for connections
Receive(socket, state);
}
}
}
catch (Exception ex)
{
// handle errors via callbacks
}
}
private async void Receive(Socket socket, ClientStateObject state)
{
try
{
while (!Shutdown)
{
var bytes = await socket.ReceiveAsync(state).ConfigureAwait(false);
var readResult = state.Read(bytes);
if (readResult == CloseConn)
{
// close connection
return;
}
if (readResult == Completed)
{
// process the message
state.Reset();
}
}
}
catch (Exception ex)
{
// handle errors via callbacks
}
}
This code seems to run fine in development builds, but sometimes behaves strange in release mode. I assume this might be related to race conditions or something that is related to threading and async/await. Does anyone see what might be the problem with the above code?
Just to be clear AcceptAsync/ReceiveAsync are wrappers around socket methods to return tasks.

Problems with asynchronous functions with TcpListener and TcpClient, function not waiting on await keyword

I am new to asynchronous socket programming, and I am having problems with my asynchronous functions.
I am trying to create a chat program that uses Windows Forms for the client, and a console application for the server.
Here is the code for handling connections on my server:
public async void StartServer()
{
TcpListener listener = new TcpListener(_ip, _port);
listener.Start();
Console.WriteLine("Server is running on IP: {0} Port: {1}", _ip.ToString(), _port);
while (true)
{
try
{
TcpClient client = await listener.AcceptTcpClientAsync();
HandleConnections(client);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
}
}
private async void HandleConnections(TcpClient client)
{
NetworkStream stream = client.GetStream();
byte[] buffer = new byte[256];
string message = null;
int x;
while(stream.DataAvailable)
{
x = await stream.ReadAsync(buffer, 0, buffer.Length);
message += Encoding.ASCII.GetString(buffer);
}
message = message.Replace('\0', ' ');
message = message.Trim();
Console.WriteLine("Message Recieved: " + message);
byte[] bytes = Encoding.ASCII.GetBytes(message);
await stream.WriteAsync(bytes, 0, bytes.Length);
stream.Close();
}
And here is the code for the client program connecting to the server:
private async void ConnectButton_Click(object sender, EventArgs e)
{
IPAddress address = IPAddress.Parse(IPInput.Text);
client = new TcpClient();
await client.ConnectAsync(address, 12345);
NetworkStream stream = client.GetStream();
string message = UsernameInput.Text + " Connected!";
Task<int> sendTask = SendMessage(stream, message);
int sendComp = await sendTask;
Task<string> recieveTask = RecieveMessage(stream);
string recieved = await recieveTask;
stream.Close();
ChatText.AppendText(recieved);
}
private async Task<int> SendMessage(NetworkStream stream, string message)
{
byte[] bytes = Encoding.ASCII.GetBytes(message + "\r\n");
await stream.WriteAsync(bytes, 0, bytes.Length);
return 1;
}
private async Task<string> RecieveMessage(NetworkStream stream)
{
byte[] buffer = new byte[256];
string message = null;
int x;
while (stream.DataAvailable)
{
x = await stream.ReadAsync(buffer, 0, buffer.Length);
message += Encoding.ASCII.GetString(buffer);
}
return message;
}
The first problem that I am having is when I run the client program and click the ConnectButton, the message gets sent to the server program which outputs Message Recieved: user Connected!, but then the client program encounters a null reference exception on the line ChatText.AppendText(recieved); saying that the recieved variable is null. It seems that the line string recieved = await recieveTask; is not waiting for the task to finish executing, and it jumps to the next line without assigning a value to recieved. If I put a breakpoint at the top of the private async Task<string> RecieveMessage(NetworkStream stream) function and step through it, then the recieved variable gets it's value and the code will complete successfully, but without the breakpoint I get the null reference exception.
The next issue that I am having, is if I leave the server running and open the client again and try connecting, the server gets a null reference exception on the line message = message.Replace('\0', ' ');. The first time I run with the client, the server receives the message successfully, but the second time, it doesn't get any data from the stream and leaves the variable null, resulting in a null reference exception.
I apologize if my code is garbage, I have been reading the MSDN documentation for hours and am unable to come up with a solution, and I feel like I am doing this completely wrong. So my questions are as follows:
What is causing these errors that I am encountering? And am I approaching this problem the right way?

Both of your issues are not related to asynchronous functions, and actually both issues are because of the same problem:
while (stream.DataAvailable)
{
// read stream here
}
If data is not yet available to read from the stream - both of your ReceiveMessage and HandleConnections functions just skip reading stream at all. What you should do instead (in your case) is:
do
{
// read your stream here
} while (stream.DataAvailable);
Then first Read (or ReadAsync) will wait until first data chunk arrives, and only after first chunk will check if more data is already available.
Also note that you use large buffer (256 bytes) while client\server send short messages (like "Client received: xxx"), which means most of the buffer is empty and when you convert it to string via Encoding.ASCII.GetString - you get a lot of whitespace at the end ("Client received: xxx ... ").

It doesn't look like a problem with async/await so much as an issue with your TCP streams.
You don't appear to be actually waiting for a response. SendMessage writes the server, then RecieveMessage expects a response to already be in the stream.
If stream.DataAvailable is false when you hit the while loop for the first time, message will remain null.
You need some way to wait for there to be data in the stream before you attempt to read from it.

await causing web socket connections to close

I need to establish multiple web socket connections that will forever receive updates unless they quit through some case. I need to have these socket connections be awaited since this loop of polling for update events is limiting the number of threads that can be run at once thus limiting the number of connections I can open.
I attempted to solve this problem by using a semaphore as a way to detect events (the events are being observed through a IObserver interface)
static SemaphoreSlim send = new SemaphoreSlim(0,1);
static object newUpdate;
static numberOfClients = 0;
and the loop to poll and possibly receive updates or terminate
while(true)
{
await send.WaitAsync() //wait the thread so others can start
//do some things with the updated valu and send data or possibly quit
}
and in the OnNext implementation that detects events
public void OnNext(object value)
{
newUpdate = value;
for (int i = 0; i < numberOfClients; i++)
{
send.Release();
Thread.Sleep(100);
}
}
and the client implementation:
do
{
try
{
WebSocketReceiveResult result = await websocket.ReceiveAsync(buffer, CancellationToken.None);
byte[] receivedData = new byte[result.Count];
Buffer.BlockCopy(buffer.Array, 0, receivedData, 0, result.Count);
if (result.MessageType == WebSocketMessageType.Text)
{
Console.WriteLine("\nReceived message: " + Encoding.UTF8.GetString(receivedData));
}
}
catch (WebSocketException e)
{
if (websocket.State == WebSocketState.Aborted)
{
break;
}
continue;
}
if (result.EndOfMessage)
{
break;
}
}
while (websocket.State == WebSocketState.Open);
When I execute this it seems that the line:
await send.WaitAsync()
immediately makes the client code crash when it tries to call await .ReceiveAsync() and cries:
"Unable to read data from transport connection. An existing connection was forcibly closed by the remote host".
Any ideas how I can accomplish something like this? This is using OWIN WebSocket Extension.