I have build a basic .NET server-client infrastructure using TcpListener and SocketClient.
It is multithreaded and asynchronious. The problem is, that the server crashes at some point when over 30 clients are connected at once.
I could not yet locate the cause for the crashes though I do use quite a few Try-Catch blocks to make sure to log all excepions.
So I'm thinking, I may be doing something wrong conceptually in the server code. I hope you guys can help me find the cause for those crashes. The code is the following:
Starting server and listening for a connection:
public void StartServer()
{
isConnected = true;
listener.Start();
connectionThread = new Thread(new ThreadStart(ListenForConnection));
connectionThread.Start();
}
private void ListenForConnection()
{
while (isConnected)
{
try
{
TcpClient client = listener.AcceptTcpClient();
ClientConnection connection = new ClientConnection(this, client);
connections.Add(connection);
}
catch (Exception ex)
{
log.Log("Exception in ListenForConnection: " + ex.Message, LogType.Exception);
}
}
}
The ClientConnection class:
public class ClientConnection : IClientConnection
{
private TcpClient client;
private ISocketServer server;
private byte[] data;
private object metaData;
public TcpClient TcpClient
{
get { return client; }
}
internal ClientConnection(ISocketServer server, TcpClient client)
{
this.client = client;
this.server = server;
data = new byte[client.ReceiveBufferSize];
lock (client.GetStream())
{
client.GetStream().BeginRead(data, 0, client.ReceiveBufferSize, ReceiveMessage, null);
}
}
internal void ReceiveMessage(IAsyncResult ar)
{
int bytesRead;
try
{
lock (client.GetStream())
{
bytesRead = client.GetStream().EndRead(ar);
}
if (bytesRead < 1)
return;
byte[] toSend = new byte[bytesRead];
for (int i = 0; i < bytesRead; i++)
toSend[i] = data[i];
// Throws an Event with the data in the GUI Dispatcher Thread
server.ReceiveDataFromClient(this, toSend);
lock (client.GetStream())
{
client.GetStream().BeginRead(data, 0, client.ReceiveBufferSize, ReceiveMessage, null);
}
}
catch (Exception ex)
{
Disconnect();
}
}
public void Disconnect()
{
// Disconnect Client
}
}
And sending data from the server to one or all clients:
public void SendDataToAll(byte[] data)
{
BinaryWriter writer;
try
{
foreach (IClientConnection connection in connections)
{
writer = new BinaryWriter(connection.TcpClient.GetStream());
writer.Write(data);
writer.Flush();
}
}
catch (Exception ex)
{
// Log
}
}
public void SendDataToOne(IClientConnection client, byte[] data)
{
BinaryWriter writer;
try
{
writer = new BinaryWriter(client.TcpClient.GetStream());
writer.Write(data);
writer.Flush();
}
catch (Exception ex)
{
// Log
}
}
At some point the server crashes and I really got no starting point where to even look for the problem. If needed I can provide more code.
Any help is very appreciated :-)
Andrej
You should make access to the connections field thread safe.
In SendData, you are iterating over connections and sending data to each client. If a new client connects when the foreach loop is executing, you will get an exception with the message "Collection was modified; enumeration operation may not execute" since the collection is modified while you are iterating over it which is not allowed.
Modify the line in SendDataToAll to
foreach (IClientConnection connection in connections.ToList())
to make the problem go away (solution is courtesy of Collection was modified; enumeration operation may not execute).
It's possible that the Disconnect call in the catch block of the ClientConnection.ReceiveMessage method is throwing an exception, which then propogates out of the catch and is unhandled.
To be sure of catching all exceptions and logging them, try registering an event handler to the AppDomain.CurrentDomain.UnhandledException event. Also, if you're running the server as a Windows Service, there may be an .NET exception entry in the Application Event Log.
Related
I'm really confused. I am able to connect a TCPClient to a tcp server asynchronously. In my callback I now want to start reading some data, but when I go stream = tcpClient.GetStream(); my program doesn't exactly hang, it just does nothing. It won't go to the next line of the method, but the UI is still running (it's Unity, maybe it is multithreaded or something).
public void SetupSocket() {
try {
tcpClient = new TcpClient(host, port);
tcpClient.BeginConnect(host, port, ConnectCallback, tcpClient);
}
catch (Exception e) {
// stuff happens
return;
}
}
private void ConnectCallback(IAsyncResult result) {
if (ConnectedToServer != null)
ConnectedToServer(this, new ServerEventArgs("Connected to server."));
Debug.Log("Where am I?"); // it does get here
try {
stream = tcpClient.GetStream();
}
catch (InvalidOperationException e){
Debug.Log(e); // no exception
}
Debug.Log("Hello?"); // never gets here
BeginReadData();
}
public void BeginReadData() {
Debug.Log(stream.CanRead); // No log here!
if (stream.CanRead) {
stream.BeginRead(tcpStateObject.buffer, 0, tcpStateObject.bufferSize, EndReadData, stream);
}
}
I'm really lost at this point. I can see on my server that I connected, and when I disconnect. I send two messages to the client. It used to work with synchronous sockets, but I wanted async.
Here is the answer
I was connecting twice
tcpClient = new TcpClient(host, port);
tcpClient.BeginConnect(host, port, ConnectCallback, tcpClient);
I have simple server that gets string from client and prints it on screen.
I also have simple client, sending data and closing:
static void Main()
{
var client = new TcpClient("localhost", 26140);
var stream = client.GetStream();
Byte[] data = System.Text.Encoding.UTF8.GetBytes("CALC qwer");
stream.Write(data, 0, data.Length);
stream.Close();
client.Close();
//Thread.Sleep(100);
}
And with uncommented string 'Thread.Sleep(100)' it works ok.
But when commenting, sometimes ( 1 of 5-10 runs ) client doesn't send the string.
Watching wireshark and netstat I've noticed that client sends SYN,ACK package, establishes connection and exits without sending anything and without closing the socket.
Could anyone explain this behaivor? Why sleep helps? What am I doing wrong?
UPD:
With this sample code adding flush() before closing really works, thanks Fox32.
But after it I returned to my initial code:
var client = new TcpClient("localhost", 26140);
client.NoDelay = true;
var stream = client.GetStream();
var writer = new StreamWriter(stream);
writer.WriteLine("CALC qwer");
writer.Flush();
stream.Flush();
stream.Close();
client.Close();
And it isn't working, even with NoDelay. It's bad - using StreamWriter over network stream?
UPD:
Here is server code:
static void Main(string[] args)
{
(new Server(26140)).Run();
}
In Server class:
public void Run()
{
var listener = new TcpListener(IPAddress.Any, port);
listener.Start();
while (true)
{
try
{
var client = listener.AcceptTcpClient();
Console.WriteLine("Client accepted: " + client.Client.RemoteEndPoint);
var stream = client.GetStream();
stream.ReadTimeout = 2000;
byte[] buffer = new byte[1000];
stream.Read(buffer, 0, 1000);
var s = Encoding.UTF8.GetString(buffer);
Console.WriteLine(s);
}
catch (Exception ex)
{
Console.WriteLine("ERROR! " + ex.Message);
}
}
}
UPD:
Adding even Sleep(1) makes crashes happen in 1 of 30-50 clients running at the same time.
And adding Sleep(10) seems to be solving it totally, I can't catch any crash.
Don't understand, why socket needs this several milliseconds to close correctly.
The TcpClient is using the Nagle's algorithm and waits for more data before sending it over the wire. If you close the socket to fast, no data is trasmitted.
You have multiple ways to solve this problem:
The NetworkStream has a Flush method for flushing the stream content (I'm not sure if this method does anything from the comment on MSDN)
Disable Nagle's algorithm: Set NoDelay of the TcpCLient to true.
The last option is to set the LingerState of the TcpClient. The Close method documentation states, that the LingerState is used while calling Close
In almost all cases you are supposed to call Shutdown on a Socket or TcpClient before disposing it. Disposing rudely kills the connection.
Your code basically contains a race condition with the TCP stack.
Setting NoDelay is also a fix for this but hurts performance. Calling Flush IMHO still results an an disorderly shutdown. Don't do it because they are just hacks which paint over the problem by hiding the symptoms. Call Shutdown.
I want to stress that Shutdown being called on the Socket is the only valid solution that I know of. Even Flush just forces the data onto the network. It can still be lost due to a network hickup. It will not be retransmitted after Close has been called because Close is a rude kill on the socket.
Unfortunately TcpClient has a bug which forces you to go to the underlying Socket to shut it down:
tcpClient.Client.Shutdown();
tcpClient.Close();
According to Reflector, if you have ever accessed GetStream this problem arises and Close does not close the underlying socket. In my estimation this bug was produced because the developer did not really know about the importance of Shutdown. Few people know and many apps are buggy because of it. A related question.
In your server side code you are only calling Read() once, but you can't assume the data will be available when you call read. You have to continue reading in a loop until no more data is available. See the full example below.
I have tried to reproduce your issue with the minimal amount of code and was not able to. The server prints out the clients message everytime. No special settings such as NoDelay and no explicit Close() or Flush(), just Using statements which ensures all resources are properly disposed.
class Program
{
static int port = 123;
static string ip = "1.1.1.1";
static AutoResetEvent waitHandle = new AutoResetEvent(false);
static void Main(string[] args)
{
StartServer();
waitHandle.WaitOne();
for (int x=0; x<1000; x++)
{
StartClient(x);
}
Console.WriteLine("Done starting clients");
Console.ReadLine();
}
static void StartClient(int count)
{
Task.Factory.StartNew((paramCount) =>
{
int myCount = (int)paramCount;
using (TcpClient client = new TcpClient(ip, port))
{
using (NetworkStream networkStream = client.GetStream())
{
using (StreamWriter writer = new StreamWriter(networkStream))
{
writer.WriteLine("hello, tcp world #" + myCount);
}
}
}
}, count);
}
static void StartServer()
{
Task.Factory.StartNew(() =>
{
try
{
TcpListener listener = new TcpListener(port);
listener.Start();
Console.WriteLine("Listening...");
waitHandle.Set();
while (true)
{
TcpClient theClient = listener.AcceptTcpClient();
Task.Factory.StartNew((paramClient) => {
TcpClient client = (TcpClient)paramClient;
byte[] buffer = new byte[32768];
MemoryStream memory = new MemoryStream();
using (NetworkStream networkStream = client.GetStream())
{
do
{
int read = networkStream.Read(buffer, 0, buffer.Length);
memory.Write(buffer, 0, read);
}
while (networkStream.DataAvailable);
}
string text = Encoding.UTF8.GetString(memory.ToArray());
Console.WriteLine("from client: " + text);
}, theClient);
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}, TaskCreationOptions.LongRunning);
}
}
UPD:
I've tested this bug on several computers and nothing crashed. Seems like it is a local bug on my computer.
ENDOFUPD
So, what I've found about reproducing this bug.
#Despertar - your code works well. But it isn't reproduce conditions of this bug. On client you need to send data and quit after it. And in your code many clients are sending data and after all application is closing.
This is how I'm testing this on my computer:
I have server ( just accepting connection and print incoming data ), client ( just sends data once end exits ) and running utility ( runs client exe several times ).
So, I starts server, copies running utility to the clients folder and runs it.
Running ulility starts 150 clients connecting to server and 5-10 of them dies ( I see error in the server console ). And uncommenting Thread.Sleep() on client works well, no errors.
Can anyone try to reproduce this version of code?
Client code:
private static void Main(string[] args)
{
try
{
using (TcpClient client = new TcpClient(ip, port))
{
using (NetworkStream networkStream = client.GetStream())
{
using (StreamWriter writer = new StreamWriter(networkStream))
{
writer.WriteLine("# hello, tcp world #");
writer.Flush();
}
networkStream.Flush();
networkStream.Close();
}
client.Close();
//Thread.Sleep(10);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
Code, running client several times ( compile it in exe file and put near client's exe - this code will run many clients one by one ):
static void Main(string[] args)
{
string path = "YOU_CLIENT_PROJECT_NAME.exe";
for (int i = 0; i < 150; i++ )
{
Console.WriteLine(i);
Process.Start(path);
Thread.Sleep(50);
}
Console.WriteLine("Done");
Console.ReadLine();
}
( don't forget to change path to corrent exe filename )
Server code:
class Program
{
static int port = 26140;
static AutoResetEvent waitHandle = new AutoResetEvent(false);
static void Main(string[] args)
{
StartServer();
waitHandle.WaitOne();
Console.ReadLine();
}
static void StartServer()
{
Task.Factory.StartNew(() =>
{
try
{
TcpListener listener = new TcpListener(port);
listener.Start();
Console.WriteLine("Listening...");
waitHandle.Set();
while (true)
{
TcpClient theClient = listener.AcceptTcpClient();
Task.Factory.StartNew(paramClient =>
{
try
{
TcpClient client = (TcpClient) paramClient;
byte[] buffer = new byte[32768];
MemoryStream memory = new MemoryStream();
using (NetworkStream networkStream = client.GetStream())
{
networkStream.ReadTimeout = 2000;
do
{
int read = networkStream.Read(buffer, 0, buffer.Length);
memory.Write(buffer, 0, read);
} while (networkStream.DataAvailable);
string text = Encoding.UTF8.GetString(memory.ToArray());
}
}
catch (Exception e)
{
Console.WriteLine("ERROR: " + e.Message);
}
}, theClient);
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}, TaskCreationOptions.LongRunning);
}
}
I've tried code, reproducing this bug on several computers. No one crashes. Seems like it's my local computer bug.
Thanks for everybody for trying to help me.
Anyway, it's so strange. If I'll found out why this bug exists on my computer, I'll write about it.
I have built a server that receives requests from a client and gives a response that depends on the request Type. If the request type is streaming, the server must send data array. While the server’s streaming data the client may send a stop request to stop the streaming. If the request and the response is transferred on the same TCP connection the server only receives the stop request when all the data has finished streaming to the client. I think I must use Asynchronous write to solve this problem. This is my code:
First I create a loop back to receive connection from clients:
while (!done)
{
try
{
Socket socket = listener.AcceptSocket();
ClientInteraction clIr = new ClientInteraction(socket, statusList);
Thread thread = new Thread(new ThreadStart(clIr.Process));
thread.Start();
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
}
In Process function of ClientInteraction class :
Public void Process()
{
ns = new NetworkStream(socket);
while (true)
{
try
{
this.myReadBuffer = new byte[socket.ReceiveBufferSize];
this.numberOfBytesRead = ns.Read(myReadBuffer, 0, myReadBuffer.Length);
}
catch
{
break;
}
if (numberOfBytesRead == 0)
{
break;
}
else
{
HandleRequest(myReadBuffer, numberOfBytesRead);
}
}
}
In HandleRequest Function, if request’s STREAM, I will send data in an array to client:
Public void HanldeRequest(……)
{
myCompleteMessage = "";
myCompleteMessage =
String.Concat(myCompleteMessage, Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));
If(myCompleteMessage == “Stream”)
{
//I get data and call SendData function
foreach(.......)
{
//get data
........
SendData(data);
}
}
}
public void SendData(byte[] data)
{
try
{
//ns.Write(data, 0, data.Length);
ns.BeginWrite(data, 0, data.Length, new AsyncCallback(StreamData), null);
}
catch
{
}
}
public void StreamData(IAsyncResult asynResult)
{
if(asynResult != null)
ns.EndWrite(asynResult);
}
With this code, I connected with client, send data to client. But I still can’t receive Stop request until all data is streamed. Please show me the correct way to fix my problem. Thank you.
I think you can try using multithread to solve this problem. I have met this circumstance and multithread is a good choice to solve. You can create a thread to write and a thread to read.
What is the best way to make a Tcpclient auto reconnect to the server when
it's disconnected (e.g. by the server itself)?
The code I'm currently using is:
public void ClientWork()
{
TcpClient client = new TcpClient();
try
{
try
{
client.Connect(ip, port);
}
catch(Exception ex)
{
logger.ErrorFormat("client.Connect: {0}", ex.Message);
return false;
}
NetworkStream ns = client.GetStream();
byte[] buff;
while (__bRunning)
{
buff = new byte[1000];
ns.Read(buff, 0, 1000);
string line = System.Text.Encoding.Default.GetString(buff);
}
//ns.Close();
client.Close();
}
catch(Exception e)
{
//Reconnect?
client.Close();
client = null;
return false;
}
}
I'm using C# .NET
There is no events available to get notification for broken connection.
There could be 2 possible solution.
Polling. You have separate thread that try to poll socket continually done in different thread. Refer Instantly detect client disconnection from server socket
If you have low level control over socket or the interface which is using socket, you can do try..catch for read and write methods or try..catch for wrappers of read and write methods and when there is any exception you can re-connect and try to read and write.
I've a thread that runs following tcpConnect method. I wish to stop it at any given time by setting Program.PrepareExit to true. However my program is stuck at:
Int32 bytes = stream.Read(data, 0, data.Length); and doesn't really react to Program.PrepareExit set to true. How can i make it to always quit when I tell it so?
public static readonly Thread MyThreadTcpClient = new Thread(ThreadTcpClient);
private static void ThreadTcpClient() {
Connections.tcpConnect(ClientTcpIp, ClientTcpPort);
}
public static void Main() {
MyThreadTcpClient.Start();
.... some code....
Program.PrepareExit = true;
}
public static bool tcpConnect(string varClientIp, int varClientPort) {
var client = new TcpClient();
try {
client = new TcpClient(varClientIp, varClientPort) {NoDelay = true};
NetworkStream stream = client.GetStream();
var data = new Byte[256];
while (!Program.PrepareExit) {
Int32 bytes = stream.Read(data, 0, data.Length);
string varReadData = Encoding.ASCII.GetString(data, 0, bytes);
if (varReadData != "" && varReadData != "echo") {
VerificationQueue.EnqueueRelease(varReadData);
}
}
} catch (ArgumentNullException e) {
MessageBox.Show(e.ToString(), "ArgumentNullException");
tcpConnect(varClientIp, varClientPort);
} catch (SocketException e) {
MessageBox.Show(e.ToString(), "SocketException");
tcpConnect(varClientIp, varClientPort);
} catch (IOException e) {
if (e.ToString() != "Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.") {
}
MessageBox.Show(e.ToString());
tcpConnect(varClientIp, varClientPort);
} finally {
client.Close();
}
return false;
}
Three options suggest themselves:
Make the thread a daemon (background) thread. The process will exit when the only live threads are daemon threads
Set a timeout on the read call, possibly having to change to use the underlying socket API. That won't be pretty, to be honest.
Use asynchronous IO. Also a bit of a pain.
Do you need this thread to do anything in terms of an orderly shutdown? If not, the daemon thread approach would probably be simplest.