Yesterday I came accross a strange problem, which gave me quite some headaches. I have a server application with a Server class, which in turn is derived from a Connection class. The Connection class provides information about the connection state and the possibility to close the connection
public bool Connected
{
get
{
if (connection != null)
{
lock (lockObject)
{
bool blockingState = connection.Blocking;
try
{
connection.Blocking = false;
connection.Send(new byte[1], 1, 0);
}
catch (SocketException e)
{
if (!e.NativeErrorCode.Equals(10035))
{
return false;
}
//is connected, but would block
}
finally
{
connection.Blocking = blockingState;
}
return connection.Connected;
}
}
return false;
}
}
public virtual void CloseConnection()
{
if (Connected)
{
lock (lockObject)
{
connection.Close();
}
}
}
The Server class is resonsible for actually sending data
private void ConnectAndPollForData()
{
try
{
TcpListener listener = new TcpListener(Port);
listener.Start();
while (true)
{
connection = listener.AcceptSocket();
string currentBuffr = string.Empty;
const int READ_BUFFER_SIZE = 1024;
byte[] readBuffr = new byte[READ_BUFFER_SIZE];
while (Connected)
{
int bytesReceived;
lock (lockObject)
{
bytesReceived = connection.Receive(readBuffr, READ_BUFFER_SIZE, SocketFlags.None);
}
currentBuffr += ASCIIEncoding.ASCII.GetString(readBuffr, 0, bytesReceived);
//do stuff
}
}
catch(ThreadAbortException)
{
Thread.ResetAbort();
}
finally
{
}
}
public void SendString(string stringToSend)
{
stringToSend += "\r\n";
if(Connected)
{
lock(lockObject)
{
connection.Send(ASCIIEncoding.UTF7.GetBytes(stringToSend));
}
}
}
There is no other explicit access to the connection object. The ConnectAndPollForData function executes in a seperate thread. Whenever I ran the host in this version (I am currently using a non thread-safe version, which causes other problems) it hang after quite a few lines received via TCP. Pausing the debugger showed me, that one thread tried to execute the code withing the lock of Connected, while the other tried to receive data in the lock of ConnectAndPollForData. This behavior seems strange to me, for I would expect to execute the code within the first lock and then do the second. There do seem to be similar problems when using callbacks like in Deadlocking lock() method or 'Deadlock' with only one locked object? but the situation here is a bit different, for in my situation (I think) the code within the locks should not emit any events that themselves try to obtain a lock on the object.
Let's assume it gets the lock in the second method first. So it is holding the lock, and waiting for data. It is unclear whether this is directly receiving the data sent by the first method, or whether this is looking for a reply from an unrelated server - a reply to the message sent in the first method. But either way, I'm assuming that there won't be data incoming until the outbound message is sent.
Now consider: the outbound message can't be sent, because you are holding an exclusive lock.
So yes, you've deadlocked yourself. Basically, don't do that. There is no need to synchronize between inbound and outbound socket operations, even on the same socket. And since it makes very little sense to have concurrent readers on the same socket, or concurrent writers, I'm guessing you don't actually need those locks at all.
Related
I'm building a server app that accepts incoming TCP connections. (roughly 300 unique clients). It's important to note that I do not have control over the clients.
I have found that some of the connecting clients remain idle for quite some time after making the initial connection and sending the first status update. When they remain idle for over 5 mins the application's CPU usage jumps to over 90% and remains there.
To address this issue I built in a cancellation token that is triggered after 4 mins. This allows me to kill the connection. The client then detects this and reconnects about a minute later. This solves the high CPU usage issue, but has the side effect of high memory usage, there seems to be a memory leak. I suspect the resources is being held by the previous socket object.
I have a client object that contains the socket connection and information about the connected client. It also manages the incoming messages. There is also a manager class which accepts the incoming connections. It then creates the client object, assigns the socket to it and adds the client object to a concurrent dictionary. Every 10 seconds it checks the dictionary for clients that have been set to _closeConnection = true and calls their dispose method.
Here is the some of client object code:
public void StartCommunication()
{
Task.Run(async () =>
{
ArraySegment<byte> buffer = new ArraySegment<byte>(new byte[75]);
while (IsConnected)
{
try
{
// This is where I suspect the memory leak is originating - this call I suspect is not properly cleaned up when the object is diposed
var result = await SocketTaskExtensions.ReceiveAsync(ClientConnection.Client, buffer, SocketFlags.None).WithCancellation(cts.Token);
if (result > 0)
{
var message = new ClientMessage(buffer.Array, true);
if(message.IsValid)
HandleClientMessage(message);
}
}
catch (OperationCanceledException)
{
_closeConnection = true;
DisconnectReason = "Client has not reported in 4 mins";
}
catch (Exception e)
{
_closeConnection = true;
DisconnectReason = "Error during receive opperation";
}
}
});
}
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (disposing)
{
_closeConnection = true;
cts.Cancel();
// Explicitly kill the underlying socket
if (UnitConnection.Client != null)
{
UnitConnection.Client.Close();
}
UnitConnection.Close();
cts.Dispose();
}
}
Task Extension Method:
public static async Task<T> WithCancellation<T>(this Task<T> task, CancellationToken cancellationToken)
{
var tcs = new TaskCompletionSource<bool>();
using (cancellationToken.Register(s => ((TaskCompletionSource<bool>)s).TrySetResult(true), tcs))
{
if (task != await Task.WhenAny(task, tcs.Task))
{
throw new OperationCanceledException(cancellationToken);
}
}
return task.Result;
}
Mananger Code:
public bool StartListener()
{
_listener = new TcpListenerEx(IPAddress.Any, Convert.ToInt32(_serverPort));
_listener.Start();
Task.Run(async () =>
{
while (_maintainConnection) // <--- boolean flag to exit loop
{
try
{
HandleClientConnection(await _listener.AcceptTcpClientAsync());
}
catch (Exception e)
{
//<snip>
}
}
});
return true;
}
private void HandleClientConnection(TcpClient client)
{
Task.Run(async () =>
{
try
{
// Create new Coms object
var client = new ClientComsAsync();
client.ClientConnection = client;
// Start client communication
client.StartCommunication();
//_clients is the ConcurrentDictionary
ClientComsAsync existingClient;
if (_clients.TryGetValue(client.ClientName, out existingClient) && existingClient != null)
{
if (existingClient.IsConnected)
existingClient.SendHeatbeat();
if (!existingClient.IsConnected)
{
// Call Dispose on existing client
CleanUpClient(existingClient, "Reconnected with new connection");
}
}
}
catch (Exception e)
{
//<snip>
}
finally
{
//<snip>
}
});
}
private void CleanUpClient(ClientComsAsync client, string reason)
{
ClientComsAsync _client;
_units.TryRemove(client.ClientName, out _client);
if (_client != null)
{
_client.Dispose();
}
}
When they remain idle for over 5 mins the application's CPU usage jumps to over 90% and remains there.
To address this issue I built in a cancellation token that is triggered after 4 mins.
The proper response is to solve the high CPU usage problem.
Looks to me like it's here:
while (IsConnected)
{
try
{
var result = await SocketTaskExtensions.ReceiveAsync(ClientConnection.Client, buffer, SocketFlags.None);
if (result > 0)
{
...
}
}
catch ...
{
...
}
}
Sockets are weird, and dealing with raw TCP/IP sockets is quite difficult to do correctly. On a side note, I always encourage devs to use something more standard like HTTP or WebSockets, but in this case you don't control the clients, so that's not an option.
Specifically, your code is not handling the case where result == 0. If the client devices gracefully closed their socket, you'd see a result of 0, immediately loop back and keep getting a result of 0 - a tight loop that uses up CPU.
This is, of course, assuming that IsConnected remains true. And that may be possible...
You don't show where IsConnected is set in your code, but I suspect it's in the error handling after sending the heartbeat message. So here's why that may not work as expected... I suspect that the client devices are closing their sending stream (your receiving stream) while keeping their receiving stream (your sending stream) open. This is one way to shut down a socket, sometimes considered "more polite" because it allows the other side to continue sending data even though this side is done sending. (This is from the client device perspective, so the "other side" is your code, and "this side" is the client device).
And this is perfectly legal socket-wise because each connected socket is two streams, not one, each of which can be independently closed. If this happens, your heartbeats will still be send and received without error (and likely just silently discarded by the client device), IsConnected will remain true, and the read loop will become synchronous and eat up your CPU.
To resolve, add a check for result == 0 in your read loop and clean up the client just the same as if a heartbeat failed to send.
I need to establish multiple web socket connections that will forever receive updates unless they quit through some case. I need to have these socket connections be awaited since this loop of polling for update events is limiting the number of threads that can be run at once thus limiting the number of connections I can open.
I attempted to solve this problem by using a semaphore as a way to detect events (the events are being observed through a IObserver interface)
static SemaphoreSlim send = new SemaphoreSlim(0,1);
static object newUpdate;
static numberOfClients = 0;
and the loop to poll and possibly receive updates or terminate
while(true)
{
await send.WaitAsync() //wait the thread so others can start
//do some things with the updated valu and send data or possibly quit
}
and in the OnNext implementation that detects events
public void OnNext(object value)
{
newUpdate = value;
for (int i = 0; i < numberOfClients; i++)
{
send.Release();
Thread.Sleep(100);
}
}
and the client implementation:
do
{
try
{
WebSocketReceiveResult result = await websocket.ReceiveAsync(buffer, CancellationToken.None);
byte[] receivedData = new byte[result.Count];
Buffer.BlockCopy(buffer.Array, 0, receivedData, 0, result.Count);
if (result.MessageType == WebSocketMessageType.Text)
{
Console.WriteLine("\nReceived message: " + Encoding.UTF8.GetString(receivedData));
}
}
catch (WebSocketException e)
{
if (websocket.State == WebSocketState.Aborted)
{
break;
}
continue;
}
if (result.EndOfMessage)
{
break;
}
}
while (websocket.State == WebSocketState.Open);
When I execute this it seems that the line:
await send.WaitAsync()
immediately makes the client code crash when it tries to call await .ReceiveAsync() and cries:
"Unable to read data from transport connection. An existing connection was forcibly closed by the remote host".
Any ideas how I can accomplish something like this? This is using OWIN WebSocket Extension.
Why I am getting program hang on the tcpClient = this.ss.AcceptTcpClient?
public virtual void Run()
{
if (this.tcpListener == null)
return;
TcpClient tcpClient = (TcpClient)null;
while (!this.m_Stop)
{
try
{
tcpClient = this.tcpListener.AcceptTcpClient();
ThreadPool.QueueUserWorkItem(new WaitCallback(this.handler.Handle), (object)tcpClient);
}
catch (Exception ex)
{
MessageBox.Show(ex.ToString());
}
}
}
Here ss is the TcpListener.
I googled for TcpListener.AcceptTcpClient Method. This is what I found.
AcceptTcpClient is a blocking method that returns a TcpClient that you
can use to send and receive data. Use the Pending method to determine
if connection requests are available in the incoming connection queue
if you want to avoid blocking.
It seems that the method waits till there is a connection request available. So either use the Pending method call or do this on a background thread (not the UI thread).
I have been writing a command line program in C# that uses multiple tcp clients that all connect to the same server. Each client resides in it's own thread. At the moment I am trying to work out an effective method of spreading say 5 requests a second efficiently between let's say 4 threads.
My code currently looks like the following but I still end up with requests overlapping each other. Does anyone have any idea how to prevent these overlaps effectively?
// Max connections is 4, interval is 200
// Loop once to give tcp clients chance to connect
var tcpClients = new TcpClient[_maxConnections];
for(int i = 0; i < _maxConnections; i++)
{
tcpClients[i] = new TcpClient();
tcpClients[i].Connect(host, port);
}
// Loop again to setup tasks
for(int i = 0; i < _maxConnections; i++)
{
Task.Factory.StartNew(TcpHandler, tcpClients[i]);
// Sleep so every task starts separate from each other.
Thread.Sleep(_interval);
}
And then the TcpHandler code looks like:
public static void TcpHandler(Object o)
{
// active is already declared
while(_active)
{
var tcpClient = (TcpClient) o;
// .. do some send and receive...
Console.WriteLine("Something here..");
Thread.Sleep(_interval * _maxConnections);
}
}
So as you can see I am sleeping to provide sufficient space between each thread executing yet now and then they still overlap.
How can I make this threads run parallel without any overlap and limit to 5 times a second spread across all 4?
Or am I going about this all wrong?
Presuming each client requires a separate thread, and that only one thread may be communicating with the server at a given time (no overlap), a lock in the TcpHandler method should suffice:
// Max connections is 4, interval is 200
// Loop once to give tcp clients chance to connect
var tcpClients = new TcpClient[_maxConnections];
// dedicated lock object
static readonly object lockObject = new object();
And then in your TcpHandler method
public static void TcpHandler(Object o)
{
// active is already declared
while(_active)
{
//DO NON-SOCKET RELATED STUFF HERE
// ... code ...
//
//DO SOCKET RELATED STUFF HERE
lock(lockObject)
{
var tcpClient = (TcpClient) o;
// .. do some send and receive...
Console.WriteLine("Something here..");
Thread.Sleep(_interval * _maxConnections);
}
}
}
I am not quite sure why you are doing this but I have used System.Timers (actually an array of timers) in windows services and have staggered the start (intervals).
In the Elapse event maybe you could use a lock(myobject) { } so they don't overlap?
Gina
I think you are using sleep to manage connection times.. Why not instead setup a "Maximum connection delay" then use BeginConnect and a Timer to look after the connection.
eg.
//setup a timer variable
TCPClient connectionOpening;
_connecting = true;
_connected = false;
connectionOpening = tcpClient;
timer.change(5000, Infinite)
tcpClient.BeginConnect(ClientConnectCallback, tcpClient)
void ClientConnectCallback(iasyncresult ar)
{
_timer.change(infinite, infinite);
TCPClient tcp = (TCPClient)ar.AsyncState;
try
{
//if we have timed out because our time will abort the connect
tcp.EndConnect(ar);
_connected = true;
_connecting = false;
//we are now connected... do the rest you want to do.
//get the stream and BeginRead etc.
}
catch (Exception ex) // use the proper exceptions IOException , socketException etc
{
if (!_connecting)
{
//We terminated the connection because our timer ticked.
}
else
{
//some other problem that we weren't expecting
}
}
void TimerTick(object state)
{
_connecting = false;
_connected = false;
connectionOpening.Close();
}
I am trying to establish a TCP connection with a number of IPs in parallel, and do that as fast as possible. I have converted some older code to use AsyncCTP for that purpose, introducing the parallelism.
Changes to Design and Speed, and Accessing Successful Connections?
My question is three-fold:
How bad is the following flow / what should I change?
i.e. the await starts a bunch of parallel TcpRequest threads,
but within each TcpRequest there is a tcpClient.BeginConnect
as well as another thread being spawn for reading (if connection is successful)
and the writing to the stream is done with a Wait / Pulse mechanism in a while loop.
Secondly, how could i make the process of connecting to a number of targets faster?
Currently, if the ip:port targets are not actually running any servers, then i get the "All Done" printed after about 18 seconds from the start, when trying to connect to about 500 local targets (that are not listening, and thus fail, on those ports).
How could i access the WriteToQueue method of successful connections, from the mothership?
Async Mothership Trying to Connect to All Targets in Parallel
// First get a bunch of IPAddress:Port targets
var endpoints = EndPointer.Get();
// Try connect to all those targets
var tasks = from t in topList select TcpRequester.ConnectAsync(t);
await TaskEx.WhenAll(tasks);
Debug.WriteLine("All Done");
Static Accessor for Individual TcpRequest Tasks
public static Task<TcpRequester> ConnectAsync(IPEndPoint endPoint)
{
var tcpRequester = Task<TcpRequester>.Factory.StartNew(() =>
{
var request = new TcpRequester();
request.Connect(endPoint);
return request;
}
);
return tcpRequester;
}
TcpRequester with BeginConnect TimeOut and new Thread for Reading
public void Connect(IPEndPoint endPoint)
{
TcpClient tcpClient = null;
Stream stream = null;
using (tcpClient = new TcpClient())
{
tcpClient.ReceiveTimeout = 1000;
tcpClient.SendTimeout = 1000;
IAsyncResult ar = tcpClient.BeginConnect(endPoint.Address, endPoint.Port, null, null);
WaitHandle wh;
wh = ar.AsyncWaitHandle;
try
{
if (!ar.AsyncWaitHandle.WaitOne(TimeSpan.FromMilliseconds(1000), false))
{
throw new TimeoutException();
}
if (tcpClient.Client != null)
{
// Success
tcpClient.EndConnect(ar);
}
if (tcpClient.Connected)
{
stream = tcpClient.GetStream();
}
// Start to read stream until told to close or remote close
ThreadStart reader = () => Read(stream);
// Reading is done in a separate thread
var thread = new Thread(reader);
thread.Start();
// See Writer method below
Writer(stream);
} finally
{
wh.Close();
}
}
} catch (Exception ex)
{
if (tcpClient != null)
tcpClient.Close();
}
}
}
Writing to Stream with Wait and Pulse
readonly Object _writeLock = new Object();
public void WriteToQueue(String message)
{
_bytesToBeWritten.Add(Convert(message));
lock (_writeLock)
{
Monitor.Pulse(_writeLock);
}
}
void Writer(Stream stream)
{
while (!_halt)
{
while (_bytesToBeWritten.Count > 0 && !_halt)
{
// Write method does the actual writing to the stream:
if (Write(stream, _bytesToBeWritten.ElementAt(0)))
{
_bytesToBeWritten.RemoveAt(0);
} else
{
Discontinue();
}
}
if (!(_bytesToBeWritten.Count > 0) && !_halt)
{
lock (_writeLock)
{
Monitor.Wait(_writeLock);
}
}
}
Debug.WriteLine("Discontinuing Writer and TcpRequester");
}
There are a few red flags that pop out at a cursory glance.
You have this Stream that is accepting reads and writes, but there is no clear indication that the operations have been synchronized appropriately. The documentation does state that a Stream's instance methods are not safe for multithreaded operations.
There does not appear to be synchronization around operations involving _bytesToBeWritten.
Acquiring a lock solely to execute Monitor.Wait and Monitor.Pulse is a little weird, if not downright incorrect. It is basically equivalent to using a ManualResetEvent.
It is almost never correct to use Monitor.Wait without a while loop. To understand why you have to understand the purpose of pulsing and waiting on a lock. That is really outside the scope of this answer.
It appears like the Writer and WriteToQueue methods are an attempt to generate a producer-consumer queue. The .NET BCL already contains the innards for this via the BlockingCollection class.
For what it is worth I see nothing flagrantly wrong with the general approach and usage of the await keyword.