We Keep Coding

TCP server connection causing processor overload

I have a TCP/IP server that is supposed to allow a connection to remain open as messages are sent across it. However, it seems that some clients open a new connection for each message, which causes the CPU usage to max out. I tried to fix this by adding a time-out but still seem to have the problem occasionally. I suspect that my solution was not the best choice, but I'm not sure what would be.
Below is my basic code with logging, error handling and processing removed.
private void StartListening()
{
try
{
_tcpListener = new TcpListener( IPAddress.Any, _settings.Port );
_tcpListener.Start();
while (DeviceState == State.Running)
{
var incomingConnection = _tcpListener.AcceptTcpClient();
var processThread = new Thread( ReceiveMessage );
processThread.Start( incomingConnection );
}
}
catch (Exception e)
{
// Unfortunately, a SocketException is expected when stopping AcceptTcpClient
if (DeviceState == State.Running) { throw; }
}
finally { _tcpListener?.Stop(); }
}
I believe the actual issue is that multiple process threads are being created, but are not being closed. Below is the code for ReceiveMessage.
private void ReceiveMessage( object IncomingConnection )
{
var buffer = new byte[_settings.BufferSize];
int bytesReceived = 0;
var messageData = String.Empty;
bool isConnected = true;
using (TcpClient connection = (TcpClient)IncomingConnection)
using (NetworkStream netStream = connection.GetStream())
{
netStream.ReadTimeout = 1000;
try
{
while (DeviceState == State.Running && isConnected)
{
// An IOException will be thrown and captured if no message comes in each second. This is the
// only way to send a signal to close the connection when shutting down. The exception is caught,
// and the connection is checked to confirm that it is still open. If it is, and the Router has
// not been shut down, the server will continue listening.
try { bytesReceived = netStream.Read( buffer, 0, buffer.Length ); }
catch (IOException e)
{
if (e.InnerException is SocketException se && se.SocketErrorCode == SocketError.TimedOut)
{
bytesReceived = 0;
if(GlobalSettings.IsLeaveConnectionOpen)
isConnected = GetConnectionState(connection);
else
isConnected = false;
}
else
throw;
}
if (bytesReceived > 0)
{
messageData += Encoding.UTF8.GetString( buffer, 0, bytesReceived );
string ack = ProcessMessage( messageData );
var writeBuffer = Encoding.UTF8.GetBytes( ack );
if (netStream.CanWrite) { netStream.Write( writeBuffer, 0, writeBuffer.Length ); }
messageData = String.Empty;
}
}
}
catch (Exception e) { ... }
finally { FileLogger.Log( "Closing the message stream.", Verbose.Debug, DeviceName ); }
}
}
For most clients the code is running correctly, but there are a few that seem to create a new connection for each message. I suspect that the issue lies around how I handle the IOException. For the systems that fail, the code does not seem to reach the finally statement until 30 seconds after the first message comes in, and each message creates a new ReceiveMessage thread. So the logs will show messages coming in, and 30 seconds in it will start to show multiple messages about the message stream being closed.
Below is how I check the connection, in case this is important.
public static bool GetConnectionState( TcpClient tcpClient )
{
var state = IPGlobalProperties.GetIPGlobalProperties()
.GetActiveTcpConnections()
.FirstOrDefault( x => x.LocalEndPoint.Equals( tcpClient.Client.LocalEndPoint )
&& x.RemoteEndPoint.Equals( tcpClient.Client.RemoteEndPoint ) );
return state != null ? state.State == TcpState.Established : false;
}

You're reinventing the wheel (in a worse way) at quite a few levels:
You're doing pseudo-blocking sockets. That combined with creating a whole new thread for every connection in an OS like Linux which doesn't have real threads can get expensive fast. Instead you should create a pure blocking socket with no read timeout (-1) and just listen on it. Unlike UDP, TCP will catch the connection being terminated by the client without you needing to poll for it.
And the reason why you seem to be doing the above is that you reinvent the standard Keep-Alive TCP mechanism. It's already written and works efficiently, simply use it. And as a bonus, the standard Keep-Alive mechanism is on the client side, not the server side, so even less processing for you.
Edit: And 3. You really need to cache the threads you so painstakingly created. The system thread pool won't suffice if you have that many long-term connections with a single socket communication per thread, but you can build your own expandable thread pool. You can also share multiple sockets on one thread using select, but that's going to change your logic quite a bit.

C# NetworkStream ReadAsync, data split between buffers

I'm currently using the ReadAsync method in NetworkStream, and thus far its been pretty stable. The problem occurs when the data received from the server is larger than the buffer itself.
ReadCallBack assumes that the buffer contains a complete and valid message, as a new buffer is created on every call to BeginAsyncRead. Thus, when the response is too large for one buffer, it gets split between multiple reads and thus multiple buffers.
My ReadCallBack then assumes each buffer is an independent message (instead of adding them together) and ends up failing the deserialization on both.
My question is, what is the correct way to handle messages that are bigger than the size of the buffer, specially if you want to do the reads asynchronously?
A character itself could be split between two buffers so this gets quite tricky.
Below is my code for ReadAsync and ReadCallBack.
public async Task ReadForever() {
while(this.IsConnected) {
await BeginAsyncRead();
}
}
public async Task BeginAsyncRead() {
if (this.Stream.CanRead) {
try {
ReceiverState readState = new ReceiverState(this.Stream);
var task = this.Stream.ReadAsync(readState.buffer, 0, ReceiverState.bufferSize);
await task.ContinueWith(bytesRead => ReadCallBack(bytesRead.Result, readState));
}
catch (Exception e) {
Console.WriteLine("Error");
}
} else {
Console.WriteLine("Unreadable stream");
}
}
private void ReadCallBack(int numBytesRead, ReceiverState state) {
if (numBytesRead > 0) {
string payload = Encoding.ASCII.GetString(state.buffer, 0, numBytesRead);
string[] responses = payload.Split(new string[] {"\n"}, StringSplitOptions.RemoveEmptyEntries);
foreach (string res in responses){
var t = Task.Run(() => this.OnRead(res));
}
} else {
Console.WriteLine("Corrupted number of bytes received");
}
}
Note : All responses sent by the server contain a newline at the end.
Note : ReadForever is called inside Task.Run, my application can receive messages from the server as notifications, thus I must always be reading for incoming messages (I don't know when a notification may arrive).

Read unknown length by DataReader

I've been working with windows app store programming in c# recently, and I've come across a problem with sockets.
I need to be able to read data with an unknown length from a DataReader().
It sounds simple enough, but I've not been able to find a solution after a few days of searching.
Here's my current receiving code (A little sloppy, need to clean it up after I find a solution to this problem. And yes, a bit of this is from the Microsoft example)
DataReader reader = new DataReader(args.Socket.InputStream);
try
{
while (true)
{
// Read first 4 bytes (length of the subsequent string).
uint sizeFieldCount = await reader.LoadAsync(sizeof(uint));
if (sizeFieldCount != sizeof(uint))
{
// The underlying socket was closed before we were able to read the whole data.
return;
}
reader.InputStreamOptions
// Read the string.
uint stringLength = reader.ReadUInt32();
uint actualStringLength = await reader.LoadAsync(stringLength);
if (stringLength != actualStringLength)
{
// The underlying socket was closed before we were able to read the whole data.
return;
}
// Display the string on the screen. The event is invoked on a non-UI thread, so we need to marshal
// the text back to the UI thread.
//MessageBox.Show("Received data: " + reader.ReadString(actualStringLength));
MessageBox.updateList(reader.ReadString(actualStringLength));
}
}
catch (Exception exception)
{
// If this is an unknown status it means that the error is fatal and retry will likely fail.
if (SocketError.GetStatus(exception.HResult) == SocketErrorStatus.Unknown)
{
throw;
}
MessageBox.Show("Read stream failed with error: " + exception.Message);
}

You are going down the right lines - read the first INT to find out how many bytes are to be sent.
Franky Boyle is correct - without a signalling mechanism it is impossible to ever know the length of a stream. Thats why it is called a stream!
NO socket implementation (including the WinSock) will ever be clever enough to know when a client has finished sending data. The client could be having a cup of tea half way through sending the data!
Your server and its sockets will never know! What are they going to do? Wait forever? I suppose they could wait until the client had 'closed' the connection? But your client could have had a blue screen and the server will never get that TCP close packet, it will just be sitting there thinking it is getting more data one day?
I have never used a DataReader - i have never even heard of that class! Use NetworkStream instead.
From my memory I have written code like this in the past. I am just typing, no checking of syntax.
using(MemoryStream recievedData = new MemoryStream())
{
using(NetworkStream networkStream = new NetworkStream(connectedSocket))
{
int totalBytesToRead = networkStream.ReadByte();
// This is your mechanism to find out how many bytes
// the client wants to send.
byte[] readBuffer = new byte[1024]; // Up to you the length!
int totalBytesRead = 0;
int bytesReadInThisTcpWindow = 0;
// The length of the TCP window of the client is usually
// the number of bytes that will be pushed through
// to your server in one SOCKET.READ method call.
// For example, if the clients TCP window was 777 bytes, a:
// int bytesRead =
// networkStream.Read(readBuffer, 0, int.Max);
// bytesRead would be 777.
// If they were sending a large file, you would have to make
// it up from the many 777s.
// If it were a small file under 777 bytes, your bytesRead
// would be the total small length of say 500.
while
(
(
bytesReadInThisTcpWindow =
networkStream.Read(readBuffer, 0, readBuffer.Length)
) > 0
)
// If the bytesReadInThisTcpWindow = 0 then the client
// has disconnected or failed to send the promised number
// of bytes in your Windows server internals dictated timeout
// (important to kill it here to stop lots of waiting
// threads killing your server)
{
recievedData.Write(readBuffer, 0, bytesReadInThisTcpWindow);
totalBytesToRead = totalBytesToRead + bytesReadInThisTcpWindow;
}
if(totalBytesToRead == totalBytesToRead)
{
// We have our data!
}
}
}

WP Background Transfer Service has Memory Leak?

Recently, I found that Windows Phone Background Transfer Service seems to have memory leak issue.
Every background transfer you add will take a memory space, which cannot be removed by GC forever.
I already read through http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh202959(v=vs.105).aspx , and still have no idea where the memory leak comes from.
What I test is very simple, add a background transfer request to the BackgroundTransferService and when that request is complete, remove it from BackgroundTransferService and add another one. If I keep doing it, I'll see the memory grows even when GC.collect being called every sec. Please download the testing code in http://hdtp.synology.me/BTS.zip and you'll know what I'm saying. The following is the testing code summary.
private int _transferCount = 1000;
private void CreateTask()
{
if (--_transferCount < 0)
{
MessageBox.Show("End");
return;
}
// Get the URI of the file to be transferred from the Tag property
// of the button that was clicked.
//string transferFileName = ((Button)sender).Tag as string;
string transferFileName = "http://hdtp.synology.me/a.jpg";
Uri transferUri = new Uri(Uri.EscapeUriString(transferFileName + "?ranNum=" + _transferCount), UriKind.RelativeOrAbsolute);
// Create the new transfer request, passing in the URI of the file to
// be transferred.
BackgroundTransferRequest transferRequest = new BackgroundTransferRequest(transferUri);
// Set the transfer method. GET and POST are supported.
transferRequest.Method = "GET";
// Get the file name from the end of the transfer Uri and create a local Uri
// in the "transfers" directory in isolated storage.
string downloadFile = transferFileName.Substring(transferFileName.LastIndexOf("/") + 1);
Uri downloadUri = new Uri("shared/transfers/" + downloadFile, UriKind.RelativeOrAbsolute);
transferRequest.DownloadLocation = downloadUri;
// Pass custom data with the Tag property. This value cannot be more than 4000 characters.
// In this example, the friendly name for the file is passed.
transferRequest.Tag = downloadFile;
// Add the transfer request using the BackgroundTransferService. Do this in
// a try block in case an exception is thrown.
try
{
BackgroundTransferService.Add(transferRequest);
}
catch (InvalidOperationException ex)
{
// TBD - update when exceptions are finalized
MessageBox.Show("Unable to add background transfer request. " + ex.Message);
}
catch (Exception)
{
MessageBox.Show("Unable to add background transfer request.");
}
InitialTansferStatusCheck();
}
private void InitialTansferStatusCheck()
{
UpdateRequestsList();
foreach (var transfer in transferRequests)
{
transfer.TransferStatusChanged += new EventHandler<BackgroundTransferEventArgs>(transfer_TransferStatusChanged);
ProcessTransfer(transfer);
}
}
private void transfer_TransferStatusChanged(object sender, BackgroundTransferEventArgs e)
{
ProcessTransfer(e.Request);
}
private void UpdateRequestsList()
{
// The Requests property returns new references, so make sure that
// you dispose of the old references to avoid memory leaks.
if (transferRequests != null)
{
foreach (var request in transferRequests)
{
request.Dispose();
}
}
transferRequests = BackgroundTransferService.Requests;
}
private void ProcessTransfer(BackgroundTransferRequest transfer)
{
switch (transfer.TransferStatus)
{
case TransferStatus.Completed:
// If the status code of a completed transfer is 200 or 206, the
// transfer was successful
if (transfer.StatusCode == 200 || transfer.StatusCode == 206)
{
// Remove the transfer request in order to make room in the
// queue for more transfers. Transfers are not automatically
// removed by the system.
RemoveTransferRequest(transfer.RequestId);
// In this example, the downloaded file is moved into the root
// Isolated Storage directory
using (IsolatedStorageFile isoStore = IsolatedStorageFile.GetUserStoreForApplication())
{
string filename = transfer.Tag;
if (isoStore.FileExists(filename))
{
isoStore.DeleteFile(filename);
}
isoStore.MoveFile(transfer.DownloadLocation.OriginalString, filename);
}
CreateTask();
}
else
{
// This is where you can handle whatever error is indicated by the
// StatusCode and then remove the transfer from the queue.
RemoveTransferRequest(transfer.RequestId);
if (transfer.TransferError != null)
{
// Handle TransferError, if there is one.
}
}
break;
}
}
private void RemoveTransferRequest(string transferID)
{
// Use Find to retrieve the transfer request with the specified ID.
BackgroundTransferRequest transferToRemove = BackgroundTransferService.Find(transferID);
// try to remove the transfer from the background transfer service.
try
{
BackgroundTransferService.Remove(transferToRemove);
}
catch (Exception ex)
{
}
}
Another few questions, according to the documentation above, we'll get the new instance from BackgroundTransferService.Requests every time, but if I called GetHashCode(), I get the same hash code every time and the hash code is even the same with the one I newed and added into BackgroundTransferService. So is it because MS override the GetHashCode method of BackgroundTransferRequest? or I misunderstand something. But in the sample code above, I did not use BackgroundTransferService.Requests to get any instance, the memory still keep growing.
Please tell me what I do wrong or any workaround, thanks...

Equivalent of Peek in C# async sockets?

I'm used to using synchronous sockets. In order to deal with messages that have not completely arrived yet, I'd set the first 4 bytes to be the expected length of the message. Then I'd use Socket.Receive(tcpRecv, 1024, SocketFlags.Peek); to take a look at the message without pulling it off the buffer. If all of it was there, I'd pull the data. If it wasn't, I'd leave it there. I had designed my protocol so that no message would ever be greater than 1024 bytes.
In asynchronous sockets, I don't see a way to peek at the data. Is there a way to do this? Is there a better approach to this than peeking at the data?
Thanks.
-Nick

You don't need to peek: .NET asynchronous sockets allow you to achieve the same type of functionality without peeking. I think you might be looking for something like this:
private void BeginReceive()
{
if ( _clientState == EClientState.Receiving)
{
if (_asyncTask.BytesReceived != 0 && _asyncTask.TotalBytesReceived <= _maxPageSize)
{
SocketAsyncEventArgs e = new SocketAsyncEventArgs();
e.SetBuffer(_asyncTask.ReceiveBuffer, 0, _asyncTask.ReceiveBuffer.Length);
e.Completed += new EventHandler<SocketAsyncEventArgs>(ReceiveCallback);
e.UserToken = _asyncTask.Host;
bool comletedAsync = false;
try
{
comletedAsync = _socket.ReceiveAsync(e);
}
catch (SocketException se)
{
Console.WriteLine("Error receiving data from: " + _asyncTask.Host);
Console.WriteLine("SocketException: {0} Error Code: {1}", se.Message, se.NativeErrorCode);
ChangeState(EClientState.Failed);
}
if (!comletedAsync)
{
// The call completed synchronously so invoke the callback ourselves
ReceiveCallback(this, e);
}
}
else
{
//Console.WriteLine("Num bytes received: " + _asyncTask.TotalBytesReceived);
ChangeState(EClientState.ReceiveDone);
}
}
}
When you get the callback you can schedule another receive:
private void ReceiveCallback(object sender, SocketAsyncEventArgs args)
{
lock (_sync) // re-entrant lock
{
// Fast fail: should not be receiving data if the client
// is not in a receiving state.
if (_clientState == EClientState.Receiving)
{
String host = (String)args.UserToken;
if (_asyncTask.Host == host && args.SocketError == SocketError.Success)
{
try
{
Encoding encoding = Encoding.ASCII;
_asyncTask.BytesReceived = args.BytesTransferred;
_asyncTask.TotalBytesReceived += _asyncTask.BytesReceived;
_asyncTask.DocSource += encoding.GetString(_asyncTask.ReceiveBuffer, 0, _asyncTask.BytesReceived);
BeginReceive();
}
catch (SocketException e)
{
Console.WriteLine("Error receiving data from: " + host);
Console.WriteLine("SocketException: {0} Error Code: {1}", e.Message, e.NativeErrorCode);
ChangeState(EClientState.Failed);
}
}
else if (_asyncTask.Host != host)
{
Console.WriteLine("Warning: received a callback for {0}, but the client is currently working on {1}.",
host, _asyncTask.Host);
}
else
{
Console.WriteLine("Socket Error: {0} when receiving from {1}",
args.SocketError,
_asyncTask.Host);
ChangeState(EClientState.Failed);
}
}
}
}
You can see the entire asynchronous client on my blog: http://codesprout.blogspot.com/2011/04/asynchronous-http-client.html

Your same data flow works without peeking:
schedule a four byte read
when it completes, save it in the buffer and decode it into length "n"
schedule a read of length "n" - 4
when it completes, append it to the four bytes already there
decode your message
The only difference from peeking is that you have to save the four bytes when you initially read them.