We Keep Coding

NetworkStream Async Read -> Cancel

Currently I try to read and write Async to/from a network stream. My software is the Client part and the server can send informations on its own or respond to commands I send him.
So I need a socket which
reads all the time (in case the server sends status informations)
stops reading when I want to send commands (commands can be sequences of data with multible Write and Read operations)
So I thought it would be a good approach to create a Semaphore and a Background Task which handles the server sent messages and in case I want to send a command I block the semaphore and have full access to read/write operations to the socket.
Here is what I do currently.
private TcpClient _tcpClient = new TcpClient();
protected SemaphoreSlim ClientSemaphore { get; } = new SemaphoreSlim(1, 1);
public async Task ConnectAsync()
{
if (_tcpClient.Connected)
{
await DisconnectAsync();
}
await _tcpClient.ConnectAsync(Hostname, RemotePort);
//here the background Task is started
_ = AutoReceiveMessages();
}
private async Task AutoReceiveMessages()
{
while (_tcpClient.Connected)
{
//enter and lock semaphore
await ClientSemaphore.WaitAsync();
try
{
//read from socket until timeout (ms)
var msg = await ReadFromSocket(2000);
foreach (var cmd in SplitMessageInTelegrams(msg))
{
Console.WriteLine("MESSAGE --> " + cmd);
}
}
catch (Exception ex)
{
}
finally
{
//release semaphore
ClientSemaphore.Release();
}
}
}
private async Task<string> ReadFromSocket(double timeout = 0)
{
var buf = new byte[4096];
var stream = _tcpClient.GetStream();
//read from stream or timeout
var amountReadTask = stream.ReadAsync(buf, 0, buf.Length);
var timeoutTask = Task.Delay(TimeSpan.FromMilliseconds(timeout));
await Task.WhenAny(timeoutTask, amountReadTask)
.ConfigureAwait(false);
//timeout
if (!amountReadTask.IsCompleted)
{
throw new TimeoutException("Timeout");
}
//no timeout
return Encoding.ASCII.GetString(buf, 0, amountReadTask.Result);
}
But this do not work as I expected...
I use this methode to send a message to the server and in WireShark I see the server resonds with the same message
protected async Task SendTelegramAsync(ITelegram telegram)
{
await ClientSemaphore.WaitAsync();
try
{
_ = telegram ?? throw new ArgumentException($"{nameof(telegram)}");
if (!_tcpClient.Connected) throw new InvalidOperationException("Socket not connected!");
var buf = new byte[4096];
var stream = _tcpClient.GetStream();
var msg = Encoding.ASCII.GetBytes("\x02" + telegram.GetCommandMessage() + "\x03");
Console.WriteLine("WRITE --> " + msg);
await stream.WriteAsync(msg, 0, msg.Length);
//comment AutoReceiveMessage and remove comment from this
//and I get responses from the server
//var test = await ReadFromSocket(2000);
}
finally
{
ClientSemaphore.Release();
}
}
I know in this case I do not need the semaphore but later I want to create sequences so one command consists of multible writes and reads and as long as the command is executed I do not want to use the AutoReceiveMessages method.
The problem now is
If I use it like this I never get a response the ReadFromSocket method always get the timeout even when wireshark tell me the server has responded
But even better if I disable AutoReceiveMessages (just comment _ = AutoReceiveMessages()) and use ReadFromSocket directly in SendTelegramAsync() everything work as expected.
So I think the problem is something related to the background task and the ReadAsync but I couldnt figure it out...

Got It!
stream.DataAvailable is your friend (or my friend :)).
If I check before the ReadAsync if DataIsAvailable then I have no problem anymore.
if (_tcpClient.GetStream().DataAvailable)
var msg = await ReadFromSocket(DEFAULT_TIMEOUT);

Wait for incoming tcp/ip server request

I'm building a TCP/IP connection for my application with a warehouse system. The communication goes like this.
I send a message to the TCP/IP(Socket) server of the warehouse system.
The warehouse system responds with a message the my local TCP/IP server.
So there are no direct response messages. Instead each application as it's own server.
Yet I want my application to wait for the response coming from the other server.
So basicly I have the following code.
public string ControllerFunction() {
startLocalTcpIpServer();
sendMessage("a message");
return clientMessage;
}
This is my own server started with the start() function
public void Start() {
// Start TcpServer background thread
tcpListenerThread = new Thread(new ThreadStart(ListenForIncommingRequests)) {
IsBackground = true
};
tcpListenerThread.Start();
}
private void ListenForIncommingRequests() {
try {
tcpListener = new TcpListener(IPAddress.Parse(serverIp), port);
tcpListener.Start();
byte[] bytes = new byte[1024];
Console.WriteLine("Server Started");
while(true) {
// Get a stream object for reading
using(NetworkStream stream = tcpListener.AcceptTcpClient().GetStream()) {
int length;
// Read incomming stream into byte arrary.
while((length = stream.Read(bytes, 0, bytes.Length)) != 0) {
byte[] incommingData = new byte[length];
Array.Copy(bytes, 0, incommingData, 0, length);
// Convert byte array to string message.
string clientMessage = Encoding.ASCII.GetString(incommingData);
}
}
}
}
catch(SocketException socketException) {
Console.WriteLine("SocketException " + socketException.ToString());
}
}
So I want to use the result string clientMessage again as a return for my ControllerFunction. But how do I get the data there in a proper way?

So what you need is to be able to wait for response coming from another place in your application (local server). Response will be fired there first. Local server should have an event you can subscribe to (OnMessage in my example). This event will forward result message to you.
Synchronization can be handled using TaskCompletionSource. You will create Task that you can use to obtain result synchronously or asynchronously.
Something like this:
public string ControllerFunction()
{
return ControllerFunctionTask().Result;
}
public Task<string> ControllerFunctionTask()
{
sendMessage("a message");
var task = new TaskCompletionSource<string>();
localServer.OnMessage = (message) =>
{
task.SetResult(message);
};
return task.Task;
}
As stated in comments, synchronous waiting for asynchronous Task may lead to deadlocks. This may happen when caller thread is context thread (UI, ASP). Therefore this should be better approach:
public async Task<string> ControllerFunction()
{
return await ControllerFunctionTask();
}
public Task<string> ControllerFunctionTask()
{
sendMessage("a message");
var task = new TaskCompletionSource<string>();
localServer.OnMessage = (message) =>
{
task.SetResult(message);
};
return task.Task;
}
OnMessage can be defined this way:
public event Action<string> OnMessage;
Then it will be called right after line where you get clientMessage string:
string clientMessage = Encoding.ASCII.GetString(incommingData);
if (OnMessage != null)
OnMessage(clientMessage);

How to build a robust/scalable async await echo server?

Im building a simple TCP client and server as a basis for my networking project. Im planning to use the async await technique for future proof and scaleable server.
If I put wrong ip address, the client cant connect to my server and throw an exception. I can catch the exception using try/catch but is that the recommended way to do?
What do you guys think of the implementation. Any comments for me to improve?
My server
private void startServer_Click(object sender, RoutedEventArgs e)
{
if (anyIP.IsChecked == true)
{
listener = new TcpListener(IPAddress.Any, Int32.Parse(serverPort.Text));
Logger.Info("Ip Address : " + IPAddress.Any + " Port : " + serverPort.Text);
}
else
{
listener = new TcpListener(IPAddress.Parse(serverIP.Text), Int32.Parse(serverPort.Text));
Logger.Info("Ip Address : " + serverIP.Text + " Port : " + serverPort.Text);
}
try
{
listener.Start();
Logger.Info("Listening");
HandleConnectionAsync(listener, cts.Token);
}
//finally
//{
//cts.Cancel();
//listener.Stop();
//Logger.Info("Stop listening");
//}
//cts.Cancel();
}
async Task HandleConnectionAsync(TcpListener listener, CancellationToken ct)
{
while (!ct.IsCancellationRequested)
{
Logger.Info("Accepting client");
//TcpClient client = await listener.AcceptTcpClientAsync();
TcpClient client = await listener.AcceptTcpClientAsync();
Logger.Info("Client accepted");
EchoAsync(client, ct);
}
}
async Task EchoAsync(TcpClient client, CancellationToken ct)
{
var buf = new byte[4096];
var stream = client.GetStream();
while (!ct.IsCancellationRequested)
{
var amountRead = await stream.ReadAsync(buf, 0, buf.Length, ct);
Logger.Info("Receive " + stream.ToString());
if (amountRead == 0) break; //end of stream.
await stream.WriteAsync(buf, 0, amountRead, ct);
Logger.Info("Echo to client");
}
}
private void stopServer_Click(object sender, RoutedEventArgs e)
{
cts.Cancel();
listener.Stop();
Logger.Info("Stop listening");
}
My client
private void connect_Click(object sender, System.Windows.RoutedEventArgs e)
{
IPAddress ipAddress;
int port;
//TODO Check if ip address is valid
ipAddress = IPAddress.Parse(serverIP.Text);
//TODO port range is 0-65000
port = int.Parse(serverPort.Text);
StartClient(ipAddress, port);
}
private static async void StartClient(IPAddress serverIpAddress, int port)
{
var client = new TcpClient();
//can i try/catch to catch await exception?
try
{
await client.ConnectAsync(serverIpAddress, port);
}
catch (Exception e)
{
Logger.Info(e);
}
Logger.Info("Connected to server");
using (var networkStream = client.GetStream())
using (var writer = new StreamWriter(networkStream))
using (var reader = new StreamReader(networkStream))
{
writer.AutoFlush = true;
for (int i = 0; i < 10; i++)
{
Logger.Info("Writing to server");
await writer.WriteLineAsync(DateTime.Now.ToLongDateString());
Logger.Info("Reading from server");
var dataFromServer = await reader.ReadLineAsync();
if (!string.IsNullOrEmpty(dataFromServer))
{
Logger.Info(dataFromServer);
}
}
}
if (client != null)
{
client.Close();
Logger.Info("Connection closed");
}
}

I have a .NET TCP/IP FAQ that I recommend to get some of the basics down.
After just a brief look at your code, these points stood out to me:
Both your client and server have times when they're only reading (not writing). This means you're subject to the half-open scenario (as I describe in my FAQ). A robust server should be writing periodically even if it has nothing to say.
Both your client and server have times when they're only writing (not reading). This means that you're subject to a deadlock (as I describe in my FAQ) if the other end is not behaving well (e.g., sending lots of data). However, you can't just read indefinitely or you'll open yourself up to a DoS; so you should decide where your limit is and establish read buffer sizes (and write timeouts) that make sense for your application.
Using ReadLineAsync leaves you open to a trivial DoS attack, since you can't specify the maximum allowed size of the line.
Your code must be prepared for an exception at any time (as I describe in my FAQ). Obviously, ReadAsync and WriteAsync may throw. What's less obvious is that any socket method may throw, including AcceptTcpClientAsync.
Your code uses a mixture of exception handling types. The async Task methods are never awaited, so exceptions there just silently end that method. The StartClient method is more problematic, since it is async void. You'll need to think through your application needs for error detection and retry strategies, and apply proper handling at every level.
In conclusion, I reiterate my comment: I strongly recommend just self-hosting SignalR. Sockets should only be used if you have no choice.

How can I send or return List<T> with socket

ther is a server and client sockets. I want to learn a way how can I send a List<T>, IEnumerable<T> object to client to server or server to client.
i want to send server-side TaskManager Threads List to client-side. this code send line by line, I want to send complate list. How can I do it?
private static IEnumerable<string> TaskManager()
{
List<string> lst = new List<string>();
foreach (System.Diagnostics.Process p in
Process.GetProcesses().OrderBy(o => o.ProcessName))
{
lst.Add(p.ProcessName + "_" + p.Id);
}
return lst.AsParallel();
}
....
....
....
while (true)
{
Socket socket = Listener.AcceptSocket();
try
{
//open stream
Stream stream = new NetworkStream(socket);
StreamReader sr = new StreamReader(stream);
StreamWriter sw = new StreamWriter(stream);
sw.AutoFlush = true;
while (true)
{
Parallel.ForEach(
TaskManager(), item=>
sw.WriteLine(item)
);
}
stream.Close();
stream.Dispose();
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
Console.WriteLine("Disconnected: {0}", socket.RemoteEndPoint);
socket.Close();
socket.Dispose();
}

You can use BinaryFormatter to serialize and deserialze a List.
using System.Runtime.Serialization.Formatters.Binary;
// Server side
Stream stream = new NetworkStream(socket);
var bin = new BinaryFormatter();
bin.Serialize(stream, TaskManager());
// Client side
Stream stream = new NetworkStream(socket);
var bin = new BinaryFormatter();
var list = (List<string>)bin.Deserialize(stream);

You could implement all aspects of a solution yourself, i.e. serialisation, compression, sending, receiving etc. Most things can be found by reading about WCF or the System.Net namespace. I can however give you a very concise solution using the network library NetworkComms.Net here.
For the client:
using System;
using NetworkCommsDotNet;
namespace Client
{
class Program
{
static void Main(string[] args)
{
//Create a connection
Connection connection = TCPConnection.GetConnection(new ConnectionInfo("127.0.0.1", 10000));
//Make a request for the threadIds and get the answer in one statement.
string[] taskManagerThreadIds = connection.SendReceiveObject<string[]>("ThreadIdRequest", "ThreadIds", 2000);
Console.WriteLine("Server provided an array containing {0} ids", taskManagerThreadIds.Length);
Console.WriteLine("Send completed. Press any key to exit client.");
Console.ReadKey(true);
NetworkComms.Shutdown();
}
}
}
For the server:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;
using NetworkCommsDotNet;
namespace Server
{
class Program
{
static void Main(string[] args)
{
NetworkComms.AppendGlobalIncomingPacketHandler<object>("ThreadIdRequest", (packetHeader, connection, incomingPlayer) =>
{
//Reply with the requested threadIds
Console.WriteLine("Received thread ID request from {0}.", connection.ToString());
connection.SendObject("ThreadIds", TaskManager().ToArray());
});
//Start listening for incoming TCP Connections
TCPConnection.StartListening(true);
Console.WriteLine("Server ready. Press any key to shutdown server.");
Console.ReadKey(true);
NetworkComms.Shutdown();
}
private static IEnumerable<string> TaskManager()
{
List<string> lst = new List<string>();
foreach (System.Diagnostics.Process p in
Process.GetProcesses().OrderBy(o => o.ProcessName))
{
lst.Add(p.ProcessName + "_" + p.Id);
}
return lst.AsParallel();
}
}
}
You will obviously need to download the NetworkComms.Net DLL from the website so that you can add it in the 'using NetworkCommsDotNet' reference. Also see the server IP address in the client example is currently "127.0.0.1", this should work if you run both the server and client on the same machine.
Disclaimer: I'm one of the developers for this library.

If you want a reliable and robust solution, use WCF instead of implementing serialization yourself.
Also, writing to stream from parallel threads would not work correctly. Parts of lines from different threads would mix together to some intangible garbage:
Parallel.ForEach(
TaskManager(), item=>
sw.WriteLine(item)
);

TcpClient.Close() works only with Thread.Sleep()

I have simple server that gets string from client and prints it on screen.
I also have simple client, sending data and closing:
static void Main()
{
var client = new TcpClient("localhost", 26140);
var stream = client.GetStream();
Byte[] data = System.Text.Encoding.UTF8.GetBytes("CALC qwer");
stream.Write(data, 0, data.Length);
stream.Close();
client.Close();
//Thread.Sleep(100);
}
And with uncommented string 'Thread.Sleep(100)' it works ok.
But when commenting, sometimes ( 1 of 5-10 runs ) client doesn't send the string.
Watching wireshark and netstat I've noticed that client sends SYN,ACK package, establishes connection and exits without sending anything and without closing the socket.
Could anyone explain this behaivor? Why sleep helps? What am I doing wrong?
UPD:
With this sample code adding flush() before closing really works, thanks Fox32.
But after it I returned to my initial code:
var client = new TcpClient("localhost", 26140);
client.NoDelay = true;
var stream = client.GetStream();
var writer = new StreamWriter(stream);
writer.WriteLine("CALC qwer");
writer.Flush();
stream.Flush();
stream.Close();
client.Close();
And it isn't working, even with NoDelay. It's bad - using StreamWriter over network stream?
UPD:
Here is server code:
static void Main(string[] args)
{
(new Server(26140)).Run();
}
In Server class:
public void Run()
{
var listener = new TcpListener(IPAddress.Any, port);
listener.Start();
while (true)
{
try
{
var client = listener.AcceptTcpClient();
Console.WriteLine("Client accepted: " + client.Client.RemoteEndPoint);
var stream = client.GetStream();
stream.ReadTimeout = 2000;
byte[] buffer = new byte[1000];
stream.Read(buffer, 0, 1000);
var s = Encoding.UTF8.GetString(buffer);
Console.WriteLine(s);
}
catch (Exception ex)
{
Console.WriteLine("ERROR! " + ex.Message);
}
}
}
UPD:
Adding even Sleep(1) makes crashes happen in 1 of 30-50 clients running at the same time.
And adding Sleep(10) seems to be solving it totally, I can't catch any crash.
Don't understand, why socket needs this several milliseconds to close correctly.

The TcpClient is using the Nagle's algorithm and waits for more data before sending it over the wire. If you close the socket to fast, no data is trasmitted.
You have multiple ways to solve this problem:
The NetworkStream has a Flush method for flushing the stream content (I'm not sure if this method does anything from the comment on MSDN)
Disable Nagle's algorithm: Set NoDelay of the TcpCLient to true.
The last option is to set the LingerState of the TcpClient. The Close method documentation states, that the LingerState is used while calling Close

In almost all cases you are supposed to call Shutdown on a Socket or TcpClient before disposing it. Disposing rudely kills the connection.
Your code basically contains a race condition with the TCP stack.
Setting NoDelay is also a fix for this but hurts performance. Calling Flush IMHO still results an an disorderly shutdown. Don't do it because they are just hacks which paint over the problem by hiding the symptoms. Call Shutdown.
I want to stress that Shutdown being called on the Socket is the only valid solution that I know of. Even Flush just forces the data onto the network. It can still be lost due to a network hickup. It will not be retransmitted after Close has been called because Close is a rude kill on the socket.
Unfortunately TcpClient has a bug which forces you to go to the underlying Socket to shut it down:
tcpClient.Client.Shutdown();
tcpClient.Close();
According to Reflector, if you have ever accessed GetStream this problem arises and Close does not close the underlying socket. In my estimation this bug was produced because the developer did not really know about the importance of Shutdown. Few people know and many apps are buggy because of it. A related question.

In your server side code you are only calling Read() once, but you can't assume the data will be available when you call read. You have to continue reading in a loop until no more data is available. See the full example below.
I have tried to reproduce your issue with the minimal amount of code and was not able to. The server prints out the clients message everytime. No special settings such as NoDelay and no explicit Close() or Flush(), just Using statements which ensures all resources are properly disposed.
class Program
{
static int port = 123;
static string ip = "1.1.1.1";
static AutoResetEvent waitHandle = new AutoResetEvent(false);
static void Main(string[] args)
{
StartServer();
waitHandle.WaitOne();
for (int x=0; x<1000; x++)
{
StartClient(x);
}
Console.WriteLine("Done starting clients");
Console.ReadLine();
}
static void StartClient(int count)
{
Task.Factory.StartNew((paramCount) =>
{
int myCount = (int)paramCount;
using (TcpClient client = new TcpClient(ip, port))
{
using (NetworkStream networkStream = client.GetStream())
{
using (StreamWriter writer = new StreamWriter(networkStream))
{
writer.WriteLine("hello, tcp world #" + myCount);
}
}
}
}, count);
}
static void StartServer()
{
Task.Factory.StartNew(() =>
{
try
{
TcpListener listener = new TcpListener(port);
listener.Start();
Console.WriteLine("Listening...");
waitHandle.Set();
while (true)
{
TcpClient theClient = listener.AcceptTcpClient();
Task.Factory.StartNew((paramClient) => {
TcpClient client = (TcpClient)paramClient;
byte[] buffer = new byte[32768];
MemoryStream memory = new MemoryStream();
using (NetworkStream networkStream = client.GetStream())
{
do
{
int read = networkStream.Read(buffer, 0, buffer.Length);
memory.Write(buffer, 0, read);
}
while (networkStream.DataAvailable);
}
string text = Encoding.UTF8.GetString(memory.ToArray());
Console.WriteLine("from client: " + text);
}, theClient);
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}, TaskCreationOptions.LongRunning);
}
}

UPD:
I've tested this bug on several computers and nothing crashed. Seems like it is a local bug on my computer.
ENDOFUPD
So, what I've found about reproducing this bug.
#Despertar - your code works well. But it isn't reproduce conditions of this bug. On client you need to send data and quit after it. And in your code many clients are sending data and after all application is closing.
This is how I'm testing this on my computer:
I have server ( just accepting connection and print incoming data ), client ( just sends data once end exits ) and running utility ( runs client exe several times ).
So, I starts server, copies running utility to the clients folder and runs it.
Running ulility starts 150 clients connecting to server and 5-10 of them dies ( I see error in the server console ). And uncommenting Thread.Sleep() on client works well, no errors.
Can anyone try to reproduce this version of code?
Client code:
private static void Main(string[] args)
{
try
{
using (TcpClient client = new TcpClient(ip, port))
{
using (NetworkStream networkStream = client.GetStream())
{
using (StreamWriter writer = new StreamWriter(networkStream))
{
writer.WriteLine("# hello, tcp world #");
writer.Flush();
}
networkStream.Flush();
networkStream.Close();
}
client.Close();
//Thread.Sleep(10);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
Code, running client several times ( compile it in exe file and put near client's exe - this code will run many clients one by one ):
static void Main(string[] args)
{
string path = "YOU_CLIENT_PROJECT_NAME.exe";
for (int i = 0; i < 150; i++ )
{
Console.WriteLine(i);
Process.Start(path);
Thread.Sleep(50);
}
Console.WriteLine("Done");
Console.ReadLine();
}
( don't forget to change path to corrent exe filename )
Server code:
class Program
{
static int port = 26140;
static AutoResetEvent waitHandle = new AutoResetEvent(false);
static void Main(string[] args)
{
StartServer();
waitHandle.WaitOne();
Console.ReadLine();
}
static void StartServer()
{
Task.Factory.StartNew(() =>
{
try
{
TcpListener listener = new TcpListener(port);
listener.Start();
Console.WriteLine("Listening...");
waitHandle.Set();
while (true)
{
TcpClient theClient = listener.AcceptTcpClient();
Task.Factory.StartNew(paramClient =>
{
try
{
TcpClient client = (TcpClient) paramClient;
byte[] buffer = new byte[32768];
MemoryStream memory = new MemoryStream();
using (NetworkStream networkStream = client.GetStream())
{
networkStream.ReadTimeout = 2000;
do
{
int read = networkStream.Read(buffer, 0, buffer.Length);
memory.Write(buffer, 0, read);
} while (networkStream.DataAvailable);
string text = Encoding.UTF8.GetString(memory.ToArray());
}
}
catch (Exception e)
{
Console.WriteLine("ERROR: " + e.Message);
}
}, theClient);
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
}, TaskCreationOptions.LongRunning);
}
}

I've tried code, reproducing this bug on several computers. No one crashes. Seems like it's my local computer bug.
Thanks for everybody for trying to help me.
Anyway, it's so strange. If I'll found out why this bug exists on my computer, I'll write about it.