We Keep Coding

How to set a correct buffer size for TCP message responses

This is how I currently send data to an external TCP server
byte[] data = new byte[0] /* the data to send */;
TcpClient client = new TcpClient("127.0.0.1", 3000); // connect to the tcp server
NetworkStream stream = client.GetStream();
await stream.WriteAsync(data, 0, data.Length);
data = new byte[256]; // set the buffer size
int responseBytes = await stream.ReadAsync(data, 0, data.Length); // store the response to the buffer
string responseData = System.Text.Encoding.ASCII.GetString(data, 0, responseBytes);
stream.Close();
client.Close();
For the response I have to setup the buffer size here new byte[256]. But what if the response is greater than this size? I can't determine the correct size because I'm just connecting to his external server, send a message to it and expect a response. Is there a way I can make this dynamic?
As a sidenote: I'm sending various HL7 messages to clinic servers and they will send back HL7 ACK messages as a response. This gives some information about HL7 ACK messages
https://healthstandards.com/blog/2007/02/01/ack-message-original-mode-acknowledgement/
An example ACK could be
MSH|^~&|CATH|StJohn|AcmeHIS|StJohn|20061019172719||ACK^O01|MSGID12349876|P|2.3
MSA|AA|MSGID12349876

For the response I have to setup the buffer size here new byte[256]. But what if the response is greater than this size?
Then you call stream.ReadAsync() and append your buffer (or the decoded string) to a larger buffer until you know you have received the entire message, which you need to do anyway: the Write() from one end of the socket does not need to correspond to one Read() on the other end. Multiple writes can be read in a single read, or the other way around.
So something like this:
data = new byte[256]; // set the buffer size
var builder = new StringBuilder();
do
{
int responseBytes = await stream.ReadAsync(data, 0, data.Length); // store the response to the buffer
string responseData = System.Text.Encoding.ASCII.GetString(data, 0, responseBytes);
builder.Append(responseData);
} while (responseBytes > 0)
Do note that this happens to work with ASCII, as it doesn't have multibyte characters. Were it UTF-8 or a similar encoding, the 256th byte could be the start of a character which continues into the next read, i.e. byte 1 (and perhaps 2) of the next read.
This code also assumes you want to keep reading until the connection is closed (then responseBytes = 0). If this protocol has a length prefix or message terminator, you have to handle those.
Usually you don't want to implement this low-level stuff yourself, aren't there libraries available that handle the HL7 protocol?

How to handle incoming protobuf message

Using TCPClient's NetworkStream and protobuf-net I send and receive protobuf messages via TCP.
Saw a similar question: How to properly handle incoming protobuf message with a NetworkStream?
But in my case there can only be one message type so i dont think i need a resolver.
So i serialized my object and send it using tcp/ip, on my server i try to deserialize it and get io exception: Unable to read data from the transport connection.
Client:
...
using (var ms = new MemoryStream())
{
Serializer.Serialize(ms, person);
data = ms.ToArray();
}
NetworkStream stream = client.GetStream();
stream.Write(data, 0, data.Length);
Server:
...
Byte[] bytes = new Byte[256];
String data = null;
while(true)
{
Console.Write("Waiting for a connection... ");
TcpClient client = server.AcceptTcpClient();
Console.WriteLine("Connected!");
data = null;
NetworkStream stream = client.GetStream();
Person newPerson = Serializer.Deserialize<Person>(stream);<--- exeption
}

I think the short version here is: use SerializeWithLengthPrefix and DeserializeWithLengthPrefix. The default protobuf behaviour is "read to the end of the stream". You shouldn't need the MemoryStream when serializing, btw; you should be fine to Serialize directly to the NetworkStream. If you need the MemoryStream for other reasons, you can save yourself a copy of the data by using:
stream.Write(ms.GetBuffer(), 0, (int)ms.Length);

Read bytes from NetworkStream (Hangs)

I'm trying to learn the basics of networking and I've built an echo server from this tutorial. I checked the server with telnet and it works perfect.
Now when I'm using some of the many client samples on the Internet:
// Create a TcpClient.
// Note, for this client to work you need to have a TcpServer
// connected to the same address as specified by the server, port
// combination.
TcpClient client = new TcpClient(server, port);
// Translate the passed message into ASCII and store it as a Byte array.
Byte[] data = System.Text.Encoding.ASCII.GetBytes(message);
// Get a client stream for reading and writing.
NetworkStream stream = client.GetStream();
// Send the message to the connected TcpServer.
stream.Write(data, 0, data.Length);
Console.WriteLine("Sent: {0}", message);
// Receive the TcpServer.response.
// Buffer to store the response bytes.
data = new Byte[256];
// String to store the response ASCII representation.
String responseData = String.Empty;
// Read the first batch of the TcpServer response bytes.
Int32 bytes = stream.Read(data, 0, data.Length);
responseData = System.Text.Encoding.ASCII.GetString(data, 0, bytes);
Console.WriteLine("Received: {0}", responseData);
// Close everything.
stream.Close();
client.Close();
It doesn't work very well. If I will comment the stream.Read line, everything works perfect (expect I can't read). I was also trying to accomplish that in a similar way using asynchronous callback method for the read. and then it only works after I terminate the program (the server handles the request)
I suspect that the way I'm reading from the stream cause this block, but I'm too clueless to understand what I'm doing wrong.

The implementation will block until at least one byte of data can be
read, in the event that no data is available.
From MSDN
Your server propably isn't sending you any data.
Edit:
I tested your client and it works perfectly fine. Try it yourself and set the following parameters:
string server = "google.com";
int port = 80;
string message = "GET /\n";
It's definitely your server which has the problem.

Send server multiple messages? C#

I have a quick and dirty question. So as it stands, i have two clients and a server running. I can communicate messages from the clients to the server without any problem. my problem appears when i want to read two messages from the client - rather than just one message.
The error which i receive is: IOException was unhandled. Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Here is my code on the server side:
private static void HandleClientComm(object client)
{
/** creating a list which contains DatabaseFile objects **/
List theDatabase = new List();
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
byte[] message = new byte[4096];
int bytesRead;
do
{
bytesRead = 0;
try
{
// Blocks until a client sends a message
bytesRead = clientStream.Read(message, 0, 4096);
}
catch (Exception)
{
// A socket error has occured
break;
}
if (bytesRead == 0)
{
// The client has disconnected from the server
break;
}
// Message has successfully been received
ASCIIEncoding encoder = new ASCIIEncoding();
Console.WriteLine("To: " + tcpClient.Client.LocalEndPoint);
Console.WriteLine("From: " + tcpClient.Client.RemoteEndPoint);
Console.WriteLine(encoder.GetString(message, 0, bytesRead));
if (encoder.GetString(message, 0, bytesRead) == "OptionOneInsert")
{
byte[] message2 = new byte[4096];
int bytesRead2 = 0;
**bytesRead2 = clientStream.Read(message, 0, 4096);** //ERROR occurs here!
Console.WriteLine("Attempting to go inside insert)");
Menu.Insert(theDatabase, bytesRead2);
}
Here is my client code:
ASCIIEncoding encoder = new ASCIIEncoding();
byte[] buffer = encoder.GetBytes("OptionOneInsert");
Console.ReadLine();
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
NetworkStream clientStream2 = client.GetStream();
String text = System.IO.File.ReadAllText("FirstNames.txt");
clientStream2.Write(buffer, 0, buffer.Length);
clientStream2.Flush();
ASCIIEncoding encoder2 = new ASCIIEncoding();
byte[] buffer2 = encoder2.GetBytes(text);
Console.WriteLine("buffer is filled with content");
Console.ReadLine();
When the client sends the message "optionOne" it is received by the server just fine. It's only when i attempt to send the string called "text" that the issues appears!
Any help would be greatly appreciated - I'm not all that familiar with Sockets, hence i've been struggling with trying to understand this for sometime now

You've got a big problem here - there's nothing to specify the end of one message and the start of another. It's quite possible that the server will receive two messages in one go, or half a message and then the other half.
The simplest way of avoiding that is to prefix each message with the number of bytes in it, e.g. as a fixed four-byte format. So to send a message you would:
Encoding it from a string to bytes (ideally using UTF-8 instead of ASCII unless you're sure you'll never need any non-ASCII text)
Write out the length of the byte array as a four-byte value
Write out the content
On the server:
Read four bytes (looping if necessary - there's no guarantee you'd even read those four bytes together, although you almost certainly will)
Convert the four bytes into an integer
Allocate a byte array of that size
Loop round, reading from "the current position" to the end of the buffer until you've filled the buffer
Convert the buffer into a string
Another alternative is simply to use BinaryReader and BinaryWriter - the ReadString and WriteString use length-prefixing, admittedly in a slightly different form.
Another alternative you could use is to have a delimiter (e.g. carriage-return) but that means you'll need to add escaping in if you ever need to include the delimiter in the text to transmit.

Problem with Socket and Serialization C#

I implemented a Client and Server model that uses Socket with thread
When I want to pass only a string from Client to the Server, it works. But I want to pass an object and it throws this error:
"Attempting to deserialize an empty stream"
Here is the code:
Client:
ASCIIEncoding asen = new ASCIIEncoding();
MemoryStream memoryStream = new MemoryStream();
BinaryFormatter binaryFormatter = new BinaryFormatter();
Command command = new Command("meuLogin", "minhaSenha");
binaryFormatter.Serialize(memoryStream, command);
stream.Write(memoryStream.ToArray(), 0, memoryStream.ToArray().Length);
Server:
byte[] message = new byte[4096];
int bytesRead = 0;
bytesRead = clientStream.Read(message, 0, 4096);
MemoryStream memoryStream = new MemoryStream(bytesRead);
BinaryFormatter bf1 = new BinaryFormatter();
memoryStream.Position = 0;
Command command = (Command)bf1.Deserialize(memoryStream);
Another question: I copied the class Command from Client and pasted at Server to deserialize. Is this correct?
Thank you

I also recommend WCF. But if you continue using sockets, the key element that you're missing in your protocol is message framing.

You never use the message that you read from the stream. The memory stream you are reading from is thus empty.
On a side note, why do you use these intermediate MemoryStreams?

To answer your second question: for maximum maintainability, the class Command should be in a separate assembly that both Client and Server reference.
To answer your first question: you are attempting to deserialize from an empty stream on your server, just as the exception tells you. You need to copy the bytes you read from the clientStream into the memoryStream before you deserialize from the memoryStream. Alternatively, use the clientStream directly rather than using the memoryStream; this may require reconsidering your protocol.
Finally, I wholeheartedly agree with #Andrey: consider using WCF. It's way way way better than raw sockets.

If you change your server code to use a different MemoryStream constructor, the problem will go away.
MemoryStream memoryStream = new MemoryStream(message, 0, bytesRead);
However, I agree with Stephen and others. Either use WCF, or use Message framing.

Another question: I copied the class Command from Client and pasted at Server to deserialize. Is this correct?
No. The namespace for your Client and Server are likely different, so the Server will be trying to deserialize a stream that matches its namespace.
You should create your Command Class using its own namespace and reference that from both your Client and the Server.
After that...
Client:
static void StreamToServer(TcpClient client, Command obj) {
using (NetworkStream ns = client.GetStream()) {
using (MemoryStream ms = new MemoryStream()) {
BinaryFormatter formatter = new BinaryFormatter();
formatter.Serialize(ms, obj);
byte[] buf = ms.ToArray();
ns.Write(buf, 0, buf.Length);
}
}
}
Server:
static Command ReadStream(TcpListener listener) {
Command obj = null;
using (TcpClient client = listener.AcceptTcpClient()) { // waits for data
using (NetworkStream ns = client.GetStream()) {
byte[] buf = new byte[client.ReceiveBufferSize];
int len = ns.Read(buf, 0, buf.Length);
using (MemoryStream ms = new MemoryStream(buf, 0, len)) {
BinaryFormatter formatter = new BinaryFormatter();
obj = formatter.Deserialize(ms) as Command;
}
}
}
return obj;
}
WCF or Message framing may be easier, but I don't often have the opportunity at work to sit around and read a book.

Instead of passing bytesRead to the MemoryStream which is actually the length of the byte stream, you should pass 'message', as it is the actual stream of bytes. Like,
MemoryStream memoryStream = new MemoryStream(message);
As you are passing an integer variable, the compiler is throwing exception that the stream is empty.
As for WCF, it is remarkable framework, but for applications that require low latency and high performance, WCF is a horrible answer because of its overheads, it is built upon sockets. So if you use sockets, that will be the lowest level implementation and thus, the fastest. That depends on your application which paradigm you should choose...