We Keep Coding

send byte[] data over network stream socket. c#

i want to send string json over networkstream . code at client side
using (var ns = new NetworkStream(socket))
{
string json = JsonConvert.SerializeObject(listCfile, Formatting.Indented);
byte[] jsonbytes = Encoding.UTF8.GetBytes(json);
byte[] jsonLength = BitConverter.GetBytes(jsonbytes.Length);
ns.Write(jsonLength, 0, jsonLength.Length);
ns.Write(jsonbytes, 0, jsonbytes.Length);
}
jsonbytes was byte[988324]
At server side
using (var ns = new NetworkStream(socket))
{
byte[] byDataLength = new byte[4];
ns.Read(byDataLength, 0, 4);
int jsonLength = BitConverter.ToInt32(byDataLength, 0);
byte[] byData = new byte[jsonLength];
ns.Read(byData, 0, jsonLength);
File.WriteAllBytes("E:\\json.txt",byData);
}
byData was byte[988324]
But byData i received isn't same as jsonbytes i sent.
i need some helps.
Update! some times it works. ByData received is same as jsonbytes i sent
Some times it doesn't work :(

You can try using raw sockets to send and receive as well as using a memory stream and end-of-packet mechanism for unknown data lengths.
Below is a snippet of methods showing using raw sockets and buffering in a memory-stream till End-Of-Packet is detected.
protected const int SIZE_RECEIVE_BUFFER = 1024; /// Receive buffer size
protected const string EOF = "!#~*|"; /// End of packet string
MemoryStream _msPacket = new MemoryStream(); /// Memory stream holding buffered data packets
int _delmtPtr = 0; /// Ranking pointer to check for EOF
Socket _baseSocket;
public event EventHandler OnReceived;
// TODO: -
// Add methods to connect or accept connections.
// When data is send to receiver, send with the same EOF defined above.
//
//
public void RegisterToReceive()
{
byte[] ReceiveBuffer = new byte[SIZE_RECEIVE_BUFFER];
_baseSocket.BeginReceive
(
ReceiveBuffer,
0,
ReceiveBuffer.Length,
SocketFlags.None,
new AsyncCallback(onReceiveData),
ReceiveBuffer
);
}
private void onReceiveData(IAsyncResult async)
{
try
{
byte[] buffer = (byte[])async.AsyncState;
int bytesCtr = 0;
try
{
if (_baseSocket != null)
bytesCtr = _baseSocket.EndReceive(async);
}
catch { }
if (bytesCtr > 0)
processReceivedData(buffer, bytesCtr);
RegisterToReceive();
}
catch{ }
}
private void processReceivedData(byte[] buffer, int bufferLength)
{
byte[] eof = Encoding.UTF8.GetBytes(EOF);
int packetStart = 0;
for (int i = 0; i < bufferLength; i++)
{
if (buffer[i].Equals(eof[_delmtPtr]))
{
_delmtPtr++;
if (_delmtPtr == eof.Length)
{
var lenToWrite = i - packetStart - (_delmtPtr - 1);
byte[] packet = new byte[lenToWrite + (int)_msPacket.Position];
if (lenToWrite > 0)
_msPacket.Write(buffer, packetStart, lenToWrite);
packetStart = i + 1;
_msPacket.Position = 0;
_msPacket.Read(packet, 0, packet.Length);
try
{
if (OnReceived != null)
OnReceived(packet, EventArgs.Empty);
}
catch { }
_msPacket.Position = 0;
_delmtPtr = 0;
}
}
else
{ _delmtPtr = 0; }
}
if (packetStart < bufferLength)
_msPacket.Write(buffer, packetStart, bufferLength - packetStart);
if (_msPacket.Position == 0)
_msPacket.SetLength(0);
}
This can receive large length of data and is independent of length by the sender.
The methods above shows how to receive data only, so will have to include other methods for connect, accept, sending-data, etc on raw sockets.

Why is this way of reading from a NetworkStream so much faster?

At my company we've run into an issue that we cannot seem to explain. We're reading from a NetworkStream using a buffer of 2048 bytes. When both the client and server use that buffersize to write and read with, it's hideously slow (1 second extra on top of other processing times). If we use a buffersize that's very large, like 32k everything works really fast. The message we're sending is usually much larger than 32k so the entire message does not entirely fit in that buffer. The way we're reading is shown in the code below in the HandleDataLargeBuffer method.
To build a testcase for this issue I've written the code below, and I've posted my output below that.
My questions:
Any idea why a larger buffer size would be faster (less read calls but those shouldn't take too much time)?
In the code below, HandleDataVariableBuffer is much much faster than the other methods of reading, why?
Which is the best way of reading anyway?
Using a Proxy class to simulate slowdowns, pasted below the main file.
class Program
{
private StreamWriter Writer;
private int Count = 0;
static void Main(string[] args)
{
new Program();
}
public Program()
{
Writer = new StreamWriter(new FileStream("C:\\test.txt", FileMode.Create));
Start();
}
public void Start()
{
Proxy p = new Proxy();
new Thread(p.Start).Start();
Thread.Sleep(500);
new Thread(SetupServer).Start();
Thread.Sleep(1000); // Wait for TCP Server setup.
for (int i = 0; i != 3; ++i)
{
TcpClient client = new TcpClient("127.0.0.1", 50001);
Send(client.GetStream());
client.Close();
Thread.Sleep(1000);
}
WriteLine("Tests done.");
Console.ReadLine();
Writer.Close();
}
private void SetupServer()
{
WriteLine("[Server] Starting server.");
TcpListener listener = new TcpListener(IPAddress.Any, 50000);
listener.Start();
WriteLine("[Server] Started listening on port 50000.");
while (true) // We'll just forcibly end, obviously you'd use the callback methods for this normally.
{
TcpClient client = listener.AcceptTcpClient();
WriteLine(String.Format("[Server] Accepted client with IP: {0}", client.Client.RemoteEndPoint.ToString()));
new Thread(HandleClient).Start(client);
}
}
private void HandleClient(object argument)
{
TcpClient client = (TcpClient)argument;
NetworkStream stream = client.GetStream();
// Now there are multiple ways to handle this data, however first we read the int at the start.
byte[] length = new byte[4];
stream.Read(length, 0, 4);
int lengthInt = BitConverter.ToInt32(length, 0);
if (lengthInt <= 0) return; // Shouldn't happen.
// Test data read in multiple ways.
Stopwatch watch = new Stopwatch();
watch.Start();
string handler = "";
if (Count == 0)
{
handler = "LargeBuffer";
HandleDataLargeBuffer(lengthInt, stream);
++Count;
}
else if (Count == 1)
{
handler = "SmallBuffer";
HandleDataSmallBuffer(lengthInt, stream);
++Count;
}
else if (Count == 2)
{
handler = "VariableBuffer";
HandleDataVariableBuffer(lengthInt, stream);
Count = 0;
}
watch.Stop();
WriteLine(String.Format("\t[Server] [{3}] Read {0} bytes from client {1} in {2} ms", lengthInt.ToString(), client.Client.RemoteEndPoint.ToString(), watch.ElapsedMilliseconds.ToString(), handler));
}
private void HandleDataLargeBuffer(int length, NetworkStream stream)
{
int read = 0;
int totalRead = 0;
MemoryStream dataBuffer = new MemoryStream(); // I'm writing to a memory stream because in my real life situation I write (stream) to another NetworkStream
byte[] buffer = new byte[8192 * 4];
while ((read = stream.Read(buffer, 0, 8192 * 4)) != 0 && totalRead < length)
{
totalRead += read;
dataBuffer.Write(buffer, 0, read);
}
}
private void HandleDataSmallBuffer(int length, NetworkStream stream)
{
int read = 0;
int totalRead = 0;
MemoryStream dataBuffer = new MemoryStream(); // I'm writing to a memory stream because in my real life situation I write (stream) to another NetworkStream
byte[] buffer = new byte[512];
while ((read = stream.Read(buffer, 0, 512)) != 0 && totalRead < length)
{
totalRead += read;
dataBuffer.Write(buffer, 0, read);
}
}
private void HandleDataVariableBuffer(int length, NetworkStream stream)
{
int read = 0;
int totalRead = 0;
MemoryStream dataBuffer = new MemoryStream(); // I'm writing to a memory stream because in my real life situation I write (stream) to another NetworkStream
while (totalRead < length)
{
byte[] buffer = new byte[length - totalRead]; // You'd obviously do some additional checks on 'length' to make sure no foul play is involved.
read = stream.Read(buffer, 0, length - totalRead);
totalRead += read;
dataBuffer.Write(buffer, 0, read);
}
}
public void Send(NetworkStream stream)
{
// Generate some random data
Random r = new Random();
byte[] buf = new byte[1024 * 420]; // Buffer of 420k to simulate a decently large message
byte[] length = BitConverter.GetBytes(1024 * 420);
r.NextBytes(buf);
// Create the total message array: [Length][Message]
byte[] totalMessage = new byte[4 + 1024 * 420];
System.Buffer.BlockCopy(length, 0, totalMessage, 0, 4);
System.Buffer.BlockCopy(buf, 0, totalMessage, 4, buf.Length);
// Use a memory stream for ease of use
Stopwatch watch = new Stopwatch();
watch.Start();
using (MemoryStream memStream = new MemoryStream(totalMessage))
{
int read = -1;
byte[] buffer = new byte[8192];
while ((read = memStream.Read(buffer, 0, 8192)) != 0)
{
stream.Write(buffer, 0, read);
}
}
stream.Flush();
watch.Stop();
WriteLine("[Send] Took " + watch.ElapsedMilliseconds + " ms to send " + totalMessage.Length + " bytes of data.");
}
public void WriteLine(string str)
{
Writer.WriteLine(str);
Writer.Flush();
}
}
Proxy.cs
public class Proxy
{
public void Start()
{
TcpListener listener = new TcpListener(IPAddress.Any, 50001);
listener.Start();
while (true)
{
TcpClient client = listener.AcceptTcpClient();
new Thread(HandleClient).Start(client);
}
}
private void HandleClient(object argument)
{
TcpClient client = (TcpClient)argument;
Thread.Sleep(1000);
NetworkStream clientStream = client.GetStream();
int read = 0;
byte[] length = new byte[4];
clientStream.Read(length, 0, 4);
int lengthInt = BitConverter.ToInt32(length, 0);
if (lengthInt <= 0)
{
client.Close();
return;
}
TcpClient serverClient = new TcpClient("127.0.0.1", 50000);
NetworkStream serverStream = serverClient.GetStream();
int totalRead = 0;
serverStream.Write(length, 0, 4);
while (totalRead < lengthInt)
{
Thread.Sleep(50);
byte[] buffer = new byte[lengthInt - totalRead]; // You'd obviously do some additional checks on 'length' to make sure no foul play is involved.
read = clientStream.Read(buffer, 0, lengthInt - totalRead);
totalRead += read;
serverStream.Write(buffer, 0, read);
}
serverStream.Flush();
Thread.Sleep(1000);
client.Close();
serverClient.Close();
}
}
I'm sorry for the heap of code posted, when I first started I hoped it'd be smaller. Here's the output on my machine.
[Server] Starting server.
[Server] Started listening on port 50000.
[Server] Accepted client with IP: 127.0.0.1:3985
[Send] Took 1047 ms to send 430084 bytes of data.
[Server] [LargeBuffer] Read 430080 bytes from client 127.0.0.1:3985 in 1148 ms
[Server] Accepted client with IP: 127.0.0.1:3987
[Send] Took 1049 ms to send 430084 bytes of data.
[Server] [SmallBuffer] Read 430080 bytes from client 127.0.0.1:3987 in 1150 ms
[Server] Accepted client with IP: 127.0.0.1:3989
[Send] Took 1051 ms to send 430084 bytes of data.
[Server] [VariableBuffer] Read 430080 bytes from client 127.0.0.1:3989 in 100 ms
Tests done.

C# sockets: can't read after writing to socket

In my client/server application my client wiil communicate with the server for 2 functions: the client will either request data from the server or it will send data so the server will save it. I'm using one socket for both methods, and the method to be used is defined by the first byte sent. If the first byte is "1" it is requesting data. If it is "2", it will send data (data bytes are sent after the "2" byte). It works perfectly for sending data. But when I'm requesting data it works, as long as I don't read the socket stream in the client. It's like if I make the client read data after sending data, the server will have no data to read, and it just crashes when trying to read the data.
Here is my server code:
private const int BufferSize = 1024;
NetworkStream netstream = null;
byte[] RecData = new byte[BufferSize];
int RecBytes;
try {
netstream = clientSocket.GetStream();
int totalrecbytes = 0;
using (MemoryStream ms = new MemoryStream()) {
//When I get here, there is no data to read
while ((RecBytes = netstream.Read(RecData, 0, RecData.Length)) > 0) {
ms.Write(RecData, 0, RecBytes);
totalrecbytes += RecBytes;
}
byte[] bytes = ms.ToArray();
byte b = bytes[0];
switch (b) {
case 1:
//Here I gather data and put it in "stream" variable
byte[] SendingBuffer = null;
int NoOfPackets = Convert.ToInt32(Math.Ceiling(Convert.ToDouble(stream.Length) / Convert.ToDouble(BufferSize)));
int TotalLength = (int)stream.Length, CurrentPacketLength, counter = 0;
for (int i = 0; i < NoOfPackets; i++) {
if (TotalLength > BufferSize) {
CurrentPacketLength = BufferSize;
TotalLength = TotalLength - CurrentPacketLength;
}
else
CurrentPacketLength = TotalLength;
SendingBuffer = new byte[CurrentPacketLength];
stream.Read(SendingBuffer, 0, CurrentPacketLength);
netstream.Write(SendingBuffer, 0, (int)SendingBuffer.Length);
}
netstream.Flush();
}
catch (Exception e) {
Console.WriteLine("EXCEPTION:\n" + e.ToString());
}
break;
case 2:
//Code to read data
break;
}
}
netstream.Close()
clientSocket.Close();
And here is my client code:
using (System.Net.Sockets.TcpClient clientSocket = new System.Net.Sockets.TcpClient()) {
string returnData = "";
IAsyncResult ar = clientSocket.BeginConnect("127.0.0.1", 8080, null, null);
System.Threading.WaitHandle wh = ar.AsyncWaitHandle;
try {
if (!ar.AsyncWaitHandle.WaitOne(TimeSpan.FromSeconds(5), false)) {
clientSocket.Close();
Console.WriteLine("Timeout");
return;
}
System.Net.Sockets.NetworkStream serverStream = clientSocket.GetStream();
byte b = 1;
byte[] outStream = { b };
serverStream.Write(outStream, 0, outStream.Length);
serverStream.Flush();
//If I comment following lines, the server can read sent data, but server can't otherwise
byte[] RecData = new byte[1024];
int RecBytes;
int totalrecbytes = 0;
MemoryStream MS = new MemoryStream();
while ((RecBytes = serverStream.Read(RecData, 0, RecData.Length)) > 0) {
MS.Write(RecData, 0, RecBytes);
totalrecbytes += RecBytes;
}
serverStream.Close();
clientSocket.Close();
clientSocket.EndConnect(ar);
}
catch (Exception ex) {
Console.WriteLine("Exceção: " + ex.ToString());
}
finally {
wh.Close();
}
}
So, how can I send data to server and read the response? (I tried even putting the thread to sleep after sending data, with no luck.)
Thanks in advance.
EDIT:
With some debug messages I discovered that the server do read the "1" byte that was sent, but somehow it gets stuck inside the while loop, like, the server just stops there, no more loops and it does not leave the while loop. I saw that after writing "loop" in console inside the while loop, and writing read bytes also in console. It wrote "loop" once, and the read byte.

This code worries me:
//When I get here, there is no data to read
while ((RecBytes = netstream.Read(RecData, 0, RecData.Length)) > 0) {
ms.Write(RecData, 0, RecBytes);
totalrecbytes += RecBytes;
}
You are reading until the client closes the connection (or shuts down sending, which you don't do). But the client only closes when the server has replied. The server reply will never come. It is a deadlock.
Solution: Read a single byte to determine the requests command (b).
Unrelated to the question, your "packetised" sending (NoOfPackets, ...) does not seem to serve any purpose. Just use Stream.Copy to write. TCP does not have packets.
An even better solution would be to abandon your custom TCP protocol and use an HTTP library. All these concerns just go away. There are various smaller problems with your code that are very typical to see in TCP code.

TcpListener truncating byte array randomly

I am writing what is essentially an image backup server to store images. It is a one way service that will not return anything beyond a basic success or failure message to the client.
The issue that I am experienceing is that when I send a byte array through the network stream, it is being cut-off before the end of the stream at random locations. I do not have this issue when I run the server on my development machine and connect locally, but rather it only occurs when the server is deployed on a remote server.
When I send very small arrays ( < 512 bytes) the server recieves the entire stream successfully, but on streams larger than 2000 bytes I experience issues. The code for the client is as follows:
try
{
TcpClient Voice = new System.Net.Sockets.TcpClient();
//Obviously I use the remote IP when it is deployed - but have altered it for privacy.
IPEndPoint BackupServer = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 57000);
Voice.Connect(BackupServer);
NetworkStream DataStream = Voice.GetStream();
byte[] buffer = new ASCIIEncoding().GetBytes(ImageData.GetXml());
DataStream.Write(buffer, 0, buffer.Length);
DataStream.Flush();
}
catch
{
}
try
{
buffer = new byte[4096];
int read = DataStream.Read(buffer, 0, buffer.Length);
MessageBox.Show(new ASCIIEncoding().GetString(buffer) + " : " + read.ToString());
}
catch
{
}
The client code executes without any errors or problems regardless of the size of data I send.
And the code for the server side is as follows:
private void BackMeUp(object voice)
{
TcpClient Voice = (TcpClient)voice;
Voice.ReceiveTimeout = 30000;
NetworkStream DataStream = Voice.GetStream();
try
{
bool ShouldLoop = true;
//int loops = 0;
int loops = -1;
byte[] input = new byte[2048];
byte[] buffer = new byte[0];
//while (ShouldLoop)
while(loops != 0)
{
loops = DataStream.Read(input, 0, 2048);
for (int x = 0; x < loops; x++)
{
Array.Resize(ref buffer, buffer.Length + 1);
buffer[buffer.Length - 1] = input[x];
}
//if (loops < 2048)
//{
//ShouldLoop = false;
//break;
//}
}
while (true)
{
StringReader Reader = new StringReader(new ASCIIEncoding().GetString(buffer, 0, buffer.Length));
DataSet DS = new DataSet();
DS.ReadXml(Reader);
if (DS.Tables.Count > 0)
{
if (DS.Tables["Images"].Rows.Count > 0)
{
foreach (DataRow row in DS.Tables["Images"].Rows)
{
//
}
}
}
string response = "Got it!";
DataStream.Write(new ASCIIEncoding().GetBytes(response), 0, response.Length);
DataStream.Flush();
Voice.Close();
break;
}
}
catch (Exception Ex)
{
File.WriteAllText("Elog.txt", Ex.Message + " " + (Ex.InnerException != null ? Ex.InnerException.ToString() : " no Inner"));
Voice.Close();
}
}
The server recieves the data fine, and closes the stream when it reaches the end, however the data is cut-off and I get an error when I try to rebuild the dataset.
I have the impression this has to do with the time it takes to send the stream, and I have played around with the Close and Flush commands but I feel like I'm just shooting in the dark. Any help would be appreciated.
Concise version of question: What factors are involved with a TcpListener that could cause a) the truncation of the stream. or b) premature closing of the stream prior to all bytes being read. When the listener in question is on a remote host rather than a local server.

The Read method doesn't have to return the number of bytes that you requested, or the entire stream at once. Especially if the stream is slow, it will be returned in small chunks.
Call the Read method repeatedly, and handle the data for each block that you get. The Read method returns zero when the stream is read to the end:
buffer = new byte[4096];
do {
int read = DataStream.Read(buffer, 0, buffer.Length);
if (read != 0) {
// handle the first "read" bytes of the buffer (index 0 to read-1)
}
} while (read != 0);
If you know that your buffer is enough for any stream, you can fill up the buffer and handle it afterwards:
buffer = new byte[4096];
int offset = 0;
do {
int read = DataStream.Read(buffer, offset, buffer.Length - offset);
offset += read;
} while (read != 0);
// handle the first "offset" bytes of the buffer (index 0 to offset-1)

TCP server client issue

First i am n00b in socket programming. So i decided to write simple data over lan tcp server
My server code that handles incomming data is
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
clientStream.ReadTimeout = 10;
int size = 4096 * 1000;
byte[] message = new byte[size];
byte[] All = new byte[0];
int bytesRead;
string error = "";
lock (this)
{
while (true)
{
All = new byte[0];
while (true)
{
bytesRead = 0;
try
{
bytesRead = clientStream.Read(message, 0, size);
All = AddBArrays(All, message, bytesRead);
}
catch
{
break;
}
if (bytesRead == 0)
{
break;
}
}
if (All.Length > 0)
{
Message m = (Message)Tools.ByteArrayToObject(All);
OnRecived(new RecivedArgs("localhost", (Message)Tools.ByteArrayToObject(All)));
}
}
tcpClient.Close();
}
}
byte[] AddBArrays(byte[] ar1, byte[] ar2, int read)
{
byte[] concat = new byte[ar1.Length + read];
if (ar1.Length != 0)
System.Buffer.BlockCopy(ar1, 0, concat, 0, ar1.Length);
System.Buffer.BlockCopy(ar2, 0, concat, ar1.Length, read);
return concat;
}
it works but have some issues. It fales receiving files bigger then 100 mbs or smthng and also if i send data very often interval < 800 then data is lost. how should i improve my code? The large file issue is not so important the primary issue is the data loss in fast data sending.
tnx for help
Ok i now updated the code by the suggestions
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
clientStream.ReadTimeout = 10;
int size = 4096 * 1000;
List<byte> Test = new List<byte>();
byte[] message = new byte[size];
byte[] All = new byte[0];
int bytesRead;
while (true)
{
//All = new byte[0];
while (true)
{
bytesRead = 0;
try
{
bytesRead = clientStream.Read(message, 0, size);
for (int i = 0; i < bytesRead; i++)
{
Test.Add(message[i]);
}
}
catch
{
break;
}
if (bytesRead == 0)
{
break;
}
}
if (Test.Count > 0)
{
Message m = (Message)Tools.ByteArrayToObject(Test.ToArray());
OnRecived(new RecivedArgs("localhost", m));
Test = new List<byte>();
}
}
tcpClient.Close();
}
but the issues still there
Edit--> large file issue fixed it was just a 'System.OutOfMemoryException' but it didn't throw a error.

The All byte array you should change to a List<byte>. You are creating instances like crazy right now. The GC is probably working a lot more than it needs to. This might be slowing it down so much that it can't keep up.
Not really related to sockets:
Make size a const
NEVER lock this. Create a private field that you can lock. In fact, I don't even think you need a lock here.
remove the error string.

OK i solved the problem.
I simple send to much data to fast so data loss was unavoidable.
My optimized code is
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
int size = 1;
byte[] message = new byte[1];
int bytesRead;
while (true)
{
bytesRead = 0;
if (clientStream.DataAvailable)
bytesRead = clientStream.Read(message, 0, 1);
if (bytesRead > 0)
{
OnRecived(new RecivedArgs("tick", null));
}
Thread.Sleep(1);
}
}
i have tested intervals as low as 1 ms and no data loss :)
thanks for your help