We Keep Coding

c# file read and send over socket

This is how I send a file using a NetworkStream.
private void go()
{
byte[] send = File.ReadAllBytes("example.txt");
ns.Write(send, 0, send.Length);
}
ns is a NetworkStream of course.
Now I would like to know how I could receive and read an incoming NetworkStream?
I know that I need to specify a buffer to read from like this,
ns.Read(buffer,0,buffer.length).
but which buffer should be there?

TCP is a stream based protocol, which means that there is no notation of application messages like in UDP. Thus you cannot really detect by TCP itself where an application message ends.
Therefore you need to introduce some kind of detection. Typically you add a suffix (new line, semicolon or whatever) or a length header.
In this case it's easier to add a length header since the chosen suffix could be found in the file data.
So sending the file would look like this:
private void SendFile(string fileName, NetworkStream ns)
{
var bytesToSend = File.ReadAllBytes(fileName);
var header = BitConverter.GetBytes(bytesToSend.Length);
ns.Write(header, 0, header.Length);
ns.Write(bytesToSend, 0, bytesToSend.Length);
}
On the receiver side it's important that you check the return value from Read as contents can come in chunks:
public byte[] ReadFile(NetworkStream ns)
{
var header = new byte[4];
var bytesLeft = 4;
var offset = 0;
// have to repeat as messages can come in chunks
while (bytesLeft > 0)
{
var bytesRead = ns.Read(header, offset, bytesLeft);
offset += bytesRead;
bytesLeft -= bytesRead;
}
bytesLeft = BitConverter.ToInt32(header, 0);
offset = 0;
var fileContents = new byte[bytesLeft];
// have to repeat as messages can come in chunks
while (bytesLeft > 0)
{
var bytesRead = ns.Read(fileContents, offset, bytesLeft);
offset += bytesRead;
bytesLeft -= bytesRead;
}
return fileContents;
}

How to write data to buffer before writing on to disk in C#

In C++ it is possible to write buffered i/o using setvbuf. how to achieve the same in C#. Is there any method available to write buffered i/o in C#

As already commented there is a BufferedStream class
Adds a buffering layer to read and write operations on another stream. This class cannot be inherited.
Example code from MSDN:
Server side:
// This is a Windows Sockets 2 error code.
const int WSAETIMEDOUT = 10060;
Socket serverSocket;
int bytesReceived, totalReceived = 0;
byte[] receivedData = new byte[2000000];
// Create random data to send to the client.
byte[] dataToSend = new byte[2000000];
new Random().NextBytes(dataToSend);
IPAddress ipAddress =
Dns.Resolve(Dns.GetHostName()).AddressList[0];
IPEndPoint ipEndpoint = new IPEndPoint(ipAddress, 1800);
// Create a socket and listen for incoming connections.
using(Socket listenSocket = new Socket(
AddressFamily.InterNetwork, SocketType.Stream,
ProtocolType.Tcp))
{
listenSocket.Bind(ipEndpoint);
listenSocket.Listen(1);
// Accept a connection and create a socket to handle it.
serverSocket = listenSocket.Accept();
Console.WriteLine("Server is connected.\n");
}
try
{
// Send data to the client.
Console.Write("Sending data ... ");
int bytesSent = serverSocket.Send(
dataToSend, 0, dataToSend.Length, SocketFlags.None);
Console.WriteLine("{0} bytes sent.\n",
bytesSent.ToString());
// Set the timeout for receiving data to 2 seconds.
serverSocket.SetSocketOption(SocketOptionLevel.Socket,
SocketOptionName.ReceiveTimeout, 2000);
// Receive data from the client.
Console.Write("Receiving data ... ");
try
{
do
{
bytesReceived = serverSocket.Receive(receivedData,
0, receivedData.Length, SocketFlags.None);
totalReceived += bytesReceived;
}
while(bytesReceived != 0);
}
catch(SocketException e)
{
if(e.ErrorCode == WSAETIMEDOUT)
{
// Data was not received within the given time.
// Assume that the transmission has ended.
}
else
{
Console.WriteLine("{0}: {1}\n",
e.GetType().Name, e.Message);
}
}
finally
{
Console.WriteLine("{0} bytes received.\n",
totalReceived.ToString());
}
}
finally
{
serverSocket.Shutdown(SocketShutdown.Both);
Console.WriteLine("Connection shut down.");
serverSocket.Close();
}
}
Client side:
public class Client
{
const int dataArraySize = 100;
const int streamBufferSize = 1000;
const int numberOfLoops = 10000;
static void Main(string[] args)
{
// Check that an argument was specified when the
// program was invoked.
if(args.Length == 0)
{
Console.WriteLine("Error: The name of the host computer" +
" must be specified when the program is invoked.");
return;
}
string remoteName = args[0];
// Create the underlying socket and connect to the server.
Socket clientSocket = new Socket(AddressFamily.InterNetwork,
SocketType.Stream, ProtocolType.Tcp);
clientSocket.Connect(new IPEndPoint(
Dns.Resolve(remoteName).AddressList[0], 1800));
Console.WriteLine("Client is connected.\n");
// Create a NetworkStream that owns clientSocket and
// then create a BufferedStream on top of the NetworkStream.
// Both streams are disposed when execution exits the
// using statement.
using(Stream
netStream = new NetworkStream(clientSocket, true),
bufStream =
new BufferedStream(netStream, streamBufferSize))
{
// Check whether the underlying stream supports seeking.
Console.WriteLine("NetworkStream {0} seeking.\n",
bufStream.CanSeek ? "supports" : "does not support");
// Send and receive data.
if(bufStream.CanWrite)
{
SendData(netStream, bufStream);
}
if(bufStream.CanRead)
{
ReceiveData(netStream, bufStream);
}
// When bufStream is closed, netStream is in turn
// closed, which in turn shuts down the connection
// and closes clientSocket.
Console.WriteLine("\nShutting down the connection.");
bufStream.Close();
}
}
static void SendData(Stream netStream, Stream bufStream)
{
DateTime startTime;
double networkTime, bufferedTime;
// Create random data to send to the server.
byte[] dataToSend = new byte[dataArraySize];
new Random().NextBytes(dataToSend);
// Send the data using the NetworkStream.
Console.WriteLine("Sending data using NetworkStream.");
startTime = DateTime.Now;
for(int i = 0; i < numberOfLoops; i++)
{
netStream.Write(dataToSend, 0, dataToSend.Length);
}
networkTime = (DateTime.Now - startTime).TotalSeconds;
Console.WriteLine("{0} bytes sent in {1} seconds.\n",
numberOfLoops * dataToSend.Length,
networkTime.ToString("F1"));
// Send the data using the BufferedStream.
Console.WriteLine("Sending data using BufferedStream.");
startTime = DateTime.Now;
for(int i = 0; i < numberOfLoops; i++)
{
bufStream.Write(dataToSend, 0, dataToSend.Length);
}
bufStream.Flush();
bufferedTime = (DateTime.Now - startTime).TotalSeconds;
Console.WriteLine("{0} bytes sent in {1} seconds.\n",
numberOfLoops * dataToSend.Length,
bufferedTime.ToString("F1"));
// Print the ratio of write times.
Console.WriteLine("Sending data using the buffered " +
"network stream was {0} {1} than using the network " +
"stream alone.\n",
(networkTime/bufferedTime).ToString("P0"),
bufferedTime < networkTime ? "faster" : "slower");
}
static void ReceiveData(Stream netStream, Stream bufStream)
{
DateTime startTime;
double networkTime, bufferedTime = 0;
int bytesReceived = 0;
byte[] receivedData = new byte[dataArraySize];
// Receive data using the NetworkStream.
Console.WriteLine("Receiving data using NetworkStream.");
startTime = DateTime.Now;
while(bytesReceived < numberOfLoops * receivedData.Length)
{
bytesReceived += netStream.Read(
receivedData, 0, receivedData.Length);
}
networkTime = (DateTime.Now - startTime).TotalSeconds;
Console.WriteLine("{0} bytes received in {1} seconds.\n",
bytesReceived.ToString(),
networkTime.ToString("F1"));
// Receive data using the BufferedStream.
Console.WriteLine("Receiving data using BufferedStream.");
bytesReceived = 0;
startTime = DateTime.Now;
int numBytesToRead = receivedData.Length;
while (numBytesToRead > 0)
{
// Read may return anything from 0 to numBytesToRead.
int n = bufStream.Read(receivedData,0, receivedData.Length);
// The end of the file is reached.
if (n == 0)
break;
bytesReceived += n;
numBytesToRead -= n;
}
bufferedTime = (DateTime.Now - startTime).TotalSeconds;
Console.WriteLine("{0} bytes received in {1} seconds.\n",
bytesReceived.ToString(),
bufferedTime.ToString("F1"));
// Print the ratio of read times.
Console.WriteLine("Receiving data using the buffered network" +
" stream was {0} {1} than using the network stream alone.",
(networkTime/bufferedTime).ToString("P0"),
bufferedTime < networkTime ? "faster" : "slower");
}
}

Both FileStream and StreamWriter are internally buffered (they have a default buffer size of 4096 and 1024 bytes and constructors able to modify it).
The complex question would be how to write to a file without using a buffer :-) And note that, thanks to how they work, a StreamWriter that writes to a file will be double-buffered (the buffer of StreamWriter is independent of the one of FileStream). At least StreamWriter has an AutoFlush property that, when set to true, will flush after each write.

Why is this way of reading from a NetworkStream so much faster?

At my company we've run into an issue that we cannot seem to explain. We're reading from a NetworkStream using a buffer of 2048 bytes. When both the client and server use that buffersize to write and read with, it's hideously slow (1 second extra on top of other processing times). If we use a buffersize that's very large, like 32k everything works really fast. The message we're sending is usually much larger than 32k so the entire message does not entirely fit in that buffer. The way we're reading is shown in the code below in the HandleDataLargeBuffer method.
To build a testcase for this issue I've written the code below, and I've posted my output below that.
My questions:
Any idea why a larger buffer size would be faster (less read calls but those shouldn't take too much time)?
In the code below, HandleDataVariableBuffer is much much faster than the other methods of reading, why?
Which is the best way of reading anyway?
Using a Proxy class to simulate slowdowns, pasted below the main file.
class Program
{
private StreamWriter Writer;
private int Count = 0;
static void Main(string[] args)
{
new Program();
}
public Program()
{
Writer = new StreamWriter(new FileStream("C:\\test.txt", FileMode.Create));
Start();
}
public void Start()
{
Proxy p = new Proxy();
new Thread(p.Start).Start();
Thread.Sleep(500);
new Thread(SetupServer).Start();
Thread.Sleep(1000); // Wait for TCP Server setup.
for (int i = 0; i != 3; ++i)
{
TcpClient client = new TcpClient("127.0.0.1", 50001);
Send(client.GetStream());
client.Close();
Thread.Sleep(1000);
}
WriteLine("Tests done.");
Console.ReadLine();
Writer.Close();
}
private void SetupServer()
{
WriteLine("[Server] Starting server.");
TcpListener listener = new TcpListener(IPAddress.Any, 50000);
listener.Start();
WriteLine("[Server] Started listening on port 50000.");
while (true) // We'll just forcibly end, obviously you'd use the callback methods for this normally.
{
TcpClient client = listener.AcceptTcpClient();
WriteLine(String.Format("[Server] Accepted client with IP: {0}", client.Client.RemoteEndPoint.ToString()));
new Thread(HandleClient).Start(client);
}
}
private void HandleClient(object argument)
{
TcpClient client = (TcpClient)argument;
NetworkStream stream = client.GetStream();
// Now there are multiple ways to handle this data, however first we read the int at the start.
byte[] length = new byte[4];
stream.Read(length, 0, 4);
int lengthInt = BitConverter.ToInt32(length, 0);
if (lengthInt <= 0) return; // Shouldn't happen.
// Test data read in multiple ways.
Stopwatch watch = new Stopwatch();
watch.Start();
string handler = "";
if (Count == 0)
{
handler = "LargeBuffer";
HandleDataLargeBuffer(lengthInt, stream);
++Count;
}
else if (Count == 1)
{
handler = "SmallBuffer";
HandleDataSmallBuffer(lengthInt, stream);
++Count;
}
else if (Count == 2)
{
handler = "VariableBuffer";
HandleDataVariableBuffer(lengthInt, stream);
Count = 0;
}
watch.Stop();
WriteLine(String.Format("\t[Server] [{3}] Read {0} bytes from client {1} in {2} ms", lengthInt.ToString(), client.Client.RemoteEndPoint.ToString(), watch.ElapsedMilliseconds.ToString(), handler));
}
private void HandleDataLargeBuffer(int length, NetworkStream stream)
{
int read = 0;
int totalRead = 0;
MemoryStream dataBuffer = new MemoryStream(); // I'm writing to a memory stream because in my real life situation I write (stream) to another NetworkStream
byte[] buffer = new byte[8192 * 4];
while ((read = stream.Read(buffer, 0, 8192 * 4)) != 0 && totalRead < length)
{
totalRead += read;
dataBuffer.Write(buffer, 0, read);
}
}
private void HandleDataSmallBuffer(int length, NetworkStream stream)
{
int read = 0;
int totalRead = 0;
MemoryStream dataBuffer = new MemoryStream(); // I'm writing to a memory stream because in my real life situation I write (stream) to another NetworkStream
byte[] buffer = new byte[512];
while ((read = stream.Read(buffer, 0, 512)) != 0 && totalRead < length)
{
totalRead += read;
dataBuffer.Write(buffer, 0, read);
}
}
private void HandleDataVariableBuffer(int length, NetworkStream stream)
{
int read = 0;
int totalRead = 0;
MemoryStream dataBuffer = new MemoryStream(); // I'm writing to a memory stream because in my real life situation I write (stream) to another NetworkStream
while (totalRead < length)
{
byte[] buffer = new byte[length - totalRead]; // You'd obviously do some additional checks on 'length' to make sure no foul play is involved.
read = stream.Read(buffer, 0, length - totalRead);
totalRead += read;
dataBuffer.Write(buffer, 0, read);
}
}
public void Send(NetworkStream stream)
{
// Generate some random data
Random r = new Random();
byte[] buf = new byte[1024 * 420]; // Buffer of 420k to simulate a decently large message
byte[] length = BitConverter.GetBytes(1024 * 420);
r.NextBytes(buf);
// Create the total message array: [Length][Message]
byte[] totalMessage = new byte[4 + 1024 * 420];
System.Buffer.BlockCopy(length, 0, totalMessage, 0, 4);
System.Buffer.BlockCopy(buf, 0, totalMessage, 4, buf.Length);
// Use a memory stream for ease of use
Stopwatch watch = new Stopwatch();
watch.Start();
using (MemoryStream memStream = new MemoryStream(totalMessage))
{
int read = -1;
byte[] buffer = new byte[8192];
while ((read = memStream.Read(buffer, 0, 8192)) != 0)
{
stream.Write(buffer, 0, read);
}
}
stream.Flush();
watch.Stop();
WriteLine("[Send] Took " + watch.ElapsedMilliseconds + " ms to send " + totalMessage.Length + " bytes of data.");
}
public void WriteLine(string str)
{
Writer.WriteLine(str);
Writer.Flush();
}
}
Proxy.cs
public class Proxy
{
public void Start()
{
TcpListener listener = new TcpListener(IPAddress.Any, 50001);
listener.Start();
while (true)
{
TcpClient client = listener.AcceptTcpClient();
new Thread(HandleClient).Start(client);
}
}
private void HandleClient(object argument)
{
TcpClient client = (TcpClient)argument;
Thread.Sleep(1000);
NetworkStream clientStream = client.GetStream();
int read = 0;
byte[] length = new byte[4];
clientStream.Read(length, 0, 4);
int lengthInt = BitConverter.ToInt32(length, 0);
if (lengthInt <= 0)
{
client.Close();
return;
}
TcpClient serverClient = new TcpClient("127.0.0.1", 50000);
NetworkStream serverStream = serverClient.GetStream();
int totalRead = 0;
serverStream.Write(length, 0, 4);
while (totalRead < lengthInt)
{
Thread.Sleep(50);
byte[] buffer = new byte[lengthInt - totalRead]; // You'd obviously do some additional checks on 'length' to make sure no foul play is involved.
read = clientStream.Read(buffer, 0, lengthInt - totalRead);
totalRead += read;
serverStream.Write(buffer, 0, read);
}
serverStream.Flush();
Thread.Sleep(1000);
client.Close();
serverClient.Close();
}
}
I'm sorry for the heap of code posted, when I first started I hoped it'd be smaller. Here's the output on my machine.
[Server] Starting server.
[Server] Started listening on port 50000.
[Server] Accepted client with IP: 127.0.0.1:3985
[Send] Took 1047 ms to send 430084 bytes of data.
[Server] [LargeBuffer] Read 430080 bytes from client 127.0.0.1:3985 in 1148 ms
[Server] Accepted client with IP: 127.0.0.1:3987
[Send] Took 1049 ms to send 430084 bytes of data.
[Server] [SmallBuffer] Read 430080 bytes from client 127.0.0.1:3987 in 1150 ms
[Server] Accepted client with IP: 127.0.0.1:3989
[Send] Took 1051 ms to send 430084 bytes of data.
[Server] [VariableBuffer] Read 430080 bytes from client 127.0.0.1:3989 in 100 ms
Tests done.

TcpListener truncating byte array randomly

I am writing what is essentially an image backup server to store images. It is a one way service that will not return anything beyond a basic success or failure message to the client.
The issue that I am experienceing is that when I send a byte array through the network stream, it is being cut-off before the end of the stream at random locations. I do not have this issue when I run the server on my development machine and connect locally, but rather it only occurs when the server is deployed on a remote server.
When I send very small arrays ( < 512 bytes) the server recieves the entire stream successfully, but on streams larger than 2000 bytes I experience issues. The code for the client is as follows:
try
{
TcpClient Voice = new System.Net.Sockets.TcpClient();
//Obviously I use the remote IP when it is deployed - but have altered it for privacy.
IPEndPoint BackupServer = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 57000);
Voice.Connect(BackupServer);
NetworkStream DataStream = Voice.GetStream();
byte[] buffer = new ASCIIEncoding().GetBytes(ImageData.GetXml());
DataStream.Write(buffer, 0, buffer.Length);
DataStream.Flush();
}
catch
{
}
try
{
buffer = new byte[4096];
int read = DataStream.Read(buffer, 0, buffer.Length);
MessageBox.Show(new ASCIIEncoding().GetString(buffer) + " : " + read.ToString());
}
catch
{
}
The client code executes without any errors or problems regardless of the size of data I send.
And the code for the server side is as follows:
private void BackMeUp(object voice)
{
TcpClient Voice = (TcpClient)voice;
Voice.ReceiveTimeout = 30000;
NetworkStream DataStream = Voice.GetStream();
try
{
bool ShouldLoop = true;
//int loops = 0;
int loops = -1;
byte[] input = new byte[2048];
byte[] buffer = new byte[0];
//while (ShouldLoop)
while(loops != 0)
{
loops = DataStream.Read(input, 0, 2048);
for (int x = 0; x < loops; x++)
{
Array.Resize(ref buffer, buffer.Length + 1);
buffer[buffer.Length - 1] = input[x];
}
//if (loops < 2048)
//{
//ShouldLoop = false;
//break;
//}
}
while (true)
{
StringReader Reader = new StringReader(new ASCIIEncoding().GetString(buffer, 0, buffer.Length));
DataSet DS = new DataSet();
DS.ReadXml(Reader);
if (DS.Tables.Count > 0)
{
if (DS.Tables["Images"].Rows.Count > 0)
{
foreach (DataRow row in DS.Tables["Images"].Rows)
{
//
}
}
}
string response = "Got it!";
DataStream.Write(new ASCIIEncoding().GetBytes(response), 0, response.Length);
DataStream.Flush();
Voice.Close();
break;
}
}
catch (Exception Ex)
{
File.WriteAllText("Elog.txt", Ex.Message + " " + (Ex.InnerException != null ? Ex.InnerException.ToString() : " no Inner"));
Voice.Close();
}
}
The server recieves the data fine, and closes the stream when it reaches the end, however the data is cut-off and I get an error when I try to rebuild the dataset.
I have the impression this has to do with the time it takes to send the stream, and I have played around with the Close and Flush commands but I feel like I'm just shooting in the dark. Any help would be appreciated.
Concise version of question: What factors are involved with a TcpListener that could cause a) the truncation of the stream. or b) premature closing of the stream prior to all bytes being read. When the listener in question is on a remote host rather than a local server.

The Read method doesn't have to return the number of bytes that you requested, or the entire stream at once. Especially if the stream is slow, it will be returned in small chunks.
Call the Read method repeatedly, and handle the data for each block that you get. The Read method returns zero when the stream is read to the end:
buffer = new byte[4096];
do {
int read = DataStream.Read(buffer, 0, buffer.Length);
if (read != 0) {
// handle the first "read" bytes of the buffer (index 0 to read-1)
}
} while (read != 0);
If you know that your buffer is enough for any stream, you can fill up the buffer and handle it afterwards:
buffer = new byte[4096];
int offset = 0;
do {
int read = DataStream.Read(buffer, offset, buffer.Length - offset);
offset += read;
} while (read != 0);
// handle the first "offset" bytes of the buffer (index 0 to offset-1)

TCP server client issue

First i am n00b in socket programming. So i decided to write simple data over lan tcp server
My server code that handles incomming data is
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
clientStream.ReadTimeout = 10;
int size = 4096 * 1000;
byte[] message = new byte[size];
byte[] All = new byte[0];
int bytesRead;
string error = "";
lock (this)
{
while (true)
{
All = new byte[0];
while (true)
{
bytesRead = 0;
try
{
bytesRead = clientStream.Read(message, 0, size);
All = AddBArrays(All, message, bytesRead);
}
catch
{
break;
}
if (bytesRead == 0)
{
break;
}
}
if (All.Length > 0)
{
Message m = (Message)Tools.ByteArrayToObject(All);
OnRecived(new RecivedArgs("localhost", (Message)Tools.ByteArrayToObject(All)));
}
}
tcpClient.Close();
}
}
byte[] AddBArrays(byte[] ar1, byte[] ar2, int read)
{
byte[] concat = new byte[ar1.Length + read];
if (ar1.Length != 0)
System.Buffer.BlockCopy(ar1, 0, concat, 0, ar1.Length);
System.Buffer.BlockCopy(ar2, 0, concat, ar1.Length, read);
return concat;
}
it works but have some issues. It fales receiving files bigger then 100 mbs or smthng and also if i send data very often interval < 800 then data is lost. how should i improve my code? The large file issue is not so important the primary issue is the data loss in fast data sending.
tnx for help
Ok i now updated the code by the suggestions
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
clientStream.ReadTimeout = 10;
int size = 4096 * 1000;
List<byte> Test = new List<byte>();
byte[] message = new byte[size];
byte[] All = new byte[0];
int bytesRead;
while (true)
{
//All = new byte[0];
while (true)
{
bytesRead = 0;
try
{
bytesRead = clientStream.Read(message, 0, size);
for (int i = 0; i < bytesRead; i++)
{
Test.Add(message[i]);
}
}
catch
{
break;
}
if (bytesRead == 0)
{
break;
}
}
if (Test.Count > 0)
{
Message m = (Message)Tools.ByteArrayToObject(Test.ToArray());
OnRecived(new RecivedArgs("localhost", m));
Test = new List<byte>();
}
}
tcpClient.Close();
}
but the issues still there
Edit--> large file issue fixed it was just a 'System.OutOfMemoryException' but it didn't throw a error.

The All byte array you should change to a List<byte>. You are creating instances like crazy right now. The GC is probably working a lot more than it needs to. This might be slowing it down so much that it can't keep up.
Not really related to sockets:
Make size a const
NEVER lock this. Create a private field that you can lock. In fact, I don't even think you need a lock here.
remove the error string.

OK i solved the problem.
I simple send to much data to fast so data loss was unavoidable.
My optimized code is
private void HandleClientComm(object client)
{
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
int size = 1;
byte[] message = new byte[1];
int bytesRead;
while (true)
{
bytesRead = 0;
if (clientStream.DataAvailable)
bytesRead = clientStream.Read(message, 0, 1);
if (bytesRead > 0)
{
OnRecived(new RecivedArgs("tick", null));
}
Thread.Sleep(1);
}
}
i have tested intervals as low as 1 ms and no data loss :)
thanks for your help