We Keep Coding

C# Read from a Networkstream

Im trying to send an object( in my case an image) over a networkstream.. however- im not getting the full image..
This is the client code:
private void Form1_Load(object sender, EventArgs e)
{
TcpClient c = new TcpClient();
c.Connect("10.0.0.4", 10);
NetworkStream ns = c.GetStream();
Bitmap f = GetDesktopImage();
byte[] buffer = imageToByteArray(f);
byte[] len = BitConverter.GetBytes(buffer.Length);
MemoryStream stream = new MemoryStream();
stream.Write(len, 0, len.Length);
stream.Write(buffer, 0, buffer.Length);
stream.Position = 0;
stream.CopyTo(ns);
}
As you can see i write the entire content to a regular MemoryStream first(because i dont want to use twice the NetworkStream- only then, i copy the MemoryStream content into the NetworkStream.
The server code:
private void Form1_Load(object sender, EventArgs e)
{
TcpListener tl = new TcpListener(IPAddress.Any, 10);
tl.Start();
TcpClient c = tl.AcceptTcpClient();
network = new NetworkStream(c.Client);
byte[] buff = new byte[4];
network.Read(buff, 0, 4);
int len = BitConverter.ToInt32(buff, 0);
buff = new byte[len];
network.Read(buff, 0, buff.Length);
pictureBox1.Image = byteArrayToImage(buff);
Thread th = new Thread(method);
}
Now when i run both application im getting only the top part of the captured image... It's even more odd because writing directly both to a network stream works perfect... For example:
ns.Write(len, 0, len.Length);
ns.Write(buffer, 0, buffer.Length);
This would work fine and i'll get the full image in the other side.. but i dont want to use it twice(it's just an example, in my real project i would have to use it alot so i would like to reduce the network usage as much as posibble ,and not trigger it for every single data).
Why it's not working using simply the CopyTo method?
I would appreciate any help!
Thanks

In your server code you are ignoring the result of NetworkStream.Read which is the actual number of bytes read. This can be less than the number of bytes you actually requested. You need to keep calling Read until you have received all the bytes you need.
The differences you seeing between 1 or multiple writes has to do with the buffering/timing in the network stack. I suspect you just getting lucky receiving the image when doing a single write - this will not always be the case! You need to handle the result of Read as explained above.
Code Example:
void ReadBuffer(byte[] buffer, int offset, int count)
{
int num;
int num2 = 0;
do
{
num2 += num = _stream.Read(buffer, offset + num2, count - num2);
}
while ((num > 0) && (num2 < count));
if (num2 != count)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", count - num2));
}

While reading large amount of data in bytes the while loop is getting tripped for the last few bytes

I'm sending byte of 8254789 bytes. It is undergoing the loop but when it reaches the at 8246597 and has to read 8192 bytes. It is going out from while loop to nowhere. Can someone explain please, what is the problem?
public static byte[] ReadFully(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, justS
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[3824726];
int read = 0;
int chunk;
try
{
while ((chunk = stream.Read(buffer, read, 3824726 - read)) > 0)
{
Console.WriteLine("Length of chunk" + chunk);
read += chunk;
Console.WriteLine("Length of read" + read);
if (read == 0)
{
stream.Close();
return buffer;
}
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
Console.WriteLine("Length of Buffer" + buffer.Length);
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Console.WriteLine("Length of newBuffer" + newBuffer.Length);
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
catch (Exception ex)
{ throw ex; }
}

Normally you don't write your stream loops like that. Instead try something like this:
byte[] buffer = new byte[BUFFER_SIZE];
int read = -1;
while((read = stream.Read(buffer, 0, buffer.Length)) > 0)
{
// ... use read bytes in buffer here
}
You're attempting to adjust your offset each time, but you don't need to because use cursors - so you're basically skipping ahead.

Asynchronous data call on TCp Client/Server for large data

I'm getting a image in bytes from a client device. A 3824726 bytes of data. This while loop will read till all the message is recieved. I'm facing a problem here. When it reads the byte till 3816534 and remaining 8192 bytes are left, it is just tripping off the while loop. No exception nothing. just going away. Please some one help reagding it. i'm not able to undersatnd what is the problem.
public static byte[] ReadFully(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, justS
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[3824726];
int read = 0;
int chunk;
try
{
while ((chunk = stream.Read(buffer, read, 3824726 - read)) > 0)
{
Console.WriteLine("Length of chunk" + chunk);
read += chunk;
Console.WriteLine("Length of read" + read);
if (read == 0)
{
stream.Close();
return buffer;
}
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
Console.WriteLine("Length of Buffer" + buffer.Length);
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Console.WriteLine("Length of newBuffer" + newBuffer.Length);
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
catch (Exception ex)
{ throw ex; }
}

CloudBlob.OpenRead() is not reading all data

With windows Azure Storage Client Library, CloudBlob.OpenRead() method reads only 4 mb of data. How can I read full stream using OpenRead method.
CloudBlob blob = container.GetBlobReference(filename);
Stream stream = blob.OpenRead();
I must use OpenRead method. Can't use DownloadToFile or DownloadToStream.
EDIT :
The consumer code which is outside my scope calls
stream.CopyTo(readIntoStream);
CopyTo is an extension method.
public static int CopyTo(this Stream source, Stream destination)
{
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
int bytesCopied = 0;
do
{
bytesRead = source.Read(buffer, 0, BUFFER_SIZE);
if (bytesRead > 0)
destination.Write(buffer, 0, bytesRead);
bytesCopied += bytesRead;
}
while (bytesRead > 0);
return bytesCopied;
}
EDIT 2 :
I have observed that when file is uploaded using blob.UploadText() method, it works fine but when blob is uploaded using OpenWrite method as done in following code sample, the OpenRead Method reads only 4194304 bytes (4 mb).
using(var input = File.OpenRead(#"C:\testFile.txt")) //5000000 bytes
using (var output = blob.OpenWrite())
{
input.CopyTo(output);
}
EDIT 3 :
Complete code that produces the issue at my end.
static void Main(string[] args)
{
var blobContainer = GetContainer("tier1");
blobContainer.CreateIfNotExist();
var containerPermissions = new BlobContainerPermissions();
containerPermissions.PublicAccess = BlobContainerPublicAccessType.Blob;
blobContainer.SetPermissions(containerPermissions);
var blob = blobContainer.GetBlobReference("testfileWithOpenWrite1.txt");
using (var input = File.OpenRead(#"C:\testFile.txt")) //5000000 bytes
using (var output = blob.OpenWrite(new BlobRequestOptions()))
input.CopyTo(output);
Console.WriteLine("uploaded");
int bytesDownloaded = 0;
byte[] buffer = new byte[65536];
using (BlobStream bs = blob.OpenRead())
{
int chunkLength;
do
{
chunkLength = bs.Read(buffer, 0, buffer.Length);
bytesDownloaded += chunkLength;
} while (chunkLength > 0);
}
Console.WriteLine(bytesDownloaded);
}
public static int CopyTo(this Stream source, Stream destination)
{
int BUFFER_SIZE = 65536;
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
int bytesCopied = 0;
do
{
bytesRead = source.Read(buffer, 0, BUFFER_SIZE);
if (bytesRead > 0)
destination.Write(buffer, 0, bytesRead);
bytesCopied += bytesRead;
}
while (bytesRead > 0);
return bytesCopied;
}
EDIT 4 - Conclusion:
This was probably a bug in Microsoft.WindowsAzure.StorageClient.dll (assembly version 6.0.6002.17043) that comes with SDK v1.2. It works with latest Microsoft.WindowsAzure.StorageClient.dll (assembly version 6.0.6002.18312 - SDK v1.4).
Thanks smarx :)

I can't reproduce what you're seeing. Things seem to work as expected:
static void Main(string[] args)
{
// I also tried a real cloud storage account. Same result.
var container = CloudStorageAccount.DevelopmentStorageAccount.CreateCloudBlobClient().GetContainerReference("testcontainer");
container.CreateIfNotExist();
var blob = container.GetBlobReference("testblob.txt");
blob.UploadText(new String('x', 5000000));
var source = blob.OpenRead();
int BUFFER_SIZE = 4000000;
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
int bytesCopied = 0;
do
{
bytesRead = source.Read(buffer, 0, BUFFER_SIZE);
bytesCopied += bytesRead;
} while (bytesRead > 0);
Console.WriteLine(bytesCopied); // prints 5000000
}
EDIT:
I've also (in response to the edited question) now tried uploading the blob using OpenWrite, and the result is the same. (I get the full blob back.) I used this code in place of blob.UploadText(...):
using (var input = File.OpenRead(#"testblob.txt")) //5000000 bytes
using (var output = blob.OpenWrite())
{
input.CopyTo(output);
}
The final WriteLine still prints 5000000.
EDIT 2:
This is getting a bit tiresome. Changing the BUFFER_SIZE to 65536 didn't change anything. The results still seem correct to me. Here's my full application for comparison:
static void Main(string[] args)
{
// I also tried a real cloud storage account. Same result.
var container = CloudStorageAccount.DevelopmentStorageAccount.CreateCloudBlobClient().GetContainerReference("testcontainer");
container.CreateIfNotExist();
var blob = container.GetBlobReference("testblob.txt");
using (var input = File.OpenRead(#"testblob.txt")) //5000000 bytes
using (var output = blob.OpenWrite())
{
input.CopyTo(output);
}
var source = blob.OpenRead();
int BUFFER_SIZE = 65536;
byte[] buffer = new byte[BUFFER_SIZE];
int bytesRead;
int bytesCopied = 0;
do
{
bytesRead = source.Read(buffer, 0, BUFFER_SIZE);
bytesCopied += bytesRead;
} while (bytesRead > 0);
Console.WriteLine(bytesCopied); // prints 5000000
}

How to split a large file into chunks in c#?

I'm making a simple file transfer sender and receiver app through the wire. What I have so far is that the sender converts the file into a byte array and sends chunks of that array to the receiver.
This works with file of up to 256mb, but this line throws a "System out of memory" exception for anything above:
byte[] buffer = StreamFile(fileName); //This is where I convert the file
I'm looking for a way to read the file in chunks then write that chunk instead of loading the whole file into a byte. How can I do this with a FileStream?
EDIT:
Sorry, heres my crappy code so far:
private void btnSend(object sender, EventArgs e)
{
Socket clientSock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
byte[] fileName = Encoding.UTF8.GetBytes(fName); //file name
byte[] fileData = null;
try
{
fileData = StreamFile(textBox1.Text); //file
}
catch (OutOfMemoryException ex)
{
MessageBox.Show("Out of memory");
return;
}
byte[] fileNameLen = BitConverter.GetBytes(fileName.Length); //length of file name
clientData = new byte[4 + fileName.Length + fileData.Length];
fileNameLen.CopyTo(clientData, 0);
fileName.CopyTo(clientData, 4);
fileData.CopyTo(clientData, 4 + fileName.Length);
clientSock.Connect("172.16.12.91", 9050);
clientSock.Send(clientData, 0, 4 + fileName.Length, SocketFlags.None);
for (int i = 4 + fileName.Length; i < clientData.Length; i++)
{
clientSock.Send(clientData, i, 1 , SocketFlags.None);
}
clientSock.Close();
}
And here's how I receive (the code was from a tutorial)
public void ReadCallback(IAsyncResult ar)
{
int fileNameLen = 1;
String content = String.Empty;
StateObject state = (StateObject)ar.AsyncState;
Socket handler = state.workSocket;
int bytesRead = handler.EndReceive(ar);
if (bytesRead > 0)
{
if (flag == 0)
{
Thread.Sleep(1000);
fileNameLen = BitConverter.ToInt32(state.buffer, 0);
string fileName = Encoding.UTF8.GetString(state.buffer, 4, fileNameLen);
receivedPath = fileName;
flag++;
}
if (flag >= 1)
{
BinaryWriter writer = new BinaryWriter(File.Open(receivedPath, FileMode.Append));
if (flag == 1)
{
writer.Write(state.buffer, 4 + fileNameLen, bytesRead - (4 + fileNameLen));
flag++;
}
else
writer.Write(state.buffer, 0, bytesRead);
writer.Close();
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
}
else
{
Invoke(new MyDelegate(LabelWriter));
}
}
I just really want to know how I can read the file in chunks so that I dont need to convert it to a byte.
Thanks for the responses so far, I think I'm starting to get it :D

Just call Read repeatedly with a small buffer (I tend to use something like 16K). Note that the call to Read may end up reading a smaller amount than you request. If you're using a fixed chunk size and need the whole chunk in memory, you could just use an array of that size of course.
Without knowing how you're sending the file, it's hard to give much advice about how to structure your code, but it could be something like this:
byte[] chunk = new byte[MaxChunkSize];
while (true)
{
int index = 0;
// There are various different ways of structuring this bit of code.
// Fundamentally we're trying to keep reading in to our chunk until
// either we reach the end of the stream, or we've read everything we need.
while (index < chunk.Length)
{
int bytesRead = stream.Read(chunk, index, chunk.Length - index);
if (bytesRead == 0)
{
break;
}
index += bytesRead;
}
if (index != 0) // Our previous chunk may have been the last one
{
SendChunk(chunk, index); // index is the number of bytes in the chunk
}
if (index != chunk.Length) // We didn't read a full chunk: we're done
{
return;
}
}
If I was more awake I'd probably find a more readable way of writing this, but it'll do for now. One option is to extract another method from the middle section:
// Attempts to read an entire chunk into the given array; returns the size of
// chunk actually read.
int ReadChunk(Stream stream, byte[] chunk)
{
int index = 0;
while (index < chunk.Length)
{
int bytesRead = stream.Read(chunk, index, chunk.Length - index);
if (bytesRead == 0)
{
break;
}
index += bytesRead;
}
return index;
}

var b = new byte[1<<15]; // 32k
while((count = inStream.Read(b, 0, b.Length)) > 0)
{
outStream.Write(b, 0, count);
}

public static IEnumerable<byte[]> SplitStreamIntoChunks(Stream stream, int chunkSize)
{
var bytesRemaining = stream.Length;
while (bytesRemaining > 0)
{
var size = Math.Min((int) bytesRemaining, chunkSize);
var buffer = new byte[size];
var bytesRead = stream.Read(buffer, 0, size);
if (bytesRead <= 0)
break;
yield return buffer;
bytesRemaining -= bytesRead;
}
}