We Keep Coding

Chunk Length from ChunkedZIP Response is not correct

So what i'm doing is:
int blockLength;
try {
blockLength = Convert.ToInt32(line, 16);
} catch (Exception ex) {
if (ex is FormatException || ex is OverflowException) {
throw new Exception("WrongChunkedBlockLength", ex);
}
throw;
}
if (blockLength == 0) {
yield break;
}
(where line = a line read from the stream that is expected to be the chunk length)
This works fine but for whatever reason im getting returned a value thats lower than what the actual chunk size is. What I mean by this, is im sending a request to Hotels.com and when I get the "chunk size" and then read from the stream with the chunk size as the length/count it reads it all, but theres actually still further data to read for that chunk.
Since the code now thinks it read the full chunk, it loops to continue the next chunk only to get hit with first chunks data.
This is how im reading from the stream:
int totalBytesRead = 0;
byte[] buffer = new byte[blockLength];
while (totalBytesRead < blockLength) {
int read = stream.Read(buffer, totalBytesRead, blockLength);
if (read == 0) {
WaitData();
continue;
}
totalBytesRead += read;
}
and then right after that this is what I do to further test the issue:
buffer = new byte[90];
while (true) {
stream.Read(buffer, 0, 10);
MessageBox.Show(Encoding.ASCII.GetString(buffer));
}

C# thread exits before receiving data on socket

I am trying to send some text over the network using sockets and memory streams. The full data length in my example is 20480 bytes long. Buffer size is 8192.
Before I can receive the last 4096 bytes, the socket receives only 3088 bytes and the whole thread exits without throwing an exception just before receiving the last chunk of data.
// Send
while (sentBytes < ms.Length)
{
if (streamSize < Convert.ToInt64(buffer.Length))
{
ms.Read(buffer, 0, Convert.ToInt32(streamSize));
count = socket.Send(buffer, 0, Convert.ToInt32(streamSize), SocketFlags.None);
sentBytes += Convert.ToInt64(count);
streamSize -= Convert.ToInt64(count);
}
else
{
ms.Read(buffer, 0, buffer.Length);
count = socket.Send(buffer, 0, buffer.Length, SocketFlags.None);
sentBytes += Convert.ToInt64(count);
streamSize -= Convert.ToInt64(count);
}
}
// Receive
while (readBytes < size)
{
if (streamSize < Convert.ToInt64(buffer.Length))
// exits after this, before receiving the last 1008 bytes
{
count = socket.Receive(buffer, 0, Convert.ToInt32(streamSize), SocketFlags.None);
if (count > 0)
{
ms.Write(buffer, 0, count);
readBytes += Convert.ToInt64(count);
streamSize -= Convert.ToInt64(count);
}
}
else
{
count = socket.Receive(buffer, 0, buffer.Length, SocketFlags.None);
if (count > 0)
{
ms.Write(buffer, 0, count);
readBytes += Convert.ToInt64(count);
streamSize -= Convert.ToInt64(count);
}
}
}
I use the exact same algorithm to send/receive files having bigger sizes (over 1 GB) and the transfer works perfectly, no files are corrupted (I use file streams for that).
Interestingly, this code works in the debugger if I add a breakpoint on the sender side.
Also works with this modification:
if (streamSize < Convert.ToInt64(buffer.Length))
{
if (count > 0)
{
ms.Write(buffer, 0, Convert.ToInt32(streamSize));
readBytes += streamSize;
streamSize -= streamSize;
}
}
but this comes with no checking on how much data is received and also doesn't work to transfer files.
Could anybody point it out what is going on here?
Thread started like this:
public ClientConnection(Socket clientSocket, Server mainForm)
{
this.clientSocket = clientSocket;
clientThread = new Thread(ReceiveData);
clientConnected = true;
this.mainForm = mainForm;
clientThread.Start(clientSocket);
}
Added from comment by OP
// text is 10240 characters long
MemoryStream ms = new MemoryStream(UnicodeEncoding.Unicode.GetBytes(text));
// streamsize is 20480, which is sent prior to text in a header to the receiver
long streamSize = ms.Length;
Update:
Tested with more files, now the file transfer fails as well. The problem is with the last 1008 bytes in all cases.

I found it... When I expected to receive the header, I hadn't prepare the software to receive exactly header sized data.
//byte[] buffer = new byte[1024];
byte[] buffer = new byte[16];
readBytes = socket.Receive(buffer, 0, buffer.Length, SocketFlags.None);
This somehow caused a rogue 16 bytes of data written on the socket every time I was receiving the last chunk of the payload, the socket disconnected and the thread exited not throwing any exceptions whatsoever. I hope this answer will help one day someone else running into the same issue. All data transfer works properly now.

Please consider the simplified implementation of the functionality using NetworkStream class to perform the I/O-operations instead of the Socket class: the NetworkStream class allows to slightly increase the level of abstraction. An instance of the NetworkStream class can be created using an instance of the Socket class.
Sender
The implementation of the Sender is pretty straightforward using the Stream.CopyTo Method:
private static void CustomSend(Stream inputStream, Socket socket)
{
using (var networkStream = new NetworkStream(socket))
{
inputStream.CopyTo(networkStream, BufferSize);
}
}
Receiver
Let's introduce the following extension methods for the Stream class which copies the exact number of bytes from one instance of the Stream class to another using the specified buffer:
using System;
using System.IO;
public static class StreamExtensions
{
public static bool TryCopyToExact(this Stream inputStream, Stream outputStream, byte[] buffer, int bytesToCopy)
{
if (inputStream == null)
{
throw new ArgumentNullException("inputStream");
}
if (outputStream == null)
{
throw new ArgumentNullException("outputStream");
}
if (buffer.Length <= 0)
{
throw new ArgumentException("Invalid buffer specified", "buffer");
}
if (bytesToCopy <= 0)
{
throw new ArgumentException("Bytes to copy must be positive", "bytesToCopy");
}
int bytesRead;
while (bytesToCopy > 0 && (bytesRead = inputStream.Read(buffer, 0, Math.Min(buffer.Length, bytesToCopy))) > 0)
{
outputStream.Write(buffer, 0, bytesRead);
bytesToCopy -= bytesRead;
}
return bytesToCopy == 0;
}
public static void CopyToExact(this Stream inputStream, Stream outputStream, byte[] buffer, int bytesToCopy)
{
if (!TryCopyToExact(inputStream, outputStream, buffer, bytesToCopy))
{
throw new IOException("Failed to copy the specified number of bytes");
}
}
}
So, the Receiver can be implemented as follows:
private static void CustomReceive(Socket socket)
{
// It seems your receiver implementation "knows" the "size to receive".
const int SizeToReceive = 20480;
var buffer = new byte[BufferSize];
var outputStream = new MemoryStream(new byte[SizeToReceive], true);
using (var networkStream = new NetworkStream(socket))
{
networkStream.CopyToExact(outputStream, buffer, SizeToReceive);
}
// Use the outputStream instance...
}
Important note
Please do not forget to call the Dispose() method of the instances of the Socket class (for both Sender and Receiver). The absence of the method call can be a root cause of the problems.

While reading large amount of data in bytes the while loop is getting tripped for the last few bytes

I'm sending byte of 8254789 bytes. It is undergoing the loop but when it reaches the at 8246597 and has to read 8192 bytes. It is going out from while loop to nowhere. Can someone explain please, what is the problem?
public static byte[] ReadFully(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, justS
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[3824726];
int read = 0;
int chunk;
try
{
while ((chunk = stream.Read(buffer, read, 3824726 - read)) > 0)
{
Console.WriteLine("Length of chunk" + chunk);
read += chunk;
Console.WriteLine("Length of read" + read);
if (read == 0)
{
stream.Close();
return buffer;
}
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
Console.WriteLine("Length of Buffer" + buffer.Length);
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Console.WriteLine("Length of newBuffer" + newBuffer.Length);
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
catch (Exception ex)
{ throw ex; }
}

Normally you don't write your stream loops like that. Instead try something like this:
byte[] buffer = new byte[BUFFER_SIZE];
int read = -1;
while((read = stream.Read(buffer, 0, buffer.Length)) > 0)
{
// ... use read bytes in buffer here
}
You're attempting to adjust your offset each time, but you don't need to because use cursors - so you're basically skipping ahead.

How to split a large file into chunks in c#?

I'm making a simple file transfer sender and receiver app through the wire. What I have so far is that the sender converts the file into a byte array and sends chunks of that array to the receiver.
This works with file of up to 256mb, but this line throws a "System out of memory" exception for anything above:
byte[] buffer = StreamFile(fileName); //This is where I convert the file
I'm looking for a way to read the file in chunks then write that chunk instead of loading the whole file into a byte. How can I do this with a FileStream?
EDIT:
Sorry, heres my crappy code so far:
private void btnSend(object sender, EventArgs e)
{
Socket clientSock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
byte[] fileName = Encoding.UTF8.GetBytes(fName); //file name
byte[] fileData = null;
try
{
fileData = StreamFile(textBox1.Text); //file
}
catch (OutOfMemoryException ex)
{
MessageBox.Show("Out of memory");
return;
}
byte[] fileNameLen = BitConverter.GetBytes(fileName.Length); //length of file name
clientData = new byte[4 + fileName.Length + fileData.Length];
fileNameLen.CopyTo(clientData, 0);
fileName.CopyTo(clientData, 4);
fileData.CopyTo(clientData, 4 + fileName.Length);
clientSock.Connect("172.16.12.91", 9050);
clientSock.Send(clientData, 0, 4 + fileName.Length, SocketFlags.None);
for (int i = 4 + fileName.Length; i < clientData.Length; i++)
{
clientSock.Send(clientData, i, 1 , SocketFlags.None);
}
clientSock.Close();
}
And here's how I receive (the code was from a tutorial)
public void ReadCallback(IAsyncResult ar)
{
int fileNameLen = 1;
String content = String.Empty;
StateObject state = (StateObject)ar.AsyncState;
Socket handler = state.workSocket;
int bytesRead = handler.EndReceive(ar);
if (bytesRead > 0)
{
if (flag == 0)
{
Thread.Sleep(1000);
fileNameLen = BitConverter.ToInt32(state.buffer, 0);
string fileName = Encoding.UTF8.GetString(state.buffer, 4, fileNameLen);
receivedPath = fileName;
flag++;
}
if (flag >= 1)
{
BinaryWriter writer = new BinaryWriter(File.Open(receivedPath, FileMode.Append));
if (flag == 1)
{
writer.Write(state.buffer, 4 + fileNameLen, bytesRead - (4 + fileNameLen));
flag++;
}
else
writer.Write(state.buffer, 0, bytesRead);
writer.Close();
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
}
else
{
Invoke(new MyDelegate(LabelWriter));
}
}
I just really want to know how I can read the file in chunks so that I dont need to convert it to a byte.
Thanks for the responses so far, I think I'm starting to get it :D

Just call Read repeatedly with a small buffer (I tend to use something like 16K). Note that the call to Read may end up reading a smaller amount than you request. If you're using a fixed chunk size and need the whole chunk in memory, you could just use an array of that size of course.
Without knowing how you're sending the file, it's hard to give much advice about how to structure your code, but it could be something like this:
byte[] chunk = new byte[MaxChunkSize];
while (true)
{
int index = 0;
// There are various different ways of structuring this bit of code.
// Fundamentally we're trying to keep reading in to our chunk until
// either we reach the end of the stream, or we've read everything we need.
while (index < chunk.Length)
{
int bytesRead = stream.Read(chunk, index, chunk.Length - index);
if (bytesRead == 0)
{
break;
}
index += bytesRead;
}
if (index != 0) // Our previous chunk may have been the last one
{
SendChunk(chunk, index); // index is the number of bytes in the chunk
}
if (index != chunk.Length) // We didn't read a full chunk: we're done
{
return;
}
}
If I was more awake I'd probably find a more readable way of writing this, but it'll do for now. One option is to extract another method from the middle section:
// Attempts to read an entire chunk into the given array; returns the size of
// chunk actually read.
int ReadChunk(Stream stream, byte[] chunk)
{
int index = 0;
while (index < chunk.Length)
{
int bytesRead = stream.Read(chunk, index, chunk.Length - index);
if (bytesRead == 0)
{
break;
}
index += bytesRead;
}
return index;
}

var b = new byte[1<<15]; // 32k
while((count = inStream.Read(b, 0, b.Length)) > 0)
{
outStream.Write(b, 0, count);
}

public static IEnumerable<byte[]> SplitStreamIntoChunks(Stream stream, int chunkSize)
{
var bytesRemaining = stream.Length;
while (bytesRemaining > 0)
{
var size = Math.Min((int) bytesRemaining, chunkSize);
var buffer = new byte[size];
var bytesRead = stream.Read(buffer, 0, size);
if (bytesRead <= 0)
break;
yield return buffer;
bytesRemaining -= bytesRead;
}
}

ArgumentOutOfRangeException when reading bytes from stream

I'm trying to read the response stream from an HttpWebResponse object. I know the length of the stream (_response.ContentLength) however I keep getting the following exception:
Specified argument was out of the range of valid values.
Parameter name: size
While debugging, I noticed that at the time of the error, the values were as such:
length = 15032 //the length of the stream as defined by _response.ContentLength
bytesToRead = 7680 //the number of bytes in the stream that still need to be read
bytesRead = 7680 //the number of bytes that have been read (offset)
body.length = 15032 //the size of the byte[] the stream is being copied to
The peculiar thing is that the bytesToRead and bytesRead variables are ALWAYS 7680, regardless of the size of the stream (contained in the length variable). Any ideas?
Code:
int length = (int)_response.ContentLength;
byte[] body = null;
if (length > 0)
{
int bytesToRead = length;
int bytesRead = 0;
try
{
body = new byte[length];
using (Stream stream = _response.GetResponseStream())
{
while (bytesToRead > 0)
{
// Read may return anything from 0 to length.
int n = stream.Read(body, bytesRead, length);
// The end of the file is reached.
if (n == 0)
break;
bytesRead += n;
bytesToRead -= n;
}
stream.Close();
}
}
catch (Exception exception)
{
throw;
}
}
else
{
body = new byte[0];
}
_responseBody = body;

You want this line:
int n = stream.Read(body, bytesRead, length);
to be this:
int n = stream.Read(body, bytesRead, bytesToRead);
You are saying the maximum number of bytes to read is the stream's length, but it isn't since it is actually only the remaining information in the stream after the offset has been applied.
You also shouldn't need this part:
if (n == 0)
break;
The while should end the reading correctly, and it is possible that you won't read any bytes before you have finished the whole thing (if the stream is filling slower than you are taking the data out of it)