So what i'm doing is:
int blockLength;
try {
blockLength = Convert.ToInt32(line, 16);
} catch (Exception ex) {
if (ex is FormatException || ex is OverflowException) {
throw new Exception("WrongChunkedBlockLength", ex);
}
throw;
}
if (blockLength == 0) {
yield break;
}
(where line = a line read from the stream that is expected to be the chunk length)
This works fine but for whatever reason im getting returned a value thats lower than what the actual chunk size is. What I mean by this, is im sending a request to Hotels.com and when I get the "chunk size" and then read from the stream with the chunk size as the length/count it reads it all, but theres actually still further data to read for that chunk.
Since the code now thinks it read the full chunk, it loops to continue the next chunk only to get hit with first chunks data.
This is how im reading from the stream:
int totalBytesRead = 0;
byte[] buffer = new byte[blockLength];
while (totalBytesRead < blockLength) {
int read = stream.Read(buffer, totalBytesRead, blockLength);
if (read == 0) {
WaitData();
continue;
}
totalBytesRead += read;
}
and then right after that this is what I do to further test the issue:
buffer = new byte[90];
while (true) {
stream.Read(buffer, 0, 10);
MessageBox.Show(Encoding.ASCII.GetString(buffer));
}
Related
Sorry for such a vague title, I really dont know what to title this issue.
Basically when I get a stream thats chunked as told by Transfer-Encoding, I then do the following code:
private IEnumerable<byte[]> ReceiveMessageBodyChunked() {
readChunk:
#region Read a line from the Stream which should be a Block Length (Chunk Body Length)
string blockLength = _receiverHelper.ReadLine();
#endregion
#region If the end of the block is reached, re-read from the stream
if (blockLength == Http.NewLine) {
goto readChunk;
}
#endregion
#region Trim it so it should end up with JUST the number
blockLength = blockLength.Trim(' ', '\r', '\n');
#endregion
#region If the end of the message body is reached
if (blockLength == string.Empty) {
yield break;
}
#endregion
int blockLengthInt = 0;
#region Convert the Block Length String to an Int32 base16 (hex)
try {
blockLengthInt = Convert.ToInt32(blockLength, 16);
} catch (Exception ex) {
if (ex is FormatException || ex is OverflowException) {
throw new Exception(string.Format(ExceptionValues.HttpException_WrongChunkedBlockLength, blockLength), ex);
}
throw;
}
#endregion
// If the end of the message body is reached.
if (blockLengthInt == 0) {
yield break;
}
byte[] buffer = new byte[blockLengthInt];
int totalBytesRead = 0;
while (totalBytesRead != blockLengthInt) {
int length = blockLengthInt - totalBytesRead;
int bytesRead = _receiverHelper.HasData ? _receiverHelper.Read(buffer, 0, length) : _request.ClientStream.Read(buffer, 0, length);
if (bytesRead == 0) {
WaitData();
continue;
}
totalBytesRead += bytesRead;
System.Windows.Forms.MessageBox.Show("Chunk Length: " + blockLengthInt + "\nBytes Read/Total:" + bytesRead + "/" + totalBytesRead + "\n\n" + Encoding.ASCII.GetString(buffer));
yield return buffer;
}
goto readChunk;
}
What this is doing is reading 1 line of data from the stream which should be the Chunk's Length, does some checks here and there but eventually converts that to a Int32 Radix16 integer.
From there it essentially creates a byte buffer of that int32 as its length size.
It then just keeps reading from the stream until its read the same amount as the Int32 we converted.
This works splendid, however, for whatever reason, its responding incorrectly on the last read.
It will read the exact amount of bytes as the chunk length perfectly fine, and all data I expect is read. BUT it's ALSO reading again another small chunk of data that was ALREADY read at the very end, resulting in lets say all data from <!DOCTYPE html> down to </html> ASWELL as some data from inside somewhere like <form> e.t.c
Here's an example of what occured:
As you can see, the highlighted red text should NOT have been returned from the read! It should have ended at </html>.
Why is the Chunk's Length lying to me and how can I find the proper size to read at?
I'm not familiar with C# but if I understand your code and the semantics of Read in C# correctly (which seem to be similar to read in C) then the problem is that you are using the same buffer again and again without resetting it first:
byte[] buffer = new byte[blockLengthInt];
int totalBytesRead = 0;
while (totalBytesRead != blockLengthInt) {
int length = blockLengthInt - totalBytesRead;
int bytesRead = _receiverHelper.HasData ? _receiverHelper.Read(buffer, 0, length) : _request.ClientStream.Read(buffer, 0, length);
...
totalBytesRead += bytesRead;
...
yield return buffer;
}
To make some example of what goes wrong here: assume that the chunk size is 10, the content you read is 0123456789 and the first read will return 6 bytes and the second read the remaining 4 bytes. In this case your buffer will be 012345 after the first read and 567845 after the second read. These 45 at the end of the buffer remain from the previous read since you only replaced the first 4 bytes in the buffer but kept the rest.
Whats odd AF is that if I hand the request to another TCPStream proxied (127.0.0.1:8888 as a proxy which is fiddler) it works perfectly fine...
Fiddler is a proxy and might change how the response gets transferred. For example it might use Content-length instead of chunked encoding or it might use smaller chunks so that you always get the full chunk with the first read.
I'm sending byte of 8254789 bytes. It is undergoing the loop but when it reaches the at 8246597 and has to read 8192 bytes. It is going out from while loop to nowhere. Can someone explain please, what is the problem?
public static byte[] ReadFully(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, justS
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[3824726];
int read = 0;
int chunk;
try
{
while ((chunk = stream.Read(buffer, read, 3824726 - read)) > 0)
{
Console.WriteLine("Length of chunk" + chunk);
read += chunk;
Console.WriteLine("Length of read" + read);
if (read == 0)
{
stream.Close();
return buffer;
}
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
Console.WriteLine("Length of Buffer" + buffer.Length);
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Console.WriteLine("Length of newBuffer" + newBuffer.Length);
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
catch (Exception ex)
{ throw ex; }
}
Normally you don't write your stream loops like that. Instead try something like this:
byte[] buffer = new byte[BUFFER_SIZE];
int read = -1;
while((read = stream.Read(buffer, 0, buffer.Length)) > 0)
{
// ... use read bytes in buffer here
}
You're attempting to adjust your offset each time, but you don't need to because use cursors - so you're basically skipping ahead.
I'm getting a image in bytes from a client device. A 3824726 bytes of data. This while loop will read till all the message is recieved. I'm facing a problem here. When it reads the byte till 3816534 and remaining 8192 bytes are left, it is just tripping off the while loop. No exception nothing. just going away. Please some one help reagding it. i'm not able to undersatnd what is the problem.
public static byte[] ReadFully(Stream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, justS
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
byte[] buffer = new byte[3824726];
int read = 0;
int chunk;
try
{
while ((chunk = stream.Read(buffer, read, 3824726 - read)) > 0)
{
Console.WriteLine("Length of chunk" + chunk);
read += chunk;
Console.WriteLine("Length of read" + read);
if (read == 0)
{
stream.Close();
return buffer;
}
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
Console.WriteLine("Length of Buffer" + buffer.Length);
int nextByte = stream.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Console.WriteLine("Length of newBuffer" + newBuffer.Length);
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
catch (Exception ex)
{ throw ex; }
}
I'm making a simple file transfer sender and receiver app through the wire. What I have so far is that the sender converts the file into a byte array and sends chunks of that array to the receiver.
This works with file of up to 256mb, but this line throws a "System out of memory" exception for anything above:
byte[] buffer = StreamFile(fileName); //This is where I convert the file
I'm looking for a way to read the file in chunks then write that chunk instead of loading the whole file into a byte. How can I do this with a FileStream?
EDIT:
Sorry, heres my crappy code so far:
private void btnSend(object sender, EventArgs e)
{
Socket clientSock = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
byte[] fileName = Encoding.UTF8.GetBytes(fName); //file name
byte[] fileData = null;
try
{
fileData = StreamFile(textBox1.Text); //file
}
catch (OutOfMemoryException ex)
{
MessageBox.Show("Out of memory");
return;
}
byte[] fileNameLen = BitConverter.GetBytes(fileName.Length); //length of file name
clientData = new byte[4 + fileName.Length + fileData.Length];
fileNameLen.CopyTo(clientData, 0);
fileName.CopyTo(clientData, 4);
fileData.CopyTo(clientData, 4 + fileName.Length);
clientSock.Connect("172.16.12.91", 9050);
clientSock.Send(clientData, 0, 4 + fileName.Length, SocketFlags.None);
for (int i = 4 + fileName.Length; i < clientData.Length; i++)
{
clientSock.Send(clientData, i, 1 , SocketFlags.None);
}
clientSock.Close();
}
And here's how I receive (the code was from a tutorial)
public void ReadCallback(IAsyncResult ar)
{
int fileNameLen = 1;
String content = String.Empty;
StateObject state = (StateObject)ar.AsyncState;
Socket handler = state.workSocket;
int bytesRead = handler.EndReceive(ar);
if (bytesRead > 0)
{
if (flag == 0)
{
Thread.Sleep(1000);
fileNameLen = BitConverter.ToInt32(state.buffer, 0);
string fileName = Encoding.UTF8.GetString(state.buffer, 4, fileNameLen);
receivedPath = fileName;
flag++;
}
if (flag >= 1)
{
BinaryWriter writer = new BinaryWriter(File.Open(receivedPath, FileMode.Append));
if (flag == 1)
{
writer.Write(state.buffer, 4 + fileNameLen, bytesRead - (4 + fileNameLen));
flag++;
}
else
writer.Write(state.buffer, 0, bytesRead);
writer.Close();
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
}
else
{
Invoke(new MyDelegate(LabelWriter));
}
}
I just really want to know how I can read the file in chunks so that I dont need to convert it to a byte.
Thanks for the responses so far, I think I'm starting to get it :D
Just call Read repeatedly with a small buffer (I tend to use something like 16K). Note that the call to Read may end up reading a smaller amount than you request. If you're using a fixed chunk size and need the whole chunk in memory, you could just use an array of that size of course.
Without knowing how you're sending the file, it's hard to give much advice about how to structure your code, but it could be something like this:
byte[] chunk = new byte[MaxChunkSize];
while (true)
{
int index = 0;
// There are various different ways of structuring this bit of code.
// Fundamentally we're trying to keep reading in to our chunk until
// either we reach the end of the stream, or we've read everything we need.
while (index < chunk.Length)
{
int bytesRead = stream.Read(chunk, index, chunk.Length - index);
if (bytesRead == 0)
{
break;
}
index += bytesRead;
}
if (index != 0) // Our previous chunk may have been the last one
{
SendChunk(chunk, index); // index is the number of bytes in the chunk
}
if (index != chunk.Length) // We didn't read a full chunk: we're done
{
return;
}
}
If I was more awake I'd probably find a more readable way of writing this, but it'll do for now. One option is to extract another method from the middle section:
// Attempts to read an entire chunk into the given array; returns the size of
// chunk actually read.
int ReadChunk(Stream stream, byte[] chunk)
{
int index = 0;
while (index < chunk.Length)
{
int bytesRead = stream.Read(chunk, index, chunk.Length - index);
if (bytesRead == 0)
{
break;
}
index += bytesRead;
}
return index;
}
var b = new byte[1<<15]; // 32k
while((count = inStream.Read(b, 0, b.Length)) > 0)
{
outStream.Write(b, 0, count);
}
public static IEnumerable<byte[]> SplitStreamIntoChunks(Stream stream, int chunkSize)
{
var bytesRemaining = stream.Length;
while (bytesRemaining > 0)
{
var size = Math.Min((int) bytesRemaining, chunkSize);
var buffer = new byte[size];
var bytesRead = stream.Read(buffer, 0, size);
if (bytesRead <= 0)
break;
yield return buffer;
bytesRemaining -= bytesRead;
}
}
I'm trying to read the response stream from an HttpWebResponse object. I know the length of the stream (_response.ContentLength) however I keep getting the following exception:
Specified argument was out of the range of valid values.
Parameter name: size
While debugging, I noticed that at the time of the error, the values were as such:
length = 15032 //the length of the stream as defined by _response.ContentLength
bytesToRead = 7680 //the number of bytes in the stream that still need to be read
bytesRead = 7680 //the number of bytes that have been read (offset)
body.length = 15032 //the size of the byte[] the stream is being copied to
The peculiar thing is that the bytesToRead and bytesRead variables are ALWAYS 7680, regardless of the size of the stream (contained in the length variable). Any ideas?
Code:
int length = (int)_response.ContentLength;
byte[] body = null;
if (length > 0)
{
int bytesToRead = length;
int bytesRead = 0;
try
{
body = new byte[length];
using (Stream stream = _response.GetResponseStream())
{
while (bytesToRead > 0)
{
// Read may return anything from 0 to length.
int n = stream.Read(body, bytesRead, length);
// The end of the file is reached.
if (n == 0)
break;
bytesRead += n;
bytesToRead -= n;
}
stream.Close();
}
}
catch (Exception exception)
{
throw;
}
}
else
{
body = new byte[0];
}
_responseBody = body;
You want this line:
int n = stream.Read(body, bytesRead, length);
to be this:
int n = stream.Read(body, bytesRead, bytesToRead);
You are saying the maximum number of bytes to read is the stream's length, but it isn't since it is actually only the remaining information in the stream after the offset has been applied.
You also shouldn't need this part:
if (n == 0)
break;
The while should end the reading correctly, and it is possible that you won't read any bytes before you have finished the whole thing (if the stream is filling slower than you are taking the data out of it)