We Keep Coding

C# - Stream.Read offset is working incorrectly

I have a test code like below. I am reading from stream, offsetting by 2 positions, and then taking next 2 bytes. I would hope that result would be an array with 2 elements. This does not work though - offset is completely ignored, and full sized array is always returned, with only offset blocks having values. But this means my result table is still very large, it just has a lot of unwanted zeroes
How can I rework below code, so that file.Read() returns only an array of 2 bytes instead of 10 when length = 2 and offset = 2? In real world scenario I am dealing with large files (>2gigs) so filtering out the result array is not an option.
Edit: As the issue is unclear - below code requires me to always define output array that is the same size as the stream. Instead I would like to have an output that is of size of length (in below example I would like to have var buffer = new byte[2], but that will throw an exception because file.Read ignores offset and length and always returns 10 elements (with only 2 of them being read, rest is dummy zeroes).
private byte[] GetFilePart(int length, int offset)
{
//build some dummy content
var content = new byte[10];
for (int i = 0; i<10; i++)
{
content[i] = 1;
}
//read the data from content
var buffer = new byte[10];
using (Stream file = new MemoryStream(content))
{
file.Read(buffer, offset, length);
}
return buffer;
}

Looks like it's working properly to me; maybe your confusion would clear a bit if you inited your content array with something like:
for (int i = 1; i<=10; i++)
{
content[i-1] = i;
}
then each byte would have a different number and the image would look like:
offset relates to where into buffer the Stream will write the bytes to (it reads from the start of content). It does not relate to what bytes are read out of content.
Imagine Read as being called WriteBytesInto(byte[] whatBuffer, int whereToStartWriting, int howManyBytesToWrite) - you provide the buffer it will write into and tell it where to start and how many to do
If you did this, having inited content to be incrementing numbers:
file.Read(buffer, 2, 3); //read 3 bytes from stream and write to buffer # index 2
file.Read(buffer, 0, 2); //read 2 bytes from stream and write to buffer # index 0
Your buffer would end up looking like:
4,5,1,2,3,0,0,0,0,0
The 1,2,3 having been written first, then the 4,5 written next
If you want to skip two bytes from the stream (i.e. read the 3rd and 4th byte from content, Seek() the stream or set its Position (or as canton7 advises in the commments, if the stream is not seekable, read and discard some bytes)
How can I rework below code, so that file.Read() returns only an array of 2 bytes instead of 10 when length = 2 and offset = 2?
Well, file.Read doesn't return an array at all; it modifies an array you give it. If you want a 2 byte array, give it a 2 byte array:
byte buf = new byte[2];
file.Read(buf, 0, buf.Length);
If you want to open a file, skip the first 7 bytes and then read bytes 8th and 9th into your length-of-2 byte array then:
byte buf = new byte[2];
file.Position = 7; //absolute skip to 8th byte
file.Read(buf, 0, buf.Length);
For more on seeking in streams see Stream.Seek(0, SeekOrigin.Begin) or Position = 0

Read Multiple Byte Arrays from File

How can I read multiple byte arrays from a file? These byte arrays are images and have the potential to be large.
This is how I'm adding them to the file:
using (var stream = new FileStream(tempFile, FileMode.Append))
{
//convertedImage = one byte array
stream.Write(convertedImage, 0, convertedImage.Length);
}
So, now they're in tempFile and I don't know how to retrieve them as individual arrays. Ideally, I'd like to get them as an IEnumerable<byte[]>. Is there a way to split these, maybe?

To retrieve multiple sets of byte arrays, you will need to know the length when reading. The easiest way to do this (if you can change the writing code) is to add a length value:
using (var stream = new FileStream(tempFile, FileMode.Append))
{
//convertedImage = one byte array
// All ints are 4-bytes
stream.Write(BitConverter.GetBytes(convertedImage.Length), 0, 4);
// now, we can write the buffer
stream.Write(convertedImage, 0, convertedImage.Length);
}
Reading the data is then
using (var stream = new FileStream(tempFile, FileMode.Open))
{
// loop until we can't read any more
while (true)
{
byte[] convertedImage;
// All ints are 4-bytes
int size;
byte[] sizeBytes = new byte[4];
// Read size
int numRead = stream.Read(sizeBytes, 0, 4);
if (numRead <= 0) {
break;
}
// Convert to int
size = BitConverter.ToInt32(sizeBytes, 0);
// Allocate the buffer
convertedImage = new byte[size];
stream.Read(convertedImage, 0, size);
// Do what you will with the array
listOfArrays.Add(convertedImage);
} // end while
}
If all saved images are the same size, then you can eliminate the first read and write call from each, and hard-code size to the size of the arrays.

Unless you can work out the number of bytes taken by each individual array from the content of these bytes themselves, you need to store the number of images and their individual lengths into the file.
There are many ways to do it: you could write lengths of the individual arrays preceding each byte array, or you could write a "header" describing the rest of the content before writing the "payload" data to the file.
Header may look as follows:
Byte offset Description
----------- -------------------
0000...0003 - Number of files, N
0004...000C - Length of file 1
000D...0014 - Length of file 2
...
XXXX...XXXX - Length of file N
XXXX...XXXX - Content of file 1
XXXX...XXXX - Content of file 2
...
XXXX...XXXX - Content of file N
You can use BitConverter methods to produce byte arrays to be written to the header, or you could use BinaryWriter.

When you read back how do you get the number of bytes per image/byte array to read?
You will need to store the length too (i.e. first 4 bytes = encoded 32 bit int byte count, followed by the data bytes.)
To read back, read the first four bytes, un-encode it back to an int, and then read that number of bytes, repeat until eof.

C# Beginner File Reading

Okay I searched for an answer to this but couldn't find it.
here's the code:
FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read);
byte[] fileText = new byte[fs.Length];
int bytesRead = fs.Read(fileText, 0, fileText.Length);
Console.WriteLine(Encoding.ASCII.GetString(fileText, 0, bytesRead));
Let me get this straight,
We declare a filestream
We Declare a byte array.. and set its CAPACITY to fs.Length
???? Why does fs.Read() return an INTEGER ???
??? How does this line display the text from the .txt file to the console? we passed in the byte[] in the getstring() method, but isnt that byte[] empty? we only set its capacity to fs.length? where did the reading happen and how?
TIA

If you are trying to read a text file and display all it's lines in console
foreach(string line in File.ReadAllLines("YourFilePath"))
{
Console.WriteLine(line);
}
In your method
FileStream fs = new FileStream("YourFilePath", FileMode.Open, FileAccess.Read);
Opens the file for reading into stream fs.
byte[] fileText = new byte[fs.Length];
Gets the number of bytes in the file content, and creates a byte array of that size
int bytesRead = fs.Read(fileText, 0, fileText.Length);
Reads the byte content, from 0 to end of content (we have length from last statement), i.e. the complete contents into the array you created. So, now your byte array fileText has all the byte contents from the file.
It returns the number of bytes read in this operation, if you need that for some reason. This can be <= the number of bytes you wanted to read (less if less bytes were available in the file content). In your case, it will be same as fileText.Length since you already calculated that.
System.Console.WriteLine(Encoding.ASCII.GetString(fileText, 0, bytesRead));
Converts the byte array into ASCII encoded text and writes to console.

Read method returns the number of bytes that were read into the buffer paramters. You passed in an array that will be filled with the data when the Read method is actually called. You have passed in the number of bytes that you want to read as well.
Read the msdn documentation for more details here

FileStream.Read() returns the number of bytes actually read. It could be that you ask for 4096 bytes, but get 0, or 1, or 1000. This is what the docs say:
Return Value
Type: System.Int32
The total number of bytes read into the buffer. This might be less than the number of bytes requested if that number of bytes are not currently available, or zero if the end of the stream is reached.
If you are reading text, you can use one of the helpful File methods: File.ReadAllText, File.ReadAllLines, or File.OpenText which gives you a StreamReader object where you can read line-by-line.
If you need to read the bytes (this is a much lower-level usage, and really you should be able to use a StreamReader), then you don't want to create a buffer the length of the stream, since this could crash your program with an OutOfMemoryException. Instead, make the buffer something like 4096 bytes, then call FileStream.Read in a loop, until it returns 0. Note, however, that you are not reading text lines here, and a line break may come in the middle of the buffer. Here's an example:
using (var fileStream = File.OpenRead("c:\\file.txt"))
{
var buffer = new Byte[4096];
var offset = 0;
var read = 0;
while ((read = fileStream.Read(buffer, offset, buffer.Length)) > 0)
{
var s = Encoding.ASCII.GetString(buffer, 0, read);
Console.Write(s);
offset += read;
}
}

3) has been answered here already.
As for 4): the Read method actually also fills the buffer with bytes and returns the number of bytes it filled into the buffer.
Passing the buffer and the number of bytes read to Encoding. GetString() interprets the bytes from the file as character codes for the given encoding, in your case ASCII, and creates an string from the byte array based on the encoding.

Split up a memorystream into bytarray

Im trying to split up a memorystream into chunks by reading parts into a byte array but i think i have got something fundamentally wrong. I can read the first chunk but when i try to read rest of memorystream i get index out of bound even if there are more bytes to read. It seems that the issue is the size of the receving byte buffer that need to be as large as the memorystrem. I need to convert it into chunks as the code is in a webservice.
Anyone knows whats wrong with this code
fb.buffer is MemoryStream
long bytesLeft = fb.Buffer.Length;
fb.Buffer.Position = 0;
int offset =0;
int BUFF_SIZE = 8196;
while (bytesLeft > 0)
{
byte[] fs = new byte[BUFF_SIZE];
fb.Buffer.Read(fs, offset, BUFF_SIZE);
offset += BUFF_SIZE;
bytesLeft -= BUFF_SIZE;
}

offset here is the offset into the array. It should be zero here. You should also be looking at the return value from Read. It is not guaranteed to fill the buffer, even if more data is available.
However, if this is a MemoryStream - a better option might be ArraySegment<byte>, which requires no duplication of data.

Please look at this code for Stream.Read from MSDN from a glance - you shouldn't be incrementing offset - it should always be zero. Unless, of course, you happen to know for a fact the exact length of the stream in advance (therefore you would create the array the exact size).
You should also always grab the amount of bytes read from Read (the return value).
If you're looking to split it into 'chunks` do you mean you want n 8k chunks? Then you might want to do something like this:
List<byte[]> chunks = new List<byte[]>();
byte chunk = new byte[BUFF_SIZE];
int bytesRead = fb.Buffer.Read(chunk, 0, BUFF_SIZE);
while(bytesRead > 0)
{
if(bytesRead != BUFF_SIZE)
{
byte[] tail = new byte[bytesRead];
Array.Copy(chunk, tail, bytesRead);
chunk = tail;
}
chunks.Add(chunk);
bytesRead = fb.Buffer.Read(chunk, 0, BUFF_SIZE);
}
Note in particular that the last chunk is more than likely not going to be exactly BUFF_SIZE in length.

Unable to read beyond the end of the stream

I did some quick method to write a file from a stream but it's not done yet. I receive this exception and I can't find why:
Unable to read beyond the end of the stream
Is there anyone who could help me debug it?
public static bool WriteFileFromStream(Stream stream, string toFile)
{
FileStream fileToSave = new FileStream(toFile, FileMode.Create);
BinaryWriter binaryWriter = new BinaryWriter(fileToSave);
using (BinaryReader binaryReader = new BinaryReader(stream))
{
int pos = 0;
int length = (int)stream.Length;
while (pos < length)
{
int readInteger = binaryReader.ReadInt32();
binaryWriter.Write(readInteger);
pos += sizeof(int);
}
}
return true;
}
Thanks a lot!

Not really an answer to your question but this method could be so much simpler like this:
public static void WriteFileFromStream(Stream stream, string toFile)
{
// dont forget the using for releasing the file handle after the copy
using (FileStream fileToSave = new FileStream(toFile, FileMode.Create))
{
stream.CopyTo(fileToSave);
}
}
Note that i also removed the return value since its pretty much useless since in your code, there is only 1 return statement
Apart from that, you perform a Length check on the stream but many streams dont support checking Length.
As for your problem, you first check if the stream is at its end. If not, you read 4 bytes. Here is the problem. Lets say you have a input stream of 6 bytes. First you check if the stream is at its end. The answer is no since there are 6 bytes left. You read 4 bytes and check again. Ofcourse the answer is still no since there are 2 bytes left. Now you read another 4 bytes but that ofcourse fails since there are only 2 bytes. (readInt32 reads the next 4 bytes).

I presume that the input stream have ints only (Int32). You need to test the PeekChar() method,
while (binaryReader.PeekChar() != -1)
{
int readInteger = binaryReader.ReadInt32();
binaryWriter.Write(readInteger);
}

You are doing while (pos < length) and length is the actual length of the stream in bytes. So you are effectively counting the bytes in the stream and then trying to read that many number of ints (which is incorrect). You could take length to be stream.Length / 4 since an Int32 is 4 bytes.

try
int length = (int)binaryReader.BaseStream.Length;

After reading the stream by the binary reader the position of the stream is at the end, you have to set the position to zero "stream.position=0;"