I am trying to read data from a file stream as shown below:
fileStream.Read(byteArray, offset, length);
The problem is that my offset and length are Unsigned Ints and above function accepts only ints. If I typecast to int, I am getting a negative value for offset which is meaningless and not acceptable by the function.
The offset and length are originally taken from another byte array as shown below:
BitConverter.ToUInt32(length, 0); //length is a 4 byte long byte-array
What is the right way to read from arbitrary locations of a file stream.
I am not sure if this is the best way to handle it, but you can change the position of the stream and use offset 0. The Position is of type long.
fileStream.Position = (long)length;
fileStream.Read(byteArray, 0, sizeToRead);
For such a filesize you should read your file in small blocks, proccess the block and read the next. int.MaxValue is about ~2GB, uint.MaxValue ~4GB. Such a size doesn't fit in most computers ram ;)
if you are having problems with conversion, something similar might help:
uint myUInt;
int i = (int)myUInt; or
int i = Convert.ToInt32(myUInt);
Related
My code
private static int readinput(byte[] buff, BinaryReader reader)
{
int size = reader.ReadInt32();
reader.Read(buff, 0, size);
return size;
}
Exception in reader.Read(buff, 0, size);
The exception is offset and length were out of bounds for the array or count is greater than the number of elements from index to the end of source collection
Take a step back and think about your code
You've written a method that takes an array of bytes. We don't know how big this array is, but it's controlled by the code calling the method. Let's assume it is 1000 bytes long
Then you read an int from somewhere else, let's assume 2000 is read
Then you attempt to read 2000 bytes into an array that can only hold 1000 bytes, you perform no checking to make sure your array is big enough, nor do you attempt to read in chunks and concatenate if it's not big enough
That's why you get the error you're getting, but as to what you should be coding, I think you need to think about that some more- maybe make the size to the buffer in response to the size int you read, or read in chunks..
The buffer buff that you passed into your function to read the data is too small. buff.Length should be bigger than or equal to your variable called size.
Set a breakpoint on "reader.Read(buff, 0, size);" and hover over buff and size and you'll see what I mean.
Make sure when you call your function, the buff you pass in is sufficient size. If you don't know what size to create a buffer for ahead of time, then change your function to look something like this:
private static byte[] ReadInput(BinaryReader reader)
{
int size = reader.ReadInt32();
return reader.ReadBytes(size);
}
Especially since you're just reading it into the beginning of a provided buffer anyways.
Summary to frame what you're currently doing:
You provided us a function which takes a binary reader (whatever position it's already at, if it's new, then position 0), it reads a 32-bit integer (4 bytes) to figure out the size of some data after it. Then you read that data of that size into a buffer. You do all of this with a buffer provided called buff. You need to be sure that whatever size data you're going to read in, the buffer provided to the function is of sufficient size. If you make the size of the buffer too large, then "reader.Read(buff, 0, size)" is only reading it into the beginning of the buffer. So if your intention was just to read the data the way you coded that function into a perfectly sized buffer, I suggest using the code above.
Just thought I'd explain it a bit more in case that helps you understand what's going on.
I am having trouble keeping the size of an array fixed. I have to read data from a file. when the number of bytes read is less then 160 the array MyPCMBuf changes to the size of bytes read.
byte[] MyPCMBuf;
MyPCMBuf = new byte[160];
BinaryReader reader = new BinaryReader(File.Open(fileNamein, FileMode.Open)) ;
Array.Clear (MyPCMBuf,0,160);
MyPCMBuf = reader.ReadBytes(160); //if bytes read =20 , size of MyPCMBuf becomes 20
What is going on and how to avoid it?
That's because ReadBytes returns a new byte array.
If you want to read bytes into an existing array, call Read.
That is:
int bytesRead = reader.Read(myBuffer, 0, 160);
You have no problem keeing the size fixed.
ReadBytes returns a new array. That simple. The old one never changes size.
If you want to use your buffer, user another method, for example:
public virtual int Read(
byte[] buffer,
int index,
int count
)
on that class. Then you keep your array.
Don't use BinaryReader at all. Use the FileStream returned by Open and call its Read method. If you don't assign to MyPCMBuf its length cannot possibly change.
Unrelated: Use Array.Clear (MyPCMBuf,0,MyPCMBuf.Length);. Less redundancy. Less potential for errors. Use using. Don't initialize MyPCMBuf if you always overwrite it. Don't clear it redundantly. I see a lot of misunderstandings here. Be more rigorous in your approach to programming. It appears you don't really understand all the language features and APIs you are using. That's dangerous.
You are overwriting MyPCMBuf in line
MyPCMBuf = reader.ReadBytes(160);
Thus, line MyPCMBuf = new byte[160]; is irrelevant in your code.
You are not really doing what you think you are.
reader.ReadBytes(160) will create a new byte array of size at most 160. It's not storing values in the array you already had. That one is send to Garbage Collection.
I want to read very large file (4GBish) chunk by chunk.
I am currently trying to use a StreamReader and the Read() read method. The syntax is:
sr.Read(char[] buffer, int index, int count)
Because the index is an int it will overflow in my case. What should i use instead?
The index is the starting index of buffer not the index of file pointer, usually it would be zero. On each Read call you will read characters equal to the count parameter of Read method. You would not read all the file at once rather read in chunks and use that chunk.
The index of buffer at which to begin writing, reference.
char[] c = null;
while (sr.Peek() >= 0)
{
c = new char[1024];
sr.Read(c, 0, c.Length);
//The output will look odd, because
//only five characters are read at a time.
Console.WriteLine(c);
}
The above example will ready 1024 bytes and will write to console. You can use these bytes, for instance sending these bytes to other application using TCP connection.
When using the Read method, it is more efficient to use a buffer that
is the same size as the internal buffer of the stream, where the
internal buffer is set to your desired block size, and to always read
less than the block size. If the size of the internal buffer was
unspecified when the stream was constructed, its default size is 4
kilobytes (4096 bytes), MSDN.
You could try the simpler version of Read which doesn't chunk the stream but instead reads it character by character. You would have to implement the chunking your self, but it would give you more control allowing you to use a Long instead.
http://msdn.microsoft.com/en-us/library/ath1fht8(v=vs.110).aspx
i want to read\write a binary file which has the following structure:
The file is composed by "RECORDS". Each "RECORD" has the following structure:
I will use the first record as example
(red)START byte: 0x5A (always 1 byte, fixed value 0x5A)
(green) LENGTH bytes: 0x00 0x16 (always 2 bytes, value can change from
"0x00 0x02" to "0xFF 0xFF")
(blue) CONTENT: Number of Bytes indicated by the decimal value of LENGTH Field minus 2. In this case LENGHT field value is 22 (0x00 0x16 converted to decimal), therefore the CONTENT will contain 20 (22 - 2) bytes.
My goal is to read each record one by one, and write it to an output file.
Actually i have a read function and write function (some pseudocode):
private void Read(BinaryReader binaryReader, BinaryWriter binaryWriter)
{
byte START = 0x5A;
int decimalLenght = 0;
byte[] content = null;
byte[] length = new byte[2];
while (binaryReader.PeekChar() != -1)
{
//Check the first byte which should be equals to 0x5A
if (binaryReader.ReadByte() != START)
{
throw new Exception("0x5A Expected");
}
//Extract the length field value
length = binaryReader.ReadBytes(2);
//Convert the length field to decimal
int decimalLenght = GetLength(length);
//Extract the content field value
content = binaryReader.ReadBytes(decimalLenght - 2);
//DO WORK
//modifying the content
//Writing the record
Write(binaryWriter, content, length, START);
}
}
private void Write(BinaryWriter binaryWriter, byte[] content, byte[] length, byte START)
{
binaryWriter.Write(START);
binaryWriter.Write(length);
binaryWriter.Write(content);
}
This way is actually working.
However since I am dealing with very large files i find it to be not performing at all, cause I Read and write 3 times foreach Record. Actually I would like to read bug chunks of data instead small amount of byte and maybe work in memory, but my experience in using Stream stops with BinaryReader and BinaryWriter. Thanks in advance.
FileStream is already buffered, so I'd expect it to work pretty well. You could always create a BufferedStream around the original stream to add extra more buffering if you really need to, but I doubt it would make a significant difference.
You say it's "not performing at all" - how fast is it working? How sure are you that the IO is where your time is going? Have you performed any profiling of the code?
I might also suggest that you read 3 (or 6?) bytes initially, instead of 2 separate reads. Put the initial bytes in a small array, check the 5a ck-byte, then the 2 byte length indicator, then the 3 byte AFP op-code, THEN, read the remainder of the AFP record.
It's a small difference, but it gets rid of one of your read calls.
I'm no Jon Skeet, but I did work at one of the biggest print & mail shops in the country for quite a while, and we did mostly AFP output :-)
(usually in C, though)
I have a binary file. It consists of 4 messages, each is inthe size of 100 bytes.
I want to read that last 2 messages again. I am using BinaryReader object.
I seek to psosition 200 and then I read: BinaryReaderObject.read(charBuffer, 0, 10000),
where charBuffer is big enougth.
I get all the time the a mount of read is always missing 1. Instead of getting 200 I get 199. Instead of getting 400 I get 399.
I checked and saw the size of the file is correct and the data that I get starts at the right place.
Thnaks,
Try this code and see what happens with your file.
String message = #"Read {0} bytes into the buffer.";
String fileName = #"TEST.DAT";
Int32 recordSize = 100;
Byte[] buffer = new Byte[recordSize];
using (BinaryReader br = new BinaryReader(File.OpenRead(fileName)))
{
br.BaseStream.Seek(2 * recordSize, SeekOrigin.Begin);
Console.WriteLine(message, br.Read(buffer, 0, recordSize));
Console.WriteLine(message, br.Read(buffer, 0, recordSize));
}
Console.ReadLine();
I get the following output with a 400 byte test file.
Read 100 bytes into the buffer.
Read 100 bytes into the buffer.
If I seek to 2 * recordSize + 1 or use a 399 byte file, I get the following output.
Read 100 bytes into the buffer.
Read 99 bytes into the buffer.
So it works as expected.
Hint: zero-based array indexes, and zero-based positions ...
First byte will start at position zero.
Seek to the end and print position. Is it as expected?
Print the position after reading the 199 -- is it as expected?
Try to read 1 more byte from the position after you get 199 -- do you get EOF?
How are you checking the size of the file?
Diff the 199 bytes with the expected ones -- what is different?
Two things I would check
CR/LF transformations
That the size is what you think it is.
The problem was that I used a wrapper to BinaryReader object.
When calling the Read method there are some function overloding. Instead os using the signeture of char[], I used byte[]. Till now it worked fine because there was only use of utf-8, but now when I entered real binary data in the beginning of each message it caused the problem.