We Keep Coding

Convert Stream to byte[] c# for large files of 2GB

Trying to convert Stream object to byte[] and using the below method for the same:
public static byte[] ReadFully(System.IO.Stream input)
{
byte[] buffer = new byte[16*1024];
using (System.IO.MemoryStream ms = new System.IO.MemoryStream())
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
ms.Write(buffer, 0, read);
}
return ms.ToArray();
}
}
However the input parameter "input" is for large file that is of 2 GB and hence the code does not get enter into while loop and hence does not convert it to byte array.
For smaller files it is working fine

That's what a Stream is for.
You don't load the whole content into a byte[], you read a small buffer from the Stream into memory and handle it, then dispose and read the next buffer.
If you still need to use a byte[]:
It seems like your app can't handle more than 2^32 Bytes Memory, meaning it's 32bit.
Try changing it to 64bit (in Project Properties go to Build and disable Prefer 32 bit)

Object limited below 32bit. (that is why all index using int)
how about use list contains byte array to deal entire data?
public List<byte[]> ReadBytesList(string fileName)
{
List<byte[]> rawDataBytes= new List<byte[]>();
byte[] buff;
FileStream fs = new FileStream(fileName,
FileMode.Open,
FileAccess.Read);
BinaryReader br = new BinaryReader(fs);
long numBytes = new FileInfo(fileName).Length;
int arrayCount= (int)(numBytes / 2100000000); //2147483648 is max
int arrayRest = (int)(numBytes % 2100000000);
if(arrayCount>0)
{
for (int i = 0; i < arrayCount; i++)
{
buff = br.ReadBytes(2100000000);
rawDataBytes.Add(buff);
}
buff = br.ReadBytes(arrayRest);
rawDataBytes.Add(buff);
}
else
{
buff = br.ReadBytes(arrayRest);
rawDataBytes.Add(buff);
}
return rawDataBytes;
}

Error decompressing gzipstream -- The magic number in GZip header is not correct

I'm using the C# System.IO class (framework 4.0) to compress an image pulled off the file system with FileDialog and then inserted into a SQL Server database as a varbinary(max) data type. The problem I'm having is when I pull the data out of the database and attempt to decompress I get the subject error with an additional message -- make sure you are passing in a gzip stream.
The code to get the file:
OpenFileDialog dlgOpen = new OpenFileDialog();
if (dlgOpen.ShowDialog() == DialogResult.OK)
{
FileStream fs = File.OpenRead(dlgOpen.FileName);
byte[] picbyte1 = new byte[fs.Length];
byte[] picbyte = Compress(picbyte1);
fs.Read(picbyte, 0, System.Convert.ToInt32(picbyte.Length));
String ImageName = dlgOpen.FileName;
//String bs64OfBytes = Convert.ToBase64String(picbyte);
fs.Close();
//additional code inserts into database
....
}
The compress method:
private static byte[] Compress(byte[] data)
{
var output = new MemoryStream();
using (var gzip = new GZipStream(output, CompressionMode.Compress, true))
{
gzip.Write(data, 0, data.Length);
gzip.Close();
}
return output.ToArray();
}
The decompress method:
private static byte[] Decompress(byte[] data)
{
var output = new MemoryStream();
var input = new MemoryStream();
input.Write(data, 0, data.Length);
input.Position = 0;
using (var gzip = new GZipStream(input, CompressionMode.Decompress, true))
{
var buff = new byte[64];//also used 32
var read = gzip.Read(buff, 0, buff.Length);//error occurs here
while (read > 0)
{
output.Write(buff, 0, read);
read = gzip.Read(buff, 0, buff.Length);
}
gzip.Close();
}
return output.ToArray();
}

You need to insert a line and remove an other:
FileStream fs = File.OpenRead(dlgOpen.FileName);
byte[] picbyte1 = new byte[fs.Length];
fs.Read(picbyte1, 0, (int)fs.Length); // <-- Add this one
byte[] picbyte = Compress(picbyte1);
// fs.Read(picbyte, 0, System.Convert.ToInt32(picbyte.Length)); // <-- And remove this one
// ...
You are reading the image in your code, but something is in the wrong order:
// Original but incorrect sequence
FileStream fs = File.OpenRead(dlgOpen.FileName); // Open the file
byte[] picbyte1 = new byte[fs.Length]; // Assign the array
byte[] picbyte = Compress(picbyte1); // Compress the assigned array, but there is no contents...
fs.Read(picbyte, 0, System.Convert.ToInt32(picbyte.Length)); // You save the file to the already compressed bytes...
So you have saved the first part of the original file, not the compressed one (but the number of saved bytes corresponds with the number of compressed bytes). If you send this to the DB and read it back, the decompressor does not find it's Magic Number.
As an improvement, you can change these lines:
FileStream fs = File.OpenRead(dlgOpen.FileName);
byte[] picbyte1 = new byte[fs.Length];
fs.Read(picbyte1, 0, (int)fs.Length); // line that I suggested to add
probably change to:
byte[] picbyte1 = File.ReadAllBytes(dlgOpen.FileName);

how to buffer an input stream until it is complete

i'm implementing a wcf service that accepts image streams. however i'm currently getting an exception when i run it. as its trying to get the length of the stream before the stream is complete. so what i'd like to do is buffer the stream until its complete. however i cant find any examples of how to do this...
can anyone help?
my code so far:
public String uploadUserImage(Stream stream)
{
Stream fs = stream;
BinaryReader br = new BinaryReader(fs);
Byte[] bytes = br.ReadBytes((Int32)fs.Length);// this causes exception
File.WriteAllBytes(filepath, bytes);
}

Rather than try to fetch the length, you should read from the stream until it returns that it's "done". In .NET 4, this is really easy:
// Assuming we *really* want to read it into memory first...
MemoryStream memoryStream = new MemoryStream();
stream.CopyTo(memoryStream);
memoryStream.Position = 0;
File.WriteAllBytes(filepath, memoryStream);
In .NET 3.5 there's no CopyTo method, but you can write something similar yourself:
public static void CopyStream(Stream input, Stream output)
{
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
However, now we've got something to copy a stream, why bother reading it all into memory first? Let's just write it straight to a file:
using (FileStream output = File.OpenWrite(filepath))
{
CopyStream(stream, output); // Or stream.CopyTo(output);
}

I'm not sure what you are returning (or not returning), but something like this might work for you:
public String uploadUserImage(Stream stream) {
const int KB = 1024;
Byte[] bytes = new Byte[KB];
StringBuilder sb = new StringBuilder();
using (BinaryReader br = new BinaryReader(stream)) {
int len;
do {
len = br.Read(bytes, 0, KB);
string readData = Encoding.UTF8.GetString(bytes);
sb.Append(readData);
} while (len == KB);
}
//File.WriteAllBytes(filepath, bytes);
return sb.ToString();
}
A string can hold up to 2 GB, I believe.

Try this :
using (StreamWriter sw = File.CreateText(filepath))
{
stream.CopyTo(sw);
sw.Close();
}

Jon Skeets answer for .Net 3.5 and below using a Buffer Read is actually done incorrectly.
The buffer isn't cleared between reads which can result in issues on any read that returns less than 8192, for example if the 2nd read, read 192 bytes, the 8000 last bytes from the first read would STILL be in the buffer which would then be returned to the stream.
My code below you supply it a Stream and it will return a IEnumerable array.
Using this you can for-each it and Write to a MemoryStream and then use .GetBuffer() to end up with a compiled merged byte[].
private IEnumerable<byte[]> ReadFullStream(Stream stream) {
while(true) {
byte[] buffer = new byte[8192];//since this is created every loop, its buffer is cleared
int bytesRead = stream.Read(buffer, 0, buffer.Length);//read up to 8192 bytes into buffer
if (bytesRead == 0) {//if we read nothing, stream is finished
break;
}
if(bytesRead < buffer.Length) {//if we read LESS than 8192 bytes, resize the buffer to essentially remove everything after what was read, otherwise you will have nullbytes/0x00bytes at the end of your buffer
Array.Resize(ref buffer, bytesRead);
}
yield return buffer;//yield return the buffer data
}//loop here until we reach a read == 0 (end of stream)
}

Most efficient way of reading data from a stream

I have an algorithm for encrypting and decrypting data using symmetric encryption. anyways when I am about to decrypt, I have:
CryptoStream cs = new CryptoStream(ms, cryptoTransform, CryptoStreamMode.Read);
I have to read data from the cs CryptoStream and place that data into a array of bytes. So one method could be:
System.Collections.Generic.List<byte> myListOfBytes = new System.Collections.Generic.List<byte>();
while (true)
{
int nextByte = cs.ReadByte();
if (nextByte == -1) break;
myListOfBytes.Add((Byte)nextByte);
}
return myListOfBytes.ToArray();
another technique could be:
ArrayList chuncks = new ArrayList();
byte[] tempContainer = new byte[1048576];
int tempBytes = 0;
while (tempBytes < 1048576)
{
tempBytes = cs.Read(tempContainer, 0, tempContainer.Length);
//tempBytes is the number of bytes read from cs stream. those bytes are placed
// on the tempContainer array
chuncks.Add(tempContainer);
}
// later do a for each loop on chunks and add those bytes
I cannot know in advance the length of the stream cs:
or perhaps I should implement my stack class. I will be encrypting a lot of information therefore making this code efficient will save a lot of time

You could read in chunks:
using (var stream = new MemoryStream())
{
byte[] buffer = new byte[2048]; // read in chunks of 2KB
int bytesRead;
while((bytesRead = cs.Read(buffer, 0, buffer.Length)) > 0)
{
stream.Write(buffer, 0, bytesRead);
}
byte[] result = stream.ToArray();
// TODO: do something with the result
}

Since you are storing everything in memory anyway you can just use a MemoryStream and CopyTo():
using (MemoryStream ms = new MemoryStream())
{
cs.CopyTo(ms);
return ms.ToArray();
}
CopyTo() will require .NET 4

Problem with C# Decompression

Have some data in a sybase image type column that I want to use in a C# app. The data has been compressed by Java using the java.util.zip package. I wanted to test that I could decompress the data in C#. So I wrote a test app that pulls it out of the database:
byte[] bytes = (byte[])reader.GetValue(0);
This gives me a compressed byte[] of 2479 length.
Then I pass this to a seemingly standard C# decompression method:
public static byte[] Decompress(byte[] gzBuffer)
{
MemoryStream ms = new MemoryStream();
int msgLength = BitConverter.ToInt32(gzBuffer, 0);
ms.Write(gzBuffer, 4, gzBuffer.Length - 4);
byte[] buffer = new byte[msgLength];
ms.Position = 0;
GZipStream zip = new GZipStream(ms, CompressionMode.Decompress);
zip.Read(buffer, 0, buffer.Length);
return buffer;
}
The value for msgLength is 1503501432 which seems way out of range. The original document should be in the range of 5K -50k. Anyway when I use that value to create "buffer" not surprisingly I get an OutOfMemoryException.
What is happening?
Jim
The Java compress method is as follows:
public byte[] compress(byte[] bytes) throws Exception {
byte[] results = new byte[bytes.length];
Deflater deflator = new Deflater();
deflater.setInput(bytes);
deflater.finish();
int len = deflater.deflate(results);
byte[] out = new byte[len];
for(int i=0; i<len; i++) {
out[i] = results[i];
}
return(out);
}

As I cant see your java code, I can only guess you are compressing your data to a zip file stream. Therefore it will obviously fail if you are trying to decompress that stream with a gzip decompression in c#. Either you change your java code to a gzip compression (Example here at the bottom of the page), or you decompress the zip file stream in c# with an appropriate library (e.g. SharpZipLib).
Update
Ok now, I see you are using deflate for the compression in java. So, obviously you have to use the same algorithm in c#: System.IO.Compression.DeflateStream
public static byte[] Decompress(byte[] buffer)
{
using (MemoryStream ms = new MemoryStream(buffer))
using (Stream zipStream = new DeflateStream(ms,
CompressionMode.Decompress, true))
{
int initialBufferLength = buffer.Length * 2;
byte[] buffer = new byte[initialBufferLength];
bool finishedExactly = false;
int read = 0;
int chunk;
while (!finishedExactly &&
(chunk = zipStream.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
if (read == buffer.Length)
{
int nextByte = zipStream.ReadByte();
// End of Stream?
if (nextByte == -1)
{
finishedExactly = true;
}
else
{
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
}
if (!finishedExactly)
{
byte[] final = new byte[read];
Array.Copy(buffer, final, read);
buffer = final;
}
}
return buffer;
}