We Keep Coding

Better way to read data from stream? [duplicate]

I have an algorithm for encrypting and decrypting data using symmetric encryption. anyways when I am about to decrypt, I have:
CryptoStream cs = new CryptoStream(ms, cryptoTransform, CryptoStreamMode.Read);
I have to read data from the cs CryptoStream and place that data into a array of bytes. So one method could be:
System.Collections.Generic.List<byte> myListOfBytes = new System.Collections.Generic.List<byte>();
while (true)
{
int nextByte = cs.ReadByte();
if (nextByte == -1) break;
myListOfBytes.Add((Byte)nextByte);
}
return myListOfBytes.ToArray();
another technique could be:
ArrayList chuncks = new ArrayList();
byte[] tempContainer = new byte[1048576];
int tempBytes = 0;
while (tempBytes < 1048576)
{
tempBytes = cs.Read(tempContainer, 0, tempContainer.Length);
//tempBytes is the number of bytes read from cs stream. those bytes are placed
// on the tempContainer array
chuncks.Add(tempContainer);
}
// later do a for each loop on chunks and add those bytes
I cannot know in advance the length of the stream cs:
or perhaps I should implement my stack class. I will be encrypting a lot of information therefore making this code efficient will save a lot of time

You could read in chunks:
using (var stream = new MemoryStream())
{
byte[] buffer = new byte[2048]; // read in chunks of 2KB
int bytesRead;
while((bytesRead = cs.Read(buffer, 0, buffer.Length)) > 0)
{
stream.Write(buffer, 0, bytesRead);
}
byte[] result = stream.ToArray();
// TODO: do something with the result
}

Since you are storing everything in memory anyway you can just use a MemoryStream and CopyTo():
using (MemoryStream ms = new MemoryStream())
{
cs.CopyTo(ms);
return ms.ToArray();
}
CopyTo() will require .NET 4

ushort array compression in C#

I've got a ushort array (actually an image coming from a camera) that I'd like to lossless compress before persistent storage. I'm using the GZipStream function provided in System.IO.Compression.GZipStream. This approach, to my knowledge, requires that I convert the ushort array to a byte array. My solution appears to function properly, but just isn't as quick as I'd like. The images are about 2 Mbytes in raw size, and the compress time ranges (on my slow machine) 200 - 400 msecs, and decompress time ranges 100 - 200 msecs. Looking for suggestions for improving my performance.
public static class Zip
{
public static ushort[] Decompress_ByteToShort(byte[] zippedData)
{
byte[] decompressedData = null;
using (MemoryStream outputStream = new MemoryStream())
{
using (MemoryStream inputStream = new MemoryStream(zippedData))
{
using (GZipStream zip = new GZipStream(inputStream, CompressionMode.Decompress))
{
zip.CopyTo(outputStream);
}
}
decompressedData = outputStream.ToArray();
}
ushort[] decompressShort = new ushort[decompressedData.Length / sizeof(ushort)];
Buffer.BlockCopy(decompressedData, 0, decompressShort, 0, decompressedData.Length);
return decompressShort;
}
public static byte[] Compress_ShortToByte(ushort[] plainData)
{
byte[] compressesData = null;
byte[] uncompressedData = new byte[plainData.Length * sizeof(ushort)];
Buffer.BlockCopy(plainData, 0, uncompressedData, 0, plainData.Length * sizeof(ushort));
using (MemoryStream outputStream = new MemoryStream())
{
using (GZipStream zip = new GZipStream(outputStream, CompressionMode.Compress))
{
zip.Write(uncompressedData, 0, uncompressedData.Length);
}
//Dont get the MemoryStream data before the GZipStream is closed
//since it doesn’t yet contain complete compressed data.
//GZipStream writes additional data including footer information when its been disposed
compressesData = outputStream.ToArray();
}
return compressesData;
}
}

The first problem in your approach I see is that you are using byte arrays instead of direcly loading and writing to files.
Using a smaller temporary buffer and reading\writing to streams and files directly in chunks should be much faster.
Here I propose some functions and overloads you can use to decompress from byte arrays, to byte arrays, from stream, to stream, from file and to file.
The performance improvement should be from 10% to 20%.
Try to adjust the constants as needed.
I used DeflateStream instead of GZipStream, this increases the performance a bit.
You can go back to a GZipStream if you prefer.
I tried just the byte to ushort and ushort to byte[] version of the code and it is about 10% faster.
Accessing directly to files instead of loading it to a big buffer should increase the performance even more.
WARNING: This approach of reading and writing images in this way is not little-endian/big-endian agnostic - it means that a file saved from a Intel/AMD machine is not compatible with an ARM machine, for example in some tablets! Just as a side note :)
/// <summary>The average file size, used to preallocate the right amount of memory for compression.</summary>
private const int AverageFileSize = 100000;
/// <summary>The default size of the buffer used to convert data. WARNING: Must be a multiple of 2!</summary>
private const int BufferSize = 32768;
/// <summary>Decompresses a byte array to unsigned shorts.</summary>
public static ushort[] Decompress_ByteToShort(byte[] zippedData)
{
using (var inputStream = new MemoryStream(zippedData))
return Decompress_File(inputStream);
}
/// <summary>Decompresses a file to unsigned shorts.</summary>
public static ushort[] Decompress_File(string inputFilePath)
{
using (var stream = new FileStream(inputFilePath, FileMode.Open, FileAccess.Read))
return Decompress_File(stream);
}
/// <summary>Decompresses a file stream to unsigned shorts.</summary>
public static ushort[] Decompress_File(Stream zippedData)
{
using (var zip = new DeflateStream(zippedData, CompressionMode.Decompress, true))
{
// Our temporary buffer.
var buffer = new byte[BufferSize];
// Read the number of bytes, written initially as header in the file.
zip.Read(buffer, 0, sizeof(int));
var resultLength = BitConverter.ToInt32(buffer, 0);
// Creates the result array
var result = new ushort[resultLength];
// Decompress the file chunk by chunk
var resultOffset = 0;
for (; ; )
{
// Read a chunk of data
var count = zip.Read(buffer, 0, BufferSize);
if (count <= 0)
break;
// Copy a piece of the decompressed buffer
Buffer.BlockCopy(buffer, 0, result, resultOffset, count);
// Advance counter
resultOffset += count;
}
return result;
}
}
/// <summary>Compresses an ushort array to a file array.</summary>
public static byte[] Compress_ShortToByte(ushort[] plainData)
{
using (var outputStream = new MemoryStream(AverageFileSize))
{
Compress_File(plainData, outputStream);
return outputStream.ToArray();
}
}
/// <summary>Compresses an ushort array directly to a file.</summary>
public static void Compress_File(ushort[] plainData, string outputFilePath)
{
using (var stream = new FileStream(outputFilePath, FileMode.OpenOrCreate, FileAccess.Write))
Compress_File(plainData, stream);
}
/// <summary>Compresses an ushort array directly to a file stream.</summary>
public static void Compress_File(ushort[] plainData, Stream outputStream)
{
using (var zip = new DeflateStream(outputStream, CompressionMode.Compress, true))
{
// Our temporary buffer.
var buffer = new byte[BufferSize];
// Writes the length of the plain data
zip.Write(BitConverter.GetBytes(plainData.Length), 0, sizeof(int));
var inputOffset = 0;
var availableBytes = plainData.Length * sizeof(ushort);
while (availableBytes > 0)
{
// Compute the amount of bytes to copy.
var bytesCount = Math.Min(BufferSize, availableBytes);
// Copy a chunk of plain data into the temporary buffer
Buffer.BlockCopy(plainData, inputOffset, buffer, 0, bytesCount);
// Write the buffer
zip.Write(buffer, 0, bytesCount);
// Advance counters
inputOffset += bytesCount;
availableBytes -= bytesCount;
}
}
}

There is GZipStream constructor (Stream, CompressionLevel) you can change CompressionLevel to speed up compression there is level which say Fastest in this enumeration.
Links to relevant documentation:
http://msdn.microsoft.com/pl-pl/library/hh137341(v=vs.110).aspx
http://msdn.microsoft.com/pl-pl/library/system.io.compression.compressionlevel(v=vs.110).aspx

System.OutOfMemory exception when trying to read large files

public static byte[] ReadMemoryMappedFile(string fileName)
{
long length = new FileInfo(fileName).Length;
using (var stream = File.Open(fileName, FileMode.OpenOrCreate, FileAccess.Read, FileShare.ReadWrite))
{
using (var mmf = MemoryMappedFile.CreateFromFile(stream, null, length, MemoryMappedFileAccess.Read, null, HandleInheritability.Inheritable, false))
{
using (var viewStream = mmf.CreateViewStream(0, length, MemoryMappedFileAccess.Read))
{
using (BinaryReader binReader = new BinaryReader(viewStream))
{
var result = binReader.ReadBytes((int)length);
return result;
}
}
}
}
}
OpenFileDialog openfile = new OpenFileDialog();
openfile.Filter = "All Files (*.*)|*.*";
openfile.ShowDialog();
byte[] buff = ReadMemoryMappedFile(openfile.FileName);
texteditor.Text = BitConverter.ToString(buff).Replace("-"," "); <----A first chance exception of type 'System.OutOfMemoryException' occurred in mscorlib.dll
I get a System.OutOfMemory exception when trying to read large files.
I've read a lot for 4 weeks in all the web... and tried a lot!!! But still, I can't seem to find a good solution to my problem.
Please help me..
Update
public byte[] FileToByteArray(string fileName)
{
byte[] buff = null;
FileStream fs = new FileStream(fileName,
FileMode.Open,
FileAccess.Read);
BinaryReader br = new BinaryReader(fs);
long numBytes = new FileInfo(fileName).Length;
buff = br.ReadBytes((int)numBytes);
//return buff;
return File.ReadAllBytes(fileName);
}
OR
public static byte[] FileToByteArray(FileStream stream, int initialLength)
{
// If we've been passed an unhelpful initial length, just
// use 32K.
if (initialLength < 1)
{
initialLength = 32768;
}
BinaryReader br = new BinaryReader(stream);
byte[] buffer = new byte[initialLength];
int read = 0;
int chunk;
while ((chunk = br.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
// If we've reached the end of our buffer, check to see if there's
// any more information
if (read == buffer.Length)
{
int nextByte = br.ReadByte();
// End of stream? If so, we're done
if (nextByte == -1)
{
return buffer;
}
// Nope. Resize the buffer, put in the byte we've just
// read, and continue
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
// Buffer is now too big. Shrink it.
byte[] ret = new byte[read];
Array.Copy(buffer, ret, read);
return ret;
}
I still get a System.OutOfMemory exception when trying to read large files.

If your file is 4GB, then BitConverter will turn each byte into XX- string, each char in string is 2 bytes * 3 chars per byte * 4 294 967 295 bytes = 25 769 803 770. You need +25Gb of free memory to fit entire string, plus you already have your file in memory as byte array.
Besides, no single object in a .Net program may be over 2GB. Theoretical limit for a string length would be 1,073,741,823 chars, but you also need to have a 64-bit process.
So solution in your case - open FileStream. Read first 16384 bytes (or how much can fit on your screen), convert to hex and display, and remember file offset. When user wants to navigate to next or previous page - seek to that position in file on disk, read and display again, etc.

You need to read the file in chunks, keep track of where you are in the file, page the contents on screen and use seek and position to move up and down in the file stream.

You will not be able to display 4Gb file reading all of it in memory first by any approach.
The approach is to virtualize the data, reading only the visible lines when user scrolls. If you need to do a read-only text viewer then you can use WPF ItemsControl with virtulizing stack panel and bind to custom IList collection which will lazily fetch lines from the file calculating file offset by for the line index.

how to buffer an input stream until it is complete

i'm implementing a wcf service that accepts image streams. however i'm currently getting an exception when i run it. as its trying to get the length of the stream before the stream is complete. so what i'd like to do is buffer the stream until its complete. however i cant find any examples of how to do this...
can anyone help?
my code so far:
public String uploadUserImage(Stream stream)
{
Stream fs = stream;
BinaryReader br = new BinaryReader(fs);
Byte[] bytes = br.ReadBytes((Int32)fs.Length);// this causes exception
File.WriteAllBytes(filepath, bytes);
}

Rather than try to fetch the length, you should read from the stream until it returns that it's "done". In .NET 4, this is really easy:
// Assuming we *really* want to read it into memory first...
MemoryStream memoryStream = new MemoryStream();
stream.CopyTo(memoryStream);
memoryStream.Position = 0;
File.WriteAllBytes(filepath, memoryStream);
In .NET 3.5 there's no CopyTo method, but you can write something similar yourself:
public static void CopyStream(Stream input, Stream output)
{
byte[] buffer = new byte[8192];
int bytesRead;
while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, bytesRead);
}
}
However, now we've got something to copy a stream, why bother reading it all into memory first? Let's just write it straight to a file:
using (FileStream output = File.OpenWrite(filepath))
{
CopyStream(stream, output); // Or stream.CopyTo(output);
}

I'm not sure what you are returning (or not returning), but something like this might work for you:
public String uploadUserImage(Stream stream) {
const int KB = 1024;
Byte[] bytes = new Byte[KB];
StringBuilder sb = new StringBuilder();
using (BinaryReader br = new BinaryReader(stream)) {
int len;
do {
len = br.Read(bytes, 0, KB);
string readData = Encoding.UTF8.GetString(bytes);
sb.Append(readData);
} while (len == KB);
}
//File.WriteAllBytes(filepath, bytes);
return sb.ToString();
}
A string can hold up to 2 GB, I believe.

Try this :
using (StreamWriter sw = File.CreateText(filepath))
{
stream.CopyTo(sw);
sw.Close();
}

Jon Skeets answer for .Net 3.5 and below using a Buffer Read is actually done incorrectly.
The buffer isn't cleared between reads which can result in issues on any read that returns less than 8192, for example if the 2nd read, read 192 bytes, the 8000 last bytes from the first read would STILL be in the buffer which would then be returned to the stream.
My code below you supply it a Stream and it will return a IEnumerable array.
Using this you can for-each it and Write to a MemoryStream and then use .GetBuffer() to end up with a compiled merged byte[].
private IEnumerable<byte[]> ReadFullStream(Stream stream) {
while(true) {
byte[] buffer = new byte[8192];//since this is created every loop, its buffer is cleared
int bytesRead = stream.Read(buffer, 0, buffer.Length);//read up to 8192 bytes into buffer
if (bytesRead == 0) {//if we read nothing, stream is finished
break;
}
if(bytesRead < buffer.Length) {//if we read LESS than 8192 bytes, resize the buffer to essentially remove everything after what was read, otherwise you will have nullbytes/0x00bytes at the end of your buffer
Array.Resize(ref buffer, bytesRead);
}
yield return buffer;//yield return the buffer data
}//loop here until we reach a read == 0 (end of stream)
}

Problem with C# Decompression

Have some data in a sybase image type column that I want to use in a C# app. The data has been compressed by Java using the java.util.zip package. I wanted to test that I could decompress the data in C#. So I wrote a test app that pulls it out of the database:
byte[] bytes = (byte[])reader.GetValue(0);
This gives me a compressed byte[] of 2479 length.
Then I pass this to a seemingly standard C# decompression method:
public static byte[] Decompress(byte[] gzBuffer)
{
MemoryStream ms = new MemoryStream();
int msgLength = BitConverter.ToInt32(gzBuffer, 0);
ms.Write(gzBuffer, 4, gzBuffer.Length - 4);
byte[] buffer = new byte[msgLength];
ms.Position = 0;
GZipStream zip = new GZipStream(ms, CompressionMode.Decompress);
zip.Read(buffer, 0, buffer.Length);
return buffer;
}
The value for msgLength is 1503501432 which seems way out of range. The original document should be in the range of 5K -50k. Anyway when I use that value to create "buffer" not surprisingly I get an OutOfMemoryException.
What is happening?
Jim
The Java compress method is as follows:
public byte[] compress(byte[] bytes) throws Exception {
byte[] results = new byte[bytes.length];
Deflater deflator = new Deflater();
deflater.setInput(bytes);
deflater.finish();
int len = deflater.deflate(results);
byte[] out = new byte[len];
for(int i=0; i<len; i++) {
out[i] = results[i];
}
return(out);
}

As I cant see your java code, I can only guess you are compressing your data to a zip file stream. Therefore it will obviously fail if you are trying to decompress that stream with a gzip decompression in c#. Either you change your java code to a gzip compression (Example here at the bottom of the page), or you decompress the zip file stream in c# with an appropriate library (e.g. SharpZipLib).
Update
Ok now, I see you are using deflate for the compression in java. So, obviously you have to use the same algorithm in c#: System.IO.Compression.DeflateStream
public static byte[] Decompress(byte[] buffer)
{
using (MemoryStream ms = new MemoryStream(buffer))
using (Stream zipStream = new DeflateStream(ms,
CompressionMode.Decompress, true))
{
int initialBufferLength = buffer.Length * 2;
byte[] buffer = new byte[initialBufferLength];
bool finishedExactly = false;
int read = 0;
int chunk;
while (!finishedExactly &&
(chunk = zipStream.Read(buffer, read, buffer.Length - read)) > 0)
{
read += chunk;
if (read == buffer.Length)
{
int nextByte = zipStream.ReadByte();
// End of Stream?
if (nextByte == -1)
{
finishedExactly = true;
}
else
{
byte[] newBuffer = new byte[buffer.Length * 2];
Array.Copy(buffer, newBuffer, buffer.Length);
newBuffer[read] = (byte)nextByte;
buffer = newBuffer;
read++;
}
}
}
if (!finishedExactly)
{
byte[] final = new byte[read];
Array.Copy(buffer, final, read);
buffer = final;
}
}
return buffer;
}