We Keep Coding

How to read contents of MemoryStream, fed by a cryptostream

I am working on some AES encryption in C#. I have a similar decryption method which functions flawlessly, however, no matter what I try I cannot read the encrypted contents of the MemoryStream
I have tried a few different ways of reading,
ms.Position = 0;
return new StreamReader(ms, Encoding.ASCII).ReadToEnd()
OR
using (StreamReader sr = new StreamReader(cs)) {
return sr.ReadToEnd();
}
OR
byte[] enc = ms.ToArray();
string ret=null;
foreach (byte b in enc) {
ret += b.ToString();
}
Here's the snippet from the code.
using (AesManaged aesMan = new AesManaged()) {
if (keystr.Length == aesSize/8)
{
//Its a valid key
aesMan.KeySize = aesSize;
aesMan.Key = Encoding.UTF8.GetBytes(keystr);
aesMan.IV = Encoding.UTF8.GetBytes(ivstr);
ICryptoTransform encryptor aesMan.CreateEncryptor(aesMan.Key, aesMan.IV);
using (MemoryStream ms = new MemoryStream())
{
using (CryptoStream cs = new CryptoStream(ms, encryptor, CryptoStreamMode.Write))
{
using (StreamWriter sw = new StreamWriter(cs))
{
sw.Write(inpstr);
using (StreamReader sr = new StreamReader(ms)) {
return sr.ReadToEnd();
}
}
}
I get various errors, such as the stream is not readable, or that it cannot read a closed stream, as well as a blank string being returned.
Has anyone got any ideas? I'm at a loss

Most of those stream functions like StreamReader, StreamWriter, etc., close the underlying stream when you Dispose() them. Try not disposing them until you are done, or using the constructors that allow you to chose not to close the underlying stream. In rare cases, I've had to implement a dummy wrapper around a stream to prevent closing the underlying stream

Writing a memory stream to a file

I have tried retrieving data in the json format as a string and writing it to a file and it worked great. Now I am trying to use MemoryStream to do the same thing but nothing gets written to a file - merely [{},{},{},{},{}] without any actual data.
My question is - how can I check if data indeed goes to memory stream correctly or if the problem occurs somewhere else. I do know that myList does contain data.
Here is my code:
MemoryStream ms = new MemoryStream();
DataContractJsonSerializer dcjs = new DataContractJsonSerializer(typeof(List<myClass>));
dcjs.WriteObject(ms, myList);
using (FileStream fs = new FileStream(Path.Combine(Application.StartupPath,"MyFile.json"), FileMode.OpenOrCreate))
{
ms.Position = 0;
ms.Read(ms.ToArray(), 0, (int)ms.Length);
fs.Write(ms.ToArray(), 0, ms.ToArray().Length);
ms.Close();
fs.Flush();
fs.Close();
}

There is a very handy method, Stream.CopyTo(Stream).
using (MemoryStream ms = new MemoryStream())
{
StreamWriter writer = new StreamWriter(ms);
writer.WriteLine("asdasdasasdfasdasd");
writer.Flush();
//You have to rewind the MemoryStream before copying
ms.Seek(0, SeekOrigin.Begin);
using (FileStream fs = new FileStream("output.txt", FileMode.OpenOrCreate))
{
ms.CopyTo(fs);
fs.Flush();
}
}
Also, you don't have to close fs since it's in a using statement and will be disposed at the end.

using (var memoryStream = new MemoryStream())
{
...
var fileName = $"FileName.xlsx";
string tempFilePath = Path.Combine(Path.GetTempPath() + fileName );
using (var fs = new FileStream(tempFilePath, FileMode.Create, FileAccess.Write))
{
memoryStream.WriteTo(fs);
}
}

//reset the position of the stream
ms.Position = 0;
//Then copy to filestream
ms.CopyTo(fileStream);

The issue is nothing to do with your file stream/ memory stream. The problem is that DataContractJsonSerializer is an OPT IN Serializer. You need to add [DataMemberAttribute] to all the properties that you need to serialize on myClass.
[DataContract]
public class myClass
{
[DataMember]
public string Foo { get; set; }
}

This line looks problematic:
ms.Read(ms.ToArray(), 0, (int)ms.Length);
You shouldn't need to read anything into the memory stream at this point, particularly when you're code is written to read ms into ms.
I'm pretty confident that simply removing this line will fix your problem.

SharpZipLib - zero bytes

I am trying to implement SharpZipLib
So I copied the below snippet from their samples:
// Compresses the supplied memory stream, naming it as zipEntryName, into a zip,
// which is returned as a memory stream or a byte array.
//
public MemoryStream CreateToMemoryStream(MemoryStream memStreamIn, string zipEntryName)
{
MemoryStream outputMemStream = new MemoryStream();
ZipOutputStream zipStream = new ZipOutputStream(outputMemStream);
zipStream.SetLevel(3); //0-9, 9 being the highest level of compression
ZipEntry newEntry = new ZipEntry(zipEntryName);
newEntry.DateTime = DateTime.Now;
zipStream.PutNextEntry(newEntry);
StreamUtils.Copy(memStreamIn, zipStream, new byte[4096]);
zipStream.CloseEntry();
zipStream.IsStreamOwner = false; // False stops the Close also Closing the underlying stream.
zipStream.Close(); // Must finish the ZipOutputStream before using outputMemStream.
outputMemStream.Position = 0;
return outputMemStream;
// Alternative outputs:
// ToArray is the cleaner and easiest to use correctly with the penalty of duplicating allocated memory.
byte[] byteArrayOut = outputMemStream.ToArray();
// GetBuffer returns a raw buffer raw and so you need to account for the true length yourself.
byte[] byteArrayOut = outputMemStream.GetBuffer();
long len = outputMemStream.Length;
}
I copy pasted that function and called it this way:
using (MemoryStream ms = new MemoryStream())
using (FileStream file = new FileStream(#"c:\file.jpg", FileMode.Open, FileAccess.Read))
{
byte[] bytes = new byte[file.Length];
file.Read(bytes, 0, (int)file.Length);
ms.Write(bytes, 0, (int)file.Length);
var result = SharpZip.CreateToMemoryStream(ms, "file.jpg");
result.WriteTo(new FileStream(#"c:\myzip.zip", FileMode.Create, System.IO.FileAccess.Write));
}
The myzip.zip is sucessfully created, but the file.jpg inside has zero bytes.
Any ideas?
Thanks a lot

It is necessary to "rewind" the input MemoryStream after writing the input file data:
using (MemoryStream ms = new MemoryStream())
using (FileStream file = File.OpenRead(#"input file path"))
{
byte[] bytes = new byte[file.Length];
file.Read(bytes, 0, (int)file.Length);
ms.Write(bytes, 0, (int)file.Length);
ms.Position = 0; // "Rewind" the stream to the beginning.
var result = SharpZip.CreateToMemoryStream(ms, "file.jpg");
using (var outputStream = File.Create(#"output file path"))
{
result.WriteTo(outputStream);
}
}
Alternative (slightly simplified) version of the implementation:
var bytes = File.ReadAllBytes(#"input file path");
using (MemoryStream ms = new MemoryStream(bytes))
{
var result = SharpZip.CreateToMemoryStream(ms, "file.jpg");
using (var outputStream = File.Create(#"output file path"))
{
result.WriteTo(outputStream);
}
}

protobuf-net returns null when calling Deserialize

My end goal is to use protobuf-net and GZipStream in an attempt to compress a List<MyCustomType> object to store in a varbinary(max) field in SQL Server. I'm working on unit tests to understand how everything works and fits together.
Target .NET framework is 3.5.
My current process is:
Serialize the data with protobuf-net (good).
Compress the serialized data from #1 with GZipStream (good).
Convert the compressed data to a base64 string (good).
At this point, the value from step #3 will be stored in a varbinary(max) field. I have no control over this. The steps resume with needing to take a base64 string and deserialize it to a concrete type.
Convert a base 64 string to a byte[] (good).
Decompress the data with GZipStream (good).
Deserialize the data with protobuf-net (bad).
Can someone assist with why the call to Serializer.Deserialize<string> returns null? I'm stuck on this one and hopefully a fresh set of eyes will help.
FWIW, I tried another version of this using List<T> where T is a custom class I created and I Deserialize<> still returns null.
FWIW 2, data.txt is a 4MB plaintext file residing on my C:.
[Test]
public void ForStackOverflow()
{
string data = "hi, my name is...";
//string data = File.ReadAllText(#"C:\Temp\data.txt");
string serializedBase64;
using (MemoryStream protobuf = new MemoryStream())
{
Serializer.Serialize(protobuf, data);
using (MemoryStream compressed = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(compressed, CompressionMode.Compress))
{
byte[] s = protobuf.ToArray();
gzip.Write(s, 0, s.Length);
gzip.Close();
}
serializedBase64 = Convert.ToBase64String(compressed.ToArray());
}
}
byte[] base64byteArray = Convert.FromBase64String(serializedBase64);
using (MemoryStream base64Stream = new MemoryStream(base64byteArray))
{
using (GZipStream gzip = new GZipStream(base64Stream, CompressionMode.Decompress))
{
using (MemoryStream plainText = new MemoryStream())
{
byte[] buffer = new byte[4096];
int read;
while ((read = gzip.Read(buffer, 0, buffer.Length)) > 0)
{
plainText.Write(buffer, 0, read);
}
// why does this call to Deserialize return null?
string deserialized = Serializer.Deserialize<string>(plainText);
Assert.IsNotNull(deserialized);
Assert.AreEqual(data, deserialized);
}
}
}
}

Because you didn't rewind plainText after writing to it. Actually, that entire Stream is unnecessary - this works:
using (MemoryStream base64Stream = new MemoryStream(base64byteArray))
{
using (GZipStream gzip = new GZipStream(
base64Stream, CompressionMode.Decompress))
{
string deserialized = Serializer.Deserialize<string>(gzip);
Assert.IsNotNull(deserialized);
Assert.AreEqual(data, deserialized);
}
}
Likewise, this should work for the serialize:
using (MemoryStream compressed = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(
compressed, CompressionMode.Compress, true))
{
Serializer.Serialize(gzip, data);
}
serializedBase64 = Convert.ToBase64String(
compressed.GetBuffer(), 0, (int)compressed.Length);
}

End of Stream encountered before parsing was completed?

I am trying to deserialize a stream but I always get this error "End of Stream encountered before parsing was completed"?
Here is the code:
//Some code here
BinaryFormatter b = new BinaryFormatter();
return (myObject)b.Deserialize(s);//s---> is a Stream object that has been fill up with data some line over here
Any one have ideas?

Try to set the position to 0 of your stream and do not use your object but the object type.
BinaryFormatter b = new BinaryFormatter();
s.Position = 0;
return (YourObjectType)b.Deserialize(s);

Make sure the serialization completed, and that the serialization type matches the de-serialization type (i.e., make sure you're serializing with a BinaryFormatter if you're de-serializing with one). Also, make sure that the stream you serialized to really finished serializing, with a Stream.Flush() or something to that effect.

I had the same exception thrown, until I added the [Serializable] tag to the class I was Serializing :)
Then it all worked perfectly.

In my case I used:
stream.Seek(0, SeekOrigin.Begin);
after i serialized the stream, and before i deserialized the stream works charm. hope this helps!

I have spent 5 hourse and have got end of stream error and lost data (Not obvious feature in GzipStream: you should use underlying stream only after flush GzipStream).
Full example of working code:
using System;
using System.IO;
using System.IO.Compression;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
namespace ConsoleApp3
{
class Program
{
static void Main(string[] args)
{
string large = LargeJsonContent.GetBigObject();
string base64;
using (var readStream = new MemoryStream())
using (var writeStream = new MemoryStream())
{
using (GZipStream compressor = new GZipStream(writeStream, CompressionMode.Compress, true)) //pay attention to leaveOpen = true
{
var formatter = new BinaryFormatter();
formatter.Serialize(readStream, large);
Console.WriteLine($"After binary serialization of JsonString: {readStream.Length} bytes");
readStream.Position = 0;
readStream.CopyTo(compressor);
}
Console.WriteLine($"Compressed stream size: {writeStream.Length} bytes");
writeStream.Position = 0;
byte[] writeBytes = writeStream.ToArray();
base64 = Convert.ToBase64String(writeBytes);
}
////
using (var stream = new MemoryStream())
{
var formatter = new BinaryFormatter();
formatter.Serialize(stream, base64);
Console.WriteLine($"Size of base64: {stream.Length} bytes");
}
Console.WriteLine("---------------------");
////
string large2;
var bytes = Convert.FromBase64String(base64);
using (var readStream = new MemoryStream())
{
readStream.Write(bytes, 0, bytes.Length);
readStream.Position = 0;
Console.WriteLine($"Compressed stream size: {readStream.Length} bytes");
using (var writeStream = new MemoryStream())
{
using (GZipStream decompressor = new GZipStream(readStream, CompressionMode.Decompress, true)) //pay attention to leaveOpen = true
{
decompressor.CopyTo(writeStream);
writeStream.Position = 0;
}
var formatter = new BinaryFormatter();
large2 = (string)formatter.Deserialize(writeStream);
}
}
Console.WriteLine(large == large2);
Console.WriteLine($"large:{large.Length} | large2:{large2.Length}");
}
}
}

Check in your sender code if you are not doing the following
NetworkStream strm = client.GetStream(); // the stream
formatter.Serialize(strm, status); // the serialization process
strm.Close();// Remove this code, this was the culprit in my case

the class which you created must has [Serializable].