We Keep Coding

Value already read, or no value when trying to read from a Stream

I've been trying this for a long time but it keeps giving me an error. I have an array of bytes that should represent a nbt document. I would like to convert this into a c# object with a library: fNbt.
Here is my code:
byte[] buffer = Convert.FromBase64String(value);
byte[] decompressed;
using (var inputStream = new MemoryStream(buffer))
{
using var outputStream = new MemoryStream();
using (var gzip = new GZipStream(inputStream, CompressionMode.Decompress, leaveOpen: true))
{
gzip.CopyTo(outputStream);
}
fNbt.NbtReader reader = new fNbt.NbtReader(outputStream, true);
var output = reader.ReadValueAs<AuctionItem>(); //Error: Value already read, or no value to read.
return output;
}
When I try this, it works:
decompressed = outputStream.ToArray();
outputStream.Seek(0, SeekOrigin.Begin);
outputStream.Read(new byte[1000], 0, decompressed.Count() - 1);
But when I try this, it doesn't:
outputStream.Seek(0, SeekOrigin.Begin);
fNbt.NbtReader reader = new fNbt.NbtReader(outputStream, true);
reader.ReadValueAs<AuctionItem>();

NbtReader, like most stream readers, begins reading from the current position of whatever stream you give it. Since you're just done writing to outputStream, then that position is the stream's end. Which means at that point there's nothing to be read.
The solution is to seek the outputStream back to the beginning before reading from it:
outputStream.Seek(0, SeekOrigin.Begin); // <-- seek to the beginning
// Do the read
fNbt.NbtReader reader = new fNbt.NbtReader(outputStream, true);
var output = reader.ReadValueAs<AuctionItem>(); // No error anymore
return output;

The solution is as follows. NbtReader.ReadValueAs does not consider a nbtCompound or nbtList as value. I made this little reader but it is not done yet (I will update the code once it is done).
public static T ReadValueAs<T>(string value) where T: new()
{
byte[] buffer = Convert.FromBase64String(value);
using (var inputStream = new MemoryStream(buffer))
{
using var outputStream = new MemoryStream();
using (var gzip = new GZipStream(inputStream, CompressionMode.Decompress, leaveOpen: true))
{
gzip.CopyTo(outputStream);
}
outputStream.Seek(0, SeekOrigin.Begin);
return new EasyNbt.NbtReader(outputStream).ReadValueAs<T>();
}
}
This is the NbtReader:
private MemoryStream MemStream { get; set; }
public NbtReader(MemoryStream memStream)
{
MemStream = memStream;
}
public T ReadValueAs<T>() where T: new()
{
return ReadTagAs<T>(new fNbt.NbtReader(MemStream, true).ReadAsTag());
}
private T ReadTagAs<T>(fNbt.NbtTag nbtTag)
{
//Reads to the root and adds to T...
}

Create zip file in memory from bytes (text with arbitrary encoding)

The application i'm developing needs to compress xml files into zip files and send them through http requests to a web service. As I dont need to keep the zip files, i'm just performing the compression in memory. The web service is denying my requests because the zip files are apparently malformed.
I know there is a solution in this question which works perfectly, but it uses a StreamWriter. My problem with that solution is that StreamWriter requires an encoding or assumes UTF-8, and I do not need to know the enconding of the xml files. I just need to read the bytes from those files, and store them inside a zip file, whatever encoding they use.
So, to be clear, this question has nothing to do with encodings, as I don't need to transform the bytes into text or the oposite. I just need to compress a byte[].
I'm using the next code to test how my zip file is malformed:
static void Main(string[] args)
{
Encoding encoding = Encoding.GetEncoding("ISO-8859-1");
string xmlDeclaration = "<?xml version=\"1.0\" encoding=\"" + encoding.WebName.ToUpperInvariant() + "\"?>";
string xmlBody = "<Test>ª!\"·$%/()=?¿\\|##~€¬'¡º</Test>";
string xmlContent = xmlDeclaration + xmlBody;
byte[] bytes = encoding.GetBytes(xmlContent);
string fileName = "test.xml";
string zipPath = #"C:\Users\dgarcia\test.zip";
Test(bytes, fileName, zipPath);
}
static void Test(byte[] bytes, string fileName, string zipPath)
{
byte[] zipBytes;
using (var memoryStream = new MemoryStream())
using (var zipArchive = new ZipArchive(memoryStream, ZipArchiveMode.Create, leaveOpen: false))
{
var zipEntry = zipArchive.CreateEntry(fileName);
using (Stream entryStream = zipEntry.Open())
{
entryStream.Write(bytes, 0, bytes.Length);
}
//Edit: as the accepted answer states, the problem is here, because i'm reading from the memoryStream before disposing the zipArchive.
zipBytes = memoryStream.ToArray();
}
using (var fileStream = new FileStream(zipPath, FileMode.OpenOrCreate))
{
fileStream.Write(zipBytes, 0, zipBytes.Length);
}
}
If I try to open that file, I get an "Unexpected end of file" error. So apparently, the web service is correctly reporting a malformed zip file. What I have tried so far:
Flushing the entryStream.
Closing the entryStream.
Both flushing and closing the entryStream.
Note that if I open the zipArchive directly from the fileStream the zip file is formed with no errors. However, the fileStream is just there as a test, and I need to create my zip file in memory.

You are trying to get bytes from MemoryStream too early, ZipArchive did not write them all yet. Instead, do like this:
using (var memoryStream = new MemoryStream()) {
// note "leaveOpen" true, to not dispose memoryStream too early
using (var zipArchive = new ZipArchive(memoryStream, ZipArchiveMode.Create, leaveOpen: true)) {
var zipEntry = zipArchive.CreateEntry(fileName);
using (Stream entryStream = zipEntry.Open()) {
entryStream.Write(bytes, 0, bytes.Length);
}
}
// now, after zipArchive is disposed - all is written to memory stream
zipBytes = memoryStream.ToArray();
}

If you use a memory stream to load your text you can control the encoding type and it works across a WCF service. This is the implementation i am using currently and it works on my WCF services
private byte[] Zip(string text)
{
var bytes = Encoding.UTF8.GetBytes(text);
using (var msi = new MemoryStream(bytes))
using (var mso = new MemoryStream())
{
using (var gs = new GZipStream(mso, CompressionMode.Compress))
{
CopyTo(msi, gs);
}
return mso.ToArray();
}
}
private string Unzip(byte[] bytes)
{
using (var msi = new MemoryStream(bytes))
using (var mso = new MemoryStream())
{
using (var gs = new GZipStream(msi, CompressionMode.Decompress))
{
CopyTo(gs, mso);
}
return Encoding.UTF8.GetString(mso.ToArray());
}
}

protobuf-net returns null when calling Deserialize

My end goal is to use protobuf-net and GZipStream in an attempt to compress a List<MyCustomType> object to store in a varbinary(max) field in SQL Server. I'm working on unit tests to understand how everything works and fits together.
Target .NET framework is 3.5.
My current process is:
Serialize the data with protobuf-net (good).
Compress the serialized data from #1 with GZipStream (good).
Convert the compressed data to a base64 string (good).
At this point, the value from step #3 will be stored in a varbinary(max) field. I have no control over this. The steps resume with needing to take a base64 string and deserialize it to a concrete type.
Convert a base 64 string to a byte[] (good).
Decompress the data with GZipStream (good).
Deserialize the data with protobuf-net (bad).
Can someone assist with why the call to Serializer.Deserialize<string> returns null? I'm stuck on this one and hopefully a fresh set of eyes will help.
FWIW, I tried another version of this using List<T> where T is a custom class I created and I Deserialize<> still returns null.
FWIW 2, data.txt is a 4MB plaintext file residing on my C:.
[Test]
public void ForStackOverflow()
{
string data = "hi, my name is...";
//string data = File.ReadAllText(#"C:\Temp\data.txt");
string serializedBase64;
using (MemoryStream protobuf = new MemoryStream())
{
Serializer.Serialize(protobuf, data);
using (MemoryStream compressed = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(compressed, CompressionMode.Compress))
{
byte[] s = protobuf.ToArray();
gzip.Write(s, 0, s.Length);
gzip.Close();
}
serializedBase64 = Convert.ToBase64String(compressed.ToArray());
}
}
byte[] base64byteArray = Convert.FromBase64String(serializedBase64);
using (MemoryStream base64Stream = new MemoryStream(base64byteArray))
{
using (GZipStream gzip = new GZipStream(base64Stream, CompressionMode.Decompress))
{
using (MemoryStream plainText = new MemoryStream())
{
byte[] buffer = new byte[4096];
int read;
while ((read = gzip.Read(buffer, 0, buffer.Length)) > 0)
{
plainText.Write(buffer, 0, read);
}
// why does this call to Deserialize return null?
string deserialized = Serializer.Deserialize<string>(plainText);
Assert.IsNotNull(deserialized);
Assert.AreEqual(data, deserialized);
}
}
}
}

Because you didn't rewind plainText after writing to it. Actually, that entire Stream is unnecessary - this works:
using (MemoryStream base64Stream = new MemoryStream(base64byteArray))
{
using (GZipStream gzip = new GZipStream(
base64Stream, CompressionMode.Decompress))
{
string deserialized = Serializer.Deserialize<string>(gzip);
Assert.IsNotNull(deserialized);
Assert.AreEqual(data, deserialized);
}
}
Likewise, this should work for the serialize:
using (MemoryStream compressed = new MemoryStream())
{
using (GZipStream gzip = new GZipStream(
compressed, CompressionMode.Compress, true))
{
Serializer.Serialize(gzip, data);
}
serializedBase64 = Convert.ToBase64String(
compressed.GetBuffer(), 0, (int)compressed.Length);
}

DeflateStream doesnt work on MemoryStream?

I have the following piece of code:
MemoryStream resultStream = new MemoryStream();
string users = ""//Really long string goes here
BinaryFormatter bFormatter = new BinaryFormatter();
using (MemoryStream assignedUsersStream = new MemoryStream())
{
bFormatter.Serialize(assignedUsersStream, users);
assignedUsersStream.Position = 0;
using (var compressionStream =
new DeflateStream(resultStream, CompressionLevel.Optimal))
{
assignedUsersStream.CopyTo(compressionStream);
Console.WriteLine("Compressed from {0} to {1} bytes.",
assignedUsersStream.Length.ToString(),
resultStream.Length.ToString());
}
}
the thing is that resultStream is always empty!
What am I doing wrong here?

Put your verification WriteLine outside of the using. The buffers haven't been flushed yet.
using (DeflateStream compressionStream = new DeflateStream(resultStream, CompressionLevel.Optimal))
{
assignedUsersStream.CopyTo(compressionStream);
//Console.WriteLine("Compressed from {0} to {1} bytes.",
// assignedUsersStream.Length.ToString(), resultStream.Length.ToString());
}
Console.WriteLine("Compressed from {0} to {1} bytes.",
assignedUsersStream.Length, resultStream.ToArray().Length);
And aside, you don't need all those ToString()s in a writeline.
PS: All a BinaryFormatter does with a string is write the bytes with length prefix. If you don't need the prefix (my guess), it could become:
string users = "";//Really long string goes here
byte[] result;
using (MemoryStream resultStream = new MemoryStream())
{
using (DeflateStream compressionStream = new DeflateStream(resultStream,
CompressionLevel.Optimal))
{
byte[] inBuffer = Encoding.UTF8.GetBytes(users);
compressionStream.Write(inBuffer, 0, inBuffer.Length);
}
result = resultStream.ToArray();
}
The reverse is just as easy but you'll need an estimate of the maximum length to create the read-buffer:
string users2 = null;
using (MemoryStream resultStream = new MemoryStream(result))
{
using (DeflateStream compressionStream = new DeflateStream(resultStream,
CompressionMode.Decompress))
{
byte[] outBuffer = new byte[2048]; // need an estimate here
int length = compressionStream.Read(outBuffer, 0, outBuffer.Length);
users2 = Encoding.UTF8.GetString(outBuffer, 0, length);
}
}

That is because the DeflateStream doesn't flush the data to the underlying stream until it is closed. After it is closed, resultStream will contain the compressed data. Note that by default, DeflateStream closes the underlying stream when it's closed, but you don't want that, so you need to pass true for the leaveOpen parameter. Also, you don't need 2 memory streams, you can just serialize directly to the compressionStream:
string users = ""; //Really long string goes here
BinaryFormatter bFormatter = new BinaryFormatter();
using (MemoryStream resultStream = new MemoryStream())
{
using (DeflateStream compressionStream = new DeflateStream(resultStream, CompressionLevel.Optimal, true))
{
bFormatter.Serialize(compressionStream, users);
Console.WriteLine(resultStream.Length); // 0 at this point
}
Console.WriteLine(resultStream.Length); // now contains the actual length
}

From the original answer (I don't have enough credits to vote down)
Put your control WriteLine outside of the using
This is incomplete and IMO therefore misleading. DeflateStream's Dispose(bool) implementation Closes the underlying resultStream when the DeflateStream is being Finalized after it's been Garbage Collected. When this happens, resultStream.Length will throw:
Unhandled Exception: System.ObjectDisposedException: Cannot access a closed Stream.
In other words, Thomas Levesque's note is critical: also set leaveOpen to true.
An interesting question with some good points raised by HH and TL.

I've come in late to this as I ran into this same problem and reading the conflicting answers. The initial response works! as does this test (over simplified to highlight the answer):
var inStream = new MemoryStream(data);
var outStream = new MemoryStream();
using (var compressor = new DeflateStream(outStream, CompressionLevel.Optimal))
{
inStream.CopyTo(compressor);
}
return outStream;
where the using block needs to complete, triggering the compressor's Dispose, which internally Flush()es so that outStream will be guaranteed to contain the complete compressed data.

End of Stream encountered before parsing was completed?

I am trying to deserialize a stream but I always get this error "End of Stream encountered before parsing was completed"?
Here is the code:
//Some code here
BinaryFormatter b = new BinaryFormatter();
return (myObject)b.Deserialize(s);//s---> is a Stream object that has been fill up with data some line over here
Any one have ideas?

Try to set the position to 0 of your stream and do not use your object but the object type.
BinaryFormatter b = new BinaryFormatter();
s.Position = 0;
return (YourObjectType)b.Deserialize(s);

Make sure the serialization completed, and that the serialization type matches the de-serialization type (i.e., make sure you're serializing with a BinaryFormatter if you're de-serializing with one). Also, make sure that the stream you serialized to really finished serializing, with a Stream.Flush() or something to that effect.

I had the same exception thrown, until I added the [Serializable] tag to the class I was Serializing :)
Then it all worked perfectly.

In my case I used:
stream.Seek(0, SeekOrigin.Begin);
after i serialized the stream, and before i deserialized the stream works charm. hope this helps!

I have spent 5 hourse and have got end of stream error and lost data (Not obvious feature in GzipStream: you should use underlying stream only after flush GzipStream).
Full example of working code:
using System;
using System.IO;
using System.IO.Compression;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
namespace ConsoleApp3
{
class Program
{
static void Main(string[] args)
{
string large = LargeJsonContent.GetBigObject();
string base64;
using (var readStream = new MemoryStream())
using (var writeStream = new MemoryStream())
{
using (GZipStream compressor = new GZipStream(writeStream, CompressionMode.Compress, true)) //pay attention to leaveOpen = true
{
var formatter = new BinaryFormatter();
formatter.Serialize(readStream, large);
Console.WriteLine($"After binary serialization of JsonString: {readStream.Length} bytes");
readStream.Position = 0;
readStream.CopyTo(compressor);
}
Console.WriteLine($"Compressed stream size: {writeStream.Length} bytes");
writeStream.Position = 0;
byte[] writeBytes = writeStream.ToArray();
base64 = Convert.ToBase64String(writeBytes);
}
////
using (var stream = new MemoryStream())
{
var formatter = new BinaryFormatter();
formatter.Serialize(stream, base64);
Console.WriteLine($"Size of base64: {stream.Length} bytes");
}
Console.WriteLine("---------------------");
////
string large2;
var bytes = Convert.FromBase64String(base64);
using (var readStream = new MemoryStream())
{
readStream.Write(bytes, 0, bytes.Length);
readStream.Position = 0;
Console.WriteLine($"Compressed stream size: {readStream.Length} bytes");
using (var writeStream = new MemoryStream())
{
using (GZipStream decompressor = new GZipStream(readStream, CompressionMode.Decompress, true)) //pay attention to leaveOpen = true
{
decompressor.CopyTo(writeStream);
writeStream.Position = 0;
}
var formatter = new BinaryFormatter();
large2 = (string)formatter.Deserialize(writeStream);
}
}
Console.WriteLine(large == large2);
Console.WriteLine($"large:{large.Length} | large2:{large2.Length}");
}
}
}

Check in your sender code if you are not doing the following
NetworkStream strm = client.GetStream(); // the stream
formatter.Serialize(strm, status); // the serialization process
strm.Close();// Remove this code, this was the culprit in my case

the class which you created must has [Serializable].