I'm trying to serialize very large object directly to a zip stream. I manage to do this by serializing to a file stream in an intermediate step, loading it back and then compressing it.
I've also tried compressing directly to a memory stream and it works. But when i use a GZipStream I'm always left with an "unfinished" object, the data is there it's correctly formatted up to the point where it ends unexpectedly.
It's not for lack of flushing buffers since I've already tried flushing everything.
Simplified sample code:
internal static byte[] SerializeAndCompress(object objectToSerialize)
{
using(var memStream = new MemoryStream())
using (var zipStream = new GZipStream(memStream, CompressionMode.Compress, true))
using (var streamWriter = new StreamWriter(zipStream))
using (var jsonWriter = new JsonTextWriter(streamWriter))
{
var jsonSerializer = new JsonSerializer { ContractResolver = new CamelCasePropertyNamesContractResolver(), Formatting = Newtonsoft.Json.Formatting.None };
jsonSerializer.Serialize(jsonWriter, objectToSerialize);
jsonWriter.Flush();
return memStream.ToArray();
}
}
Thanks.
Rather than flushing the writer, I suggest you close it completely. That way the GZipStream knows there's no more data to write and can add any appropriate checksums or whatever it needs to do.
You could either call Close explicitly, or put it between the closing parts of using statements:
using(var memStream = new MemoryStream())
{
using (var zipStream = new GZipStream(memStream, CompressionMode.Compress, true))
using (var streamWriter = new StreamWriter(zipStream))
using (var jsonWriter = new JsonTextWriter(streamWriter))
{
var jsonSerializer = new JsonSerializer { ContractResolver = new CamelCasePropertyNamesContractResolver(), Formatting = Newtonsoft.Json.Formatting.None };
jsonSerializer.Serialize(jsonWriter, objectToSerialize);
}
return memStream.ToArray();
}
Note that by the time you call ToArray, the MemoryStream will already be closed, but that's okay - the data will still be there.
Related
I'm trying to create a middleware that changes the request in a certain way. I am able to read it and change the content but I cannot figure out how to correctly setup the stream writers to create a new body. When I call normalized.WriteToAsync(jsonWriter) the MemoryStream remains empty and consequently I receive the A non-empty request body is required. exception. What am I missing here? This is what I have so far:
public async Task Invoke(HttpContext context)
{
if (context.Request.ContentType == "application/json" && context.Request.ContentLength > 0)
{
using var scope = _logger.BeginScope("NormalizeJson");
try
{
using var requestReader = new HttpRequestStreamReader(context.Request.Body, Encoding.UTF8);
using var jsonReader = new JsonTextReader(requestReader);
var json = await JToken.LoadAsync(jsonReader);
var normalized = _normalize.Visit(json); // <-- Modify json and return JToken
// Create new Body
var memoryStream = new MemoryStream();
var requestWriter = new StreamWriter(memoryStream);
var jsonWriter = new JsonTextWriter(requestWriter);
await normalized.WriteToAsync(jsonWriter); // <-- At this point the MemoryStream has still 0 length.
var content = new StreamContent(memoryStream.Rewind()); // <-- Use helper extension to Seek.Begin = 0
context.Request.Body = await content.ReadAsStreamAsync();
}
catch (Exception e)
{
_logger.Scope().Exceptions.Push(e);
}
}
await _next(context);
}
Demo for LINQPad etc.:
async Task Main()
{
var token = JToken.FromObject(new User { Name = "Bob" });
var memoryStream = new MemoryStream();
var requestWriter = new StreamWriter(memoryStream);
var jsonWriter = new JsonTextWriter(requestWriter);
await token.WriteToAsync(jsonWriter);
memoryStream.Length.Dump(); // <-- MemoryStream.Length = 0
}
public class User
{
public string Name { get; set; }
}
You need to properly flush and close your JsonTextWriter and StreamWriter in order to fully populate the memoryStream, like so:
var memoryStream = new MemoryStream();
// StreamWriter implements IAsyncDisposable
// Leave the underlying stream open
await using (var requestWriter = new StreamWriter(memoryStream, leaveOpen: true))
{
var jsonWriter = new JsonTextWriter(requestWriter); // But JsonTextWriter does not implement IAsyncDisposable, only IDisposable!
try
{
await token.WriteToAsync(jsonWriter);
}
finally
{
await jsonWriter.CloseAsync();
}
}
Demo fiddle #1 here.
Or, since you're writing to a MemoryStream, there's really no nead to use async at all, and instead you can do:
var memoryStream = new MemoryStream();
using (var requestWriter = new StreamWriter(memoryStream, leaveOpen: true)) // Leave the underlying stream open
using (var jsonWriter = new JsonTextWriter(requestWriter))
{
token.WriteTo(jsonWriter);
}
Demo fiddle #2 here.
Notes:
Note the use of await using for the StreamWriter. This syntax guarantees that the StreamWriter will be flushed and closed asynchronously, and can be used on any object that implements IAsyncDisposable. (This only really matters if you were writing to a file stream or other non-memory stream.)
It seems that neither JsonTextWriter nor the base class JsonWriter implement IAsyncDisposable, so I had to asynchronously close the JSON writer manually rather than via a using statement. The outer await using should ensure that the underlying StreamWriter is not left open in the event of an exception.
JSON RFC 8259 specifies that Implementations MUST NOT add a byte order mark (U+FEFF) to the beginning of a networked-transmitted JSON text. Thus, when constructing a StreamWriter, it is recommended to pass an encoding such as new UTF8Encoding(false) that does not prepend a BOM. Alternatively, if you just want UTF-8, the StreamWriter constructors will create a StreamWriter with UTF-8 encoding without a Byte-Order Mark (BOM) if you do not specify one yourself and leave a default value for that parameter as is shown in the code above.
I need to use this:
http://www.newtonsoft.com/json/help/html/SerializeToBson.htm
This is code to convert object to BSON format. The code which interests me is this:
System.IO.MemoryStream stream = new System.IO.MemoryStream();
using (Newtonsoft.Json.Bson.BsonWriter writer = new Newtonsoft.Json.Bson.BsonWriter(stream))
{
Newtonsoft.Json.JsonSerializer serializer = new Newtonsoft.Json.JsonSerializer();
serializer.Serialize(writer, message);
}
However, I want the result in a string. So do I really have to use a stream or a file to write stuff in, then read it to put it in the string?
There must be a better way to do this?
You can get the string from the stream using StreamReader.ReadToEnd():
string bsonText = "";
using(MemoryStream stream = new MemoryStream())
using(StreamReader reader = new StreamReader(stream))
using (BsonWriter writer = new Newtonsoft.Json.Bson.BsonWriter(stream))
{
JsonSerializer serializer = new JsonSerializer();
serializer.Serialize(writer, message);
stream.Position = 0;
bsonText = reader.ReadToEnd();
}
Or also, Encoding.UTF8.GetString():
using(MemoryStream stream = new MemoryStream())
using (BsonWriter writer = new Newtonsoft.Json.Bson.BsonWriter(stream))
{
JsonSerializer serializer = new JsonSerializer();
serializer.Serialize(writer, message);
bsonText = Encoding.UTF8.GetString(stream.ToArray());
}
BTW who knows what you're going to get from this, since BSON is a binary object representation, it's not like JSON!
I want to use the DataContractJsonSerializer to serialize to file in JsonFormat.
The problem is that the WriteObjectmethod only has 3 options XmlWriter, XmlDictionaryWriter and Stream.
To get what I want I used the following code:
var js = new DataContractJsonSerializer(typeof(T), _knownTypes);
using (var ms = new MemoryStream())
{
js.WriteObject(ms, item);
ms.Position = 0;
using (var sr = new StreamReader(ms))
{
using (var writer = new StreamWriter(path, false))
{
string jsonData = sr.ReadToEnd();
writer.Write(jsonData);
}
}
}
Is this the only way or have I missed something?
Assuming you're just trying to write the text to a file, it's not clear why you're writing it to a MemoryStream first. You can just use:
var js = new DataContractJsonSerializer(typeof(T), _knownTypes);
using (var stream = File.Create(path))
{
js.WriteObject(stream, item);
}
That's rather simpler, and should do what you want...
I am actually quite terrified to claim to know something that Jon Skeet doesn't, but I have used code similar to the following which produces the Json text file and maintains proper indentation:
var js = new DataContractJsonSerializer(typeof(T), _knownTypes);
using (var stream = File.Create(path))
{
using (var writer = JsonReaderWriterFactory.CreateJsonWriter(stream, Encoding.UTF8, true, true, "\t"))
{
js.WriteObject(writer, item);
writer.Flush();
}
}
(as suggested here.)
I have an XmlTextWriter that gets written to file using an XmlSerializer that looks like the following:
using (XmlTextWriter writer = new XmlTextWriter(path, null))
{
writer.Formatting = Formatting.Indented;
writer.Indentation = 3;
MyFileObj.ourSerializer.Serialize(writer, xmlFile, ourXmlNamespaces);
}
where "ourSerializer" is just a reference to an System.Xml.Serialization.XmlSerializer object. However, I have an instance where this XML must be encrypted to disk so that the end user cannot read its contents, and I am unsure of the proper way to go about it using the existing code since there are many places where this code is called and does not need to be encrypted. Can anyone shed some insight into this for me?
An alternative way would be to use a CryptoStream, like this:
using (var fs = new FileStream(path, System.IO.FileMode.Create))
{
using (var cs = new CryptoStream(fs, _Provider.CreateEncryptor(), CryptoStreamMode.Write))
{
using (var writer = XmlWriter.Create(cs))
{
writer.Formatting = Formatting.Indented;
writer.Indentation = 3;
MyFileObj.ourSerializer.Serialize(writer, xmlFile, ourXmlNamespaces);
}
}
}
Where _Provider is an AesCryptoServiceProvider properly initialized.
Here is how I ended up solving the issue:
MemoryStream ms = new MemoryStream();
XmlSerializer ourSerializer.Serialize(ms, xmlFile, ourXmlNamespaces);
ms.Position = 0;
//Encrypt the memorystream
using (TextReader reader = new StreamReader(ms, Encoding.ASCII))
using (StreamWriter writer = new StreamWriter(path))
{
string towrite = Encrypt(reader.ReadToEnd());
writer.Write(towrite);
}
Basically serialized the XML to a MemoryStream, read the text back out into a TextReader, encrypted the TextReader contents and then saved the resulting encrypted string to a file.
I am trying to deserialize a stream but I always get this error "End of Stream encountered before parsing was completed"?
Here is the code:
//Some code here
BinaryFormatter b = new BinaryFormatter();
return (myObject)b.Deserialize(s);//s---> is a Stream object that has been fill up with data some line over here
Any one have ideas?
Try to set the position to 0 of your stream and do not use your object but the object type.
BinaryFormatter b = new BinaryFormatter();
s.Position = 0;
return (YourObjectType)b.Deserialize(s);
Make sure the serialization completed, and that the serialization type matches the de-serialization type (i.e., make sure you're serializing with a BinaryFormatter if you're de-serializing with one). Also, make sure that the stream you serialized to really finished serializing, with a Stream.Flush() or something to that effect.
I had the same exception thrown, until I added the [Serializable] tag to the class I was Serializing :)
Then it all worked perfectly.
In my case I used:
stream.Seek(0, SeekOrigin.Begin);
after i serialized the stream, and before i deserialized the stream works charm. hope this helps!
I have spent 5 hourse and have got end of stream error and lost data (Not obvious feature in GzipStream: you should use underlying stream only after flush GzipStream).
Full example of working code:
using System;
using System.IO;
using System.IO.Compression;
using System.Runtime.Serialization;
using System.Runtime.Serialization.Formatters.Binary;
namespace ConsoleApp3
{
class Program
{
static void Main(string[] args)
{
string large = LargeJsonContent.GetBigObject();
string base64;
using (var readStream = new MemoryStream())
using (var writeStream = new MemoryStream())
{
using (GZipStream compressor = new GZipStream(writeStream, CompressionMode.Compress, true)) //pay attention to leaveOpen = true
{
var formatter = new BinaryFormatter();
formatter.Serialize(readStream, large);
Console.WriteLine($"After binary serialization of JsonString: {readStream.Length} bytes");
readStream.Position = 0;
readStream.CopyTo(compressor);
}
Console.WriteLine($"Compressed stream size: {writeStream.Length} bytes");
writeStream.Position = 0;
byte[] writeBytes = writeStream.ToArray();
base64 = Convert.ToBase64String(writeBytes);
}
////
using (var stream = new MemoryStream())
{
var formatter = new BinaryFormatter();
formatter.Serialize(stream, base64);
Console.WriteLine($"Size of base64: {stream.Length} bytes");
}
Console.WriteLine("---------------------");
////
string large2;
var bytes = Convert.FromBase64String(base64);
using (var readStream = new MemoryStream())
{
readStream.Write(bytes, 0, bytes.Length);
readStream.Position = 0;
Console.WriteLine($"Compressed stream size: {readStream.Length} bytes");
using (var writeStream = new MemoryStream())
{
using (GZipStream decompressor = new GZipStream(readStream, CompressionMode.Decompress, true)) //pay attention to leaveOpen = true
{
decompressor.CopyTo(writeStream);
writeStream.Position = 0;
}
var formatter = new BinaryFormatter();
large2 = (string)formatter.Deserialize(writeStream);
}
}
Console.WriteLine(large == large2);
Console.WriteLine($"large:{large.Length} | large2:{large2.Length}");
}
}
}
Check in your sender code if you are not doing the following
NetworkStream strm = client.GetStream(); // the stream
formatter.Serialize(strm, status); // the serialization process
strm.Close();// Remove this code, this was the culprit in my case
the class which you created must has [Serializable].