We Keep Coding

Serializing to MemoryStream causes an OutOfmemoryException, but serializing to a FileStream does not. Can anyone tell me why?

I'm using Newtonsoft Json.Net to serialize objects as json. I continually run into an OutOfMemoryException when I try to serialize to a MemoryStream, but not when I serialize to a FileStream. Could someone explain why this might be happening? These are the two methods that I am using to serialize.
Throws an OutOfMemoryException
private static MemoryStream _serializeJson<T>(T obj)
{
try
{
var stream = new MemoryStream();
var streamWriter = new StreamWriter(stream);
var jsonWriter = new JsonTextWriter(streamWriter);
var serializer = new JsonSerializer();
serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();
serializer.Formatting = Formatting.Indented;
serializer.Serialize(jsonWriter, obj);
streamWriter.Flush();
stream.Position = 0;
return stream;
}
catch (Exception e)
{
//Logger.WriteError(e.ToString());
Console.WriteLine(e.ToString());
return null;
}
}
Doesn't throw an OutOfMemoryException
private static void _serializeJsonToFile<T>(T obj, string path)
{
try
{
using (FileStream fs = File.Open(path, FileMode.Create, FileAccess.ReadWrite))
using (StreamWriter sw = new StreamWriter(fs))
using (JsonWriter jw = new JsonTextWriter(sw))
{
jw.Formatting = Formatting.Indented;
JsonSerializer serializer = new JsonSerializer();
serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();
serializer.Serialize(jw, obj);
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
P.S. some might ask why I would want to return a stream instead of simply serializing to the file stream. This is because I want to keep serialization in one class and file handling in another, so I'm passing the memory stream to a WriteFile method in another class later.

You are getting OutOfMemoryExceptions because the memory stream is very aggressive about it's growth. Every time it needs to re-size it doubles it's internal buffer.
//The code from MemoryStream http://referencesource.microsoft.com/mscorlib/system/io/memorystream.cs.html#1416df83d2368912
private bool EnsureCapacity(int value) {
// Check for overflow
if (value < 0)
throw new IOException(Environment.GetResourceString("IO.IO_StreamTooLong"));
if (value > _capacity) {
int newCapacity = value;
if (newCapacity < 256)
newCapacity = 256;
// We are ok with this overflowing since the next statement will deal
// with the cases where _capacity*2 overflows.
if (newCapacity < _capacity * 2)
newCapacity = _capacity * 2;
// We want to expand the array up to Array.MaxArrayLengthOneDimensional
// And we want to give the user the value that they asked for
if ((uint)(_capacity * 2) > Array.MaxByteArrayLength)
newCapacity = value > Array.MaxByteArrayLength ? value : Array.MaxByteArrayLength;
Capacity = newCapacity;
return true;
}
return false;
}
With a 17.8 MB file that is a worst case scenario of a 35.6 MB byte array being used. The old byte arrays that where discarded during the resizing process can also cause Memory Fragmentation depending on how long they live, this can easily get your program to throw a OOM error before you get to the 32 bit memory limit.
Writing directly to a FileStream does not require any large buffers to be created in memory so it uses much less space.
There is a way to separate the logic of the saving from the serializing, just pass in the stream to the function instead of creating it in the function itself.
private static void _serializeJson<T>(T obj, Stream stream)
{
try
{
using(var streamWriter = new StreamWriter(stream, Encoding.UTF8, 1024, true))
using(var jsonWriter = new JsonTextWriter(streamWriter))
{
var serializer = new JsonSerializer();
serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();
serializer.Formatting = Formatting.Indented;
serializer.Serialize(jsonWriter, obj);
}
}
catch (Exception e)
{
//Logger.WriteError(e.ToString());
Console.WriteLine(e.ToString());
}
}
I also dispose of the StreamWriter that is created, the constructor I used has a leaveOpen flag which causes the underlying stream to not be closed when you dispose of the StreamWriter.

Finding the perfect fit for a serialized object written to a MemoryMappedFile

Currently, I use the code below to find a fit for a serialized object in a MemoryMappedFile.
Basically, I increase the size of the MemoryMappedFile until the NotSupportedException (cannot extend length of stream) is not thrown anymore.
MemoryMappedFile mmf = MemoryMappedFile.CreateNew(blobName, this.size);
bool serializeOk = false;
while (!serializeOk)
{
try
{
using (var stream = mmf.CreateViewStream())
{
IFormatter formatter = new BinaryFormatter();
formatter.Serialize(stream, obj);
}
serializeOk = true;
}
catch (NotSupportedException)
{
mmf.Dispose();
this.size *= 2;
mmf = MemoryMappedFile.CreateNew(blobName, this.sizeMemMapFile);
}
}
Q. Can somebody come up with a solution which provides a better fit and is more efficient?

Load File Not working - The magic number in GZip header is not correct

I am attempting to create a Save/Load class that has the option for saving & load files compressed files. Below is what I have so far. Stepping through it seems to work just fine, except that I get a "The magic number in GZip header is not correct" exception. I don't understand how this can be as I am checking to make sure that the number is there before I pass it over, and I have verified via an external program that it is a GZip file.
Any assistance in finding out where I went wrong would be appreciated. Constructive criticism of my code is always welcome - Thanks!
public static class SaveLoad
{
public static void Save(string fileName, object savefrom, bool compress)
{
FileStream stream = new FileStream(fileName, FileMode.Create);
BinaryFormatter formatter = new BinaryFormatter();
if (compress)
{
GZipStream compressor = new GZipStream(stream, CompressionMode.Compress);
formatter.Serialize(compressor, savefrom);
compressor.Close();
}
else { formatter.Serialize(stream, savefrom); }
stream.Close();
}
public static object Load(string fileName)
{
object loadedObject = null;
try
{
FileStream stream = new FileStream(fileName, FileMode.Open);
BinaryFormatter formatter = new BinaryFormatter();
if (stream.Length > 4)
{
byte[] data = new byte[4];
stream.Read(data, 0, 4);
if (BitConverter.ToUInt16(data, 0) == 0x8b1f) //GZIP_LEAD_BYTES == 0x8b1f
{
GZipStream decompressor = new GZipStream(stream, CompressionMode.Decompress);
loadedObject = formatter.Deserialize(decompressor); //Exception
decompressor.Close();
}
else { loadedObject = formatter.Deserialize(stream); }
}
stream.Close();
}
catch (Exception e)
{
Logger.StaticLog.AddEvent(new Logger.lEvent(null, Logger.lEvent.EventTypes.Warning, "Failed to load file: " + fileName, e)
{
SendingObject = "SaveLoad"
});
Logger.StaticLog.WriteLog();
throw;
}
return loadedObject;
}
}

It seems that you read the magic number before passing the stream to decompressor (which won't read the magic number then, because you've already read it).
Use stream.Seek(0,SeekOrigin.Begin) before you decompress.

Image.FromStream throws ArgumentException: Parameter is not valid

I'm trying to output an image to the output stream through an HttpHandler but it keeps throwing an ArgumentException. I've googled the issue and tried so many things but I still couldn't fix the problem. Anyway, here's the code:
public void ProcessRequest(HttpContext context)
{
Int32 imageId = context.Request.QueryString["id"] != null ?
Convert.ToInt32(context.Request.QueryString["id"]) : default(Int32);
if (imageId != 0)
{
//context.Response.ContentType = "image/jpeg";
Byte[] imageData = this._imageInfoManager.GetTradeMarkImage(imageId);
using (MemoryStream ms = new MemoryStream(imageData, 0, imageData.Length))
{
using (Image image = Image.FromStream(ms, true, true)) //this line throws
{
image.Save(context.Response.OutputStream, ImageFormat.Jpeg);
}
}
}
throw new ArgumentException("Image could not be found.");
}
Note that the imageData byte array is not empty and the memory stream is being filled up correctly. Any thoughts?
UPDATE:
Here's the code for GetTradeMarkImage... Note that the images are stored in the an SQL Server database in the image format.
public Byte[] GetTradeMarkImage(Int32 id)
{
object result = DB.ExecuteScalar(SqlConstants.SqlProcedures.Request_GetImageById, id);
if (result != null)
{
using (MemoryStream ms = new MemoryStream())
{
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(ms, result);
return ms.ToArray();
}
}
return null;
}

Okay, now you've posted the GetTradeMarkImage code, that's almost certainly the problem:
using (MemoryStream ms = new MemoryStream())
{
BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(ms, result);
return ms.ToArray();
}
Why would you expect the value of BinaryFormatter to be a valid image stream? It's not clear what's in your database (a BLOB?) nor what the execution-time type of result is here (which you should find out by debugging) but you shouldn't be using BinaryFormatter here. I suspect you just want to get the raw data out of the database, and put that into the byte array.
If you're lucky, you may just be able to cast result to byte[] to start with. (I don't know what ExecuteScalar does with blobs, and this obviously isn't the "normal" ExecuteScalar method anyway). Otherwise, you may need to use an alternative approach, opening a DataReader and getting the value that way.

Serializing / Deserializing image to / from XElement

I am trying to serialize image into XElement and afterwards deserialize it for further use.
I am using this method to serialize:
public XElement subElement = new XElement("Element");
private void Serialize(System.Windows.Forms.Button button) {
if (button.Image != null) {
var bf = new BinaryFormatter();
var ms = new MemoryStream();
bf.Serialize(ms, button.Image);
var textWriter = new StringWriter();
var writer = new XmlTextWriter(textWriter);
byte[] imageBytes = ms.ToArray();
writer.WriteBase64(imageBytes, 0, imageBytes.Length);
subElement.Add(new XAttribute("Image", imageBytes));
}
}
But I can't figure out how to deserialize. I tried something like this:
private void Deserialize(XElement element) {
if (element.Attribute("Image") != null) {
//tried XmlReader reader = XmlReader.Create(new StringReader(element.Attribute("Image").Value));
//but reader is empty
//when I try: XmlReader reader = XmlReader.Create(element.Attribute("Image").Value);
//exception is thrown because XmlReader expects path, not element
}
}
I basically only need to get byte array from XElement, later I know how to handle it.

Once you have your byte array you can do Convert.ToBase64String(byteArray). The result of that function (a string) is what goes in the XAttribute's value.
Then when it comes to reading, you'd just do byteArray = Convert.FromBase64String(element.Attribute("Image").Value)
This should prevent the issues with saving the string within the XML file.

Don't use serialization, simple save/load it from memory stream.