We Keep Coding

XMLSerializer - issue with UTF-8 vs UTF-16 Code

I am trying to serialize a simple object (5 string properties) into XML to save to a DB Image field. Then I need to DeSerialize it back into a string later in the program.
However, I am getting some errors - caused by the XML being saved thinking it is in UTF-16 - however, when I load it from the DB back into a string - it thinks it is a UTF 8 String.
The error I get is
InnerException {"There is no Unicode byte order mark. Cannot switch to Unicode."} System.Exception {System.Xml.XmlException}
-- Message "There is an error in XML document (0, 0)." string
Is this happening because of the two different ways I save and load the string to/from the DB? On the save I am using a StringBuilder - but on the load from DB I am using just a String.
Thoughts?
Serialize and Save to DB
// Now Save the OBject XML to the Query Tables
var serializer = new XmlSerializer(ExportConfig.GetType());
StringBuilder StringResult = new StringBuilder();
using (var writer = XmlWriter.Create(StringResult))
{
serializer.Serialize(writer, ExportConfig);
}
//MessageBox.Show("XML : " + StringResult);
// Now Save to the Query
try
{
string UpdateSQL = "Update ZQryRpt "
+ " Set ExportConfig = " + TAGlobal.QuotedStr(StringResult.ToString())
+ " where QryId = " + TAGlobal.QuotedStr(((DataRowView)bindingSource_zQryRpt.Current).Row["QryID"].ToString())
;
ExecNonSelectSQL(UpdateSQL, uniConnection_Config);
}
catch (Exception Error)
{
MessageBox.Show("Error Setting ExportConfig: " + Error.Message);
}
Load from DB And Deserialize
byte[] binaryData = (byte[])((DataRowView)bindingSource_zQryRpt.Current).Row["ExportConfig"];
string XMLStored = System.Text.Encoding.UTF8.GetString(binaryData, 0, binaryData.Length);
if (XMLStored.Length > 0)
{
IIDExportObject ExportConfig = new IIDExportObject();
var serializer = new XmlSerializer(ExportConfig.GetType());
//StringBuilder StringResult = new StringBuilder(XMLStored);
// Load the XML from the Query into the StringBuilder
// Now we need to build a Stream from the String to use in the XMLReader
byte[] byteArray = Encoding.UTF8.GetBytes(XMLStored);
MemoryStream stream = new MemoryStream(byteArray);
using (var reader = XmlReader.Create(stream))
{
ExportConfig = (IIDExportObject)serializer.Deserialize(reader);
}
}

John - thank you very much for the comment! It allowed me to complete the code and find a solution.
As you noted - using a stream reader was the solution - but I could not read the first line because there was only one 'line' in my string. However, I could use the line
using (StreamReader sr = new StreamReader(stream, false))
Which allows me to read the stream and ignore the "Byte Order Mark Detection" set to false.
string XMLStored = MainFormRef.GetExportConfigForCurrentQuery();
if (XMLStored.Length > 0)
{
IIDExportObject ExportConfig = new IIDExportObject();
try
{
var serializer = new XmlSerializer(ExportConfig.GetType());
// Now we need to build a Stream from the String to use in the XMLReader
byte[] byteArray = Encoding.UTF8.GetBytes(XMLStored);
MemoryStream stream = new MemoryStream(byteArray);
// Now we need to use a StreamReader to get around UTF8 vs UTF16 issues
// A little cumbersome - but it works
using (StreamReader sr = new StreamReader(stream, false))
{
using (var reader = XmlReader.Create(sr))
{
ExportConfig = (IIDExportObject)serializer.Deserialize(reader);
}
}
}
catch
{
}
I am not sure this is the best solution - but it works. I will be curious to see if anyone else has a better way of dealing with this.

Thanks to G Bradley, I took his answer and generalized it a bit to make it a bit easier to call.
public static string SerializeToXmlString<T>(T objectToSerialize)
{
XmlSerializer serializer = new XmlSerializer(typeof(T));
XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = false;
settings.Encoding = Encoding.UTF8;
StringBuilder builder = new StringBuilder();
using (XmlWriter writer = XmlWriter.Create(builder, settings))
{
serializer.Serialize(writer, objectToSerialize);
}
return builder.ToString();
}
public static T DeserializeFromXmlString<T>(string xmlString)
{
if (string.IsNullOrWhiteSpace(xmlString))
return default;
var serializer = new XmlSerializer(typeof(T));
byte[] byteArray = Encoding.UTF8.GetBytes(xmlString);
MemoryStream stream = new MemoryStream(byteArray);
using (StreamReader sr = new StreamReader(stream, false))
{
using (var reader = XmlReader.Create(sr))
{
return (T)serializer.Deserialize(reader);
}
}
}

Writing to Filestream and copying to MemoryStream

I want to overwrite or create an xml file on disk, and return the xml from the function. I figured I could do this by copying from FileStream to MemoryStream. But I end up appending a new xml document to the same file, instead of creating a new file each time.
What am I doing wrong? If I remove the copying, everything works fine.
public static string CreateAndSave(IEnumerable<OrderPage> orderPages, string filePath)
{
if (orderPages == null || !orderPages.Any())
{
return string.Empty;
}
var xmlBuilder = new StringBuilder();
var writerSettings = new XmlWriterSettings
{
Indent = true,
Encoding = Encoding.GetEncoding("ISO-8859-1"),
CheckCharacters = false,
ConformanceLevel = ConformanceLevel.Document
};
using (var fs = new FileStream(filePath, FileMode.OpenOrCreate, FileAccess.ReadWrite))
{
try
{
XmlWriter xmlWriter = XmlWriter.Create(fs, writerSettings);
xmlWriter.WriteStartElement("PRINT_JOB");
WriteXmlAttribute(xmlWriter, "TYPE", "Order Confirmations");
foreach (var page in orderPages)
{
xmlWriter.WriteStartElement("PAGE");
WriteXmlAttribute(xmlWriter, "FORM_TYPE", page.OrderType);
var outBound = page.Orders.SingleOrDefault(x => x.FlightInfo.Direction == FlightDirection.Outbound);
var homeBound = page.Orders.SingleOrDefault(x => x.FlightInfo.Direction == FlightDirection.Homebound);
WriteXmlOrder(xmlWriter, outBound, page.ContailDetails, page.UserId, page.PrintType, FlightDirection.Outbound);
WriteXmlOrder(xmlWriter, homeBound, page.ContailDetails, page.UserId, page.PrintType, FlightDirection.Homebound);
xmlWriter.WriteEndElement();
}
xmlWriter.WriteFullEndElement();
MemoryStream destination = new MemoryStream();
fs.CopyTo(destination);
Log.Progress("Xml string length: {0}", destination.Length);
xmlBuilder.Append(Encoding.UTF8.GetString(destination.ToArray()));
destination.Flush();
destination.Close();
xmlWriter.Flush();
xmlWriter.Close();
}
catch (Exception ex)
{
Log.Warning(ex, "Unhandled exception occured during create of xml. {0}", ex.Message);
throw;
}
fs.Flush();
fs.Close();
}
return xmlBuilder.ToString();
}
Cheers
Jens

FileMode.OpenOrCreate is causing the file contents to be overwritten without shortening, leaving any 'trailing' data from previous runs. If FileMode.Create is used the file will be truncated first. However, to read back the contents you just wrote you will need to use Seek to reset the file pointer.
Also, flush the XmlWriter before copying from the underlying stream.
See also the question Simultaneous Read Write a file in C Sharp (3817477).
The following test program seems to do what you want (less your own logging and Order details).
using System;
using System.IO;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using System.Threading.Tasks;
namespace ReadWriteTest
{
class Program
{
static void Main(string[] args)
{
string filePath = Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.Personal),
"Test.xml");
string result = CreateAndSave(new string[] { "Hello", "World", "!" }, filePath);
Console.WriteLine("============== FIRST PASS ==============");
Console.WriteLine(result);
result = CreateAndSave(new string[] { "Hello", "World", "AGAIN", "!" }, filePath);
Console.WriteLine("============== SECOND PASS ==============");
Console.WriteLine(result);
Console.ReadLine();
}
public static string CreateAndSave(IEnumerable<string> orderPages, string filePath)
{
if (orderPages == null || !orderPages.Any())
{
return string.Empty;
}
var xmlBuilder = new StringBuilder();
var writerSettings = new XmlWriterSettings
{
Indent = true,
Encoding = Encoding.GetEncoding("ISO-8859-1"),
CheckCharacters = false,
ConformanceLevel = ConformanceLevel.Document
};
using (var fs = new FileStream(filePath, FileMode.Create, FileAccess.ReadWrite))
{
try
{
XmlWriter xmlWriter = XmlWriter.Create(fs, writerSettings);
xmlWriter.WriteStartElement("PRINT_JOB");
foreach (var page in orderPages)
{
xmlWriter.WriteElementString("PAGE", page);
}
xmlWriter.WriteFullEndElement();
xmlWriter.Flush(); // Flush from xmlWriter to fs
xmlWriter.Close();
fs.Seek(0, SeekOrigin.Begin); // Go back to read from the begining
MemoryStream destination = new MemoryStream();
fs.CopyTo(destination);
xmlBuilder.Append(Encoding.UTF8.GetString(destination.ToArray()));
destination.Flush();
destination.Close();
}
catch (Exception ex)
{
throw;
}
fs.Flush();
fs.Close();
}
return xmlBuilder.ToString();
}
}
}
For the optimizers out there, the StringBuilder was unnecessary because the string is formed whole and the MemoryStream can be avoided by just wrapping fs in a StreamReader. This would make the code as follows.
public static string CreateAndSave(IEnumerable<string> orderPages, string filePath)
{
if (orderPages == null || !orderPages.Any())
{
return string.Empty;
}
string result;
var writerSettings = new XmlWriterSettings
{
Indent = true,
Encoding = Encoding.GetEncoding("ISO-8859-1"),
CheckCharacters = false,
ConformanceLevel = ConformanceLevel.Document
};
using (var fs = new FileStream(filePath, FileMode.Create, FileAccess.ReadWrite))
{
try
{
XmlWriter xmlWriter = XmlWriter.Create(fs, writerSettings);
xmlWriter.WriteStartElement("PRINT_JOB");
foreach (var page in orderPages)
{
xmlWriter.WriteElementString("PAGE", page);
}
xmlWriter.WriteFullEndElement();
xmlWriter.Close(); // Flush from xmlWriter to fs
fs.Seek(0, SeekOrigin.Begin); // Go back to read from the begining
var reader = new StreamReader(fs, writerSettings.Encoding);
result = reader.ReadToEnd();
// reader.Close(); // This would just flush/close fs early(which would be OK)
}
catch (Exception ex)
{
throw;
}
}
return result;
}

I know I'm late, but there seems to be a simpler solution. You want your function to generate xml, write it to a file and return the generated xml. Apparently allocating a string cannot be avoided (because you want it to be returned), same for writing to a file. But reading from a file (as in your and SensorSmith's solutions) can easily be avoided by simply "swapping" the operations - generate xml string and write it to a file. Like this:
var output = new StringBuilder();
var writerSettings = new XmlWriterSettings { /* your settings ... */ };
using (var xmlWriter = XmlWriter.Create(output, writerSettings))
{
// Your xml generation code using the writer
// ...
// You don't need to flush the writer, it will be done automatically
}
// Here the output variable contains the xml, let's take it...
var xml = output.ToString();
// write it to a file...
File.WriteAllText(filePath, xml);
// and we are done :-)
return xml;
IMPORTANT UPDATE: It turns out that the XmlWriter.Create(StringBuider, XmlWriterSettings) overload ignores the Encoding from the settings and always uses "utf-16", so don't use this method if you need other encoding.

Serializing to MemoryStream causes an OutOfmemoryException, but serializing to a FileStream does not. Can anyone tell me why?

I'm using Newtonsoft Json.Net to serialize objects as json. I continually run into an OutOfMemoryException when I try to serialize to a MemoryStream, but not when I serialize to a FileStream. Could someone explain why this might be happening? These are the two methods that I am using to serialize.
Throws an OutOfMemoryException
private static MemoryStream _serializeJson<T>(T obj)
{
try
{
var stream = new MemoryStream();
var streamWriter = new StreamWriter(stream);
var jsonWriter = new JsonTextWriter(streamWriter);
var serializer = new JsonSerializer();
serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();
serializer.Formatting = Formatting.Indented;
serializer.Serialize(jsonWriter, obj);
streamWriter.Flush();
stream.Position = 0;
return stream;
}
catch (Exception e)
{
//Logger.WriteError(e.ToString());
Console.WriteLine(e.ToString());
return null;
}
}
Doesn't throw an OutOfMemoryException
private static void _serializeJsonToFile<T>(T obj, string path)
{
try
{
using (FileStream fs = File.Open(path, FileMode.Create, FileAccess.ReadWrite))
using (StreamWriter sw = new StreamWriter(fs))
using (JsonWriter jw = new JsonTextWriter(sw))
{
jw.Formatting = Formatting.Indented;
JsonSerializer serializer = new JsonSerializer();
serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();
serializer.Serialize(jw, obj);
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
P.S. some might ask why I would want to return a stream instead of simply serializing to the file stream. This is because I want to keep serialization in one class and file handling in another, so I'm passing the memory stream to a WriteFile method in another class later.

You are getting OutOfMemoryExceptions because the memory stream is very aggressive about it's growth. Every time it needs to re-size it doubles it's internal buffer.
//The code from MemoryStream http://referencesource.microsoft.com/mscorlib/system/io/memorystream.cs.html#1416df83d2368912
private bool EnsureCapacity(int value) {
// Check for overflow
if (value < 0)
throw new IOException(Environment.GetResourceString("IO.IO_StreamTooLong"));
if (value > _capacity) {
int newCapacity = value;
if (newCapacity < 256)
newCapacity = 256;
// We are ok with this overflowing since the next statement will deal
// with the cases where _capacity*2 overflows.
if (newCapacity < _capacity * 2)
newCapacity = _capacity * 2;
// We want to expand the array up to Array.MaxArrayLengthOneDimensional
// And we want to give the user the value that they asked for
if ((uint)(_capacity * 2) > Array.MaxByteArrayLength)
newCapacity = value > Array.MaxByteArrayLength ? value : Array.MaxByteArrayLength;
Capacity = newCapacity;
return true;
}
return false;
}
With a 17.8 MB file that is a worst case scenario of a 35.6 MB byte array being used. The old byte arrays that where discarded during the resizing process can also cause Memory Fragmentation depending on how long they live, this can easily get your program to throw a OOM error before you get to the 32 bit memory limit.
Writing directly to a FileStream does not require any large buffers to be created in memory so it uses much less space.
There is a way to separate the logic of the saving from the serializing, just pass in the stream to the function instead of creating it in the function itself.
private static void _serializeJson<T>(T obj, Stream stream)
{
try
{
using(var streamWriter = new StreamWriter(stream, Encoding.UTF8, 1024, true))
using(var jsonWriter = new JsonTextWriter(streamWriter))
{
var serializer = new JsonSerializer();
serializer.ContractResolver = new CamelCasePropertyNamesContractResolver();
serializer.Formatting = Formatting.Indented;
serializer.Serialize(jsonWriter, obj);
}
}
catch (Exception e)
{
//Logger.WriteError(e.ToString());
Console.WriteLine(e.ToString());
}
}
I also dispose of the StreamWriter that is created, the constructor I used has a leaveOpen flag which causes the underlying stream to not be closed when you dispose of the StreamWriter.

How to Deserialize using binary Deserialization from file [duplicate]

This question already has an answer here:
How to Deserialize using binary Deserialization from file text file
(1 answer)
Closed 8 years ago.
public static void SaveRestaurantList(List<Restaurant> restaurantList)
{
FileStream fs = new FileStream("Restaurant.txt", FileMode.Create, FileAccess.Write);
BinaryFormatter bf = new BinaryFormatter();
for (int i = 0; i < restaurantList.Count; i++)
{
Restaurant r = new Restaurant();
r = (Restaurant)restaurantList[i];
bf.Serialize(fs, r);
fs.Flush();
}
fs.Close();
Console.WriteLine("\n\n\t\t File Get Serialized.., \n\t\t Close the Promt and Check in Application Debug Folder..!!");
}
I have Serailze the generic list which I have, into "Restaurant.txt" file.
now I want to Deserialize the same and return it into a Generic List, I have tried
but its not working and it is giving error "Invalid Cast Expression".
Can anyone please help in solving out this.

You should serialize the complete list itself.
using (Stream stream = File.Open("data.bin", FileMode.Create))
{
BinaryFormatter bin = new BinaryFormatter();
bin.Serialize(stream, restaurantList);
}
You can later deserialize the complete list like this
using (Stream stream = File.Open("data.bin", FileMode.Open))
{
BinaryFormatter bin = new BinaryFormatter();
var restaurantList=(List<Restaurant>)bin.Deserialize(stream);
}

Kapadni,
I am storing my list/BindingList of object in .xml file and may be below functions/code will help you to serialize and de-serialize object and store/retrieve from .xml file
BindingList<IntradayData> objIntradayDataList;
SerializeObject(objIntradayDataList, filepath);
objIntradayDataList = DeSerializeObject<BindingList<IntradayData>>(filepath);
public void SerializeObject<T>(T serializableObject, string fileName)
{
if (serializableObject == null) { return; }
try
{
XmlDocument xmlDocument = new XmlDocument();
XmlSerializer serializer = new XmlSerializer(serializableObject.GetType());
using (MemoryStream stream = new MemoryStream())
{
serializer.Serialize(stream, serializableObject);
stream.Position = 0;
xmlDocument.Load(stream);
xmlDocument.Save(fileName);
stream.Close();
}
}
catch (Exception ex)
{
//Log exception here
log.Error("SerializeObject ", ex);
}
}
public T DeSerializeObject<T>(string fileName)
{
if (string.IsNullOrEmpty(fileName)) { return default(T); }
T objectOut = default(T);
try
{
string attributeXml = string.Empty;
XmlDocument xmlDocument = new XmlDocument();
xmlDocument.Load(fileName);
string xmlString = xmlDocument.OuterXml;
using (StringReader read = new StringReader(xmlString))
{
Type outType = typeof(T);
XmlSerializer serializer = new XmlSerializer(outType);
using (XmlReader reader = new XmlTextReader(read))
{
objectOut = (T)serializer.Deserialize(reader);
reader.Close();
}
read.Close();
}
}
catch (Exception ex)
{
//Log exception here
log.Error("DeSerializeObject ", ex);
}
return objectOut;
}

Not disposing MemoryStream has side affects when disposing the StreamReader that uses it?

Given the following code snippet:
try
{
var myTxt = "";
var serializer = new DataContractSerializer(myObject.GetType());
var memoryStream = new MemoryStream()
serializer.WriteObject(memoryStream, myObject);
memoryStream.Position = 0;
using (var reader = new StreamReader(memoryStream))
{
myTxt = reader.ReadToEnd();
}
.
.
.
}
catch (IOException ioEx)
{
//log or whatever...
throw;
}
Typically I'd have a using statement around my memory stream,
but from my understanding disposing the StreamReader will close the Memory Stream.
So the question is, is there anything REALLY wrong with the above?

You could simplify your code as I don't see the point of writing, rewinding and reading:
var serializer = new DataContractSerializer(facets.GetType());
using (var stream = new MemoryStream())
{
serializer.WriteObject(stream, facets);
string xml = Encoding.UTF8.GetString(stream.ToArray());
}

I think its fine because MemoryStream doesn't hold unmanaged resources anyway, but if you wanted to be extra safe you could do this:
var memoryStream = new MemoryStream()
StreamReader reader = null;
try{
serializer.WriteObject(memoryStream, myObject);
memoryStream.Position = 0;
reader = new StreamReader(memoryStream)
//...
}
finally
{
if(reader != null)
reader.Dispose();
else
memoryStream.Dispose();
}

It is a good practice to use either
using (var stream = new MemoryStream(...))
using (var reader = new StreamReader(stream))
{
myTxt = reader.ReadToEnd();
}
Here both stream and reader will be disposed by runtime
or Close() manually both reader(writer) and stream

To test your scenario, I've added
memoryStream.Seek(0, SeekOrigin.Begin);
to the end and received an System.ObjectDisposedException, your code seems valid.
Of course vcsjones's comment is also valid.