We Keep Coding

XMLSerializer - issue with UTF-8 vs UTF-16 Code

I am trying to serialize a simple object (5 string properties) into XML to save to a DB Image field. Then I need to DeSerialize it back into a string later in the program.
However, I am getting some errors - caused by the XML being saved thinking it is in UTF-16 - however, when I load it from the DB back into a string - it thinks it is a UTF 8 String.
The error I get is
InnerException {"There is no Unicode byte order mark. Cannot switch to Unicode."} System.Exception {System.Xml.XmlException}
-- Message "There is an error in XML document (0, 0)." string
Is this happening because of the two different ways I save and load the string to/from the DB? On the save I am using a StringBuilder - but on the load from DB I am using just a String.
Thoughts?
Serialize and Save to DB
// Now Save the OBject XML to the Query Tables
var serializer = new XmlSerializer(ExportConfig.GetType());
StringBuilder StringResult = new StringBuilder();
using (var writer = XmlWriter.Create(StringResult))
{
serializer.Serialize(writer, ExportConfig);
}
//MessageBox.Show("XML : " + StringResult);
// Now Save to the Query
try
{
string UpdateSQL = "Update ZQryRpt "
+ " Set ExportConfig = " + TAGlobal.QuotedStr(StringResult.ToString())
+ " where QryId = " + TAGlobal.QuotedStr(((DataRowView)bindingSource_zQryRpt.Current).Row["QryID"].ToString())
;
ExecNonSelectSQL(UpdateSQL, uniConnection_Config);
}
catch (Exception Error)
{
MessageBox.Show("Error Setting ExportConfig: " + Error.Message);
}
Load from DB And Deserialize
byte[] binaryData = (byte[])((DataRowView)bindingSource_zQryRpt.Current).Row["ExportConfig"];
string XMLStored = System.Text.Encoding.UTF8.GetString(binaryData, 0, binaryData.Length);
if (XMLStored.Length > 0)
{
IIDExportObject ExportConfig = new IIDExportObject();
var serializer = new XmlSerializer(ExportConfig.GetType());
//StringBuilder StringResult = new StringBuilder(XMLStored);
// Load the XML from the Query into the StringBuilder
// Now we need to build a Stream from the String to use in the XMLReader
byte[] byteArray = Encoding.UTF8.GetBytes(XMLStored);
MemoryStream stream = new MemoryStream(byteArray);
using (var reader = XmlReader.Create(stream))
{
ExportConfig = (IIDExportObject)serializer.Deserialize(reader);
}
}

John - thank you very much for the comment! It allowed me to complete the code and find a solution.
As you noted - using a stream reader was the solution - but I could not read the first line because there was only one 'line' in my string. However, I could use the line
using (StreamReader sr = new StreamReader(stream, false))
Which allows me to read the stream and ignore the "Byte Order Mark Detection" set to false.
string XMLStored = MainFormRef.GetExportConfigForCurrentQuery();
if (XMLStored.Length > 0)
{
IIDExportObject ExportConfig = new IIDExportObject();
try
{
var serializer = new XmlSerializer(ExportConfig.GetType());
// Now we need to build a Stream from the String to use in the XMLReader
byte[] byteArray = Encoding.UTF8.GetBytes(XMLStored);
MemoryStream stream = new MemoryStream(byteArray);
// Now we need to use a StreamReader to get around UTF8 vs UTF16 issues
// A little cumbersome - but it works
using (StreamReader sr = new StreamReader(stream, false))
{
using (var reader = XmlReader.Create(sr))
{
ExportConfig = (IIDExportObject)serializer.Deserialize(reader);
}
}
}
catch
{
}
I am not sure this is the best solution - but it works. I will be curious to see if anyone else has a better way of dealing with this.

Thanks to G Bradley, I took his answer and generalized it a bit to make it a bit easier to call.
public static string SerializeToXmlString<T>(T objectToSerialize)
{
XmlSerializer serializer = new XmlSerializer(typeof(T));
XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = false;
settings.Encoding = Encoding.UTF8;
StringBuilder builder = new StringBuilder();
using (XmlWriter writer = XmlWriter.Create(builder, settings))
{
serializer.Serialize(writer, objectToSerialize);
}
return builder.ToString();
}
public static T DeserializeFromXmlString<T>(string xmlString)
{
if (string.IsNullOrWhiteSpace(xmlString))
return default;
var serializer = new XmlSerializer(typeof(T));
byte[] byteArray = Encoding.UTF8.GetBytes(xmlString);
MemoryStream stream = new MemoryStream(byteArray);
using (StreamReader sr = new StreamReader(stream, false))
{
using (var reader = XmlReader.Create(sr))
{
return (T)serializer.Deserialize(reader);
}
}
}

Convert LINQ result to CSV or Excel

Using a LINQ query I need to export to Excel when a WebApi method is called. I have built the LINQ query that will return the correct data, now I need it to export to .csv or Excel file format.
I have tried using MemoryStream and StreamWriter but I think I am just chasing my tail now.
[HttpGet]
[Route("Download")]
public Task<IActionResult> Download(int memberId)
{
var results = (from violations in _db.tblMappViolations
where violations.MemberID == memberId
select new IncomingViolations
{
Contact = violations.ContactName,
Address = violations.str_Address,
City = violations.str_City,
State = violations.str_State,
Zip = violations.str_Zipcode,
Country = violations.str_Country,
Phone = violations.str_Phone,
Email = violations.str_Email,
Website = violations.str_WebSite,
}).FirstOrDefault();
MemoryStream stream = new MemoryStream(results);
StreamWriter writer = new StreamWriter(stream);
writer.Flush();
stream.Position = 0;
FileStreamResult response = File(stream, "application/octet-stream");
response.FileDownloadName = "violations.csv";
return response;
}

Here is how you can send CSV file to the user from server.
string attachment = "attachment; filename=MyCsvLol.csv";
HttpContext.Current.Response.Clear();
HttpContext.Current.Response.ClearHeaders();
HttpContext.Current.Response.ClearContent();
HttpContext.Current.Response.AddHeader("content-disposition", attachment);
HttpContext.Current.Response.ContentType = "text/csv";
HttpContext.Current.Response.AddHeader("Pragma", "public");
var sb = new StringBuilder();
// Add your data into stringbuilder
sb.Append(results.Contact);
sb.Append(results.Address);
sb.Append(results.City);
// and so on
HttpContext.Current.Response.Write(sb.ToString());
For Sending it from API
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(stream);
// Write Your data here in writer
writer.Write("Hello, World!");
writer.Flush();
stream.Position = 0;
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType = new MediaTypeHeaderValue("text/csv");
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = "Export.csv" };
return result;
Update:-
public HttpResponseMessage Download()
{
var results = (from violations in _db.tblMappViolations
where violations.MemberID == memberId
select new IncomingViolations
{
Contact = violations.ContactName,
Address = violations.str_Address,
City = violations.str_City,
State = violations.str_State,
Zip = violations.str_Zipcode,
Country = violations.str_Country,
Phone = violations.str_Phone,
Email = violations.str_Email,
Website = violations.str_WebSite,
});
var sb = new StringBuilder();
MemoryStream stream = new MemoryStream();
StreamWriter writer = new StreamWriter(stream);
foreach(var tempResult in results)
{
sb.Append(tempResult.Contact+",");
sb.Append(tempResult.Address+",");
sb.Append(tempResult.City+",");
sb.Append(tempResult.State+",");
sb.Append(tempResult.Zip+",");
sb.Append(tempResult.Country+",");
sb.Append(tempResult.Phone+",");
sb.Append(tempResult.Email+",");
sb.Append(tempResult.Website+",");
sb.Append(Enviroment.NewLine);
}
writer.Write(sb.ToString());
writer.Flush();
stream.Position = 0;
HttpResponseMessage result = new HttpResponseMessage(HttpStatusCode.OK);
result.Content = new StreamContent(stream);
result.Content.Headers.ContentType = new MediaTypeHeaderValue("text/csv");
result.Content.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = "Export.csv" };
return result;
}

First, to reuse the code in other areas, always create helper classes.
I adopted this method of converting list into a stream with headers as property names, if you want a file from this, essentially, I would just add another step to this:
STEP 1:
public static Stream ConvertToCSVStream<T>(IEnumerable<T> objects)
{
Type itemType = typeof(T);
var properties = itemType.GetProperties();
var mStream = new MemoryStream();
StreamWriter sWriter = new StreamWriter(mStream);
var values = objects.Select(o =>
{
return string.Join(",", properties.Select(p =>
{
var value = p.GetValue(o).ToString();
if (!Regex.IsMatch(value, "[,\"\\r\\n]"))
{
return value;
}
value = value.Replace("\"", "\"\"");
return string.Format("\"{0}\"", value);
})) + sWriter.NewLine;
});
var valuesInStrings = values.Aggregate((current, next) => current + next);
try
{
sWriter.Write(string.Join(",", properties.Select(x => x.Name.Replace("_", " "))) + sWriter.NewLine);
sWriter.Write(valuesInStrings);
}
catch (Exception e)
{
mStream.Close();
throw e;
}
sWriter.Flush();
mStream.Position = 0;
return mStream;
}
if your data is text, just convert it directly to a file result but if not, you must convert it to binary array and write it to stream, refer to this article for converting it to binary data, in our case, for csv, you could just use the FileStream result that you've implemented in a separate method:
STEP 2:
public FileStreamResult CreateFile(MemoryStream mStream, string path, string name)
{
//set values, names, content type, etc
//return filestream
}
or any other method you find better.

Save your result in a DataTable and then just use this
XLWorkbook workbook = new XLWorkbook();
DataTable table = GetYourTable();
workbook.Worksheets.Add(table);
And you should definitely use stream writer for this if you know which file its going to write to from the start, else stream reader and then stream writer.

I want to serialize my object to a string, but I did get an error "System.OutOfMemoryException"

I want to serialize my object to a string, but I did get an error:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Text.StringBuilder.ToString()
at System.IO.StreamReader.ReadToEnd()
at TEG.USBSharing.Utility.FileUtility.ValidateReadByStreamReader(StreamReader streamReader)
This is my code:
(1) Serialization
public static string Serialization<T>(T obj)
{
using (MemoryStream memoryStream = new MemoryStream())
using (StreamReader reader = new StreamReader(memoryStream))
{
DataContractSerializer serializer = new DataContractSerializer(obj.GetType());
serializer.WriteObject(memoryStream, obj);
memoryStream.Position = 0;
return FileUtility.ValidateReadByStreamReader(reader);
}
}
(2) ReadToEnd
public string ValidateReadByStreamReader(StreamReader streamReader)
{
string m_Data = string.Empty;
if (streamReader != null)
{
m_Data = streamReader.ReadToEnd();
}
return m_Data;
}
(3) My object
DataPacKet m_DataPacKet = new DataPacKet() {
Command = DataPacKet.COMMAND_SEND_FILE_TRANSFER_CONTENT,
GuidId = m_Guid,
Data = new DataPacKetContent()
{
Sequence = i,
DataBase64 = Convert.ToBase64String(m_SendingBuffer),
},
TransferType = dataPacketRequest.TransferType }; string m_MessageFileDataPacKet =
JsonUtility.Serialization<DataPacKet>(m_DataPacKet);
Thanks for your help.

If you are sure your system has enough memory to contain your object, you can try compiling for x64:
If you are using Visual Studio, you can go to the project's properties and set the Platform target to x64 under Build tab.

Using XmlDiffPatch when writing to stream

I am trying to use xmldiffpatch to write to a stream.
The first method is to write my xml to a memory stream.
The second method loads an xml from a file and creates a stream for the patched file to be written into. The third method actually compares the two files. I'm always getting that both files are identical, even though they are not, so I know that I am missing something.
Any help is appreciated!
public MemoryStream FirstXml()
{
string[] names = { "John", "Mohammed", "Marc", "Tamara", "joy" };
MemoryStream ms = new MemoryStream();
XmlTextWriter xtw= new XmlTextWriter(ms, Encoding.UTF8);
xtw.WriteStartDocument();
xtw.WriteStartElement("root");
foreach (string s in names)
{
xtw.WriteStartElement(s);
xtw.WriteEndElement();
}
xtw.WriteEndElement();
xtw.WriteEndDocument();
return ms;
}
public Stream SecondXml()
{
XmlReader finalFile =XmlReader.Create(#"c:\......\something.xml");
MemoryStream ms = FirstXml();
XmlReader originalFile = XmlReader.Create(ms);
MemoryStream ms2 = new MemoryStream();
XmlTextWriter dgw = new XmlTextWriter(ms2, Encoding.UTF8);
GenerateDiffGram(originalFile, finalFile, dgw);
return ms2;
}
public void GenerateDiffGram(XmlReader originalFile, XmlReader finalFile,
XmlWriter dgw)
{
XmlDiff xmldiff = new XmlDiff();
bool bIdentical = xmldiff.Compare(originalFile, finalFile, dgw);
dgw.Close();
StreamReader sr = new StreamReader(SecondXml());
string xmlOutput = sr.ReadToEnd();
if(xmlOutput.Contains("</xd:xmldiff>"))
{Console.WriteLine("Xml files are not identical");
Console.Read();}
else
{Console.WriteLine("Xml files are identical");Console.Read();}
}

The following modified version works.
static void Main()
{
SecondXml();
}
public static string FirstXml()
{
string[] names = { "John", "Mohammed", "Marc", "Tamara", "joy" };
var sw = new StringWriter();
var xtw = new XmlTextWriter(sw);
xtw.WriteStartDocument();
xtw.WriteStartElement("root");
foreach (string s in names)
{
xtw.WriteStartElement(s);
xtw.WriteEndElement();
}
xtw.WriteEndElement();
xtw.WriteEndDocument();
return sw.ToString();
}
public static void SecondXml()
{
string secondXml = File.ReadAllText(#"t:\something.xml");
string firstXml = FirstXml();
Console.WriteLine("Comparing...");
string result = GenerateDiffGram(firstXml, secondXml);
Console.WriteLine(result);
Console.WriteLine();
Console.WriteLine("Finished compare");
Console.Out.Write(firstXml);
Console.WriteLine();
Console.WriteLine();
Console.WriteLine(secondXml);
}
public static string GenerateDiffGram(string originalFile, string finalFile)
{
var xmldiff = new XmlDiff();
var r1 = XmlReader.Create(new StringReader(originalFile));
var r2 = XmlReader.Create(new StringReader(finalFile));
var sw = new StringWriter();
var xw = new XmlTextWriter(sw) {Formatting = Formatting.Indented};
bool bIdentical = xmldiff.Compare(r1, r2, xw);
Console.WriteLine();
Console.WriteLine("bIdentical: " + bIdentical);
return sw.ToString();
}
I'm actually not entirely sure what's wrong with your original code. The XML being compared is an empty string in both the first and second readers. Since you're using memory streams as the backing stores anyways, then you won't lose anything by just using strings as the above does.

Persist a DataContract as XML in a database

I'm working on a kind of "store and forward" application for WCF services. I want to save the message in a database as a raw XML blob, as XElement. I'm having a bit of trouble converting the datacontract into the XElement type I need for the database call. Any ideas?

this returns it as a string, which you can put into the db into an xml column. Here is a good generic method you can use to serialize datacontracts.
public static string Serialize<T>(T obj)
{
StringBuilder sb = new StringBuilder();
DataContractSerializer ser = new DataContractSerializer(typeof(T));
ser.WriteObject(XmlWriter.Create(sb), obj);
return sb.ToString();
}
btw, are you using linq to sql? The reason i ask is because of the XElement part of your question. if thats the case, you can modify this in the .dbml designer to use a string as the CLR type, and not the default XElement.

The most voted on answer (Jason W. posted) did not work for me. I dont know why that answer got the most votes. But after searching around I found this
http://billrob.com/archive/2010/02/09/datacontractserializer-converting-objects-to-xml-string.aspx
Which worked for my project. I just had a few classes and put the datacontract and datamemeber attributes on classes and properties and then wanted to get an XML string which I could write to the database.
Code from the link above incase it goes 404:
Serializes:
var serializer = new DataContractSerializer(tempData.GetType());
using (var backing = new System.IO.StringWriter())
using (var writer = new System.Xml.XmlTextWriter(backing))
{
serializer.WriteObject(writer, tempData);
data.XmlData = backing.ToString();
}
Deserializes:
var serializer = new DataContractSerializer(typeof(T));
using (var backing = new System.IO.StringReader(data.XmlData))
using (var reader = new System.Xml.XmlTextReader(backing))
{
return serializer.ReadObject(reader) as T;
}

If your database is SQL Server 2005 or above, you can use the XML data type:
private readonly DataContractToSerialize _testContract =
new DataContractToSerialize
{
ID = 1,
Name = "One",
Children =
{
new ChildClassToSerialize {ChildMember = "ChildOne"},
new ChildClassToSerialize {ChildMember = "ChildTwo"}
}
};
public void SerializeDataContract()
{
using (var outputStream = new MemoryStream())
{
using (var writer = XmlWriter.Create(outputStream))
{
var serializer =
new DataContractSerializer(_testContract.GetType());
if (writer != null)
{
serializer.WriteObject(writer, _testContract);
}
}
outputStream.Position = 0;
using (
var conn =
new SqlConnection(Settings.Default.ConnectionString))
{
conn.Open();
const string INSERT_COMMAND =
#"INSERT INTO XmlStore (Data) VALUES (#Data)";
using (var cmd = new SqlCommand(INSERT_COMMAND, conn))
{
using (var reader = XmlReader.Create(outputStream))
{
var xml = new SqlXml(reader);
cmd.Parameters.Clear();
cmd.Parameters.AddWithValue("#Data", xml);
cmd.ExecuteNonQuery();
}
}
}
}
}

I'm not sure about the most efficient way to get it to an XElement, but to get it to a string just run:
DataContractSerializer serializer = new DataContractSerializer(typeof(Foo));
using (MemoryStream memStream = new MemoryStream())
{
serializer.WriteObject(memStream, fooInstance);
byte[] blob = memStream.ToArray();
}

I tried to use Jason w'Serialize function that uses StringBuilder , but it returns empty string for LingToSQL Designer generated table class
with [DataContract()] attribute
However if I serialze to byte array as suggested by AgileJon
and then use UTF7Encoding to convert to string , it creates readable XML string.
static string DataContractSerializeUsingByteArray<T>(T obj)
{
string sRet = "";
DataContractSerializer serializer = new DataContractSerializer(typeof(T));
using (MemoryStream memStream = new MemoryStream())
{
serializer.WriteObject(memStream, obj);
byte[] blob = memStream.ToArray();
var encoding= new System.Text.UTF7Encoding();
sRet = encoding.GetString(blob);
}
return sRet;
}
Not sure why stringBuilder solution not working.