We Keep Coding

Writing compact xml with XmlDictionaryWriter.CreateBinaryWriter and a XmlDictionary

I want to write an xml document to disk in a compact format. To this end, I use the net framework method XmlDictionaryWriter.CreateBinaryWriter(Stream stream,IXmlDictionary dictionary)
This method writes a custom compact binary xml representation, that can later be read by XmlDictionaryWriter.CreateBinaryReader. The method accepts an XmlDictionary that can contain common strings, so that those strings do not have to be printed in the output each time. Instead of the string, the dictionary index will be printed in the file. CreateBinaryReader can later use the same dictionary to reverse the process.
However the dictionary I pass is apparently not used. Consider this code:
using System.IO;
using System.Xml;
using System.Xml.Linq;
class Program
{
public static void Main()
{
XmlDictionary dict = new XmlDictionary();
dict.Add("myLongRoot");
dict.Add("myLongAttribute");
dict.Add("myLongValue");
dict.Add("myLongChild");
dict.Add("myLongText");
XDocument xdoc = new XDocument();
xdoc.Add(new XElement("myLongRoot",
new XAttribute("myLongAttribute", "myLongValue"),
new XElement("myLongChild", "myLongText"),
new XElement("myLongChild", "myLongText"),
new XElement("myLongChild", "myLongText")
));
using (Stream stream = File.Create("binaryXml.txt"))
using (var writer = XmlDictionaryWriter.CreateBinaryWriter(stream, dict))
{
xdoc.WriteTo(writer);
}
}
}
The produced output is this (binary control characters not shown)
#
myLongRootmyLongAttribute˜myLongValue#myLongChild™
myLongText#myLongChild™
myLongText#myLongChild™
myLongText
So apparently the XmlDictionary has not been used. All strings appear in their entirety in the output, even multiple times.
This is not a problem limited to XDocument. In the above minimal example I used a XDocument to demonstrate the problem, but originally I stumbled upon this while using XmlDictionaryWriter in conjunction with a DataContractSerializer, as it is commonly used. The results were the same:
[Serializable]
public class myLongChild
{
public double myLongText = 0;
}
...
using (Stream stream = File.Create("binaryXml.txt"))
using (var writer = XmlDictionaryWriter.CreateBinaryWriter(stream, dict))
{
var dcs = new DataContractSerializer(typeof(myLongChild));
dcs.WriteObject(writer, new myLongChild());
}
The resulting output did not use my XmlDictionary.
How can I get XmlDictionaryWriter to use the suplied XmlDictionary?
Or have I misunderstood how this works?
with the DataContractSerializer approach, I tried debugging the net framework code (visual studio/options/debugging/enable net. framework source stepping). Apparently the Writer does attempt to lookup each of the above strings in the dictionary, as expected. However the lookup fails in line 356 of XmlbinaryWriter.cs, for reasons that are not clear to me.
Alternatives I have considered:
There is an overload for XmlDictionaryWriter.CreatebinaryWriter, that also accepts a XmlBinaryWriterSession. The writer then adds any new strings it encounters into the session dictionary. However, I want to only use a static dictionary for reading and writing, which is known beforehand.
I could wrap the whole thing into a GzipStream and let the compression take care of the multiple instances of strings. However, this would not compress the first instance of each string, and seems like a clumsy workaround overall.

Yes there is a misunderstanding. XmlDictionaryWriter is primarily used for serialization of objects and it is child class of XmlWriter. XDocument.WriteTo(XmlWriter something) takes XmlWriter as argument. The call XmlDictionaryWriter.CreateBinaryWriter will create an instance of System.Xml.XmlBinaryNodeWriter internally. This class has both methods for "regular" writing:
// override of XmlWriter
public override void WriteStartElement(string prefix, string localName)
{
// plain old "xml" for me please
}
and for dictionary based approach:
// override of XmlDictionaryWriter
public override void WriteStartElement(string prefix, XmlDictionaryString localName)
{
// I will use dictionary to hash element names to get shorter output
}
The later is mostly used if you serialize object via DataContractSerializer (notice its method WriteObject takes argument of both XmlDictionaryWriter and XmlWriter type), while XDocument takes just XmlWriter.
As for your problem - if I were you I'd make my own XmlWriter:
class CustomXmlWriter : XmlWriter
{
private readonly XmlDictionaryWriter _writer;
public CustomXmlWriter(XmlDictionaryWriter writer)
{
_writer = writer;
}
// override XmlWriter methods to use the dictionary-based approach instead
}
UPDATE (based on your comment)
If you indeed use DataContractSerializer you have few mistakes in your code.
1) POC classes have to be decorated with [DataContract] and [DataMember] attribute, the serialized value should be property and not field; also set namespace to empty value or you'll have to deal with namespaces in your dictionary as well. Like:
namespace XmlStuff {
[DataContract(Namespace = "")]
public class myLongChild
{
[DataMember]
public double myLongText { get; set; }
}
[DataContract(Namespace = "")]
public class myLongRoot
{
[DataMember]
public IList<myLongChild> Items { get; set; }
}
}
2) Provide instance of session as well; for null session the dictionary writer uses default (XmlWriter-like) implementation:
// order matters - add new items only at the bottom
static readonly string[] s_Terms = new string[]
{
"myLongRoot", "myLongChild", "myLongText",
"http://www.w3.org/2001/XMLSchema-instance", "Items"
};
public class CustomXmlBinaryWriterSession : XmlBinaryWriterSession
{
private bool m_Lock;
public void Lock() { m_Lock = true; }
public override bool TryAdd(XmlDictionaryString value, out int key)
{
if (m_Lock)
{
key = -1;
return false;
}
return base.TryAdd(value, out key);
}
}
static void InitializeWriter(out XmlDictionary dict, out XmlBinaryWriterSession session)
{
dict = new XmlDictionary();
var result = new CustomXmlBinaryWriterSession();
var key = 0;
foreach(var term in s_Terms)
{
result.TryAdd(dict.Add(term), out key);
}
result.Lock();
session = result;
}
static void InitializeReader(out XmlDictionary dict, out XmlBinaryReaderSession session)
{
dict = new XmlDictionary();
var result = new XmlBinaryReaderSession();
for (var i = 0; i < s_Terms.Length; i++)
{
result.Add(i, s_Terms[i]);
}
session = result;
}
static void Main(string[] args)
{
XmlDictionary dict;
XmlBinaryWriterSession session;
InitializeWriter(out dict, out session);
var root = new myLongRoot { Items = new List<myLongChild>() };
root.Items.Add(new myLongChild { myLongText = 24 });
root.Items.Add(new myLongChild { myLongText = 25 });
root.Items.Add(new myLongChild { myLongText = 27 });
byte[] buffer;
using (var stream = new MemoryStream())
{
using (var writer = XmlDictionaryWriter.CreateBinaryWriter(stream, dict, session))
{
var dcs = new DataContractSerializer(typeof(myLongRoot));
dcs.WriteObject(writer, root);
}
buffer = stream.ToArray();
}
XmlBinaryReaderSession readerSession;
InitializeReader(out dict, out readerSession);
using (var stream = new MemoryStream(buffer, false))
{
using (var reader = XmlDictionaryReader.CreateBinaryReader(stream, dict, new XmlDictionaryReaderQuotas(), readerSession))
{
var dcs = new DataContractSerializer(typeof(myLongRoot));
var rootCopy = dcs.ReadObject(reader);
}
}
}

How do i get a list of json objects from json file

I could serialize one node of json file like below
using (JsonTextReader jsonReader = new JsonTextReader(file))
{
deserializedJson = eerializer.Deserialize<Person>(jsonReader);
}
JsonNRadio is autogenerated class with json file.
Now my json file has multiple json objects.
so, for this i created a class called Persons
public class Persons
{
private Person[] person;
public Person[] person
{
get
{
return this.person;
}
set
{
this.person= value;
}
}
}
public partial class Person
{
public string id { get; set; }
public string name { get; set; }
}
Now I should get a aaray of json objects but when I am trying with below code, I am getting only the first node.
JsonSerializer serializer = new JsonSerializer();
using (StreamReader file = File.OpenText(#"D:\\Sample.json"))
// deserializedXml = serialized.Deserialize(xmlStream);
using (JsonTextReader jsonReader = new JsonTextReader(file))
{
Persons deserializedXml = serializer.Deserialize<Persons> (jsonReader);
}
And Person is object is coming null.
Could anybody help with an example or how to get a list of json objects.

For receiving json data with your Persons class above, your json file have to look like this:
{"Person":[{"id":"6397486515343190","name":"Henry"},{"id":"6397486515343192","name":"Paul"}]}
But since you have no influence on the design of the json data, you could simply deserialize it into a list of Person:
using (JsonTextReader jsonReader = new JsonTextReader(file))
{
var deserializedXml = serializer.Deserialize<List<Person>> (jsonReader);
}
This should work for you.
For editing your json string before deserialize it because of the missing "[" and "]". You can use something like that if you don't want to add much more new code:
using (JsonTextReader jsonReader = new JsonTextReader(file))
{
var json = "[" + jsonReader.Value.ToString() + "]";
var deserializedXml = JsonConvert.DeserializeObject<List<Person>>(json);
}
Or you can read the file with a normal FileReader into a string object and adding the "[", "]" at the beginning and ending.

Xml being deserialized into base class instead of derived classes

I know this is a popular topic and I have researched extensively without finding an answer to my problem.
I have a base class IntroductionAction and 2 derived classes IntroductionActionComplex and IntroductionActionSimple. I have a list of IntroductionAction objects to which I have added objects of both of the derived types. My classes are as follows:
[XmlInclude(typeof(IntroductionActionComplex))]
[XmlInclude(typeof(IntroductionActionSimple))]
public class IntroductionAction
{
public IntroductionAction() { }
}
public class IntroductionActionComplex : IntroductionAction
{
[XmlIgnore]
public string name { get; set; }
[XmlElement(ElementName = "QuestionString")]
public string question { get; set; }
[XmlElement(ElementName = "AnswerString")]
public List<string> answerStrings { get; set; }
public IntroductionActionComplex()
{
name = string.Empty;
question = null;
answerStrings = new List<string>();
}
}
public class IntroductionActionSimple : IntroductionAction
{
[XmlIgnore]
public string name { get; set; }
[XmlText]
public string Value { get; set; }
public IntroductionActionSimple()
{
Value = string.Empty;
}
}
I then create the List as follows
[XmlElement("IntroductionAction")]
public List<IntroductionAction> introductionActions { get; set; }
I am using XmlSerializer and everything serializes correctly. This is the resulting XML of the list containing one of each of the derived classes which is correct.
<IntroductionAction>
<QuestionString>
test
</QuestionString>
<AnswerString>
test
</AnswerString>
<AnswerString>
test
</AnswerString>
</IntroductionAction>
<IntroductionAction>
test
</IntroductionAction>
This XML file is going onto a device which doesn't read it as XML but just searches for the tags and does whatever work it needs to do and because of that the file can't contain any XSI or XSD tags, indentation, etc that is usually associated with proper XML.
My deserialization code is straight forward:
public static T Deserialize_xml_Config<T>(string file1, T obj)
{
XmlSerializer deserializer = new XmlSerializer(obj.GetType());
using (TextReader reader = new StreamReader(file1))
{
return (T)deserializer.Deserialize(reader);
}
}
Finally to my problem. When I deserialize, it is being deserialized to the base class IntroductionAction and not to the derived classes.
These IntroductionAction classes are just part of a much larger object that I am serializing/deserializing. I have tried making the base class abstract since it contains no functionality but I get an error on deserialization saying
The specified type is abstract: name='IntroductionAction'
Despite my XmlIncludes it seems unable to find the derived classes.
I have tried adding the types to the serializer but that didn't work.
Any help is much appreciated.
Edit:
This is what I mean by adding the types to the serializer
XmlSerializer deserializer = new XmlSerializer(obj.GetType(), new Type [] { typeof(IntroductionActionComplex), typeof(IntroductionActionSimple) });
using (TextReader reader = new StreamReader(file1))
{
return (T)deserializer.Deserialize(reader);
}
Also my attempt at using XmlAttributeOverrides:
XmlAttributeOverrides attrOverrides = new XmlAttributeOverrides();
var attrs = new XmlAttributes();
XmlElementAttribute attr = new XmlElementAttribute();
attr.ElementName = "IntroductionAction";
attr.Type = typeof(IntroductionActionComplex);
attrs.XmlElements.Add(attr);
attr.ElementName = "IntroductionAction";
attr.Type = typeof(IntroductionActionSimple);
attrs.XmlElements.Add(attr);
attrOverrides.Add(typeof(IntroductionAction), "IntroductionAction", attrs);
XmlSerializer deserializer = new XmlSerializer(obj.GetType(), attrOverrides);
using (TextReader reader = new StreamReader(file1))
{
return (T)deserializer.Deserialize(reader);
}

I think you are pretty close. Below is the full example of saving and loading the XML file based on derived class types. This will save the nodes as the derived type itself, so loading back in will keep the desired type, rather than convert back to the base type. You'll probably need to add exception handling, this was just a quick solution. I did not change your base IntroductionAction or the derived IntroductionActionComplex / IntroductionActionSimple classes.
public class RootNode
{
[XmlElement("IntroductionAction")]
public List<IntroductionAction> introductionActions { get; set; }
public RootNode()
{
introductionActions = new List<IntroductionAction>();
}
private static XmlAttributeOverrides GetXmlAttributeOverrides()
{
XmlAttributeOverrides xml_attr_overrides = new XmlAttributeOverrides();
XmlAttributes xml_attrs = new XmlAttributes();
xml_attrs.XmlElements.Add(new XmlElementAttribute(typeof(IntroductionActionComplex)));
xml_attrs.XmlElements.Add(new XmlElementAttribute(typeof(IntroductionActionSimple)));
xml_attr_overrides.Add(typeof(RootNode), "introductionActions", xml_attrs);
return xml_attr_overrides;
}
// Add exception handling
public static void SaveToFile(RootNode rootNode, string fileName)
{
using (MemoryStream mem_stream = new MemoryStream())
{
XmlSerializer serializer = new XmlSerializer(rootNode.GetType(), RootNode.GetXmlAttributeOverrides());
serializer.Serialize(mem_stream, rootNode);
using (BinaryWriter output = new BinaryWriter(new FileStream(fileName, FileMode.Create)))
{
output.Write(mem_stream.ToArray());
}
}
}
// Add exception handling
public static RootNode LoadFromFile(string fileName)
{
if (File.Exists(fileName))
{
using (FileStream file = new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using (TextReader reader = new StreamReader(file))
{
XmlSerializer serializer = new XmlSerializer(typeof(RootNode), RootNode.GetXmlAttributeOverrides());
return (RootNode)serializer.Deserialize(reader);
}
}
}
return null;
}
}
Test program:
internal class Program
{
private static void Main(string[] args)
{
RootNode obj = new RootNode();
obj.introductionActions.Add(new IntroductionActionComplex() { question = "qTest", answerStrings = { "aTest1", "aTest2" }, name = "aName1" });
obj.introductionActions.Add(new IntroductionActionSimple() { name = "aName2", Value = "aValue" });
RootNode.SaveToFile(obj, "Test.xml");
RootNode obj2 = RootNode.LoadFromFile("Test.xml");
}
}

How to serialize/de-serialize a Dictionary<string, object> into XML

I am trying to write a save routine for my application where several parts of the application add items to a Dictionary and then the save function writes them to a XML file. The open routine needs to read those files and re-populate the Dictionary and I can then place those objects back into my application. I am struggling with the de-serialization of the routine I have now. My save routine is as follows
XmlDocument xmlDoc = new XmlDocument();
// Write down the XML declaration
XmlDeclaration xmlDeclaration = xmlDoc.CreateXmlDeclaration("1.0", "utf-8", null);
// Create the root element
XmlElement rootNode = xmlDoc.CreateElement("TireStudy");
xmlDoc.InsertBefore(xmlDeclaration, xmlDoc.DocumentElement);
xmlDoc.AppendChild(rootNode);
foreach (var saveItem in _SaveItems)
{
XPathNavigator nav = rootNode.CreateNavigator();
using (var writer = nav.AppendChild())
{
var serializer = new XmlSerializer(saveItem.Value.GetType());
writer.WriteWhitespace("");
serializer.Serialize(writer, saveItem.Value);
writer.Close();
}
}
xmlDoc.Save(fileName);
This routine works to create a file, but I would like the key value of the dictionary to be saved in the file as well and I am not sure how to de-serialize the file this creates because I do not know the types of the objects before I read them.
Part 2 (I hate adding new parts to a question, but I don't see a better way to address the problems going forward)
I now have the following code,
var knownTypes = new List<Type>
{
typeof(ObservableCollection<string>),
typeof(ObservableCollection<Segments>),
typeof(Segments),
typeof(List<string>)
};
var serialized = _SaveItems.Serialize(knownTypes);
but I get the following exception
Type 'System.Collections.Generic.List`1[System.String]' cannot be added to list of known types since another type 'System.Collections.ObjectModel.ObservableCollection`1[System.String]' with the same data contract name 'http://schemas.microsoft.com/2003/10/Serialization/Arrays:ArrayOfstring' is already present. If there are different collections of a particular type - for example, List<Test> and Test[], they cannot both be added as known types. Consider specifying only one of these types for addition to the known types list.
If I delete either the typeof(ObservableCollection) or the typeof(List) it exceptions complaining it needs the one I deleted.

You could use DataContractSerializer as explained in this post but you may have to pass the known types as a parameter to the serializer to support nested object typed classes:
using System;
using System.Collections.Generic;
using System.Runtime.Serialization;
using System.IO;
using System.Xml;
public static class SerializationExtensions
{
public static string Serialize<T>(this T obj, IEnumerable<Type> knownTypes)
{
var serializer = new DataContractSerializer(obj.GetType(), knownTypes);
using (var writer = new StringWriter())
using (var stm = new XmlTextWriter(writer))
{
serializer.WriteObject(stm, obj);
return writer.ToString();
}
}
public static T Deserialize<T>(this string serialized, IEnumerable<Type> knownTypes)
{
var serializer = new DataContractSerializer(typeof(T), knownTypes);
using (var reader = new StringReader(serialized))
using (var stm = new XmlTextReader(reader))
{
return (T)serializer.ReadObject(stm);
}
}
}
public class Address
{
public string Country { get; set; }
public string City { get; set; }
}
public class CodedAddress
{
public int CountryCode { get; set; }
public int CityCode { get; set; }
}
class Program
{
static void Main(string[] args)
{
var persons1 = new Dictionary<string, Address>();
persons1.Add("John Smith", new Address { Country = "US", City = "New York" });
persons1.Add("Jean Martin", new Address { Country = "France", City = "Paris" });
// no need to provide known types to the serializer
var serializedPersons1 = persons1.Serialize(null);
var deserializedPersons1 = serializedPersons1.Deserialize<Dictionary<string, Address>>(null);
var persons2 = new Dictionary<string, object>();
persons2.Add("John Smith", new Address { Country = "US", City = "New York" });
persons2.Add("Jean Martin", new CodedAddress { CountryCode = 33, CityCode = 75 });
// must provide known types to the serializer
var knownTypes = new List<Type> { typeof(Address), typeof(CodedAddress) };
var serializedPersons2 = persons2.Serialize(knownTypes);
var deserializedPersons2 = serializedPersons2.Deserialize<Dictionary<string, object>>(knownTypes);
}
}

How to convert a List<T> to specific Json format

I want to be able to convert a List<T> into a specific JSON table-like format. In my case, the T will always be a simple object (no nested properties). Here are two examples to illustrate what I want.
Example #1: List<Person> to JSON
// C# list of Persons
var list = new List<Person>() {
new Person() { First = "Jesse", Last = "Gavin", Twitter = "jessegavin" },
new Person() { First = "John", Last = "Sheehan", Twitter = "johnsheehan" }
};
// I want to transform the list above into a JSON object like so
{
columns : ["First", "Last", "Twitter"],
rows: [
["Jesse", "Gavin", "jessegavin"],
["John", "Sheehan", "johnsheehan"]
]
}
Example #2: List<Address> to JSON
// C# list of Locations
var list = new List<Location>() {
new Location() { City = "Los Angeles", State = "CA", Zip = "90210" },
new Location() { City = "Saint Paul", State = "MN", Zip = "55101" },
};
// I want to transform the list above into a JSON object like so
{
columns : ["City", "State", "Zip"],
rows: [
["Los Angeles", "CA", "90210"],
["Saint Paul", "MN", "55101"]
]
}
Is there a way to tell JSON.net to serialize an object in this manner? If not, how could I accomplish this? Thanks.
UPDATE:
Thanks to #Hightechrider's answer, I was able to write some code that solves the problem.
You can view a working example here https://gist.github.com/1153155

Using reflection you can get a list of properties for the type:
var props = typeof(Person).GetProperties();
Given an instance of a Person p you can get an enumeration of the property values thus:
props.Select(prop => prop.GetValue(p, null))
Wrap those up in a generic method, add your favorite Json serialization and you have the format you want.

Assuming your using .Net 4 this should do everything you want. The class actually lets you convert to either XML or JSON. The Enum for CommunicationType is at the bottom. The serializer works best if the class your passing it has been decorated with DataContract & DataMember attributes. I've included a sample at the bottom. It will also take an anonymous type so long as it's all simple types.
Reflection would work as well but then you have to understand all the JSON nuances to output complex data types, etc. This used the built-in JSON serializer in .Net 4. One more note, because JSON does not define a date type .Net puts dates in a funky ASP.Net custom format. So long as your deserializing using the built-in deserializer it works just fine. I can dig up the documentation on that if you need.
using System;
using System.Xml.Serialization;
using System.Text;
using System.Runtime.Serialization.Json;
using System.IO;
using System.Xml.Linq;
internal class Converter
{
public static string Convert<T>(T obj, CommunicationType format, bool indent = false, bool includetype = false)
{
if (format == CommunicationType.XML)
{
return ToXML<T>(obj, includetype, indent);
}
else if (format == CommunicationType.JSON)
{
return ToJSON<T>(obj);
}
else
{
return string.Empty;
}
}
private static string ToXML<T>(T obj, bool includetype, bool indent = false)
{
if (includetype)
{
XElement xml = XMLConverter.ToXml(obj, null, includetype);
if(indent) {
return xml.ToString();
}
else
{
return xml.ToString(SaveOptions.DisableFormatting);
}
}
else
{
System.Xml.Serialization.XmlSerializerNamespaces ns = new System.Xml.Serialization.XmlSerializerNamespaces();
XmlSerializer xs = new XmlSerializer(typeof(T));
StringBuilder sbuilder = new StringBuilder();
var xmlws = new System.Xml.XmlWriterSettings() { OmitXmlDeclaration = true, Indent = indent };
ns.Add(string.Empty, string.Empty);
using (var writer = System.Xml.XmlWriter.Create(sbuilder, xmlws))
{
xs.Serialize(writer, obj, ns);
}
string result = sbuilder.ToString();
ns = null;
xs = null;
sbuilder = null;
xmlws = null;
return result;
}
}
private static string ToJSON<T>(T obj)
{
DataContractJsonSerializer ser = new DataContractJsonSerializer(typeof(T));
using (MemoryStream ms = new MemoryStream())
{
string result = string.Empty;
System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
ser.WriteObject(ms, obj);
result = encoding.GetString(ms.ToArray());
ms.Close();
encoding = null;
ser = null;
return result;
}
}
}
[DataContract()]
public enum CommunicationType : int
{
[XmlEnum("0"), EnumMember(Value = "0")]
XML = 0,
[XmlEnum("1"), EnumMember(Value = "1")]
JSON = 1
}
[DataContract(Namespace = "")]
public partial class AppData
{
[DataMember(Name = "ID")]
public string ID { get; set; }
[DataMember(Name = "Key")]
public string Key { get; set; }
[DataMember(Name = "Value")]
public string Value { get; set; }
[DataMember(Name = "ObjectType")]
public string ObjectType { get; set; }
}

Any specific reason why you don't need the standard format?
To actually answer the question:
Since this is something that is outside of JSON syntax I can't think of a way to implement this within the default framework.
One solution would be to leverage attributes decorate the properties you want transported over the wired with a custom attribute and using Reflection cycle through the properties and output their property names as the column headers and then cycle throw the objects and write the values. Generic enough so it could be applied across other objects as well.
public class Location
{
[JsonFooAttribute("City")]
public string city {get;set;}
[JsonFooAttribute("State")]
public string state {get;set;}
}