We Keep Coding

Serialize multiple streams into a single one

I've got a scenario where I need to backup thousands of rather small (1-3KB) files (in Azure blob storage, but that's not the point) which I have as a list of these models (source code of which I don't own - it's Azure SDK):
class Model {
public string Name {get;set;}
public Stream Data {get;}
//bunch of other things I'd like to ignore
}
My best attempt (from performance point of view) is now merging them all in a single file using JsonTextWriter from Json.NET, but obviously I don't need text format here and it introduces a lot of overheads. I'm wondering if there is any (binary?) serializer available which won't require me to decorate existing model with some specific attributes and will also have a nice API including something like
var writer = new MagicWriterThatImLookingFor();
foreach(var model in models){
writer.WriteString(model.Name);
writer.WriteByteArray(model.Data.ToArray());
}
with corresponding deserialize?

Create XML in memory and pass to another method to transform via XSLT

This is what I'm trying to do. I'm sure the solution is simple, I just haven't found it yet (and believe me I've been trying).
I have an object that I want to use to create XML data. (I'm using an XmlTextWriter to create the elements.)
I then want to pass the XML data to another method.
This method should take the XML data and uses it as the input for an XSLT transformation to get HTML.
The Transform method works with an XMLReader, so basically I want to first write something with an XmlTextWriter and pass this to an XmlReader.
Of course I could just write to a file and then read the file again, but I thought it would be easier to just handle the data in memory and not actually write it to a file.
The reason why I'm not directly using an HtmlTextWriter is simple: As far as I understand this only works in .NET 4.5 and for various reasons the project needs to stay in 4.0.
There is also the option to pass an XML from another source later on, which is why I'd like to stick with XSLT transformation. Also, this will make changing the layout of the HTML form easier later on, since I would only have to change the XSLT template and not change the actual code.

you could:
StringBuilder sb = new StringBuilder();
var writer = new XmlWriter(sb);
WriteStuff( writer );
writer.Flush();
string s = sb.ToString();
XmlReader reader = new XmlReader(s);
DoStuff(reader);
Not sure what you want to do, or if that's the best way to do it...
But you can.
This will also work in a similar way with MemoryStream, Which will probably be more appropriate for a lot of data... just don't forget to Flush...

Whenever you want to sendan ordered amount of bytes to another "thingy", which needs to interpret the bytes in the same order to which it was sent, it is wise to use a System.IO.Stream class.
Using this class you declare that you don't really care where the bytes are going to, as long as the order remains the same. So both the writer and the reader of the bytes don't care whether the stream of bytes is put in a file, a piece of memory, even a papertape reader, as long as the order isn't changed, and no byte is being added or lost.
In practice: whenever you think you could write them on a file, after which you'd give the filename to someone else, you could also use a stream. Either you, or the object that ordered you to write it to the file, creates the stream with the destination of the stream (memory / papertape / file). The stream is than passed to the other, who can read the stream.
Example:
MySpecialObject has some data: MySpecialData. There are classes to serialize and deserialize this special data. These classes are not interested where the data is fysically serialized: patertape? diskette? memory. They only need to know that they can (de)serialize it by writing and reading a stream of bytes. IN that case you use an instance of System.IO.Stream.
(I write the namespaces, to make it easier to find the classes)
class MySerializer
{
private System.IO.Stream myStream = null;
public MySerializer(system.IO.Stream stream)
{
this.myStream = stream;
}
public void Serialize(object data)
{
System.Xml.Serialization.XmlSerializer serializer =
new System.Xml.SerializationXmlSerializer(data.GetType());
System.XML.XmlWriter writer = System.XML XmlWriter.Create(this.MyStream);
serializer.Serialize(writer, data);
}
}
If you look closely, you'll see that the only difference with using a file is the parameter in XMLWriter.Create()
Now the usage of this in mySpecialObject:
public void SerializeMySpecialData()
{
System.IO.Stream myStream = new System.IO.MemoryStream();
// in this case: read and write using only memory.
// if you want to write to something else, for instance a file,
// create a file stream.
// or any other subclass from Sytem.IO.Stream. The serializer won't notice
// where it is actually stored.
MySerializer mySerializer = new MySerializer(myStream);
mySerializer.Serialize(mySpecialData);
myStream.Flush();
myStream.Close();
myStream.Dispose(); // this function will probably also flush and close
}
Reading XML from a stream is similar as reading XML from a file:
public object Deserialize(System.Type expectedDataType)
{
object data = null;
System.Xml.Serialization.XmlSerializer deserializer =
new System.Xml.SerializationXmlSerializer(expectedDataType);
System.XML.XmlReader reader = System.XML XmlReader.Create(this.MyStream);
deserializer.Deserialize(reader, data);
return data;
}
Once again: the advantage is that the caller decides where the data actually is stored. If the data is stored on a different type of media the serialization classes won't see the difference, and thus won't have to change.
You could also decide to hide to the user of the serialization classes where the data is stored. In that case the constructor of the serialization class creates the stream. This approach would be useful if you want all data to be serialized all on the same medium.

To save on xml parsing, I would suggest to write all data into XmlDocument using XmlWriter, and then create XmlReader from it:
//create xml data
XmlDocument doc = new XmlDocument();
using (XmlWriter writer = doc.CreateNavigator().AppendChild())
{
CreateXmlData(writer);
}
//pass xml data to transform as reader
using (XmlReader reader = new XmlNodeReader(doc))
{
Transform(reader);
}

How to use XmlReader to read data in a List Part 2

Okay, it appeared that I had an answer for this question, but I was wrong. Probably because I asked the wrong thing. Here is what I want to do:
I want to take an xml document and pull it into memory. Then, at will, I want to run queries on that document and get information out of it. I want to do this whenever I need data WITHOUT loading the document into memory again. After all, the point is to stop hitting up the disk when I need the data. The methods I've been using only work ONCE, and I need to do this multiple times. Here is the code that I tried:
public static class GrabFile
{
public static XDocument configData = XDocument.Load(#"myxml.xml");
public static XmlReader templateReader = configData.CreateReader();
}
I thought this would copy the document into memory, and I'd be able to use the templateReader to query the configData any time I wanted to by using this:
while (GrabFile.templateReader.Reader())
{
//get the data I wanted
}
I thought I could put that while statement, and create others to do specific queries in a method, and call them whenever I needed the info. But it only works ONCE.
As I said, I'm not overly familiar with this, so there's probably a way to do it that's easy.
Also, people in the other thread wanted an example of what was in the xml document. That's irrelevant. The point is I want to put the document into memory, then query it as many times as needed with out accessing it from the disk and creating another reader. And yes, I want to use a reader for this.
Perhaps I need to move the pointer in the file in memory back to the top so it'll read it again?
Thanks for any help.

Why do you not want to create another reader?
Just calling GrabFile.configData.CreateReader() each time you need it is the simplest approach. (That won't load it from disk again, but it will create a separate XmlReader each time you call it.)
From the docs for XmlReader:
Represents a reader that provides fast, non-cached, forward-only access to XML data.
(Emphasis mine.) I don't see anything around resetting, and I wouldn't expect to.
Given that you've got the information in an XDocument to start with, I would personally try to do all the querying with that rather than using the rather-harder-to-work-with XmlReader anyway, but obviously that's your call. It would help if you'd give some justification though - as you say you're "not overly familiar with this", so it's worth revisiting your assumptions about how you're tackling whatever the higher-level task is.

Are you doing this in a server environment, like with an ASP.NET application? That might introduce some more steps, but the general idea is to use a static object to hold the XML Document that you load into memory. Much like you have done. And that's really all there is to it. Could you provide some more context?
You also don't really need to use the XML reader (unless you only want that sort of forward only access);
public static class Globals
{
public static XDocument ConfigFile = XDocument.Load("C:/somefile.xml");
}
public void SomeOtherFunctionSomewhere()
{
var configName = Globals.ConfigFile
.Descendants("someconfigsection")
.Descendants("configName")
.First().Value;
}
If you are storing your program's configuration in an XML file - consider reading the entire configuration into an in-memory object that you create yourself at startup. This will be an even faster way to access the config sections during runtime than having to query the XML every time (even if the XML is already in memory). Contrived example below:
public class Configuration
{
//just some made up config settings
public string Name {get;set;}
public int Id {get;set;}
public int XDimension {get;set;}
public int YDimension {get;set;}
}
class Program
{
static Configuration Config;
static void Main(string[] args)
{
Config = ReadConfig();
}
private static Configuration ReadConfig()
{
var config = new Configuration();
//read entire XML and set properties on the config object
return config;
}
}

C# Persisting Objects Between User Sessions Without using a database

I'm making a simple Winforms that reads in a text file and does some parsing to put it into C# objects that are then manipulated by my app.
I would like to parse the file once. Then go away for a week, come back and load all the objects back into my app.
I don't want to persist to database thus was hoping there was some sort of simple object persistance framework I could hook into.

You are talking about serialization.
Look into the DataContractSerializer for this. If you want a file that isn't XML you can try the DataContractJsonSerializer.
You simply add the DataContract and DataMember attributes to tell the serializer how to manage the serialization.
If your objects will not change (new/old/renamed properties and visibility) and you don't care to read the serialization format, you can try the BinaryFormatter.
There are several older serializers in the frame work (XmlSerializer, BinaryFormatter and more), but the best control over the serialization format will be with the above).

You could use application settings or user settings. Its pretty easy with the wizard that can be found under the projects properties. here is the tutorial for it.

You could just serialize them to an XML file. Simple, quick and easy (unless someone hacks the file).

The simplest non-DB method of storing object state is serialization. Mark your class as either Serializable or a DataContract, then use a BinaryFormatter (or DataContractSerializer if you used the DataContract) to convert the object into a persistable state. That can then be saved to a file that you can load when the app starts back up and deserialize in a similar fashion. Be aware that the class's members, including child classes, must be serializable in the same way, and that you will lose any references to non-serialized elements such as pointers and delegate references.

You can use serialization as sugested before.
A good place to save the data in is Windows's LocalData folder -
Environment.SpecialFolder.LocalApplicationData

Probably the simplest way to do this would be to serialize the objects directly to a file, using the BinaryFormatter class. You'll need to make sure that all your types are marked as [Serializable] but this approach is fast and easy.
Here is a simple example of a "read" and "write" method for an arbitrary object graph:
private void SerializeToFile(string fileName, object o)
{
var binaryFormatter = new BinaryFormatter();
using (var fileStream = new FileStream(fileName, FileMode.Create, FileAccess.Write, FileShare.None))
{
binaryFormatter.Serialize(fileStream, o);
}
}
private object DeserializeFromFile(string fileName)
{
var binaryFormatter = new BinaryFormatter();
using (var fileStream = new FileStream(fileName, FileMode.Open, FileAccess.Read, FileShare.None))
{
return binaryFormatter.Deserialize(fileStream);
}
}
Other approaches include XmlSerialization (the XmlSerializer class) - better if you want to the serialized objects to be human readable or editable, or using the newer DataContractSerializer.

What is wrong with a database ? If you do not want to have a full engine with its setup and associated problems, maybe you can take a look at "file" based databases...
My first toughs are :
SqlCompact
Access databases
SqlLite
...
Depending the complexity of your objects to persist, it can be a good solution.

What is the fastest way to convert a class to XML

I would like to know what is the fastest and most lightweight technique to convert a fairly large class to XML. The class will have lists and arrays in it. I need to convert all this data to XML
Here is what my application does:
it will get all the information from the database using linq to enties. Then store the data in a class. Then I want to convert this class to XML. When the data is in XML I will send the XML to the browser along with the xsl stylesheet to be displayed to the user. What is the fastest way to do this.

The XmlSerializer actually creates an assembly (with an XmlSerializationWriter) that is custom made to serialize your class. You can look at the generated code by following these steps.
You only pay the price the first time it encounters a new type.
So I think that you should really go with the XmlSerializer, not only for performance, but for maintainability.
You can use a mixin-like serializer class:
public interface MXmlSerializable { }
public static class XmlSerializable {
public static string ToXml(this MXmlSerializable self) {
if (self == null) throw new ArgumentNullException();
var serializer = new XmlSerializer(self.GetType());
using (var writer = new StringWriter()) {
serializer.Serialize(writer, self);
return writer.GetStringBuilder().ToString();
}
}
}
public class Customer : MXmlSerializable {
public string Name { get; set; }
public bool Preferred { get; set; }
}
// ....
var customer = new Customer {
Name = "Guybrush Threepwood",
Preferred = true };
var xml = customer.ToXml();

The fastest way is to write the code for it yourself. That will remove any overhead, like the need to use reflection to read the properties of the object, as you can access the properties directly.
Add a method to the class that returns it's data as XML, either by returning an XDocument, the XML already formatted as a string, or you can pass an XmlWriter to the method.

By "fastest" do you mean you want the approach which will be fastest to develop? Or do you want the approach which will have the fastest execution speed?
If it's the former, I recommend just using .NET's XmlSerializer class: http://msdn.microsoft.com/en-us/library/system.xml.serialization.xmlserializer.aspx
Serializing a class to XML output is as simple as:
XmlSerializer serializer = new XmlSerializer(myObject.GetType());
serializer.Serialize(Response.OutputStream, myObject);
And there are various attributes you can decorate your class with to control things like whether individual properties are serialized as elements or attributes, etc.
There's a good FAQ at http://www.devolutions.net/articles/serialization.aspx also

You could use XML serialization, for example:
Foo foo = new Foo();
XmlSerializer serializer = new XmlSerializer(typeof(Foo));
TextWriter writer = new StringWriter();
serializer.Serialize(writer, foo);
string xml = writer.ToString();

The fastest method would depend on the class, because it would be hand-written to take advantage of knowledge of the specifics of that class in ways a more general approach couldn't do.
I'd probably use XmlTextWriter rather than straight to TextWriter though. While the latter would allow for some further savings, these would be minimal compared to the better structure of XmlTextWriter, and you've already sacrificed a good bit in terms of structure and ease of maintenance as it is.
You can always slot in your super-optimised implementation of XmlWriter afterwards ;)

It sounds like a rather convoluted set-up, when you could just display the class's information on a webpage using ASP.NET MVC. Why take the extra two steps of converting it to XML, send it to the browser, and use an XSL stylesheet to display it to the user? It doesn't make sense.

I wrote a program that serialized one simple object graph to XML in different ways:
1. Using XmlSerializer
2. Using hardcoded xml serializer
30,000 documents:
XmlSerializer took : 0.9 sec
Hardcoded serializer took: 0.45 sec
I relied on XmlWriter in both cases and that adds some overhead.
Note that you can instruct Visual Studio to generate the XmlSerializer assembly during compile time in order to reduce the serialization for that first instance (otherwise an assembly is generated in runtime).