We Keep Coding

Write out huge XElement as XML string - OutOfMemory Exception

I need to create a large XML file which unfortunately I have no control over.
One of the elements of the XML contains hundreds of thousands of elements (up to 500k), so the resulting XML of the XElement is quite large, however still smaller than what I would expect for an OutOfMemory exception.
The below code is being used to serialise objects to XML, then use an XmlReader to stream that XML and pull out the element I need (which is huge). I'm then trying to compress the element (using GZip to Base64) however, I can't even get this far because xmlReader.ReadOuterXml throws an OutOfMemory exception.
I have tried various ways of doing this, including using XmlDocument (.InnerXml/.OuterXml) and XDocument with XElements (.ToString()) but all of them throw OutOfMemory exceptions.
The reason I'm trying to convert my element to a string in the first place is because my compression expects a byte[] - I'm not sure if my approach is flawed in this case.
Code:
private string GetCompressed(object obj)
{
XmlSerializer serialiser = new XmlSerializer(obj.GetType());
// Serialise the XML to an XDocument so that we can manipulate it
using (MemoryStream memoryStream = new MemoryStream())
{
// Serialise to the memory stream
serialiser.Serialize(memoryStream, obj);
// Reset the position to 0.
memoryStream.Seek(0, SeekOrigin.Begin);
// Create an XML reader to stream out the results
using (XmlReader xmlReader = XmlReader.Create(memoryStream))
{
while (xmlReader.Read())
{
if (xmlReader.Name == "ElementIWant")
{
return CompressXML(xmlReader.ReadOuterXml()); //<= ReadOuterXml() throws OutOfMemory
}
}
}
}
return string.Empty;
}

XML Serialize large collections of objects to xml

What is the best way to serialize a large collection of objects?
I need to serialize 10 millions of collections from the database to xml.
I always have OutOfMemoryException.
Can you help me with it?

You need to do this in a Read loop one by one. Read one from DB and insert one to XML file by using XmlTextWriter.
Something like below should work for your case.
using (var xml = new XmlTextWriter ("filename", Encoding.UTF8))
{
xml.WriteStartElement ("container_name");
var cmd = new SqlCommand ("your query", dbConnection);
using (var reader = cmd.ExecuteReader ())
{
while (reader.Read ())
{
//
// Read from DB write to xml
//
}
}
xml.WriteEndElement (); // Close your container
}
For XmlTextWriter reference check MSDN

close var XDocument.Load method/way

how do I close this document that was called this way:
var xmlDoc = XDocument.Load(new XmlTextReader(Server.MapPath("Nc.xml")));
thanks

XmlTextReader implements IDisposable. In general, you should call IDisposable.Dispose() as soon as you no longer need the resource to allow the system to close open handles, etc.
The best use pattern for IDisposable is to use the using syntax, which will call IDisposable.Dispose() automatically in an implicit try..finally wrapper:
using (var reader = new XmlTextReader(Server.MapPath("Nc.xml")))
{
var xdoc = XDocument.Load(reader);
{ .. do xdoc work here .. }
} // reader disposed here
or if you want to keep the xdoc around a long time for other work but want to close the file as soon as possible, do it this way:
XDocument xdoc = null;
using (var reader = new XmlTextReader(Server.MapPath("Nc.xml")))
{
xdoc = XDocument.Load(reader);
} // reader disposed here
{ .. do xdoc work here .. }

Once the reader is done, it will close the what it has read automatically.
otherwise hang the reference out for GC by
xmlDoc = null;
which will tear down any internal open items.

Do I need to Dispose XmlReader if I Dispose its underlying Stream?

I have the following method GetData that creates a StreamReader from a file.
private void GetData(string file)
{
string filename = Path.GetFileNameWithoutExtension(file);
XmlDocument xmldoc = new XmlDocument();
using (StreamReader sr = new StreamReader(file))
{
Stream bs = sr.BaseStream;
Stream cl = mainParser.CleanMarkup(bs);
try
{
xmldoc = mainParser.LoadDocument(bs);
}
catch (XmlException ex)
{
// Exceptions are usually caused by non-compliant documents.
// These errors are not critical to the operation of this program.
Console.WriteLine(filename + " " + ex.Message);
}
}
Msdn msdnParser = new Msdn(xmldoc);
ListViewItem lvitem = new ListViewItem(filename);
lvitem.SubItems.Add(filename);
foreach (string item in msdnParser.Subitems)
{
lvitem.SubItems.Add(item);
}
listView.Items.Add(lvitem);
}
mainParser.LoadDocument(bs) calls the following:
public XmlDocument LoadDocument(Stream file)
{
XmlDocument xmldoc = new XmlDocument();
XmlReader xmlread = XmlReader.Create(file);
xmldoc.Load(xmlread);
return xmldoc;
}
StreamReader is disposed by GetData. Does this mean that I don't have to dispose of XmlReader since (I believe) this would dispose of its only unmanaged resource?

The best "rule of thumb" to work by is:
If something implements IDisposable, always wrap it in a using() block to ensure that any unmanaged resources it owns are disposed of correctly.
Relying on the fact that the current implementation of "something" disposes of an underlying resource is dangerous and it won't hurt to wrap everything in a using, just to be on the safe side =)

You're right, you don't have to dispose the reader. But in the code given, it wouldn't hurt either.
I would not put a using block inside LoadDocument() because it is designed so that it 'borrows' it's stream (it does not create it).
But there are arguments to Dispose the XmlReader anyway, just because it's IDisposable. I don't think there is a clear winner here because of the disputable design of the Reader (and Writer) family: They Dispose their baseStreams without clearly being the owner of those streams.

How to determine if XML is well formed?

I've got a large xml document in a string. What's the best way to determine if the xml is well formed?

Something like:
static void Main() {
Test("<abc><def/></abc>");
Test("<abc><def/><abc>");
}
static void Test(string xml) {
using (XmlReader xr = XmlReader.Create(
new StringReader(xml))) {
try {
while (xr.Read()) { }
Console.WriteLine("Pass");
} catch (Exception ex) {
Console.WriteLine("Fail: " + ex.Message);
}
}
}
If you need to check against an xsd, then use XmlReaderSettings.

Simply run it through a parser. That will perform the appropriate checks (whether it parses ok).
If it's a large document (as indicated) then an event-based parser (e.g. SAX) will be appropriate since it won't store the document in memory.
It's often useful to have XML utilities around to check this sort of stuff. I use XMLStarlet, which is a command-line set of tools for XML checking/manipulation.

XmlReader seems a good choice as it should stream the data (not load the whole xml in one go)
http://msdn.microsoft.com/en-us/library/9d83k261.aspx

Try using an XmlReader with an XmlReaderSettings that has ConformanceLevel.Document set.