OutOfMemoryException trying to write Xml from DataSet - c#

In C# i'm trying to get the xml string from a big DataSet:
private string GetXmlFromDecomposedPortfolio(string dataSetName, DecomposedPortfolio ptf)
{
StringWriter writer = new StringWriter();
System.Data.DataSet ds = new System.Data.DataSet(dataSetName);
ds.Tables.Add(ptf.Security.Copy());
ds.WriteXml((TextWriter)writer, XmlWriteMode.IgnoreSchema);
return writer.ToString();
}
but i've got the exception:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity)
at System.Text.StringBuilder.GetNewString(String currentString, Int32 requiredLength)
at System.Text.StringBuilder.Append(Char value) at System.IO.StringWriter.Write(Char value)
at System.Xml.XmlTextWriter.WriteStartElement(String prefix, String localName, String ns)
at System.Data.DataTextWriter.WriteStartElement(String prefix, String localName, String ns)
at System.Data.XmlDataTreeWriter.XmlDataRowWriter(DataRow row, String encodedTableName)
at System.Data.XmlDataTreeWriter.Save(XmlWriter xw, Boolean writeSchema)
at System.Data.DataSet.WriteXml(XmlWriter writer, XmlWriteMode mode)
at System.Data.DataSet.WriteXml(TextWriter writer, XmlWriteMode mode)
at Decompose.Library.Render.GetXmlFromDecomposedPortfolio(String dataSetName, DecomposedPortfolio ptf)
at Decompose.Library.Render.SavePE()
at Decompose.Library.WorkFlow.ProcessBatch()
Any suggestion?

First you create huge (apparently) xml data
ds.WriteXml((TextWriter)writer, XmlWriteMode.IgnoreSchema);
After clone it, into the string object writer.ToString();, so you almost double the memory you need.
What you can do, is to create XML row-per-row, so create kind of XmlDataSetRowEnumerator, that retrieves XML-per-row and yield returns the resulting XML.

Related

Which XML message generates an exception while doing ".LoadXml(xml_string)"?

I'm receiving text string messages, which should be XML-like.
However, sometimes that's not the case but I don't see what those messages look like:
Current code:
var xmlDocument = new XmlDocument();
xmlDocument.LoadXml(xml);
The Exception errormessage is quite elaborated (it mentions the wrong tags), but it does not mention the xml string itself:
Exception during data receiving event: [System.Xml.XmlException: Unexpected end of file has occurred. The following elements are not closed: ID, VM, HMS. Line 1, position 156.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ThrowUnclosedElements()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at ... Information.FromString(String xml) in C:\...\Information.cs:line 40
at ....<ExtractInformation>d__71.MoveNext() in C:\...\Extraction.cs:line 161
at some_other_function(Byte[] data) in C:\...\Extraction.cs:line 134]
System.Xml.XmlException: Unexpected end of file has occurred. The following elements are not closed: ID, VM, HMS. Line 1, position 156.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ThrowUnclosedElements()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at ... Information.FromString(String xml) in C:\...\Information.cs:line 40
at ....<ExtractInformation>d__71.MoveNext() in C:\...\Extraction.cs:line 161
at some_other_function(Byte[] data) in C:\...\Extraction.cs:line 134]
My idea was to add the xml-string to the Exception's Message property, but this seems to be read-only:
public static Information FromString(string xml)
{
var xmlDocument = new XmlDocument();
try
{
xmlDocument.LoadXml(xml);
}
catch (Exception ex)
{
ex.Message += "[" + xml + "]"; // <= compiler error CS0200
throw ex;
}
At that depth I don't have any logging possibilities and as I'm working with a server application, I can't show something on screen.
How can I add information to this Exception message (or to other properties/fields of the Exception)?

Exception while reading XML, DataSet.ReadXML, Object reference not set to an instance of an object

It occurs when multiple calls to read xml files using dataSet.ReadXML.
Object reference not set to an instance of an object.
at System.Data.DataTable.NewRow(Int32 record)
at System.Data.DataRowCollection.AddWithColumnEvents(Object[] values)
at System.Data.XmlDataLoader.LoadTable(DataTable table, Boolean isNested)
at System.Data.XmlDataLoader.LoadData(XmlReader reader)
at System.Data.DataSet.ReadXml(XmlReader reader, XmlReadMode mode, Boolean denyResolving)
at System.Data.DataSet.ReadXml(String fileName, XmlReadMode mode)

XML file as input to LoadXml returns xml exception c#

In my unit testing project, I have merged all my xml files into single xml file and need to send to webservice to seperate it into modules.
So, I have used the below code in my webservice,
//xmlString is my xml file content
try
{
XElement XmlDocument = new XElement("XMLDocument");
XmlDocument testDoc = new XmlDocument();
testDoc.LoadXml(xmlString);
XmlDocument = XElement.Load(new XmlNodeReader(testDoc));
}
catch (Exception ex)
{
string returnString = ex.StackTrace.ToString();
returnString += "XML breaks the code";
return returnString;
}
EDIT: While running this in finalbuilder, I am getting below exception
Exception Message: at ModuleSeparator.GetDetails(String xmlString) System.Xml.XmlException: Data at the root level is invalid. Line 1, position 1.
Results
Return value of 'ModuleSeparator' : at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at ModuleSeparator.GetDetails(String xmlString)XML breaks the code
Setting variable 'ModuleWiseTestCaseResults' to value ' at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at ModuleSeparator.GetDetails(String xmlString)XML breaks the code'.
My XML will look like,
<?xml version="1.0" encoding="utf-8"?>
<test-results name="Merged results" total="418" errors="0" failures="0" not-run="0" inconclusive="128" ignored="0" skipped="7" invalid="0" date="2016-03-08" time="10:47:33">
<test-suite type="Test Project" name="" executed="True" result="Skipped" success="True" time="311.143731" asserts="0">
<results></results>
</test-suite>
</test-results>

Replacing values in XML file

Our application needs to process XML files. Some times we receive XMLs with values as follows:
<DiagnosisStatement>
<StmtText>ST &</StmtText>
</DiagnosisStatement>
Because of &< my application is not able to load XML correctly and throwing exception as follows:
An error occurred while parsing EntityName. Line 92, position 24.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.Throw(String res)
at System.Xml.XmlTextReaderImpl.ParseEntityName()
at System.Xml.XmlTextReaderImpl.ParseEntityReference()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.Load(String filename)
at Transformation.GetEcgTransformer(String filePath, String fileType, String Manufacture, String Producer) in D:\Transformation.cs:line 160
Now I need to replace all occurrences of &< with 'and<' so that XML can get processed successfully without any exceptions.
This is what I did in order to load XML with the help of answer given by Botz3000.
string oldText = File.ReadAllText(filePath);
string newText = oldText.Replace("&<", "and<");
File.WriteAllText(filePath, newText, Encoding.UTF8);
xmlDoc = new XmlDocument();
xmlDoc.Load(filePath);
The Xml file is invalid, because & needs to be escaped as &, so you cannot just load the xml without getting an error. You can do it if you load the file as plain text though:
string invalid = File.ReadAllText(filename);
string valid = invalid.Replace("&<", "and<");
File.WriteAllText(filename, valid);
If you have control over how the Xml file is generated though, you should fix that issue by either escaping the & as & or by replacing it with "and" as you said.

How to resolve XSL includes in a Transformation that loads XSL from a String?

.NET 2.0/VS2005
I am trying to use the XslCompiledTransform class to perform a XSL Transformation. I have two XSL files, the first of which includes a reference to the other in the form of an <xsl:include> statement :
Main.xsl:
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:include href="Included.xsl" />
...
...
</xsl:stylesheet>
Now, If I could load the "Main.xsl" file itself as a URI, my transformation code would be as simple as :
// This is a function that works. For demo only.
private string Transform(string xslFileURI)
{
XslCompiledTransform xslt = new XslCompiledTransform();
// This load works just fine, if I provide the path to "Main.xsl".
// The xsl:include is automatically resolved.
xslTransform.Load(xslFileURI);
StringWriter sw = new StringWriter();
xslt.Transform(Server.MapPath("~/XML/input.xml"), null, sw);
return sw.ToString();
}
The problem is that I receive the contents of the Main.xsl file as a string and need to load the string as an XmlReader/IXpathNavigable. This is a necessary restriction at this time. When I try to do the same using an XmlReader/XpathDocument, it fails because the code looks for "Included.xsl" in the C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\ folder! Obviously, the XmlResolver is not able to resolve the relative URL because it only receives a string as input XSL.
My efforts in this direction look like:
// This doesn't work! Halp!
private string Transform(string xslContents)
{
XslCompiledTransform xslt = new XslCompiledTransform();
XmlUrlResolver resolver = new XmlUrlResolver();
resolver.Credentials = CredentialCache.DefaultCredentials;
//METHOD 1: This method does not work.
XmlReaderSettings settings = new XmlReaderSettings();
settings.XmlResolver = resolver;
XmlReader xR = XmlReader.Create(new StringReader(xslContents), settings);
xslt.Load(xR); // fails
// METHOD 2: Does not work either.
XPathDocument xpDoc = new XPathDocument(new StringReader(xslContents));
xslt.Load(xpDoc, new XsltSettings(true, true), resolver); //fails.
StringWriter sw = new StringWriter();
xslt.Transform(Server.MapPath("~/XML/input.xml"), null, sw);
return sw.ToString();
}
I have tried to use the ResolveUri method of the XmlUrlResolver to obtain a Stream referencing the XSL file to be included, but am confused as to how to use this Stream. IOW, how do I tell the XslCompiledTransform object to use this stream in addition to the Main.xsl XmlReader:
Uri mainURI = new Uri(Request.PhysicalApplicationPath + "Main.xsl");
Uri uri = resolver.ResolveUri(mainURI, "Included.xsl");
// I can verify that the Included.xsl file loads in the Stream below.
Stream s = resolver.GetEntity(uri, null, typeof(Stream)) as Stream;
// How do I use this Stream in the function above??
Any help is greatly appreciated. Sorry for the long post!
For your reference, the Exception StackTrace looks like this:
[FileNotFoundException: Could not find file 'C:\Program Files\Microsoft Visual Studio 8\Common7\IDE\Included.xsl'.]
System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) +328
System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy) +1038
System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize) +113
System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials) +78
System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn) +51
System.Xml.Xsl.Xslt.XsltLoader.CreateReader(Uri uri, XmlResolver xmlResolver) +22
System.Xml.Xsl.Xslt.XsltLoader.LoadStylesheet(Uri uri, Boolean include) +33
System.Xml.Xsl.Xslt.XsltLoader.LoadInclude() +349
System.Xml.Xsl.Xslt.XsltLoader.LoadRealStylesheet() +704
System.Xml.Xsl.Xslt.XsltLoader.LoadDocument() +293
System.Xml.Xsl.Xslt.XsltLoader.LoadStylesheet(XmlReader reader, Boolean include) +173
Use a custom XmlUrlResolver
class MyXmlUrlResolver : XmlUrlResolver
{
public override Uri ResolveUri(Uri baseUri, string relativeUri)
{
if (baseUri != null)
return base.ResolveUri(baseUri, relativeUri);
else
return base.ResolveUri(new Uri("http://mypath/"), relativeUri);
}
}
And use it in load function of XslCompiledTransform,
resolver=new MyXmlUrlResolver();
xslt.Load(xR,null,resolver);
As Gee's answer mentions, you want to use a custom XmlResolver (of which XmlUrlResolver is already derived), but if you also override the method GetEntity you can resolve references in the primary XSLT document in fun and interesting ways. A deliberately simple example of how you could resolve the reference to Included.xsl:
public class CustomXmlResolver : XmlResolver
{
public CustomXmlResolver() { }
public override ICredentials Credentials
{
set { }
}
public override object GetEntity(Uri absoluteUri, string role, Type ofObjectToReturn)
{
MemoryStream entityStream = null;
switch (absoluteUri.Scheme)
{
case "custom-scheme":
string absoluteUriOriginalString = absoluteUri.OriginalString;
string ctgXsltEntityName = absoluteUriOriginalString.Substring(absoluteUriOriginalString.IndexOf(":") + 1);
string entityXslt = "";
// TODO: Replace the following with your own code to load data for referenced entities.
switch (ctgXsltEntityName)
{
case "Included.xsl":
entityXslt = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<xsl:stylesheet version=\"1.0\" xmlns:xsl=\"http://www.w3.org/1999/XSL/Transform\">\n <xsl:template name=\"Included\">\n\n </xsl:template>\n</xsl:stylesheet>";
break;
}
UTF8Encoding utf8Encoding = new UTF8Encoding();
byte[] entityBytes = utf8Encoding.GetBytes(entityXslt);
entityStream = new MemoryStream(entityBytes);
break;
}
return entityStream;
}
public override Uri ResolveUri(Uri baseUri, string relativeUri)
{
// You might want to resolve all reference URIs using a custom scheme.
if (baseUri != null)
return base.ResolveUri(baseUri, relativeUri);
else
return new Uri("custom-scheme:" + relativeUri);
}
}
When you load the Main.xsl document you'd change the relevant code to the following:
xslt.Load(xpDoc, new XsltSettings(true, true), new CustomXmlResolver());
The above example was based on info I picked-up in the MSDN article Resolving the Unknown: Building Custom XmlResolvers in the .NET Framework.
I am probably missing the obvious but is there a reason you don't just change the URI of Included.xsl to be a true URL? This could either be done in the XSL doc if you have access or using string manipulation otherwise?
I already have success with doing transformations using all in memory:
Having a xslt containing the following includes:
import href="Common.xslt" and
import href="Xhtml.xslt"
private string Transform(string styleSheet, string xmlToParse)
{
XslCompiledTransform xslt = new XslCompiledTransform();
MemoryResourceResolver resolver = new MemoryResourceResolver();
XmlTextReader xR = new XmlTextReader(new StringReader(styleSheet));
xslt.Load(xR, null, resolver);
StringWriter sw = new StringWriter();
using (var inputReader = new StringReader(xmlToParse))
{
var input = new XmlTextReader(inputReader);
xslt.Transform(input,
null,
sw);
}
return sw.ToString();
}
public class MemoryResourceResolver : XmlResolver
{
public override object GetEntity(Uri absoluteUri,
string role, Type ofObjectToReturn)
{
if (absoluteUri.ToString().Contains("Common"))
{
return new MemoryStream(Encoding.UTF8.GetBytes("Xml with with common data"));
}
if (absoluteUri.ToString().Contains("Xhtml"))
{
return new MemoryStream(Encoding.UTF8.GetBytes("Xml with with xhtml data"));
}
return "";
}
}
Note that absolutely all content is as strings: styleSheet, xmlToParse and the content of the "Common" and "Xhtml" imports

Categories