We Keep Coding

Generate XML File From Generated Object

I started with three (3) XSD files provided from an external party (one XSD links to the other two). I used the xsd.exe tool to generate a .NET object by running the following command: xsd.exe mof-simpleTypes.xsd mof-isp.xsd esf-submission.xsd /c and it generated a single CS file with a handful of partial objects.
I've created an XmlSerializerNamespaces object and fill with the namespaces required (two directly used in the provided sample XML file as well as two others that don't appear to be referenced). I have successfully generated an XML file using the following method:
private XmlDocument ConvertEsfToXml(ESFSubmissionType type)
{
var xml = new XmlDocument();
var serializer = new XmlSerializer(type.GetType());
string result;
using (var writer = new Utf8StringWriter()) //override of StringWriter to force UTF-8
{
serializer.Serialize(writer, type, _namespaces); //_namespaces object holds all 4 namespaces
result = writer.ToString();
}
xml.LoadXml(result);
return xml;
}
My problem that I'm facing is in the generated CS file, one of the objects has a property (another generated partial object) that is of type XmlElement. I have successfully built the object in code, and I'm having an issue converting the object to an XmlElement. The questions and answers I have found here on SO say convert it to an XmlDocument first and then take the DocumentElement property. This works, however the returned XML has namespaces embedded in the element as follows:
<esf:ESFSubmission xmlns:isp="http://www.for.gov.bc.ca/schema/isp" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:esf="http://www.for.gov.bc.ca/schema/esf">
<esf:submissionMetadata>
<esf:emailAddress>test#test.com</esf:emailAddress>
<esf:telephoneNumber>1234567890</esf:telephoneNumber>
</esf:submissionMetadata>
<esf:submissionContent>
<isp:ISPSubmission xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:esf="http://www.for.gov.bc.ca/schema/esf" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:isp="http://www.for.gov.bc.ca/schema/isp">
<isp:ISPMillReport>
<isp:reportMonth>12</isp:reportMonth>
<isp:reportYear>2014</isp:reportYear>
<isp:reportComment>comment</isp:reportComment>
<isp:ISPLumberDetail>
<isp:species>FI</isp:species>
Note: this is just a partial of the generated XML file (for illustration purposes).
As you can see, each XML node is prefixed with the namespace variable. My question is: how can I do this in code? Is my approach sound and if so, then how do NOT include the namespaces in the ISPSubmission node OR if there is a better way to approach this problem that I overlooked, please provide insight. My desired outcome is to have all namespace definitions at the top of the document (their appropriate location) and not on the sub elements - as well as maintain the namespace variables on each element as illustrated above.
EDIT (after reggaeguitar's comment)
Here is the sample XML document I was provided
<?xml version="1.0" encoding="UTF-8"?>
<esf:ESFSubmission xmlns:esf="http://www.for.gov.bc.ca/schema/esf"
xmlns:isp="http://www.for.gov.bc.ca/schema/isp" xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.for.gov.bc.ca/schema/esf esf-submission.xsd
http://www.for.gov.bc.ca/schema/isp mof-isp.xsd">
<esf:submissionMetadata>
<esf:emailAddress>mailto:eric.murphy#cgi.com</esf:emailAddress>
<esf:telephoneNumber>6044445555</esf:telephoneNumber>
</esf:submissionMetadata>
<esf:submissionContent>
<isp:ISPSubmission>
<isp:ISPMillReport>
<isp:reportMonth>06</isp:reportMonth>
<isp:reportYear>2014</isp:reportYear>
<isp:reportComment>Up to 4000 characters is permitted for notes in this element.</isp:reportComment>
<isp:ISPLumberDetail>
<isp:species>FI</isp:species>
<isp:lumberGrade>EC</isp:lumberGrade>
<isp:gradeDescription/>
<isp:size>2x4</isp:size>
<isp:finishType/>
<isp:length>10</isp:length>
<isp:thickWidthUom>IN</isp:thickWidthUom>
<isp:volumeUnitOfMeasure>MBM</isp:volumeUnitOfMeasure>
<isp:volume>11543.987</isp:volume>
<isp:amount>1467893.98</isp:amount>
<isp:invoiceNumber>837261</isp:invoiceNumber>
</isp:ISPLumberDetail>
<isp:ISPLumberDetail>
<isp:species>CE</isp:species>
<isp:lumberGrade/>
<isp:gradeDescription/>
<isp:size/>
<isp:finishType>D</isp:finishType>
<isp:thickness>40</isp:thickness>
<isp:width>100</isp:width>
<isp:thickWidthUom>MM</isp:thickWidthUom>
<isp:volumeUnitOfMeasure>MBM</isp:volumeUnitOfMeasure>
<isp:volume>9743.987</isp:volume>
<isp:amount>1247893.98</isp:amount>
<isp:invoiceNumber/>
</isp:ISPLumberDetail>
<isp:ISPChipDetail>
<isp:species>CE</isp:species>
<isp:unitOfMeasure>BDT</isp:unitOfMeasure>
<isp:wholeLogInd>N</isp:wholeLogInd>
<isp:destinationCode>FBCO</isp:destinationCode>
<isp:destinationDescription/>
<isp:volume>563</isp:volume>
<isp:amount>54463</isp:amount>
<isp:invoiceNumber>12345679</isp:invoiceNumber>
</isp:ISPChipDetail>
</isp:ISPMillReport>
<isp:ISPSubmitter>
<isp:millNumber>103</isp:millNumber>
<isp:contactName>Dave Marotto</isp:contactName>
<isp:contactEmail>eric.murphy#cgi.com</isp:contactEmail>
<isp:contactPhone>2507775555</isp:contactPhone>
<isp:contactPhoneExtension>1234</isp:contactPhoneExtension>
</isp:ISPSubmitter>
</isp:ISPSubmission>
</esf:submissionContent>
</esf:ESFSubmission>

Solved my problem by doing the whole thing in code and not even using the xsd.exe to generate a .NET object.

How to search through XML to find bad nodes

I have a large XML file (68Mb), I am using SQL Server Business Intelligence Studio 2008 to extract the XML data into a database. There is an error in the XML file some where that prevents it from executing. Possibly a missing tag or something like that. The file is so large I cant manually sort through it looking for the error.
Below is a sample of the the XML schema used.
How can I use XPath to sort through the XML in VS 2012 using C#?
An example would be great!
-<PhoneNumberList>
<PhoneNumber value="1234567890" type="Phone"/>
</PhoneNumberList>
-<YearsOfServiceList>
<YearsOfService experienceInMonths="24" description="SuperAdmin" objectCode="049"/>
</YearsOfServiceList>
</Person>
-<Person dob="1960-01-09T00:00:00" lastName="Smith" middleName="Will" firstName="John" id="9999-9999-9999">
-<SiteList>
-<Site id="2014" siteLongName="HA" siteCode="1255" systemCode="999">
-<StaffPositionList>
<StaffPosition id="73" staffPosition="Administrator"/>
</StaffPositionList>
</Site>
</SiteList>
-<ProgramList>
<Program id="1234" siteLongName="ABC" siteCode="0000" systemCode="205"/>
<Program id="5678" siteLongName="DEF" siteCode="0000" systemCode="357"/>
</ProgramList>
-<TypeList>
<Type Description="Leader" certificateType="D"/>
<Type Description="Professional" certificateType="P"/>
</TypeList>
-<EmailList>
<Email value="jsmith#somesite.com" type="Email"/>
</EmailList>
-<PhoneNumberList>
<PhoneNumber value="1234567890" type="Phone"/>
</PhoneNumberList>
-<YearsOfServiceList>
<YearsOfService experienceInMonths="24" description="SuperAdmin" objectCode="049"/>
</YearsOfServiceList>
</Person>
</PersonList>
</GetPersonDetail>

If you want to do it in code then create an XSD file describing a valid format for the data, embed it as a resource in your app and then use code like this
var errors = new List<string>();
var schemaSet = new XmlSchemaSet();
schemaSet.Add("", XmlReader.Create(new StringReader(Properties.Resources.NameOfXSDResource)));
document.Validate(schemaSet, (sender, args) =>
{
errors.Add(args.Message);
}
);
This will give you a list of validation errors.

You don't need to search "by hand" if you use a competent text editor. NotePad++'s XML plugin, for instance, can determine if your XML as a whole is well-formed or valid, and both instances will provide separate error messages.
If you don't have a schema and the file is well-formed, you can use the CLR's System.XML namespace to read in the document and then iterate through its nodes using LINQ-to-XML, which would allow you to very finely control which nodes go where. With LINQ, you could either create a new XML file with only the valid entries, procedurally correct the invalid entries as you determine where they are, or even just write to your SQL server database directly.
Your troubleshooting process should be something as follows:
Is the XML well-formed? I..e, does it comport to the fundamental rules of XML?
Is the XML valid? I.e., does it have the elements and attributes you expect?
Is your import query accurate?

For things like this I usually have luck checking and fixing the data in Notepad++. Install the XmlTools plugin and that has a menu for checking the xml syntax and tags.
Also, those dashes will give you problems, it's best to save out the xml file directly without copying by hand.

A 68MB XML file is no problem for XML editors such as XMLBlueprint 64-bit (http://www.xmlblueprint.com/) or Stylus Studio (http://www.stylusstudio.com/). Just check the well-formedness of your xml file (F7 in XMLBlueprint) and the editor will display the errors.

Programmatic XML Diff / Merge in C#

At this moment, I am managing a piece of software that has multiple XML configuration files. When a new version of software is released, sometimes the base config files change, we currently have the software call KDiff on startup. If it detects a change, it prompts the user to choose the changes.
The problem with this approach is that KDiff is a line comparing program and not aware of the ways of XML (like Nodes, etc.)
Ideally, I would like to programmatically work with a library in C# (since we're a MS shop) that can Diff two XML files: a Source XML and a Current Working XML.
And then Merge the two together using a few simple rules:
If the Current Working XML has a node that the Source XML does not, remove it.
If the Source XML has a node that the Current Working XML does not, add it.
If both have the same node and the values differ, favor the Source XML's value, unless it the Source XML's value is set to "UseExistingValue".
For example, here's the "Source" XML:
<Configuration>
<Items>
<Item Id="1" Position="true">
<Location X="UseExistingValue" Y="UseExistingValue" Z="UseExistingValue" />
<Something/>
<SomethingElse/>
</Item>
</Items>
</Configuration>
And here's the "Current Working" XML:
<Configuration>
<Items>
<Item Id="1" Position="false">
<Location X="123" Y="234" Z="345" />
<Another/>
<Something/>
</Item>
</Items>
</Configuration>
And the merged version would look like:
<Configuration>
<Items>
<Item Id="1" Position="true">
<Location X="123" Y="234" Z="345" />
<Something/>
<SomethingElse/>
</Item>
</Items>
</Configuration>
I've looked at the MS XML Diff and Patch Tool and it definitely merges the files together, but doesn't allow for the programmatic rules that I want to define.
XMLUnit for Java devs seems promising, but the .NET version of it seems underdeveloped, which is unfortunate.
Anyone have any suggestions for either scriptable XML Diff/Merge tools and/or .NET libraries (paid or free)?
Thanks.

After a couple days of messing around, I found a solution that I think works for me. Maybe it could work for other people as well.
The MS XML Diff and Patch tool was a viable option. When you Diff first file against the second file it creates an XML "DiffGram" listing what changes it detected between the two XML files.
To take care of all 3 rules that I listed above, I Diff'd the two files in one direction, then opened the DiffGram file using Linq-to-XML and Removed all the "Add" and "Remove" lines.
XNamespace xd = "http://schemas.microsoft.com/xmltools/2002/xmldiff";
var doc = XDocument.Load(_diffGramFile);
doc.Root.DescendantsAndSelf(xd + "add").Remove();
doc.Root.DescendantsAndSelf(xd + "remove").Remove();
Then I patched up (merged) this edited diffgram against the first file and created a partially merged temporary file. This takes care of Rules 1 and 2.
Next, I Diff'd the partially merged file against the first file used. Then opened the new DiffGram and removed all Change references to "UseExistingValue".
var newdoc = XDocument.Load(_diffGramFile);
newdoc.Root.DescendantsAndSelf(xd + "change")
.Where(x => x.Value == "UseExistingValue").Remove();
And merged this edited DiffGram against the partially merged file which takes care of Rule 3. Saving this out to XML then produces the final XML merged according to the rules defined above.
Hopefully this can help out other people.
HINT: After installing the XmlDiffPatch library, the XmlDiffPatch DLL can be found in C:\Windows\assembly\GAC\XmlDiffPatch\1.0.8.28__b03f5f7f11d50a3a\XmlDiffPatch.dll

XML DocumentElement is trashing the innerXml

I have a simple XML file, shown below, which when read-in via a basic XmlDocument.Load(filename.xml). If I load the file, and inspect it's innerXML, it all looks normal. However, when I inspect the value of DocumentElement, it's a mess!!! I kept the example small, so you can easily see there is no mal-formation:
<?xml version="1.0" encoding="UTF-8"?>
<fax:FaxService xmlns:fax="http://www.hp.com/schemas/imaging/con/service/fax/2009/02/11/" xmlns:dd="http://www.hp.com/schemas/imaging/con/dictionaries/1.0/">
<fax:ServiceDefaults>
<fax:ServiceSendDefaults>
<fax:InternetFaxSettings>
<dd:FaxFileFormat>MTIFFG4</dd:FaxFileFormat>
<dd:UseEmailAsFaxAcctAddr>false</dd:UseEmailAsFaxAcctAddr>
<dd:AutoCompleteToNANP>false</dd:AutoCompleteToNANP>
<dd:RetryInterval>0</dd:RetryInterval>
<dd:MaxRetryAttempts>0</dd:MaxRetryAttempts>
</fax:InternetFaxSettings>
</fax:ServiceSendDefaults>
</fax:ServiceDefaults>
</fax:FaxService>
Now, try this in C# with this simple code:
...
XmlDocument xDoc = new XmlDocument();
xDoc.Load("*XMLSAMPLE.XML*");
textBox1.Text = xDoc.InnerXml;
textBox2.Text = xDoc.DocumentElement.InnerXml;
...
It's completely mangled, with the 2nd namespace repeated with every dd tag, and not even included in the top-most tag.
What am I doing wrong? This is driving me nuts!

The content returned by xDoc.DocumentElement.InnerXml is semantically identical to your original ServiceDefaults tag - if the first fragment conforms to your XML schema, the InnerXml fragment will also conform to the definition of the inner element. Just because the framework has re-arranged the namespace declarations does not change the semantics of the document.
Compare the output of your the two XmlDocument properties:
xDoc.DocumentElement:
<?xml version="1.0" encoding="UTF-8"?>
<fax:FaxService xmlns:fax="http://www.hp.com/schemas/imaging/con/service/fax/2009/02/11/" xmlns:dd="http://www.hp.com/schemas/imaging/con/dictionaries/1.0/">
<fax:ServiceDefaults>
<fax:ServiceSendDefaults>
<fax:InternetFaxSettings>
<dd:FaxFileFormat>MTIFFG4</dd:FaxFileFormat>
<dd:UseEmailAsFaxAcctAddr>false</dd:UseEmailAsFaxAcctAddr>
<dd:AutoCompleteToNANP>false</dd:AutoCompleteToNANP>
<dd:RetryInterval>0</dd:RetryInterval>
<dd:MaxRetryAttempts>0</dd:MaxRetryAttempts>
</fax:InternetFaxSettings>
</fax:ServiceSendDefaults>
</fax:ServiceDefaults>
</fax:FaxService>
xDoc.DocumentElement.InnerXml:
<fax:ServiceDefaults xmlns:fax="http://www.hp.com/schemas/imaging/con/service/fax/2009/02/11/">
<fax:ServiceSendDefaults>
<fax:InternetFaxSettings>
<dd:FaxFileFormat xmlns:dd="http://www.hp.com/schemas/imaging/con/dictionaries/1.0/">MTIFFG4</dd:FaxFileFormat>
<dd:UseEmailAsFaxAcctAddr xmlns:dd="http://www.hp.com/schemas/imaging/con/dictionaries/1.0/">false</dd:UseEmailAsFaxAcctAddr>
<dd:AutoCompleteToNANP xmlns:dd="http://www.hp.com/schemas/imaging/con/dictionaries/1.0/">false</dd:AutoCompleteToNANP>
<dd:RetryInterval xmlns:dd="http://www.hp.com/schemas/imaging/con/dictionaries/1.0/">0</dd:RetryInterval>
<dd:MaxRetryAttempts xmlns:dd="http://www.hp.com/schemas/imaging/con/dictionaries/1.0/">0</dd:MaxRetryAttempts>
</fax:InternetFaxSettings>
</fax:ServiceSendDefaults>
</fax:ServiceDefaults>

A look at the following link in MSDN will help shed light on your situation:
http://msdn.microsoft.com/en-us/library/system.xml.xmldocument.innerxml.aspx
Basically, xDoc.DocumentElement.InnerXml is looking at the <fax:ServiceDefaults> node, whereas xDoc.InnerXml is looking one level higher (FaxService node). This is crucial to understanding your problem - because all of your xmlns is on the FaxService node.
Make the following change to your XML document, and notice what happens (basically, copy over the xmlns info to the ServiceDefaults node:
<?xml version="1.0" encoding="UTF-8"?>
<fax:FaxService xmlns:fax="http://www.hp.com/schemas/imaging/con/service/fax/2009/02/11/" xmlns:dd="http://www.hp.com/schemas/imaging/con/dictionaries/1.0/">
<fax:ServiceDefaults xmlns:fax="http://www.hp.com/schemas/imaging/con/service/fax/2009/02/11/" xmlns:dd="http://www.hp.com/schemas/imaging/con/dictionaries/1.0/">
<fax:ServiceSendDefaults>
<fax:InternetFaxSettings>
<dd:FaxFileFormat>MTIFFG4</dd:FaxFileFormat>
<dd:UseEmailAsFaxAcctAddr>false</dd:UseEmailAsFaxAcctAddr>
<dd:AutoCompleteToNANP>false</dd:AutoCompleteToNANP>
<dd:RetryInterval>0</dd:RetryInterval>
<dd:MaxRetryAttempts>0</dd:MaxRetryAttempts>
</fax:InternetFaxSettings>
</fax:ServiceSendDefaults>
</fax:ServiceDefaults>
</fax:FaxService>
Suddenly your code will behave according to your expectations. So hopefully this helps you towards understanding the issue. What the permanent fix should be, that's up to you.
HTH!

Parse an xml document when namespace is no-longer available

I have a number rather large, complex xml documents that I need to loop through. An xmlns is defined at the top of the document however the url this points to is no longer available.
What's the best way to parse the file to get the important data from it using C#?
I tried to load it into a Dataset but would occasionally receive the errors:
The table (endpoint) cannot be the child table to itself in nested relations.
or
Cannot add a SimpleContent column to a table containing element columns or nested relations.
XPath was my next port of call but I had problems because of the lack of namespace.
I suspect this is seriously limiting my options, but does anyone have any suggestions?
Snippet of the XML document:
<?xml version="1.0" encoding="UTF-8"?>
<cdr:cdr_set xmlns:cdr="http://www.naturalconvergence.com/schema/cdr/v3/cdr">
<!-- Copyright (c) 2001-2009, all rights reserved -->
<cdr:cdr xmlns:cdr="http://www.naturalconvergence.com/schema/cdr/v3/cdr">
<cdr:call_id>2040-1247062136726-5485131</cdr:call_id>
<cdr:cdr_id>1</cdr:cdr_id>
<cdr:status>Normal</cdr:status>
<cdr:responsibility>
<cdr:tenant id="17">
<cdr:name>SpiriTel plc</cdr:name>
</cdr:tenant>
<cdr:site id="45">
<cdr:name>KWS</cdr:name>
<cdr:time_zone>GB</cdr:time_zone>
</cdr:site>
</cdr:responsibility>
<cdr:originator type="sipGateway">
<cdr:sipGateway id="3">
<cdr:name>Audiocodes-91</cdr:name>
</cdr:sipGateway>
</cdr:originator>
<cdr:terminator type="group">
<cdr:group>
<cdr:tenant id="17">
<cdr:name>SpiriTel plc</cdr:name>
</cdr:tenant>
<cdr:type>Broadcast</cdr:type>
<cdr:extension>6024</cdr:extension>
<cdr:name>OLD PMS DDIS DO NOT USE</cdr:name>
</cdr:group>
</cdr:terminator>
<cdr:initiation>Dialed</cdr:initiation>
<cdr:calling_number>02087893850</cdr:calling_number>
<cdr:dialed_number>01942760142</cdr:dialed_number>
<cdr:target>6024</cdr:target>
<cdr:direction>Inbound</cdr:direction>
<cdr:disposition>No Answer</cdr:disposition>
<cdr:timezone>GB</cdr:timezone>
<cdr:origination_timestamp>2009-07-08T15:08:56.727+01:00</cdr:origination_timestamp>
<cdr:release_timestamp>2009-07-08T15:09:26.493+01:00</cdr:release_timestamp>
<cdr:release_cause>Normal Clearing</cdr:release_cause>
<cdr:call_duration>PT29S</cdr:call_duration>
<cdr:redirected>false</cdr:redirected>
<cdr:conference>false</cdr:conference>
<cdr:transferred>false</cdr:transferred>
<cdr:estimated>false</cdr:estimated>
<cdr:interim>false</cdr:interim>
<cdr:segments>
<cdr:segment>
<cdr:originationTimestamp>2009-07-08T15:08:56.727+01:00</cdr:originationTimestamp>
<cdr:initiation>Dialed</cdr:initiation>
<cdr:call_id>2040-1247062136726-5485131</cdr:call_id>
<cdr:originator type="sipGateway">
<cdr:sipGateway id="3">
<cdr:name>Audiocodes-91</cdr:name>
</cdr:sipGateway>
</cdr:originator>
<cdr:termination_attempt>
<cdr:termination_timestamp>2009-07-08T15:08:56.728+01:00</cdr:termination_timestamp>
<cdr:terminator type="group">
<cdr:group>
<cdr:tenant id="17">
<cdr:name>SpiriTel plc</cdr:name>
</cdr:tenant>
<cdr:type>Broadcast</cdr:type>
<cdr:extension>6024</cdr:extension>
<cdr:name>OLD PMS DDIS DO NOT USE</cdr:name>
</cdr:group>
</cdr:terminator>
<cdr:provided_address>01942760142</cdr:provided_address>
<cdr:direction>Inbound</cdr:direction>
<cdr:disposition>No Answer</cdr:disposition>
</cdr:termination_attempt>
</cdr:segment>
</cdr:segments>
</cdr:cdr>
...
</cdr:cdr_set>
Each entry is essentially the same but there are sometimes differences such as some of the fields may be missing, if they aren't required.

These values in an xml file are identifiers, not locators. Unless you are expecting to download a schema, it is not needed at all, and can be "flibble" if needed. I expect the best thing would be to just load it into XmlDocument / XDocument and try to access the data.
For example:
XmlDocument doc = new XmlDocument();
doc.Load("cdr.xml");
XmlNamespaceManager ns = new XmlNamespaceManager(doc.NameTable);
ns.AddNamespace("cdr", "http://www.naturalconvergence.com/schema/cdr/v3/cdr");
XmlElement el = (XmlElement)doc.SelectSingleNode(
"cdr:cdr_set/cdr:cdr/cdr:originator", ns);
Console.WriteLine(el.GetAttribute("type"));
or to loop over the cdr elements:
foreach (XmlElement cdr in doc.SelectNodes("/cdr:cdr_set/cdr:cdr", ns))
{
Console.WriteLine(cdr.SelectSingleNode("cdr:call_id", ns).InnerText);
}
Note that the aliases used in the document are largely unrelated to the aliases used in the XmlNamespaceManager, hence you need to re-declare it. I could have used x as my alias in the C# just as easily.
Of course, if you prefer to work with an object model; run it through xsd (where cdr.xml is your example file):
xsd cdr.xml
xsd cdr.xsd /classes
Now you can load it with XmlSerializer.

alternativley load it into an Xdocument and use linq2XML? ... although you might just get the same error.
I don't know what data you want, so its hard to suggest a query.
I personally prefer the use of XDocument to xmlDocument now in most cases.
the only problem with the automatic generation of an XSD is that it can get your datatypes pretty badly wrong if you are not using a good sized chunk of sample data.