We Keep Coding

Querying very large xml files

I have a merged very large xml file on scale of GB's. I am using following code with xpath queries to read and process data.
IColumn column = output.Schema.FirstOrDefault(col => col.Type != typeof(string));
if (column != null)
{
throw new ArgumentException(string.Format("Column '{0}' must be of type 'string', not '{1}'", column.Name, column.Type.Name));
}
XmlReaderSettings settings = new XmlReaderSettings();
settings.ConformanceLevel = ConformanceLevel.Auto;//.Fragment;
XmlReader r = XmlReader.Create(input.BaseStream, settings);
XmlDocument xmlDocument = new XmlDocument();
xmlDocument.Load(r);
//xmlDocument.LoadXml("<root/>");
//xmlDocument.DocumentElement.CreateNavigator().AppendChild(r);
//xmlDocument.Load(input.BaseStream);
XmlNamespaceManager nsmgr = new XmlNamespaceManager(xmlDocument.NameTable);
if (this.namespaces != null)
{
foreach (Match nsdef in xmlns.Matches(this.namespaces))
{
string prefix = nsdef.Groups[1].Value;
string uri = nsdef.Groups[3].Value;
nsmgr.AddNamespace(prefix, uri);
}
}
foreach (XmlNode xmlNode in xmlDocument.DocumentElement.SelectNodes(this.rowPath, nsmgr))
{
foreach (IColumn col in output.Schema)
{
var explicitColumnMapping = this.columnPaths.FirstOrDefault(columnPath => columnPath.Value == col.Name);
XmlNode xml = xmlNode.SelectSingleNode(explicitColumnMapping.Key ?? col.Name, nsmgr);
output.Set(explicitColumnMapping.Value ?? col.Name, xml == null ? null : xml.InnerXml);
}
yield return output.AsReadOnly();
}
However it only works well for smaller files on scale of MBs. It works fine locally but fails for ADLA. I need to use the namespace manager as well. How can i scale it so i can process bigger files. On submitting job with huge file I always get this error with no information.
VertexFailedError

Copying answer I gave in MSDN Forum to same question:
U-SQL Extractors by default are scaled out to work in parallel over smaller parts of the input files, called extents. These extents are about 250MB in size each.
If the data you are processing cannot fit into an extent, you have to tell the extractor with a C# attribute, that the extractor has to see the file in its entirety. You do that by adding the following part ahead of your extractor class:
[SqlUserDefinedExtractor(AtomicFileProcessing = true)]
Now in your case, XML documents obviously cannot be split since the parser needs to see the beginning and end of a document. This is especially true if you only have a single XML document (side note: Having GBs of a single XML document or JSON document is in my opinion often a bad idea).
Furthermore, I would suggest that you look at the sample XML extractor that we provide on our GitHub site here: https://github.com/Azure/usql/tree/master/Examples/DataFormats

How to use LINQ to XML when XML parent and child nodes have the same name

I am trying to extract some SQL data to XML from a Microsoft Dynamics environment, I am currently using LINQ To XML in C# to read and write to my XML files. One piece of data I need is from a view called SECURITYSUBROLE. Looking at the structure of this view shows that there is a column also named SECURITYSUBROLE. My normal method of extraction has given me this XML.
<SECURITYSUBROLE>
<SECURITYROLE>886301</SECURITYROLE>
<SECURITYSUBROLE>886317</SECURITYSUBROLE>
<VALIDFROM>1900-01-01T00:00:00-06:00</VALIDFROM>
<VALIDFROMTZID>0</VALIDFROMTZID>
<VALIDTO>1900-01-01T00:00:00-06:00</VALIDTO>
<VALIDTOTZID>0</VALIDTOTZID>
<RECVERSION>1</RECVERSION>
<RECID>886317</RECID>
</SECURITYSUBROLE>
When I try to import this data later on, I am getting errors because the parent XML node has the same name as a child node. Here is a snippet of the import method:
XmlReaderSettings settings = new XmlReaderSettings();
settings.CheckCharacters = false;
XmlReader reader = XmlReader.Create(path, settings);
reader.MoveToContent();
int count = 1;
List<XElement> xmlSubset = new List<XElement>();
while (reader.ReadToFollowing(xmlTag))
{
if (count % 1000 == 0)
{
xmlSubset.Add(XElement.Load(reader.ReadSubtree()));
XDocument xmlTemp = new XDocument(new XElement(xmlTag));
foreach (XElement elem in xmlSubset)
{
xmlTemp.Root.Add(elem);
}
xmlSubset = new List<XElement>();
ImportTableByName(connectionString, tableName, xmlTemp);
count = 1;
}
else
{
xmlSubset.Add(XElement.Load(reader.ReadSubtree()));
count++;
}
}
}
It's currently failing on the XmlReader.ReadToFollowing, where it doesn't know where to go next because of the name confusion. So my question has two parts:
1) Is there some better way to be extracting this data other than to XML?
2) Is there a way through LINQ To XML that I can somehow differentiate between the parent and child nodes named exactly the same?

To get the elements (in your case) for SECURITYSUBROLE you can check to see if the element's have children:
XElement root = XElement.Load(path);
var subroles = root.Descendants("SECURITYSUBROLE") // all subroles
.Where(x => !x.HasElements); // only subroles without children

I'm going to suggest a different approach:
1) VS2013 (possibly earlier versions too) has a function to create a class from an XML source. So get one of your XML files and copy the content to your clipboard. Then in a new class file Edit --> Paste Special --> Paste XML as Classes
2) Look into XmlSerialization which will allow you to convert an XML file into an in memory object with a strongly typed class.
XmlSerializer s = new XmlSerializer(yourNewClassTYPE);
TextReader r = new StreamReader(XmlFileLocation);
var dataFromYourXmlAsAStronglyTypedClass = (yourNewlyClassTYPE) s.Deserialize(r);
r.Close();

Can't parse XML data with C#

I have been trying to extract one value from a XML file, and then store it on the same file but in another node, I tried all the examples i've found on the net, read XPath Syntax documentation like hell and still can't get it to work.
I must take the <Documento ID="F60T33"> ID Value (which may vary) and copy it into <Reference URI="#F60T33">.
But I can't even do that if I can't manage to parse the lines, most of times node/variables/"", or I get an "object reference not established as object instance error".
Here's the code:
// Create a new XML document.
XmlDocument xmlDoc = new XmlDocument();
// Load an XML file into the XmlDocument object.
xmlDoc.PreserveWhitespace = true;
xmlDoc.Load(pfile);
//TEST !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! PROBLEMA
XmlNodeList Documentos = xmlDoc.GetElementsByTagName("//Documento");
XmlNodeList DatosDocumento = ((XmlElement)Documentos[0]).GetElementsByTagName("ID");
foreach (XmlElement nodo in DatosDocumento)
{
int I = 0;
XmlNodeList ID = nodo.GetElementsByTagName("ID");
Console.WriteLine("Elemento nombre ... {0}}", ID[i].InnerText);
}
//
XmlNodeList nodes = xmlDoc.SelectNodes("EnvioDTE");
XmlNode nodesimple = xmlDoc.SelectSingleNode("EnvioDTE");
Console.WriteLine("Lista Nodos: " + nodes.Count);
Console.WriteLine("Nodo Simple: " + nodesimple.InnerText);
foreach (XmlNode node in nodes)
{
string id = node.Attributes["ID"].InnerText;
Console.WriteLine(id);
}
I am almost certain the problem is on the XPath Syntax, but I can't get it to work.
Sadly I can't use XDocument as im using .NET 3.5 for this task, I would really appreciate some help on this, by behand apologize my bad english
As the XML file is too big, I'll put it here on this URL
http://puu.sh/bVNDc/31e4da5a26.xml

If you can get your references right for using System.Linq.XDocument you can do:
string idValue = xDocument.XPathSelectElement("//EnvioDTE/SetDTE")
.Attributes().SingleOrDefault(a => a.Name == "ID").Value;

Trouble reading iTunes XML feed

I am trying to read an XML feed from http://itunes.apple.com/us/rss/topsongs/limit=10/genre=2/xml.
I want to access the fields like this:
<im:price amount="1.29000" currency="USD">$1.29</im:price>
<im:releaseDate label="December 31, 1960">1960-12-31T16:00:00-07:00</im:releaseDate>
Here is what I have done so far:
var xml = "http://itunes.apple.com/us/rss/topsongs/limit=10/genre=2/xml";
XmlDocument doc = new XmlDocument();
doc.Load(xml);
XmlNodeList items = doc.SelectNodes("//entry");
foreach (var item in items) {
// Do something with item.
}
No luck, though. items is null. Why? What am I doing wrong?

You need to create a namespace manager to map the RSS and also the iTunes custom tags namespace URIs to short prefixes (itunes and im in the example below):
var xml = "http://itunes.apple.com/us/rss/topsongs/limit=10/genre=2/xml";
XmlDocument doc = new XmlDocument();
doc.Load(xml);
var namespaceManager = new XmlNamespaceManager(doc.NameTable);
namespaceManager.AddNamespace("itunes", "http://www.w3.org/2005/Atom");
namespaceManager.AddNamespace("im", "http://itunes.apple.com/rss");
XmlNodeList items = doc.SelectNodes("//itunes:entry", namespaceManager);
foreach (XmlNode item in items)
{
var price = item.SelectSingleNode("im:price", namespaceManager);
var releaseDate = item.SelectSingleNode("im:releaseDate", namespaceManager);
if (price != null)
{
Console.WriteLine(price.Attributes["amount"].InnerText);
}
if (releaseDate != null)
{
Console.WriteLine(releaseDate.Attributes["label"].InnerText);
}
}
For that specific feed you should get 10 entries.
It's in the docs as well:
If the XPath expression does not include a prefix, it is assumed that
the namespace URI is the empty namespace. If your XML includes a
default namespace, you must still use the XmlNamespaceManager and add
a prefix and namespace URI to it; otherwise, you will not get any
nodes selected. For more information, see Select Nodes Using XPath
Navigation.
Alternatively you can use a namespace-agnostic XPath (from here):
XmlNodeList items = doc.SelectNodes("//*[local-name() = 'entry']");
Finally, not sure why you said items is null. It cannot be. When running your original code you should get this:

How to read XML attributes in C#?

I have several XML files that I wish to read attributes from. My main objective is to apply syntax highlighting to rich text box.
For example in one of my XML docs I have: <Keyword name="using">[..] All the files have the same element: Keyword.
So, how can I get the value for the attribute name and put them in a collection of strings for each XML file.
I am using Visual C# 2008.

The other answers will do the job - but the syntax highlighting thingy and the several xml files you say you have makes me thinks you need something faster, why not use a lean and mean XmlReader?
private string[] getNames(string fileName)
{
XmlReader xmlReader = XmlReader.Create(fileName);
List<string> names = new List<string>();
while (xmlReader.Read())
{
//keep reading until we see your element
if (xmlReader.Name.Equals("Keyword") && (xmlReader.NodeType == XmlNodeType.Element))
{
// get attribute from the Xml element here
string name = xmlReader.GetAttribute("name");
// --> now **add to collection** - or whatever
names.Add(name);
}
}
return names.ToArray();
}
Another good option would be the XPathNavigator class - which is faster than XmlDoc and you can use XPath.
Also I would suggest to go with this approach only IFF after you try with the straightforward options you're not happy with performance.

You could use XPath to get all the elements, then a LINQ query to get the values on all the name atttributes you find:
XDocument doc = yourDocument;
var nodes = from element in doc.XPathSelectElements("//Keyword")
let att = element.Attribute("name")
where att != null
select att.Value;
string[] names = nodes.ToArray();
The //Keyword XPath expression means, "all elements in the document, named "Keyword".
Edit: Just saw that you only want elements named Keyword. Updated the code sample.

Like others, I would suggest using LINQ to XML - but I don't think there's much need to use XPath here. Here's a simple method to return all the keyword names within a file:
static IEnumerable<string> GetKeywordNames(string file)
{
return XDocument.Load(file)
.Descendants("Keyword")
.Attributes("name")
.Select(attr => attr.Value);
}
Nice and declarative :)
Note that if you're going to want to use the result more than once, you should call ToList() or ToArray() on it, otherwise it'll reload the file each time. Of course you could change the method to return List<string> or string[] by -adding the relevant call to the end of the chain of method calls, e.g.
static List<string> GetKeywordNames(string file)
{
return XDocument.Load(file)
.Descendants("Keyword")
.Attributes("name")
.Select(attr => attr.Value)
.ToList();
}
Also note that this just gives you the names - I would have expected you to want the other details of the elements, in which case you'd probably want something slightly different. If it turns out you need more, please let us know.

You could use LINQ to XML.
Example:
var xmlFile = XDocument.Load(someFile);
var query = from item in xmlFile.Descendants("childobject")
where !String.IsNullOrEmpty(item.Attribute("using")
select new
{
AttributeValue = item.Attribute("using").Value
};

You'll likely want to use XPath. //Keyword/#name should get you all of the keyword names.
Here's a good introduction: .Net and XML XPath Queries

**<Countries>
<Country name ="ANDORRA">
<state>Andorra (general)</state>
<state>Andorra</state>
</Country>
<Country name ="United Arab Emirates">
<state>Abu Z¸aby</state>
<state>Umm al Qaywayn</state>
</Country>**
public void datass(string file)
{
string file = HttpContext.Current.Server.MapPath("~/App_Data/CS.xml");
XmlDocument doc = new XmlDocument();
if (System.IO.File.Exists(file))
{
//Load the XML File
doc.Load(file);
}
//Get the root element
XmlElement root = doc.DocumentElement;
XmlNodeList subroot = root.SelectNodes("Country");
for (int i = 0; i < subroot.Count; i++)
{
XmlNode elem = subroot.Item(i);
string attrVal = elem.Attributes["name"].Value;
Response.Write(attrVal);
XmlNodeList sub = elem.SelectNodes("state");
for (int j = 0; j < sub.Count; j++)
{
XmlNode elem1 = sub.Item(j);
Response.Write(elem1.InnerText);
}
}
}