We Keep Coding

Big Data Xml file (file size over 20GB) convert to Json File

I have an XML file. I want to convert it to JSON with C#. However, the XML file is over 20 GB.
I have tried to read XML with XmlReader, then append every node to a JSON file. I wrote the following code:
var path = #"c:\result.json";
TextWriter tw = new StreamWriter(path, true, Encoding.UTF8);
tw.Write("{\"A\":");
using (XmlTextReader xmlTextReader = new XmlTextReader("c:\\muslum.xml"))
{
while (xmlTextReader.Read())
{
if (xmlTextReader.Name == "A")
{
var xmlDoc = new XmlDocument();
var v = xmlTextReader.ReadInnerXml();
string json = Newtonsoft.Json.JsonConvert.SerializeXmlNode(xmlDoc, Newtonsoft.Json.Formatting.None, true);
tw.Write(json);
}
}
}
tw.Write("}");
tw.Close();
This code not working. I am getting error while converting json. Is there any best way to perform the conversion?

I would do it the following way
generate classes out of xsd schema using xsd.exe
open file and read top level ( i e your document level ) tags one by one (with XmlTextReader or XmlReader)
serialize each tag into object using generated classes
deserialize resulting object to json and save to whatever
consider saving in batches of 1000-2000 tags
you are right about serialize/deserialize being slow. still doing work in several threads, preferably using TPL will give you good speed. Also consider using json.net serializer, it is really a lot faster than standard ones ( it is standard for web.api though)
I can put some code snippets in the morning if you need them.
We are processing big ( 1-10gigs) files this manner in order to save data to sql server database.

How to create a component by providing the XML source file as input

I want to create a component by giving XML source input directly using core service 2011, in SDL Tridion 2011.
I want to write code to create a component by uploading source XML. Using the core service 2011.
Say name of the xml file is helloworld.xml and location is D:\abcd\cdef\all\helloworld.xml.
I have written the code like this, but its not working.
XmlDocument contentxml = new XmlDocument();
contentxml.LoadXml(#"D:\abcd\cdef\all\helloworld.xml");
Response.Write("<BR>" + contentxml.ToString());
component.Content = contentxml.ToString();
ComponentData comp = (ComponentData)client.Create(component, new ReadOptions());
The Response.write is displaying nothing. Correct me if I missed any thing.
It's not creating any component and error is coming.
When i tried this:
XmlDocument contentxml = new XmlDocument();
try
{
contentxml.LoadXml(#"D:\abcd\cdef\all\helloworld.xml");
}
catch (XmlException exp)
{
Console.WriteLine(exp.Message);
}
StringWriter sw = new StringWriter();
XmlTextWriter xw = new XmlTextWriter(sw);
contentxml.WriteTo(xw);
Response.Write("<BR>" + sw.ToString());
component.Content = sw.ToString();
ComponentData comp = (ComponentData)client.Create(component, new ReadOptions());
This time it's showing unable to find UUId: some thing like that.
My helloworld.xml looks like this.
<Content xmlns="uuid:1111eb85-0X11-11f9-1e2X-1X82X78fX920">
<first>Hello World.This is Fisrt field</first>
<second>Hello World.This is second field</second>
</Content>
It would be great if some one share some sample code to do it.

Loading any XML from a a file and trying to create a component won't work unless the XML uses the format the CMS is expecting.
The XML structure of a component in SDL Tridion has some fixed parts (Example Nodes Content, Metadata) plus some flexible parts
(The way you define the fields).
First you need to have the XML with the same structure that the CMS is expecting.
Typically the nodes that should be in your xml are the CONTENT and METADATA, load those in an XML Document and use the Core Service API to
create a component using the content contained in those nodes.
The best way to know how which is the structure of a component based on an schema is to create a sample component using the Tridion UI and
see how the XML is constructed. After that you need to create your XML Sources using that structure.
I posted recently an example of how to create a component using the Core Service, please have a look at that.
Faulted State error while creating component with Core Service
Following this code, you can access the nodes Content and Metadata
componentData.Content = xmlUtil.GetNewXmlNode("Content", schemaData.NamespaceUri);
componentData.Metadata = xmlUtil.GetNewXmlNode("Metadata", schemaData.NamespaceUri);
And replace those with your content

The general outline:
Load the XML from the file into an XDocument / XmlDocument.
Create a new Component by calling GetDefaultData on the client.
Set the Content property of the Component to the XML.
Save the Component by calling Save on the client.
If you haven't already, please have a look at the Core Service API documentation available on SDL Tridion World.
If you have trouble implementing this, please post the code that you have and what you have tried in order to make it work.

Using XmlDocument.LoadXML() expects an XML string as input, as commented by Peter you should use XMLDocument.Load() instead, see here for more details http://msdn.microsoft.com/en-us/library/a8ta6tz4.aspx
When you have passed that hurdle you will need the information Miguel gave in his answer to continue.

Parse Weather Forecast Data (from the NDFD) in c#

I am using > http://www.weather.gov/forecasts/xml/DWMLgen/wsdl/ndfdXML.wsdl webservice to get the weather detials by calling GmlTimeSeries webmethod. Now I just want to read temparature, weather icon link details from the xml. The xml has huge data. Can anybody give an idea to fetch the required data from the xml?
NDFD HOme Page
The XML looks almost like below : Full XML File is Here
I want to fetch Temparature from below xml data:
<gml:featureMember>
<app:Forecast_Gml2Point>
<gml:position>
<gml:Point srsName="EPSG:4326">
<gml:coordinates>-87.8859170,41.7450495</gml:coordinates>
</gml:Point>
</gml:position>
<app:validTime>2011-06-07T12:00:00</app:validTime>
<app:temperature>77.0</app:temperature>
</app:Forecast_Gml2Point>
</gml:featureMember>
<gml:featureMember>
<app:Forecast_Gml2Point>
<gml:position>
<gml:Point srsName="EPSG:4326">
<gml:coordinates>-87.8859170,41.7450495</gml:coordinates>
</gml:Point>
</gml:position>
<app:validTime>2011-06-07T15:00:00</app:validTime>
<app:temperature>90.0</app:temperature>
</app:Forecast_Gml2Point>
</gml:featureMember>
And weather phrase from below:
<gml:featureMember>
<app:Forecast_Gml2Point>
<gml:position>
<gml:Point srsName="EPSG:4326">
<gml:coordinates>-87.8859170,41.7450495</gml:coordinates>
</gml:Point>
</gml:position>
<app:validTime>2011-06-08T03:00:00</app:validTime>
<app:weatherPhrase>Mostly Clear</app:weatherPhrase>
</app:Forecast_Gml2Point>
</gml:featureMember>
<gml:featureMember>
<app:Forecast_Gml2Point>
<gml:position>
<gml:Point srsName="EPSG:4326">
<gml:coordinates>-87.8859170,41.7450495</gml:coordinates>
</gml:Point>
</gml:position>
<app:validTime>2011-06-08T06:00:00</app:validTime>
<app:weatherPhrase>Mostly Clear</app:weatherPhrase>
</app:Forecast_Gml2Point>
</gml:featureMember>
The above is a piece of xml file. Like this I have large data for 7 days weather details. I need to read for the temp and weather condition from above xml.
Full XML File is Here

I think you will find your answer here
Edit:
you need to use namespace, for example:
XNamespace app = "http://www.weather.gov/forecasts/xml/OGC_services";
var result = from i in doc.Descendants(app+"Forecast_Gml2Point")
select new
{
temperature = i.Element(app + "temperature"),
icon = i.Element(app+"weatherIcon")
};
Edit 2:
if you need to get Element with other namespace, here is another example:
XNamespace gml ="http://www.opengis.net/gml"
i.Element(gml+"coordinates" )

It would be easier if you use the "Add Web Reference" feature of Visual Studio. In this way, Visual Studio generates all the (proxy) classes for you base on the WSDL, and you can then program against the classes like how you'll do normally. In other words, there is no parsing of XML needed.
As pointed out in this link:
Visual Studio.Net Web References are proxy classes created on the client to connect to the Web Service running on the server. Inside of the IDE Web references automatically generate code and insert hidden files into your project. This is required because .Net is type safe and in order to compile code that uses the Web Service, the client has to know the method signature of each method that is called.
You might want to refer to the above link about consuming WSDL in details.

How to cache XML url?

I am retrieving an XML string through a URL. My code works great, but I do not know how to add caching to it. I am not sure if I am able to cache XML streams or if that is even the right approach. What is the best way to add caching here?
XmlTextReader xmlTextReader = new XmlTextReader(this.RssUrl);
XmlDataDocument xdoc1 = new XmlDataDocument();
xdoc1.DataSet.ReadXml(xmlTextReader, XmlReadMode.InferSchema);
return xdoc1.DataSet.Tables["item"];

You could save the XML together with a timestamp, and if you need to reread the XML, you read from the stored XML until the timestamp gets older than a preconfigured value. Most RSS-readers do this to avoid flooding the RSS-service with requests.
Or, if you control the RSSUrl, you could implement the caching there. That would utilize HTTP caching and the fact that the web server can return 304 Not modified if no new items are added to the feed.

Cache the entire XmlDataDocument. If you only cache the XML, you'll have to parse it all the time.

Tool that auto-generate a code for accesing a xml-file

My application have a configuration xml-file. That file contains more than 50 program settings. At the present time I read and save each program setting separately. I guess It is not effiсiently for such tasks.
I need something that can auto-generate a code for load and save my program settings using predefined xml-schema.
I found a dataset in Add New Item dialog. Unfortunately, i cannot add new code to dataset1 such as events in set-accessors of properties because of this
// Changes to this file may cause incorrect behavior and will be lost if
// the code is regenerated.
Maybe, there is a tool that allows a user to generate a wrapper for accesing a xml-file ? Such as DataSet1, but with availability to add events.
Edit: I didn't mark a useful answer because i read an articles (link) which you give me. I will mark useful answer later.

If you are not willing to use app.config/web.config or the properties file (which Oded and Bruno recommend and I recommend as well), I highly recommend this utility:
Web Services Contract First (WSCF) Blue for VS2008 and VS2010
If you're on VS2005, you'll want this version of the tool: http://www.thinktecture.com/resourcearchive/tools-and-software/wscf (don't use the VS2008 version on this site. I could never get it to work right.)
Once you have the plugin installed into Visual Studio, you'll need an XSD schema of your XML file. (Google for an online XSD Generator.) Following the instructions found on the WSCF website, you can generate a wrapper class that will deserialize and reserialize your XML and give you an abstracted view of your XML.
I figure it is impossible (or at least very hard) to add new node/element TYPES, but adding new instances of existing node/element types, accessing to your data, editing the data, reordering nodes, and then saving back out are all easy.
Deserialization code looks like this:
private MyGeneratedXMLconfigClass config;
using (StreamReader sr = new StreamReader(filename))
{
XmlSerializer cXml = new XmlSerializer(typeof(MyGeneratedXMLconfigClass));
config = (MyGeneratedXMLconfigClass)cXml.Deserialize(sr);
}
Now your XML has been de-serialized into the "config" instance of your custom class. Then you can access the whole class as a series of nested values and Lists.
For example:
string errorFile = config.errorsFile;
List<string> actions = config.actionList;
var specialActions = from action in config.actionList
where action.contains("special")
select action;
Etc., etc. Then once you're done manipulating your data, you can re-serialize with this code:
using (StreamWriter wr = new StreamWriter(filename, false))
{
XmlSerializer cXml = new XmlSerializer(typeof(MyGeneratedXMLconfigClass));
cXml.Serialize(wr, config);
}
One of the very nice things about this tool is that it auto-generates all classes as "partial" classes, so that you can feel free to extend each class on your own without fear of your code getting stomped on in case you ever need to re-generate because the XSD/XML was changed.
I imagine this might sound like a lot, but the learning curve is actually pretty easy and once you get it installed and working, you'll realize how stupidly easy it is. It's worth it. I swear. :-)

If you have an appropriate xsd schema for your xml file microsoft provides xsd.exe, a small tool which auto-generates c# classes for this schema.
For details see: http://msdn.microsoft.com/en-us/library/x6c1kb0s%28VS.71%29.aspx

Why are you using hand rolled XML for configuration? What is wrong with the existing app.config and web.config schemas?

Why not use a .Settings file?

You can follow these steps:
1) generate an XSD file from your XML file. For There used to be a tool to infer schema from an XML file, I forgot what it's called. Currently I use my own utility, which basically runs this core routine to read an xml file and generate the corresponding xsd:
static void InferSchema(string fileName)
{
XmlWriter writer = null;
XmlSchemaInference infer = new XmlSchemaInference();
XmlSchemaSet sc = new XmlSchemaSet();
string outputXsd = fileName.Replace(".xml", ".xsd");
sc = infer.InferSchema(new XmlTextReader(fileName));
using (writer = XmlWriter.Create(new StreamWriter(outputXsd)))
{
foreach(XmlSchema schema in sc.Schemas())
{
schema.Write(writer);
Console.WriteLine(">> found schema - generated to {0}",
outputXsd);
}
}
}
2) run xsd.exe to generate a serializable class from the XSD file.
xsd.exe /c /n:MyNameSpaceHere MyGenerated.xsd
Next, you can read the XML file into the serializable class using XmlSerializer.Serialize() method. Something like this:
public static void Serialize<T>(T data, TextWriter writer)
{
try
{
XmlSerializer xs = new XmlSerializer(typeof(T));
xs.Serialize(writer, data);
}
catch (Exception e)
{
throw;
}
}
Finally, you can write back to the XML file from the class using the XmlSerializer.Deserialize() method, like this for instance:
public static void Deserialize<T>(out T data, XmlReader reader)
{
try
{
XmlSerializer xs = new XmlSerializer(typeof(T));
data = (T)xs.Deserialize(reader);
}
catch (Exception e)
{
reader.Close();
throw;
}
}

This is called a properties file. C# should have something similar to Java's Properties class where you can load all properties without hard-coding their names.
EDIT:
There's apparently no built-in properties parsing solution for C#. But you can easily implement your own. See here.

If you have an XSD file, you can generate classes from that. Besides the already mentioned xsd.exe from Microsoft (which hasn't been updated for quite some time), there are other tools for this. I am using XSD2Code, which allows generating strongly typed collections, lazy initialization, etc.
If you do not have an XSD, you can point the xsd.exe at your xml-file, and it will generate an XSD from that. The schema usually needs some work, but generally is a good starting point.
xsd.exe (instance).xml

You can use System.Xml.Serialization - it's very easy and you can serialize even class objects directly, like MyCustomClass (it even saves MyCustomClass public fields).
Deserializing the XML file will get a new instance of MyCustomClass, so such a feature is priceless.
Note one thing: you should add EVERY SINGLE TYPE you use in the class, but that's easy though.
I attached a complete project that does the thing you want. just change classes and objects and that will be all.
source code
for example (i'm shortening the code):
using System.Xml;
using System.Xml.Serialization;
using System.IO;
[XmlRootAttribute("Vendor")]
class Vendor{
[XmlAttribute]
Product prod;
}
[XmlRootAttribute("Product")]
class Product{
[XmlAttribute]
public string name="";
}
class Test{
Vendor v=new Vendor();
Product p=new Product();
p.name="a cake";
v.prod=p;
//add EVERY SINGLE TYPE you use in the serialized class.
Type[] type_list = { typeof(Product) };
XmlSerializer packer = new XmlSerializer(v.GetType(),type_list);
XmlWriter flusher = XmlWriter.Create(#"c:\bak.xml");
packer.Serialize(flusher, v);
flusher.Close();
XmlReader restorer = XmlReader.Create(#"c:\bak.xml");
Vendor v2 = (Vendor)packer.Deserialize(restorer);
//v2.prod.name is now "cake"
//COOL was my first impression :P
}