Find and append to file without loading it in memory? - c#

Im writing a simple XML file for logging that looks like this :
<root>
<objects>
</objects>
</root>
The file is created the first time like this
using (xmlWriter = new XmlTextWriter(filePathAndName, System.Text.Encoding.UTF8))
{
xmlWriter.Formatting = Formatting.Indented;
xmlWriter.WriteStartDocument();
xmlWriter.WriteStartElement("root");
xmlWriter.WriteStartElement("objects");
xmlWriter.WriteEndElement();
xmlWriter.WriteEndElement();
xmlWriter.WriteEndDocument();
xmlWriter.Close();
}
Now I need to place objects(serialized data contracts in string format) within the objects tag without loading it to memory.
I have found a lot of suggestions on how to do this but all loads the entire file into memory and thats no good when the files is large.
My thought is this :
Open file with some kind of reader
Search from end of file to </objects> tag
Store the index and close the reader/file
Open the file again but this time as a writer
Write the serlized datacontract to the index(just before the
Close file
I'm however not sure how to do this properly in C#?

The solution for me in this case was XML Inclusion Technique, it is not perfect but the login is much simplier. I can use appendToFile instead of manipulate or read into memory.

Related

Clear XML file before writing data back in

Is there a way to clear an XML document out before I write back into it with the Variable.Save(Stream) command?
FileStream savingFs = new FileStream("AutoUpdaterCheckList.xml", FileMode.OpenOrCreate);
savingFs.Seek(0,SeekOrigin.Begin);
xmlDoc.Save(savingFs);
Currently, when I write the stream back into the file I get a few leftover characters from the old file that cause the XML to break.

C# XmlTextWriter can't write to file

I have a problem with the XmlTextWriter class, I can't write to a file.
It is a console application I'm creating and the function looks like this.
private void WriteXML()
{
Console.WriteLine("Writing");
using (XmlTextWriter writer = new XmlTextWriter("DataW.xml", System.Text.Encoding.UTF8))
{
writer.Formatting = Formatting.Indented;
writer.WriteStartDocument();
writer.WriteComment("This is a comment");
writer.WriteStartElement("WeeklyReview");
writer.WriteStartElement("DateTime");
writer.WriteStartAttribute("DateTime","09/04/2018 05:00");
writer.WriteElementString("Activity", "At Theodor");
writer.WriteElementString("Social", "Not Theodor,Theodor");
}
}
The result I'm getting is just a file with nothing in it. And in VS 2017 the file is shown with a white background and nothing in it.
The file I'm writing to is in the solution explorer and located in the same place as the file I'm reading from (which works).
I'm guessing the issue you're seeing is that the XML file created is blank or incomplete. If so, add a using statement around your code like so:
using (XmlTextWriter writer = new XmlTextWriter("DataW.xml", System.Text.Encoding.UTF8))
{
writer.Formatting = Formatting.Indented;
writer.WriteStartDocument();
writer.WriteComment("This is a comment");
writer.WriteStartElement("Weekly Review"); //NB: A space in an element name!
writer.WriteStartElement("DateTime");
writer.WriteStartAttribute("09/04/2018 05:00"); //NB: a value given as an attribute name (consider: WriteAttributeString)!
writer.WriteElementString("Activity", "At Theodor");
writer.WriteElementString("Social", "Not Theodor,Theodor");
}
The reason for the issue is the methods you're calling write to an underlying stream rather than to the file system. To ensure that data is written to the file system you need to call the Flush method. Flush is called automatically when the writer is closed/disposed (but not when it's finalized). The using statement ensures that your writer is always disposed of correctly, so will ensure that your stream is flushed to disk.
For any other issues, you'd likely have an exception thrown (e.g. were there file access issues writing to the target location). If you're receiving an exception, please share that message and we can recommend a solution for that.
ps. The XML created by this is given below / it's not valid XML, so you'll have other issues down the line. I'll leave those for you to resolve, or for you to ask as another question should you have issues there. In the meantime, you can get some hints from the example in the documentation: https://learn.microsoft.com/en-us/dotnet/api/system.xml.xmltextwriter.writestartattribute?view=netframework-4.7.2
<?xml version="1.0" encoding="utf-8"?>
<!--This is a comment-->
<Weekly Review>
<DateTime 09/04/2018 05:00="">
<Activity>At Theodor</Activity>
<Social>Not Theodor,Theodor</Social>
</DateTime>
</Weekly Review>
I know the Question is old, but in VB.net i got the same result.
I had to call XmlTextWriter.Flush() at the End to get Content in my File.

Getting specific XML Node from XmlWriter C#

I would like to know if there is any chance of getting specific XML node from XmlWriter.
I've got a method, which exports employees to XML document and has a structure, something like:
<header>
<agency>
...
<employees>
<employee>
....
</employee>
<empolyee>
...
</employee>
</employees>
...
in for cycle in this method each employee is created and filled with elements. The whole method for exporting this XML has already been written.
Now, at the end of the loop, I would like to create a string, which contains the node of the employee, but I don't know, how to do it with the writer which is filled with the data in the loop.
I've tried to search for some solution, but I didn't find anything good for my case. I've considered something like MemoryStream, but I think that you need to pass the stream while you are creating the XmlWriter, which in my case firstly writes the header node etc.
Could you please help me with this? I don't want to create another Writer or Reader and write the nodes twice. At first, I would like to find some better solution.
Thank you!
EDIT: I'm pasting some of the code as you want:
XmlWriterSettings settings = new XmlWriterSettings();
settings.Indent = true;
settings.IndentChars = ("\t");
settings.OmitXmlDeclaration = false;
XmlWriter xmlWriter = XmlWriter.Create(filenameXml, settings);
xmlWriter.WriteStartElement("message");
xmlWriter.WriteStartElement("data");
xmlWriter.WriteStartElement("header");
xmlWriter.WriteElementString("agency", agencyNumber);
xmlWriter.WriteElementString("date", DateTime.Now.ToShortDateString());
xmlWriter.WriteEndElement();
xmlWriter.WriteStartElement("employees");
//Loading of employees from DB and some other work
for (int i = 0; i < employeesDs.RowCount(); i++)
{
xmlWriter.WriteStartElement("employee");
//A lot of work - xmlWriter.WriteElementString so many times or WriteStartElement (address, numbers, dates etc)
xmlWriter.WriteEndElement(); // employee
//HERE I would like to add new method which stores whole Employee element created in this iteration i to the string
//After that I need to store this string to DB to VARCHAR MAX, but I don't know how to get the string of the Employee
//from the writer in this part of the code
}
//another work - closing and flushing xmlwriter, disposing etc and exporting whole xml document

Best approach to write huge xml data to file?

I'm currently exporting a database table with huge data (100000+ records) into an xml file using XmlTextWriter class and I'm writing directly to a file on the physical drive.
_XmlTextWriterObject = new XmlTextWriter(_xmlFilePath, null);
While my code runs ok, my question is that is it the best approach? Should I instead write the whole xml in memory stream first and then write the xml document in physical file from memory stream? And what are the effects on memory/ performance in both cases?
EDIT
Sorry that I could not actually convey what I meant to say.Thanks Ash for pointing out.
I will indeed be using XmlTextWriter but I meant to say whether to pass a physical file path string to the XmlTextWriter constructor (or, as John suggested, to the XmlTextWriter.Create() method) or use stream based api. My current code looks like the following:
XmlWriter objXmlWriter = XmlTextWriter.Create(new BufferedStream(new FileStream(#"C:\test.xml", FileMode.Create, System.Security.AccessControl.FileSystemRights.Write, FileShare.None, 1024, FileOptions.SequentialScan)), new XmlWriterSettings { Encoding = Encoding.Unicode, Indent = true, CloseOutput = true });
using (objXmlWriter)
{
//writing xml contents here
}
The rule of thumb is to use XmlWriter when the document need only be written and not worked with in memory, and to use XmlDocument (or the DOM) where you do need to work with it in memory.
Remember though, XmlWriter implements IDisposable, so do the following:
using (XmlWriter _XmlTextWriterObject = XmlWriter.Create(_xmlFilePath))
{
// Code to do the write here
}
While my code runs ok, my question is that is it the best approach?
As mentioned and your update, XmlWriter.Create is fine.
Or should I write the whole xml in memory stream first and then write the xml document in physical file from memory stream?
Do you have the memory to write the entire file in-memory? If you do then that approach will be faster, otherwise stream it using a FileStream which will take care of it for you.
And what are the effects on memory/ performance in both cases?
Reading the entire XML file in will use more memory, and spike the processor to start with. Streaming to disk will use more processor. But you'll need to be using a huge file for this to be noticeable given even desktop hardware now. If you're worried about the size increasing even more in the future, stick to the FileStream technique to future proof it.
As John Saunders mentioned it is better to use XmlWriter.Create(). That is the recommendation from the MSDN. The XmlWriter.Create() method can also take an XmlWriterSettings object. There you can customize your behavior quite a bit. If you don't need validation and character checking then you can turn it off and get a bit more speed. For example
XmlWriterSettings settings = new XmlWriterSettings();
settings.CheckCharacters = false;
using (XmlWriter writer = XmlWriter.Create("path", settings))
{
//writing code
writer.Flush();
}
Otherwise I think everything is okay.

xml and files on disc interaction

Its been a while since i've needed to do this so i was looking at the old school methods of writing XMLDocument from code down to a File.
In my application i am writing alot to an XMLdocument with new elements and values and periodically saving it down to disc and also reading from the file and depending on the data i am doing things.
I am using methods like File.Exists(...) _xmldoc.LoadFile(..) etc...
Im wondering probably now a days there are better methods for this with regards
Parsing the XML
Checking its format for saving down
rather then the data being saved down being treated as text but as XML
maybe what i am doing is fine but its been a while and wondered if there are other methods :)
thanks
Well there's LINQ to XML which is a really nice XML API introduced in .NET 3.5. I don't think the existing XMLDocument API has changed much, beyond some nicer ways of creating XmlReaders and XmlWriters (XmlReader.Create/XmlWriter.Create).
I'm not sure what you really mean by your second and third bullets though. What exactly are you doing in code which feels awkward?
Have you looked at the Save method of your XmlDocument? It will save whatever is in your XmlDocument as a valid formatted file.
If your program is able to use the XmlDocument class, the XmlDocument class will be able to save your file. You won't need to worry about validating before saving, and you can give it whatever file extension you want. As to your third point... an XML file is really just a text file. It won't matter how the OS sees it.
I was a big fan of XmlDocument due to its facility to use but recently I got a huge memory problem with that class so I started to use XmlReader and XmlWriter.
XmlReader can be a little bit tricky to use if your Xml file is complex because you read the Xml file sequentially. In that case, the method ReadSubTree of XmlReader can be very useful because this method returns only the xml tree under the current node so you send the new xmlreader to a function to parse the subnode content and once it is done, you continue to the next node.
XmlReader Example:
string xmlcontent = "<BigXml/>";
using(StringReader strContent = new StringReader(xmlcontent))
{
using (XmlReader reader = XmlReader.Create(strContent))
{
while (reader.Read())
{
if (reader.Name == "SomeName" && reader.NodeType == XmlNodeType.Element)
{
//Send the XmlReader created by ReadSubTree to a function to read it.
ReadSubContentOfSomeName(reader.ReadSubtree());
}
}
}
}
XmlWriter Example:
StringBuilder builder = new StringBuilder();
using (XmlWriter writer = XmlWriter.Create(builder))
{
writer.WriteStartDocument();
writer.WriteStartElement("BigXml");
writer.WriteAttributeString("someAttribute", "42");
writer.WriteString("Some Inner Text");
//Write nodes under BigXml
writer.WriteStartElement("SomeName");
writer.WriteEndElement();
writer.WriteEndElement();
writer.WriteEndDocument();
}

Categories