I need to create a large text document. I currently use StringBuilder to make the document and then call File.WriteallText(filename,sb.ToString). Unfortunately, this is now starting to throw out of memory exceptions.
Is there a better way to stream a StringBuilder to file or is there some other technique I should be using?
Instead of using StringBuilder, try using TextWriter (which has a broadly similar API, but which can write to a number of underlying destinations, including files) - i.e.
using(TextWriter writer = File.CreateText(path))
{
// loop etc
writer.Write(...);
}
More generally, it is worth separating the code that knows about files from the code that knows about how to write the data, i.e.
using(var writer = File.CreateText(path))
{
Serialize(writer);
}
...
void Serialize(TextWriter writer)
{
...
}
this makes it easier to write to different targets. For example, you can now do in-memory too:
var sw = new StringWriter();
Serialize(sw);
string text = sw.ToString();
The point being: your Serialize code didn't need to change to accomodate a different target. This could also be writing directly to a network, or writing through a compression/encryption stream. Very versatile.
Just use a StreamWriter that writes to a FileStream:
using (StreamWriter writer = new StreamWriter("filename.txt")) {
...
}
This will of course mean that you can't change the text that is already written, like you can do in a StringBuilder, but I assume that you are not using that anyway.
Why not streaming directly into the stream?
You could use the TextWriter.
You can use StreamWriter and write to the file directly.
Related
The program I am working on is currently using a StreamWriter to create one or many text files in a target folder. Off StreamWriter class, I am using WriteLine and its IDisposable interface via Using directive (for implicit .Close).
I need to add an option to create one or many text files in a zip archive inside a target folder. I was going to change existing code to use streams, so it's possible to use a ZIP file as an output (planning to use DotNetZip).
I was thinking to create some GetOutputStream function and feed that into the currently existing method. This function would determine whether archive option is set, and either create plain files, or archive them. Problem is that MemoryStream, which looks like a good buffer class to use with DotNetZip, does not intersect with StreamWriter in the inheritance hierarchy.
Looks like my only option is to create some IWriteLine interface, which would implement WriteLine and IDisposable. Then branch two new child classes from StreamWriter and MemoryStream, and implement IWriteLine in them.
Is there a better solution?
The current code conceptually looks like this:
Using sw As StreamWriter = File.CreateText(fullPath)
sw.WriteLine(header)
sw.WriteLine(signature)
While dr.Read 'dr=DataReader
Dim record As String = GetDataRecord(dr)
sw.WriteLine(record)
End While
End Using
For code samples, either VB.NET or C# is fine, although this is more of a conceptual question.
EDIT: Cannot use .NET 4.5's System.IO.Compression.ZipArchive, have to stick with .NET 4.0. We still need to support clients running on Windows 2003.
Use the StreamWriter(Stream) constructor to have it write to a MemoryStream. Set the Position back to 0 so you can then save the written text to the archive with ZipFile.Save(Stream). Check the ZipIntoMemory helper method in the project's sample code for guidance.
First of all, with .NET 4.5 System.IO.Compression.ZipArchive class (see http://msdn.microsoft.com/en-us/library/system.io.compression.ziparchive.aspx) you no longer need DotNetZip at least for common zipping tasks.
It could look like this:
string filePath = "...";
//Create file.
using (FileStream fileStream = File.Create(filePath))
{
//Create archive infrastructure.
using (ZipArchive archive = new ZipArchive(fileStream, ZipArchiveMode.Create, true, Encoding.UTF8))
{
SqlDataReader sqlReader = null;
//Reading each row into a separate text file in the archive.
while(sqlReader.Read())
{
string record = sqlReader.GetString(0);
//Archive entry is a file inside archive.
ZipArchiveEntry entry = archive.CreateEntry("...", CompressionLevel.Optimal);
//Get stream to write the archive item body.
using (Stream entryStream = entry.Open())
{
//All you need here is to write data into archive item stream.
byte[] recordData = Encoding.Unicode.GetBytes(record);
MemoryStream recordStream = new MemoryStream(recordData);
recordStream.CopyTo(entryStream);
//Flush the archive item to avoid data loss on dispose.
entryStream.Flush();
}
}
}
}
I use this code to write a string to a file.
var fs = File.Open(path, FileMode.OpenOrCreate, FileAccess.Write);
var sw = new StreamWriter(fs, Encoding.UTF8);
sw.Write(dataString);
fs.Flush();
sw.Flush();
sw.Close();
When I debug it, I see the dataString correct. However, after writing to a file, there are some strings have wrong orders. Should I add a thread Sleep after writing data to the file because right after writing data to the file, I read it back for other process.
OR probably there is better way to write a string to a file?
Thanks in advance.
OR probably there is better way to write a string to a file?
Yes, there's a simpler way. Use File.WriteAllText or File.AppendAllText.
File.WriteAllText("output.txt", text, Encoding.UTF8);
Apart from being more concise, another advantage is that the file will be correctly closed even if there is an exception. However, I doubt that it will change the order in which the data is written to the file.
Let's say I have the following code:
StreamWriter sw = new StreamWriter(File.OpenWrite(Path));
sw.Write("Some stuff here");
sw.Dispose();
This code replaces the contents of the file with "Some stuff here," however I would like to add text to the file rather than replacing the text. How would I do this?
You could use the File.AppendAllText method and replace the 3 lines of code you have with:
File.AppendAllText(Path, "blah");
This way you don't have to worry about streams and disposing them (even in the event of an exception which by the way you are not doing properly) and the code is pretty simple and straightforward to the point.
You need to tell the StreamWriter you want to append:
var sw = new StreamWriter(path, true);
File.OpenWrite doesn't support appending.
Check out System.IO.File.AppendAllText
http://msdn.microsoft.com/en-us/library/ms143356.aspx
You can do what you want doing something like
File.AppendAllText(path, text);
There is a StreamWriter constructor which takes a bool as the 2nd parameter, which instructs the writer to append if true.
Use the append parameter:
StreamWriter sw = new StreamWriter(Path, true);
http://msdn.microsoft.com/en-us/library/36b035cb.aspx
To append text to a file, you can open it using FileMode.Append.
StreamWriter sw = new StreamWriter(File.Open(Path, System.IO.FileMode.Append));
This is just one way to do it. Although, if you don't explicitly need the StreamWriter object, I would recommend using what others have suggested: File.AppendAllText.
I would like to write to a text file, but I have small problem. When use the code below, it writes just once.
StreamWriter fileWriter = new StreamWriter("test.txt");
fileWriter.WriteLine(jointHead.Position.X);
fileWriter.Close();
When I write like this:
Debug.WriteLine(jointHead.Position.X);
it writes the X position until I close the application. How can I write to a text file like I write in Debug mode until I close the application. If I take fileWrite.Close() from where it stays, the program doesnt work.
Thank you...
From your description I am assuming that the code snippets you give are in a loop.
It's likely that you will get better performance by moving the file open/close outside of the loop (which will also cure your problem).
I you really want to keep opening/closing the file every time, then specify the append flag.
using (var fileWriter = new StreamWriter("test.txt", true))
{
fileWriter.WriteLine(jointHead.Position.X);
}
Try
StreamWriter fileWriter = new StreamWriter("test.txt", true);
fileWriter.WriteLine(jointHead.Position.X);
fileWriter.Close();
This will allow text written to be appended to the end of the file.
At the moment your writing from the beginning each time.
edit
If you wish to clear the file at the start of the application then just perform the following:
StreamWriter fileWriter = new StreamWriter("test.txt");
fileWriter.Write("");
fileWriter.Close();
I'd rather suggest you to use some sort of logger with a stringbuilder.
public class Logger {
private StringBuilder sb;
public Logger() {
sb = new StringBuilder();
}
public Log(String log) {
sb.Append(log).AppendLine();
}
public void Flush() {
File.WriteAllText(String.Format(#"D:\Logs\Log at {0:yyyy-MM-dd HH:mm:ss}.txt", DateTime.Now), sb.ToString());
sb.Clear();
}
}
This class is much more elegant and reusable solution. It is really acceptable if your target log is not very big.
Keep the stream writer open. But call Flush after the WriteLine call.
Alternatively you can open the file for appending, instead of recreating it for each line.
You might also want to look into an existing logging framework. There are many existing ones, no need to reinvent the wheel. Personally I'd use Common.Logging with a backend of your choice.
that constructor of the streamwriter will delete test.txt if it already existed. so every time that bit of code gets executed it will delete the file text.txt that it created earlier. instead, use the overload for the constructor of streamwriter that takes an additional bool to append to the existing test.txt file instead of replacing it:
StreamWriter fileWriter = new StreamWriter("test.txt", true);
alternatively, you could go with the File.AppendAllLines method to append your text to the file. then you don't need to worry about closing the file handle and the method name itself clearly states what's going to happen. to me this would be more convenient and not as obscure as the overloaded streamwriter constructor.
or alternatively, you could go for a logging framework like NLog. in that case NLog will take care of all your file operations so you're free of worries there then. also, you could configure NLog to write to whatever you like, like your file or, as you mentioned, the debug output window, or the event log, etc etc. also, you can bet on any file operations probably being a whole lot more efficient than your own implementation.
I'm currently exporting a database table with huge data (100000+ records) into an xml file using XmlTextWriter class and I'm writing directly to a file on the physical drive.
_XmlTextWriterObject = new XmlTextWriter(_xmlFilePath, null);
While my code runs ok, my question is that is it the best approach? Should I instead write the whole xml in memory stream first and then write the xml document in physical file from memory stream? And what are the effects on memory/ performance in both cases?
EDIT
Sorry that I could not actually convey what I meant to say.Thanks Ash for pointing out.
I will indeed be using XmlTextWriter but I meant to say whether to pass a physical file path string to the XmlTextWriter constructor (or, as John suggested, to the XmlTextWriter.Create() method) or use stream based api. My current code looks like the following:
XmlWriter objXmlWriter = XmlTextWriter.Create(new BufferedStream(new FileStream(#"C:\test.xml", FileMode.Create, System.Security.AccessControl.FileSystemRights.Write, FileShare.None, 1024, FileOptions.SequentialScan)), new XmlWriterSettings { Encoding = Encoding.Unicode, Indent = true, CloseOutput = true });
using (objXmlWriter)
{
//writing xml contents here
}
The rule of thumb is to use XmlWriter when the document need only be written and not worked with in memory, and to use XmlDocument (or the DOM) where you do need to work with it in memory.
Remember though, XmlWriter implements IDisposable, so do the following:
using (XmlWriter _XmlTextWriterObject = XmlWriter.Create(_xmlFilePath))
{
// Code to do the write here
}
While my code runs ok, my question is that is it the best approach?
As mentioned and your update, XmlWriter.Create is fine.
Or should I write the whole xml in memory stream first and then write the xml document in physical file from memory stream?
Do you have the memory to write the entire file in-memory? If you do then that approach will be faster, otherwise stream it using a FileStream which will take care of it for you.
And what are the effects on memory/ performance in both cases?
Reading the entire XML file in will use more memory, and spike the processor to start with. Streaming to disk will use more processor. But you'll need to be using a huge file for this to be noticeable given even desktop hardware now. If you're worried about the size increasing even more in the future, stick to the FileStream technique to future proof it.
As John Saunders mentioned it is better to use XmlWriter.Create(). That is the recommendation from the MSDN. The XmlWriter.Create() method can also take an XmlWriterSettings object. There you can customize your behavior quite a bit. If you don't need validation and character checking then you can turn it off and get a bit more speed. For example
XmlWriterSettings settings = new XmlWriterSettings();
settings.CheckCharacters = false;
using (XmlWriter writer = XmlWriter.Create("path", settings))
{
//writing code
writer.Flush();
}
Otherwise I think everything is okay.