We Keep Coding

Best practice for writing big files

I need to write a big file in my project.
What I learned:
I should NOT write the big file directly to the destination path,
because this may leave a incomplete file in case the app crash while writing it.
Instead, I should write to a temporary file and move (rename) it. (called atomic file operation)
My code snippet:
[NotNull]
public static async Task WriteAllTextAsync([NotNull] string path, [NotNull] string content)
{
string temporaryFilePath = null;
try {
temporaryFilePath = Path.GetTempFileName();
using (var stream = new StreamWriter(temporaryFilePath, true)) {
await stream.WriteAsync(content).ConfigureAwait(false);
}
File.Delete(path);
File.Move(temporaryFilePath, path);
}
finally {
if (temporaryFilePath != null) File.Delete(temporaryFilePath);
}
}
My Question:
The file will be missing if the app crashes between File.Delete and File.Move. Can I avoid this?
Is there any other best practice for writing big files?
Is there any suggestion on my code?

The file will be missing if the app crashes between File.Delete and File.Move. Can I avoid this?
Not that I'm aware of, but you can detect it - and if you use a more predictable filename, you can recover from that. It helps if you tweak the process somewhat to use three file names: the target, a "new" file and an "old" file. The process becomes:
Write to "new" file (e.g. foo.txt.new)
Rename the target file to the "old" file (e.g. foo.txt.old)
Rename the "new" file to the target file
Delete the "old" file
You then have three files, each of which may be present or absent. That can help you to detect the situation when you come to read the new file:
No files: Nothing's written data yet
Just target: All is well
Target and new: App crashed while writing new file
Target and old: App failed to delete old file
New and old: App failed after the first rename, but before the second
All three, or just old, or just new: Something very odd is going on! User may have interfered
Note: I was unaware of File.Replace before, but I suspect it's effectively just a simpler and possibly more efficient way of doing the code you're already doing. (That's great - use it!) The recovery process would still be the same though.

You can use File.Replace instead of deleting and moving files. In case of hard fault (electricity cut or something like this) you will always lost data, you have to count with that.

If no text to write via StreamWriter, then discard (do not create) the file

I am using the StreamWriter to create a file and to write some text to that file. In some cases I have no text to write via StreamWriter, but the file was already created when StreamWriter was initialized.
using (StreamWriter sw = new StreamWriter(#"C:\FileCreated.txt"))
{
}
Currently I am using the following code, when StreamWriter is closed, to check if the FileCreated.txt content is empty, if it is delete it. I am wondering if there is a more elegant approach than this (an option within StreamWriter perhaps)?
if (File.Exists(#"C:\FileCreated.txt"))
{
if (new FileInfo(#"C:\FileCreated.txt").Length == 0)
{
File.Delete(#"C:\FileCreated.txt");
}
}
By the way, I must open a stream to write before I can check if there is any text because of some other logic in the code.

If you want to take input from the user bit by bit, you can make your source a StringBuilder, and then just commit to disk when you're done
StringBuilder SB = new StringBuilder();
...
SB.AppendLine("text");
...
if(SB.Length > 0)
File.WriteAllLines(SB.ToString());

Delaying opening the file until the first output would solve this problem, but it might create a new one (if there's a permission error creating the file, you won't find out until later, maybe when the operator is no longer at the computer).
Your current approach is decent. I don't see the need to test File.Exists, though, if you just closed a stream to it. Also consider the race condition:
You find that the file is zero-length
Another process writes to the file
You delete the file
Also consider that you might have permission to create a file, and not to delete it afterwards!
Doing this correctly requires using the raw Win32 API, as I described in a previous answer. Do note that a .NET stream could be used for the first file handle, as long as you specify the equivalent of FILE_SHARE_WRITE.

Revisit your assumptions, i.e. that you must open the stream before checking for content. Simply reorganize your logic.

UnauthorizedAccessException StreamWriter

I have the following code:
public WriteToFile(string path, List<string> text)
{
File.Delete(path);
using (TextWriter writer = new StreamWriter(path, true))
{
foreach(string t in text)
{
writer.WriteLine(text);
}
}
}
Most of the time it works fine, the file is deleted and then created again with the text inside. However every so often the using statement throws an UnauthorizedAccessException. Any idea why? I have admin rights and the program is run as admin.

This is normal, it became undiagnosable because you used File.Delete(). Which is unnecessary, just use the StreamWriter(string) constructor.
This goes wrong because deleting a file doesn't provide a guarantee that the file will actually be deleted. It may be opened by another process. Which has opened the file with delete sharing, programs like virus scanners and file indexers commonly do this. Which makes the Delete() call succeed but the file doesn't disappear until all handles on the file are closed. You got the UnauthorizedAccessException exception because the file didn't get deleted yet.
Get ahead by removing the File.Delete() call. You still need to assume that the StreamReader() constructor can fail. Less often, it is bound to happen sooner or later. You'll get a better exception message. Such are the vagaries of a multi-tasking operating system.

How to write to a text file until the application is closed

I would like to write to a text file, but I have small problem. When use the code below, it writes just once.
StreamWriter fileWriter = new StreamWriter("test.txt");
fileWriter.WriteLine(jointHead.Position.X);
fileWriter.Close();
When I write like this:
Debug.WriteLine(jointHead.Position.X);
it writes the X position until I close the application. How can I write to a text file like I write in Debug mode until I close the application. If I take fileWrite.Close() from where it stays, the program doesnt work.
Thank you...

From your description I am assuming that the code snippets you give are in a loop.
It's likely that you will get better performance by moving the file open/close outside of the loop (which will also cure your problem).
I you really want to keep opening/closing the file every time, then specify the append flag.
using (var fileWriter = new StreamWriter("test.txt", true))
{
fileWriter.WriteLine(jointHead.Position.X);
}

Try
StreamWriter fileWriter = new StreamWriter("test.txt", true);
fileWriter.WriteLine(jointHead.Position.X);
fileWriter.Close();
This will allow text written to be appended to the end of the file.
At the moment your writing from the beginning each time.
edit
If you wish to clear the file at the start of the application then just perform the following:
StreamWriter fileWriter = new StreamWriter("test.txt");
fileWriter.Write("");
fileWriter.Close();

I'd rather suggest you to use some sort of logger with a stringbuilder.
public class Logger {
private StringBuilder sb;
public Logger() {
sb = new StringBuilder();
}
public Log(String log) {
sb.Append(log).AppendLine();
}
public void Flush() {
File.WriteAllText(String.Format(#"D:\Logs\Log at {0:yyyy-MM-dd HH:mm:ss}.txt", DateTime.Now), sb.ToString());
sb.Clear();
}
}
This class is much more elegant and reusable solution. It is really acceptable if your target log is not very big.

Keep the stream writer open. But call Flush after the WriteLine call.
Alternatively you can open the file for appending, instead of recreating it for each line.
You might also want to look into an existing logging framework. There are many existing ones, no need to reinvent the wheel. Personally I'd use Common.Logging with a backend of your choice.

that constructor of the streamwriter will delete test.txt if it already existed. so every time that bit of code gets executed it will delete the file text.txt that it created earlier. instead, use the overload for the constructor of streamwriter that takes an additional bool to append to the existing test.txt file instead of replacing it:
StreamWriter fileWriter = new StreamWriter("test.txt", true);
alternatively, you could go with the File.AppendAllLines method to append your text to the file. then you don't need to worry about closing the file handle and the method name itself clearly states what's going to happen. to me this would be more convenient and not as obscure as the overloaded streamwriter constructor.
or alternatively, you could go for a logging framework like NLog. in that case NLog will take care of all your file operations so you're free of worries there then. also, you could configure NLog to write to whatever you like, like your file or, as you mentioned, the debug output window, or the event log, etc etc. also, you can bet on any file operations probably being a whole lot more efficient than your own implementation.

how to check if a file is being accessed by any other processs and release it?

I am trying to delete/open/edit some files in my C# .Net application.Sometimes i get exception stating the file/directory is being accessed by another process.Is there a way to check if a file/directory is being accessed by process and try to release the file from that process?

No. The only way to do this is to try to access the file, and handle the IOException.
Realistically this is the only safe way anyway. Suppose there was a IsFileInUse() method, and you called it, and it returned "nope, nobody's using that file," and you went ahead and accessed the file. The problem is that in the meantime some other process might have locked or deleted the file. So you'd need to put exception handling around your attempt to access the file anyway. The "test by acquiring" model is the only one that is 100% reliable.
If a file is in use by another process, .NET doesn't provide a way of determining which other process that might be. I believe this would require some pretty low-level unmanaged code though I could be wrong. It is a very low-level operation, if it is possible at all, to "release the file from that process" because that would violate the other process' expectations -- e.g. it thinks it is allowed to write to the file but you have deleted the file and garbaged the handle. I believe you would need to terminate the other process if it's not willing to give up its lock voluntarily.

First, I suppose there are 2 things that may help you:
consider using FileAccess and FileShare flags when opening files
if data from the file is needed only withing the scope of the function use the construction
using(FileStream stream = File.Open(...)) { <file operations> }
this will ensure that file is closed immediately after exiting 'using' block, and not when FileStream object is collected by GC.
Second, there is an unsafe way to get processes that use the file. It is based on debugging features provided by windows. The main idea is to get all system handles and iterate through them to find which are the files handle and additional information. This is done using functions that I'm not sure are documented. If you are interested use google to find more information, but I do not think it is not a good way.

public bool IsInUse(string path)
{
bool IsFree = true;
try
{
//Just opening the file as open/create
using (FileStream fs = new FileStream(path, FileMode.OpenOrCreate))
{
//we can check by using
fs.CanRead // or
fs.CanWrite
}
}
catch (IOException ex)
{
IsFree = false;
}
return IsFree;
}
string path = "D:\\test.doc";
bool IsFileFree = IsInUse(path);