We Keep Coding

How can I log low-level OS file transactions in C#?

Since File/Process Monitor falls short in terms of filtering and unnecessary duplication when logging, I want to recreate what that program does and log all Windows file operations live in realtime.
I want to record various attributes such as the time, process name, source path, destination path, operation, result, and detail, just like Process Monitor does.
How can I get C# to extract this information from the OS?
EDIT: As zett42 pointed out, the FileSystemWatcher won't quite work as for example, file events created from processes themselves won't be intercepted. For instance, none of these transactions show up, even though I added the events: Changed, Created, Renamed, and Deleted to the FileSystemWatcher and set the EnableRaisingEvents flag to true.
EDIT 2: Using SimonMourier's suggestion of the Microsoft.Diagnostics.Tracing.TraceEvent nuget package, I managed to knock up the code below.
This section is put into a background worker:
Console.CancelKeyPress += (sender, e) => session.Stop();
session.EnableKernelProvider(KernelTraceEventParser.Keywords.All);
session.Source.Kernel.FileIOWrite += Kernel_FileIOWrite;
session.Source.Process();
And then the FileIOWrite event created runs the following when called (automatically):
private void Kernel_FileIOWrite(Microsoft.Diagnostics.Tracing.Parsers.Kernel.FileIOReadWriteTraceData obj)
{
string filename = obj.FileName;
string processpath = "";
if (obj.ProcessID == 0) processpath = "System Idle Process";
else if (obj.ProcessID == 4) processpath = "System";
else
{
try { processpath = "ID: " + obj.ProcessID + ": "+ Process.GetProcessById(obj.ProcessID).MainModule.FileName; }
catch { processpath = "error ID: "+ obj.ProcessID; }
}
richTextBox1.AppendText(filename + " ............."+ processpath +"\n");
}
Unfortunately, FileIOReadWriteTraceData.FileName is not picking up things Procmon picks up such as (for example), very common Chrome stuff such as writing to C:\Users\Dan\AppData\Local\Google\Chrome\User Data\Default\Cookies-journal or C:\Users\Dan\AppData\Local\Google\Chrome\User Data\Default\Current Session

You can never capture all the things that Process Monitor captures in C#. One of the reasons that ProcMon is so good at capturing all of those things is because ProcMon contains a driver that is loaded in kernel mode and hooks itself and listens for those events. If you want to replicate the process monitor, you will have to write your own driver to capture all the things that you want to. A windows driver cannot be written in C# and you will have to write the driver in C/C++.
The other option is to get Process Monitor to log everything to file and filter the events yourself.

Did you try to use WMI? The ManagementEventWatcher could provide all information you need, even though it's not that handy as the FileSytemWatcher.
I'm not sure it will work and unfortunately cannot try it myself at the moment, but this is the point where I would start.
Take a look at the anwer of RRUZ in this thread, which does exactly what you want. You will just need to find out if WMI provides all required information.

Using Directory.Delete() and Directory.CreateDirectory() to overwrite a folder

In my WebApi action method, I want to create/over-write a folder using this code:
string myDir = "...";
if(Directory.Exists(myDir))
{
Directory.Delete(myDir, true);
}
Directory.CreateDirectory(myDir);
// 1 - Check the dir
Debug.WriteLine("Double check if the Dir is created: " + Directory.Exists(myDir));
// Some other stuff here...
// 2 - Check the dir again
Debug.WriteLine("Check again if the Dir still exists: " + Directory.Exists(myDir));
Issue
Strangely, sometimes right after creating the directory, the directory does not exist!
Sometimes when checking the dir for the first time (where the number 1 is); Directory.Exist() returns true, other times false. Same happens when checking the dir for the second time (where the number 2 is).
Notes
None of this part of code throw any exception.
Only can reproduce this when publishing the website on server. (Windows server 2008)
Happens when accessing the same folder.
Questions
Is this a concurrency issue race condition?
Doesn't WebApi or the Operating System handle the concurrency?
Is this the correct way to overwrite a folder?
Should I lock files manually when we have many API requests to the same file?
Or in General:
What's the reason for this strange behavior?
UPDATE:
Using DirectoryInfo and Refresh() instead of Directory does not solve the problem.
Only happens when the recursive option of Delete is true. (and the directory is not empty).

Many filesystem operations are not synchonous on some filesystems (in case of windows - NTFS). Take for example RemoveDirectory call (which is called by Directory.DeleteDirectory at some point):
The RemoveDirectory function marks a directory for deletion on close. Therefore, the directory is not removed until the last handle to the directory is closed.
As you see, it will not really delete directory until all handles to it are closed, but Directory.DeleteDirectory will complete fine. In your case that is also most likely such concurrency problem - directory is not really created while you executing Directory.Exists.
So, just periodically check what you need and don't consider filesystem calls in .NET to be synchronous. You can also use FileSystemWatcher in some cases to avoid polling.
EDIT: I was thinking how to reproduce it, and here is the code:
internal class Program {
private static void Main(string[] args) {
const string path = "G:\\test_dir";
while (true) {
if (Directory.Exists(path))
Directory.Delete(path);
Directory.CreateDirectory(path);
if (!Directory.Exists(path))
throw new Exception("Confirmed");
}
}
}
You see that if all filesystem calls were synchronous (in .NET), this code should run without problem. Now, before running that code, create empty directory at specified path (preferrably don't use SSD for that) and open it with windows explorer. Now run the code. For me it either throws Confirmed (which exactly reproduces your issue) or throws on Directory.Delete saying that directory does not exist (almost the same case). It does it 100% of the time for me.
Here is another code which when running on my machine confirms that it's certainly possible for File.Exists to return true directly after File.Delete call:
internal class Program {
private static void Main(string[] args) {
while (true) {
const string path = #"G:\test_dir\test.txt";
if (File.Exists(path))
File.Delete(path);
if (File.Exists(path))
throw new Exception("Confirmed");
File.Create(path).Dispose();
}
}
}
To do this, I opened G:\test_dir folder and during execution of this code tried to open constantly appearing and disappearing test.txt file. After couple of tries, Confirmed exception was thrown (while I didn't create or delete that file, and after exception is thrown, it's not present on filesystem already). So race conditions are possible in multiple cases and my answer is correct one.

I wrote myself a little C# method for synchronous folder deletion using Directory.Delete(). Feel free to copy:
private bool DeleteDirectorySync(string directory, int timeoutInMilliseconds = 5000)
{
if (!Directory.Exists(directory))
{
return true;
}
var watcher = new FileSystemWatcher
{
Path = Path.Combine(directory, ".."),
NotifyFilter = NotifyFilters.DirectoryName,
Filter = directory,
};
var task = Task.Run(() => watcher.WaitForChanged(WatcherChangeTypes.Deleted, timeoutInMilliseconds));
// we must not start deleting before the watcher is running
while (task.Status != TaskStatus.Running)
{
Thread.Sleep(100);
}
try
{
Directory.Delete(directory, true);
}
catch
{
return false;
}
return !task.Result.TimedOut;
}
Note that getting task.Result will block the thread until the task is finished, keeping the CPU load of this thread idle. So that is the point where it gets synchronous.

Sounds like race condition to me. Not sure why - you did not provide enough details - but what you can do is to wrap everything in lock() statement and see if the problem is gone. For sure this is not production-ready solution, it is only a quick way to check. If it's indeed a race condition - you need to rethink your approach of rewriting folders. May be create "GUID" folder and when done - update DB with the most recent GUID to point to the most recent folder?..

How to check that a file can be read successfully? [duplicate]

This question already has answers here:
Detecting whether a file is locked by another process (or indeed the same process) [duplicate]
(4 answers)
Closed 9 years ago.
In my program I pass the path to a file which may be still being written to by another process (independent external non-c#).
How can I check the file is fully readable and accessible before I actually send the path back to the client? FileIOPermission with Demand for reading did not work for me, so I am wondering if there is some other way of doing this without attempting to read the whole file upfront.
Thank you.

The trouble with checking to see if a file is accessible and fully readable, and then opening the file for reading is that potentially, the state could change between the check and the open where you actually want to be doing something with it.
My suggestion is to go ahead and open the file for reading, being sure to catch the appropriate exception if a problem occurs. You can then wait a little and try again, or do something else.
Remember that just because your program has a sequence of events that are logical to you, many things are going on in an operating system and the state can easily change between two lines of code that seem watertight for you, but have an epoch between them at a multitasking level.

Moo-Juice is right. You can't check whether the file can be read successfully and then open it for reading... things can change and you might get an exception anyway. Best to just read it and catch the exception.
public bool TryReadFile(String path, out String contentsOfFile)
{
try
{
// Try reading file
contentsOfFile = File.ReadAllText(path);
// Success! Yay!
return true;
}
catch (IOException)
{
// Oops! Can't read that file!
// Return some default value and let the caller know we failed
contentsOfFile = String.Empty;
return false;
}

Best way would be to read the file in a normal try catch block, and then apply the logic to continue or not, based on if an exception was thrown or not.
Also, a secondary way is to check if a file is not of zero size, but purely as a secondary check.

Create a loop and try to open the file
If exception occur make your thread sleep for some seconds
and repeat the process

When the external process is complete, does it close? If so, you could modify the solution found here. It would look something like the following:
//While the process is found running
while (System.Diagnostics.Process.GetProcessesByName(process).Length != 0)
{
//Sleep for three seconds
System.Threading.Thread.Sleep(3000);
}
//open file for reading
If the process doesn't close when complete, however, the above won't work. You could try placing the following in a loop that tries to open the file exclusively until it's successful:
System.IO.File.Open(PathToFile, FileMode.Open, FileAccess.Read, FileShare.None);
Both of these methods should also have some kind of count added to ensure that they eventually stop trying and error out, or else you could end up with an infinite loop.

Check If File Is In Use By Other Instances of Executable Run

Before I go into too detail, my program is written in Visual Studio 2010 using C# .Net 4.0.
I wrote a program that will generate separate log files for each run. The log file is named after the time, and accurate up at millisecond (for example, 20130726103042375.log). The program will also generate a master log file for the day if it has not already exist (for example, *20130726_Master.log*)
At the end of each run, I want to append the log file to a master log file. Is there a way to check if I can append successfully? And retry after Sleep for like a second or something?
Basically, I have 1 executable, and multiple users (let's say there are 5 users).
All 5 users will access and run this executable at the same time. Since it's nearly impossible for all user to start at the exact same time (up to millisecond), there will be no problem generate individual log files.
However, the issue comes in when I attempt to merge those log files to the master log file. Though it is unlikely, I think the program will crash if multiple users are appending to the same master log file.
The method I use is
File.AppendAllText(masterLogFile, File.ReadAllText(individualLogFile));
I have check into the lock object, but I think it doesn't work in my case, as there are multiple instances running instead of multiple threads in one instance.
Another way I look into is try/catch, something like this
try
{
stream = file.Open(FileMode.Open, FileAccess.ReadWrite, FileShare.None);
}
catch {}
But I don't think this solve the problem, because the status of the masterLogFile can change in that brief millisecond.
So my overall question is: Is there a way to append to masterLogFile if it's not in use, and retry after a short timeout if it is? Or if there is an alternative way to create the masterLogFile?
Thank you in advance, and sorry for the long message. I want to make sure I get my message across and explain what I've tried or look into so we are not wasting anyone's time.
Please let me know if there's anymore information I can provide to help you help me.

Your try/catch is the way to do things. If the call to File.Open succeeds, then you can write to to the file. The idea is to keep the file open. I would suggest something like:
bool openSuccessful = false;
while (!openSuccessful)
{
try
{
using (var writer = new StreamWriter(masterlog, true)) // append
{
// successfully opened file
openSuccessful = true;
try
{
foreach (var line in File.ReadLines(individualLogFile))
{
writer.WriteLine(line);
}
}
catch (exceptions that occur while writing)
{
// something unexpected happened.
// handle the error and exit the loop.
break;
}
}
}
catch (exceptions that occur when trying to open the file)
{
// couldn't open the file.
// If the exception is because it's opened in another process,
// then delay and retry.
// Otherwise exit.
Sleep(1000);
}
}
if (!openSuccessful)
{
// notify of error
}
So if you fail to open the file, you sleep and try again.
See my blog post, File.Exists is only a snapshot, for a little more detail.

I would do something along the lines of this as I think in incurs the least overhead. Try/catch is going to generate a stack trace(which could take a whole second) if an exception is thrown. There has to be a better way to do this atomically still. If I find one I'll post it.

C# move file as soon as it becomes available

I need to accomplish the following task:
Attempt to move a file. If file is locked schedule for moving as soon as it becomes available.
I am using File.Move which is sufficient for my program. Now the problems are that:
1) I can't find a good way to check if the file I need to move is locked. I am catching System.IO.IOException but reading other posts around I discovered that the same exception may be thrown for different reasons as well.
2) Determining when the file gets unlocked. One way of doing this is probably using a timer/thread and checking the scheduled files lets say every 30 seconds and attempting to move them. But I hope there is a better way using FileSystemWatcher.
This is a .net 3.5 winforms application. Any comments/suggestions are appreciated. Thanks for attention.

You should really just try and catch an IOException. Use Marshal.GetHRForException to check for the cause of the exception.
A notification would not be reliable. Another process might lock the file again before File.Move is executed.

One possible alternative is by using MoveFileEx with a MOVEFILE_DELAY_UNTIL_REBOOT flag. If you don't have access to move the file right now, you can schedule it to be moved on the next reboot when it's guaranteed to be accessible (the moving happens very early in the boot sequence).
Depending on your specific application, you could inform the user a reboot is necessary and initiate the reboot yourself in addition to the moving scheduling.

It's simple:
static void Main(string[] args)
{
//* Create Watcher object.
FileSystemWatcher watcher = new FileSystemWatcher(#"C:\MyFolder\");
//* Assign event handler.
watcher.Created += new FileSystemEventHandler(watcher_Created);
//* Start watching.
watcher.EnableRaisingEvents = true;
Console.ReadLine();
}
static void watcher_Created(object sender, FileSystemEventArgs e)
{
try
{
File.Move(e.FullPath, #"C:\MyMovedFolder\" + e.Name);
}
catch (Exception)
{
//* Something went wrong. You can do additional proceesing here, like fire-up new thread for retry move procedure.
}
}

This is not specific to your problem, but generally you will always need to retain the 'try it and gracefully deal with a failure' mode of operation for this sort of action.
That's because however clever your 'detect that the file is available' mechanism is, there will always be some amount of time between you detecting that the file is available and moving it, and in that time someone else might mess with the file.

The scheduled retry on exception (probably increasing delays - up to a point) is probably the simplest way to achieve this (your (2) ).
To do it properly you're going to have to drop to system level (with Kernel code) hooks to trap the file close event - which has its own idiosynchrases. It's a big job - several orders of magnitude more complex than the scheduled retry method. It's up to you and your application case to make that call, but I don't know of anything effective in between.

Very old question, but google led me here, so when I found a better answer I decided to post it:
There's a nice code I found in the dotnet CLI repo:
/// <summary>
/// Run Directory.Move and File.Move in Windows has a chance to get IOException with
/// HResult 0x80070005 due to Indexer. But this error is transient.
/// </summary>
internal static void RetryOnMoveAccessFailure(Action action)
{
const int ERROR_HRESULT_ACCESS_DENIED = unchecked((int)0x80070005);
int nextWaitTime = 10;
int remainRetry = 10;
while (true)
{
try
{
action();
break;
}
catch (IOException e) when (e.HResult == ERROR_HRESULT_ACCESS_DENIED)
{
Thread.Sleep(nextWaitTime);
nextWaitTime *= 2;
remainRetry--;
if (remainRetry == 0)
{
throw;
}
}
}
}
There is also a method for just IOException. Here's the usage example:
FileAccessRetrier.RetryOnMoveAccessFailure(() => Directory.Move(packageDirectory.Value, tempPath));
Overall, this repo contains a lot of interesting ideas for file manipulations and installation/removal logic, like TransactionalAction, so I recommend it for reviewing. Unfortunately, these functions are not available as NuGet package.

Have a look at the FileSystemWatcher.
http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher(VS.90).aspx
Listens to the file system change
notifications and raises events when a
directory, or file in a directory,
changes