We Keep Coding

What's the best practice to recover from a FileSystemWatcher error?

After a FileSystemWatcher.Error event was raised, I have no clue about what to do next.
The exception can be a [relatively] minor one, such as
too many changes at once in directory
which doesn't affect the watcher's watching process, but it can also be a big issue - such as the watched directory being deleted, in which case the watcher is no longer functional.
My question is what is the best way to handle the Error event?

Depends on the error surely?
If it is too much data because the buffer was overrun (many changes) do a list directory and grab the changes you're after.
If it is too much data because you're not processing the FileSystemWatcher events quickly enough, ensure you're processing it efficiently.
Deleted directory, can't do anything about it other than disposing the FileSystemWatcher, or maybe watching the parent for a recreation of that directory name again.

I would simply get the inner exception type, then decide on a per-error basis what to do ( restart or fail ).
So
myWatcher.Error += new ErrorEventHandler(OnError);
Followde by
private static void OnError(object source, ErrorEventArgs e)
{
if (e.GetException().GetType() == typeof(InternalBufferOverflowException))
{
// This can happen if Windows is reporting many file system events quickly
// and internal buffer of the FileSystemWatcher is not large enough to handle this
// rate of events. The InternalBufferOverflowException error informs the application
// that some of the file system events are being lost.
Console.WriteLine(("The file system watcher experienced an internal buffer overflow: " + e.GetException().Message));
}
}

FileSystemWatcher with Samba on Linux

I'm using a FileSystemWatcher on my C# application (running on Windows) in order to update in my app the files that I'm currently browsing.
It's work well when I browse a local directory. I am notified when a file is renamed, deleted or added.
But for example when I rename a file on the network drive the first time, the FileSystemWatcher notifies me of a rename action and then, when I rename the same file or another file, the FileSystemWatcher notifies me of an error :
the specified server cannot perform the requested operation.
Then the FileSystemWatcher not notify me about anything.
Sometimes I can rename twice before the FileSystemWatcher not notify me nothing...
Here is my test code :
static void Main(string[] args)
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = #"N:\prive\defFolder";
watcher.NotifyFilter = NotifyFilters.FileName | NotifyFilters.LastWrite;
watcher.Changed += new FileSystemEventHandler(watcher_Changed);
watcher.Created += new FileSystemEventHandler(watcher_Changed);
watcher.Deleted += new FileSystemEventHandler(watcher_Changed);
watcher.Renamed += new RenamedEventHandler(watcher_Renamed);
watcher.Error += new ErrorEventHandler(watcher_Error);
watcher.EnableRaisingEvents = true;
Console.Read();
watcher.Dispose();
}
static void watcher_Error(object sender, ErrorEventArgs e)
{
Console.WriteLine("error : " + e.GetException().Message);
}
static void watcher_Renamed(object sender, RenamedEventArgs e)
{
Console.WriteLine("rename success");
}
static void watcher_Changed(object sender, FileSystemEventArgs e)
{
Console.WriteLine("change success");
}

First of all, file system monitoring of remote shares is always going to be somewhat unreliable. You should not depend on your app getting all the events - in fact, I would suggest you provide a backup polling mechanism to check for changes that you may have missed. A refresh button in the GUI could be another option, depending on your app.
That said, your particular problem doesn't seem to be that uncommon. I googled around a bit and found these things:
Problem with FileSystemWatcher on 2003 server box - seems to be the same problem as you have, with a samba share and that "The specified server cannot perform the requested operation" error message.
SO Question regarding the same issue and one of the answers indicate that it worked for some customers and not for others
My guess is that this is a problem with certain versions (or configuration) of Samba combined with Windows. Are there anything in the Samba logs on the linux server that could point you towards the cause of the problem?
As an immediate workaround, I suggest you try these things:
Add a polling mechanism that ensures that you do get the folder changes even if the FSW breaks down
When the FSW breaks, try to "restart" it by creating a new one. You may also want to check if EnableRaisingEvents is set to false when you get the error - maybe you can just set it to true to start receiving events again.
(Grasping for straws here) try playing around with the internal buffer size in case that's the problem (I doubt it, but it's worth a shot)

if (Directory.Exists(monitorPath))
{
watcher.Path = monitorPath;
watcher.IncludeSubdirectories = true;
watcher.NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.CreationTime;
watcher.InternalBufferSize = 65536;
watcher.Filter = "test_prod-Pg1_ICT*";
watcher.Changed += new FileSystemEventHandler(fileChangedEvent);
watcher.EnableRaisingEvents = true;
LogEvent("Folder Syncronization Service is Started!");
}
I created a Window Service FileSystemWatcher based class to Monitor a Samba Shared Folder onChanges and used DifferenceEngine from CodeProject to check the different and copy over to Window Shared Folder Path if there is a changes. I also added a Timer to check every 10 seconds to handle when network is fail. There is a List Array To store the changes count. Added into List when File Changed event is Raised and remove the List when is success.
I have tested two HP Laptop on Window 7 Pro OS, working fine.
But Failed to working on other Window 7 Pro Laptop and also Window XP Pro SP3 Desktop. (We are on same Company Network/VLAN and Service Pack)
The failed is mean if i amended something in Samba Shared Folder, it will not Sync the Latest content to Window Share Path.
I also has added these
[PermissionSetAttribute(SecurityAction.LinkDemand, Name = "FullTrust")]
[PermissionSetAttribute(SecurityAction.InheritanceDemand, Name = "FullTrust")]
in the top of the coding, it seem does not work

For people recently having this kind of problem, quoting from my answer to this question:
We are yet to test that this is a suitable alternative in our
situation, but it seems that Microsoft introduced
PhysicalFileProvider some time ago.
It has a UsePollingFileWatcher property specifically for this
kind of problem.
Then for the file watching, it seems you want the Watch method.
The documentation is currently showing it as not available on certain
platforms or target frameworks, but in practice the NuGet package
claims to be .NET Standard 2.0 compatible, so that should cover
basically everything.

FileSystemWatcher changed event (for "LastWrite") is unreliable

I am trying to get a notification when a file is updated on disk. I am interested in getting this notifications as soon as a flush occurs, yet, it seems that the FileSystemWatcher would only send an event when the stream opens or closes.
In the code below I am writing to a file and repeatedly flush the buffer to the disk. Yet, the FileSystemWatcher only notified me once in the beginning of the writes and once when they end.
Is there another way to get these notifications? Or should I use polling for that?
The code:
class Program
{
static void Main(string[] args)
{
FileSystemWatcher watcher = new FileSystemWatcher(Environment.CurrentDirectory, "Test.txt");
watcher.Changed += watcher_Changed;
watcher.EnableRaisingEvents = true;
using(TextWriter writer = new StreamWriter("Test.txt"))
{
WriteData(writer);
WriteData(writer);
WriteData(writer);
WriteData(writer);
}
Thread.Sleep(10000);
}
private static void WriteData(TextWriter writer)
{
writer.WriteLine("Hello!");
writer.Flush();
Console.WriteLine(DateTime.Now.ToString("T") + "] Wrote data");
Thread.Sleep(3000);
}
static void watcher_Changed(object sender, FileSystemEventArgs e)
{
Console.WriteLine(DateTime.Now.ToString("T") + "] Watcher changed!");
}
}
UPDATE 1
I've corrected the ToString function of the DateTime to show seconds. Here is the output with the code above:
11:37:47 AM] Watcher changed!
11:37:47 AM] Wrote data
11:37:50 AM] Wrote data
11:37:53 AM] Wrote data
11:37:56 AM] Wrote data
11:37:59 AM] Watcher changed!
Thanks!

It has nothing to do with the FileSystemWatcher. The watcher reacts to updates on the LastWrite attribute for the filesystem.
E.g. NTFS does not update the LastWrite on every write. The value is cached and only written when the stream is closed or at some other unspecified time. This document says
Timestamps are updated at various times and for various reasons. The only guarantee about a file timestamp is that the file time is correctly reflected when the handle that makes the change is closed. [...] The NTFS file system delays updates to the last access time for a file by up to 1 hour after the last access
I assume a similar caching applies for write

I think one of your issues here is that all your writes get executed before the event gets a slice of that cpu-time.
MSDN states
The Changed event is raised when changes are made to the size, system attributes, last write time, last access time, or security permissions of a file or directory in the directory being monitored.
I did a test and inserted Sleeps after each call to WriteData(...), what i got was
09:32] Watcher changed!
09:32] Watcher changed!
09:32] -----> Wrote data
09:32] -----> Wrote data
09:32] -----> Wrote data
09:32] -----> Wrote data
09:32] Watcher changed!
09:32] Watcher changed!
I guess this kind of proves that the even is fired right after you call Flush(), it's just a question of when the event-handler gets executed (and I assume it groups events as well).
I don't know the specific needs of your project, but I wouldn't poll. Seems like a waste since FileSystemWatcher does what you want it to do in my opinion.
Edit: Ok, I guess my brain wasn't quite ready yet for thinking when I posted this.
Your conclusion that it fires when you open and close the stream seems more logical and right.
I guess I was looking for "prove" that it fires when you call flush and therefore found it - somehow.
Update
I just had a poke at the USN-Journal and it seems that wont get you what you want either as it writes the record only when the file is closed. -> http://msdn.microsoft.com/en-us/library/aa363803(VS.85).aspx
I also found a USN-Viewer in C# and http://social.msdn.microsoft.com/Forums/en/csharpgeneral/thread/c1550294-d121-4511-ac32-31551497f64e might be interesting to read as well.
I also ran DiskMon to see if it gets the changes in realtime. It doesn't, but I don't know if that's intentional or not. However, the problem with both is that they require admin-rights to run.
So I guess you are stuck with FileSystemWatcher. (What is it you need the updates for anyway, it's not like you can read the file while it's open/locked by another program.)
ps: I just noticed you are a dev of BugAid, which I was made aware of it only recently - it looks awesome :)

I know this is an old thread but the problem still exists 10 years later. I solved this by using a timer:
private Timer _checkTimer;
private DateTime _lastSaved;
Before initializing the timer, I get the last write time of the file:
_lastSaved = File.GetLastWriteTime(_watchedFile);
_checkTimer = new Timer() { Interval = 1000, AutoReset = true };
_checkTimer.Elapsed += CheckTimer_Elapsed;
_checkTimer.Start();
Then on each elapsed event I check if the last write time is newer than the one that I knew to be the last write time:
private void CheckTimer_Elapsed(object sender, ElapsedEventArgs e)
{
DateTime lastWriteTime = File.GetLastWriteTime(_watchedFile);
if (lastWriteTime > _lastSaved)
{
// Stop the timer to avoid concurrency problems
_syncTimer.Stop();
// Do here your processing
// Now, this is the new last known write time
_lastSaved = File.GetLastWriteTime(_watchedFile);
// Start again the timer
_syncTimer.Start();
}
}
This is how I personally solved this problem that I also had with FileSystemWatcher. This method is not as expensive as reading the file content or computing the file hash every second to check if there is a difference.

FileSystemWatcher - only the change event once firing once?

I'm using the below code to listen for change events of a file i download off a server and open. however the change event only gets fired the first time the file is saved, then on subsequent saves the file watcher does not fire the change events?
Can anyone see whats going on?
private FileSystemWatcher StartWatchingFile()
{
fw = new FileSystemWatcher();
fw.Path = this.directoryLocation;
fw.Filter = this.Filename;
fw.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite;
// Add event handler
fw.Changed += new FileSystemEventHandler(fw_Changed);
// Open file
System.Diagnostics.Process.Start(this.CreateAbsoluteFilePath(this.Filename));
// Begin watching.
fw.EnableRaisingEvents = true;
return fw;
}
//************************
void fw_Changed(object sender, FileSystemEventArgs e)
{
MessageBox.Show("File: " + e.FullPath + " " + e.ChangeType);
}
EDIT: The StartWatchingFile() now returns the filewatcher which is kept in a class that will not be garbage collected, just to make sure i'm holding the whole class as thought the fw_changed() function might not be able to be called. So the whole class is now not getting garbage collectioned. The class is held in a ArrayList which is a public member of a class
Regards,
Jon

I'm sorry I can't answer your question specifically. In general, I'll contribute that, if you use it enough, you'll learn that FileSystemWatcher is not reliable. Microsoft Connect shows multiple problems with it. I agree with Jason Jackson's take on FileSystemWatcher.

Is it reproduceable that it always works for the first time?
If not, the FileSystemWatcher might have been collected by the garbage collector in the meantime after StartWatchingFile has finished because it's declared locally.
If so, does the process you're starting probably lock the file so the file actually isn't modified?

FileSystemWatcher vs polling to watch for file changes

I need to setup an application that watches for files being created in a directory, both locally or on a network drive.
Would the FileSystemWatcher or polling on a timer would be the best option. I have used both methods in the past, but not extensively.
What issues (performance, reliability etc.) are there with either method?

I have seen the file system watcher fail in production and test environments. I now consider it a convenience, but I do not consider it reliable. My pattern has been to watch for changes with the files system watcher, but poll occasionally to catch missing file changes.
Edit: If you have a UI, you can also give your user the ability to "refresh" for changes instead of polling. I would combine this with a file system watcher.

The biggest problem I have had is missing files when the buffer gets full. Easy as pie to fix--just increase the buffer. Remember that it contains the file names and events, so increase it to the expected amount of files (trial and error). It does use memory that cannot be paged out, so it could force other processes to page if memory gets low.
Here is the MSDN article on buffer :
FileSystemWatcher..::.InternalBufferSize Property
Per MSDN:
Increasing buffer size is expensive, as it comes from non paged memory that cannot be swapped out to disk, so keep the buffer as small as possible. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties to filter out unwanted change notifications.
We use 16MB due to a large batch expected at one time. Works fine and never misses a file.
We also read all the files before beginning to process even one...get the file names safely cached away (in our case, into a database table) then process them.
For file locking issues I spawn a process which waits around for the file to be unlocked waiting one second, then two, then four, et cetera. We never poll. This has been in production without error for about two years.

The FileSystemWatcher may also miss changes during busy times, if the number of queued changes overflows the buffer provided. This is not a limitation of the .NET class per se, but of the underlying Win32 infrastructure. In our experience, the best way to minimize this problem is to dequeue the notifications as quickly as possible and deal with them on another thread.
As mentioned by #ChillTemp above, the watcher may not work on non-Windows shares. For example, it will not work at all on mounted Novell drives.
I agree that a good compromise is to do an occasional poll to pick up any missed changes.

Also note that file system watcher is not reliable on file shares. Particularly if the file share is hosted on a non-windows server. FSW should not be used for anything critical. Or should be used with an occasional poll to verify that it hasn't missed anything.

Personally, I've used the FileSystemWatcher on a production system, and it has worked fine. In the past 6 months, it hasn't had a single hiccup running 24x7. It is monitoring a single local folder (which is shared). We have a relatively small number of file operations that it has to handle (10 events fired per day). It's not something I've ever had to worry about. I'd use it again if I had to remake the decision.

I currently use the FileSystemWatcher on an XML file being updated on average every 100 milliseconds.
I have found that as long as the FileSystemWatcher is properly configured you should never have problems with local files.
I have no experience on remote file watching and non-Windows shares.
I would consider polling the file to be redundant and not worth the overhead unless you inherently distrust the FileSystemWatcher or have directly experienced the limitations everyone else here has listed (non-Windows shares, and remote file watching).

I have run into trouble using FileSystemWatcher on network shares. If you're in a pure Windows environment, it might not be an issue, but I was watching an NFS share and since NFS is stateless, there was never a notification when the file I was watching changed.

I'd go with polling.
Network issues cause the FileSystemWatcher to be unreliable (even when overloading the error event).

Returning from the event method as quickly as possible, using another thread, solved the problem for me:
private void Watcher_Created(object sender, FileSystemEventArgs e)
{
Task.Run(() => MySubmit(e.FullPath));
}

I had some big problems with FSW on network drives: Deleting a file always threw the error event, never the deleted event. I did not find a solution, so I now avoid the FSW and use polling.
Creation events on the other hand worked fine, so if you only need to watch for file creation, you can go for the FSW.
Also, I had no problems at all on local folders, no matter if shared or not.

Using both FSW and polling is a waste of time and resources, in my opinion, and I am surprised that experienced developers suggest it. If you need to use polling to check for any "FSW misses", then you can, naturally, discard FSW altogether and use only polling.
I am, currently, trying to decide whether I will use FSW or polling for a project I develop. Reading the answers, it is obvious that there are cases where FSW covers the needs perfectly, while other times, you need polling. Unfortunately, no answer has actually dealt with the performance difference(if there is any), only with the "reliability" issues. Is there anyone that can answer that part of the question?
EDIT : nmclean's point for the validity of using both FSW and polling(you can read the discussion in the comments, if you are interested) appears to be a very rational explanation why there can be situations that using both an FSW and polling is efficient. Thank you for shedding light on that for me(and anyone else having the same opinion), nmclean.

Working solution for working with create event instead of change
Even for copy, cut, paste, move.
class Program
{
static void Main(string[] args)
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
FileSystemWatcher FileSystemWatcher = new FileSystemWatcher();
FileSystemWatcher.Path = SourceFolderPath;
FileSystemWatcher.IncludeSubdirectories = false;
FileSystemWatcher.NotifyFilter = NotifyFilters.FileName; // ON FILE NAME FILTER
FileSystemWatcher.Filter = "*.txt";
FileSystemWatcher.Created +=FileSystemWatcher_Created; // TRIGGERED ONLY FOR FILE GOT CREATED BY COPY, CUT PASTE, MOVE
FileSystemWatcher.EnableRaisingEvents = true;
Console.Read();
}
static void FileSystemWatcher_Created(object sender, FileSystemEventArgs e)
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
try
{
// DO SOMETING LIKE MOVE, COPY, ETC
File.Copy(e.FullPath, DestinationFolderPath + #"\" + e.Name);
}
catch
{
}
}
}
Solution for this file watcher while file attribute change event using static storage
class Program
{
static string IsSameFile = string.Empty; // USE STATIC FOR TRACKING
static void Main(string[] args)
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
FileSystemWatcher FileSystemWatcher = new FileSystemWatcher();
FileSystemWatcher.Path = SourceFolderPath;
FileSystemWatcher.IncludeSubdirectories = false;
FileSystemWatcher.NotifyFilter = NotifyFilters.LastWrite;
FileSystemWatcher.Filter = "*.txt";
FileSystemWatcher.Changed += FileSystemWatcher_Changed;
FileSystemWatcher.EnableRaisingEvents = true;
Console.Read();
}
static void FileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
if (e.Name == IsSameFile) //SKIPS ON MULTIPLE TRIGGERS
{
return;
}
else
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
try
{
// DO SOMETING LIKE MOVE, COPY, ETC
File.Copy(e.FullPath, DestinationFolderPath + #"\" + e.Name);
}
catch
{
}
}
IsSameFile = e.Name;
}
}
This is a workaround solution for this problem of multiple triggering event.

I would say use polling, especially in a TDD scenario, as it is much easier to mock/stub the presence of files or otherwise when the polling event is triggered than to rely on the more "uncontrolled" fsw event. + to that having worked on a number of apps which were plagued by fsw errors.