I want to ask if there is a possibility to ignore changes within the
file system changed event made by the own process the FileSystemWatcher
is running in.
For example my own process creates a file in a watched directory and the
FileSystemWatcher should recognize that it was created by the process or
do not raise the event from the outset.
You can use a list of files the application operates on:
static private readonly List<string> WatcherFilesToIgnore = new List<string>();
try
{
WatcherFilesToIgnore.Add(filePath1);
WatcherFilesToIgnore.Add(filePath2);
// some file operation
}
finally
{
WatcherFilesToIgnore.Remove(filePath1);
WatcherFilesToIgnore.Remove(filePath2);
}
Hence in the watcher event handler you write at the beginning:
if ( WatcherFilesToIgnore.Contains(e.FullPath) ) return;
// watcher process
Thus when your application acts on some files, the watcher will do nothing for the specified to ignore.
If you use multithreading, consider using some synchronization.
As mentionned by #CodeCaster there may be a lag between the application and the windows shell event raised.
A simple thing could be to add a Sleep(2000) for example but it can be hazardous.
Thus we can use a Dictionary<string, bool> to indicate that the process is terminated and let the watcher clean itself the list:
WatcherFilesToIgnore.Add(filePath1, false);
// file operation
WatcherFilesToIgnore[filePath1] = true;
if ( WatcherFilesToIgnore.Contains(e.FullPath) )
{
if ( WatcherFilesToIgnore[e.FullPath] )
WatcherFilesToIgnore.Remove(e.FullPath);
return;
}
// watcher process
This only work if a real watcher event is raised else the file will remain in the list.
So we need to ensure that in case of nothing is done we remove the file:
WatcherFilesToIgnore.Add(filePath1, false);
if ( ProcessFile(filePath1) )
WatcherFilesToIgnore[filePath1] = true;
else
WatcherFilesToIgnore.Remove(e.FullPath);
The ProcessFile method (I factored like that for the explanation) do the operation needed and returns true if some change successed, else false (nothing done, exception raised, process aborted...).
After a FileSystemWatcher.Error event was raised, I have no clue about what to do next.
The exception can be a [relatively] minor one, such as
too many changes at once in directory
which doesn't affect the watcher's watching process, but it can also be a big issue - such as the watched directory being deleted, in which case the watcher is no longer functional.
My question is what is the best way to handle the Error event?
Depends on the error surely?
If it is too much data because the buffer was overrun (many changes) do a list directory and grab the changes you're after.
If it is too much data because you're not processing the FileSystemWatcher events quickly enough, ensure you're processing it efficiently.
Deleted directory, can't do anything about it other than disposing the FileSystemWatcher, or maybe watching the parent for a recreation of that directory name again.
I would simply get the inner exception type, then decide on a per-error basis what to do ( restart or fail ).
So
myWatcher.Error += new ErrorEventHandler(OnError);
Followde by
private static void OnError(object source, ErrorEventArgs e)
{
if (e.GetException().GetType() == typeof(InternalBufferOverflowException))
{
// This can happen if Windows is reporting many file system events quickly
// and internal buffer of the FileSystemWatcher is not large enough to handle this
// rate of events. The InternalBufferOverflowException error informs the application
// that some of the file system events are being lost.
Console.WriteLine(("The file system watcher experienced an internal buffer overflow: " + e.GetException().Message));
}
}
I have a FileSystemWatcher monitoring a file on a network share. If an event occurs to make the share unavailable, maybe due to a network problem, the FileSystemWatcher becomes disconnected.
Obviously I can handle the "Error" event, maybe do some logging and lots of articles suggest reconnecting the FSW inside the error event handler.
But what if the network share is still unavailable inside the error event. I then need to introduce a timer to test if the network share is available and attempt to reconnect the FSW.
1) Is there a better approach?
2) Is there a property that allows me to determine that the FSW has become disconnected from the file? I notice there is a non-public member of the FSW "stopListening", which appears to be set to true when the FSW becomes disconnected. But this is not publicly exposed
Any help would be appreciated ...
Thanks
Kevin
A couple of comments and suggestions...(which grew and grew as I was typing...sorry)
The FileSystemWatcher.Error event is fired when the FileSystemWatcher gets so many events happening so quickly that it can't handle them all. It doesn't get fired when an error occurs in watching the file system (such as the network dropping out).
I believe I've had a similar situation. The problem is that when the network connection drops out, the FileSystemWatcher will never have an event triggered, because it actually can't see what it's supposed to be watching, but it doesn't seem to be aware of the fact. When the network connection comes back, the FileSystemWatcher doesn't recover - i.e. it still can't see the (restored) connection. The only solution that we came up with that seemed to work reliably was to have a timer that regularly dropped the whole FileSystemWatcher object and created a new one, setting all of the events and watch folder etc. Since dropping and creating a new FileSystemWatcher is (relatively) quick (i.e. milliseconds) you could set the timer to activate every 10 seconds or so without using up too much of the processor. Of course, if the network is still out, the FileSystemWatcher is not going to be able to see the network no matter what you do. But that's OK, it will try again in another 10 seconds.
Two things to watch out for with this solution:
When the timer activates, it needs to check that the FileSystemWatcher isn't currently processing any events, and it needs to wait if it is. So in the timer event, stop the Timer, stop the FileSystemWatcher from raising events, then wait for any FileSystemWatcher events to finish (using lock (...) {...} is a good way of doing this).
After dropping and recreating the FileSystemWatcher, you need to manually check for any events that might have occurred while the FileSystemWatcher was being refreshed (or while the network was down). For example, if you're watching for files being created, and a file gets created while refreshing the FileSystemWatcher or while the network connection is out, you won't get events for those files when you start up the new instance of the FileSystemWatcher (since the files have already been created).
I hope that helps.
Follow-up in this. At the suggestion of a Microsoft resource on the MSDN forums, I added this to Microsoft Connect.
Key points from the feedback from Microsoft:
- Error event is not only for internal buffer overflows
- They will add the possibility of exposing the stopListening property to their list of customer suggestions
Link here: http://connect.microsoft.com/VisualStudio/feedback/details/727934/filesystemwatcher-error-handling
Wouldn't something like this work? Seems to work for my simple test case.
var fsw = new FileSystemWatcher("[folder]", "*.*") { IncludeSubdirectories = true};
var fsw_processing = false;
fsw.Deleted += (s, e) =>
{
fsw_processing = true;
fsw.EnableRaisingEvents = false;
//......
fsw.EnableRaisingEvents = true;
fsw_processing = false;
};
fsw.Changed += (s, e) =>
{
fsw_processing = true;
fsw.EnableRaisingEvents = false;
//......
fsw.EnableRaisingEvents = true;
fsw_processing = false;
};
//governor thread to check FileSystemWatcher is still connected.
//It seems to disconnects on network outages etc.
Task.Run(() =>
{
while (true)
{
if (fsw.EnableRaisingEvents == false && fsw_processing == false)
{
try
{fsw.EnableRaisingEvents = true;}
catch (Exception) { fsw.EnableRaisingEvents = false; }
}
System.Threading.Thread.Sleep(1000 * 10);//sleep 10 secs
}
});
I'm using the below code to listen for change events of a file i download off a server and open. however the change event only gets fired the first time the file is saved, then on subsequent saves the file watcher does not fire the change events?
Can anyone see whats going on?
private FileSystemWatcher StartWatchingFile()
{
fw = new FileSystemWatcher();
fw.Path = this.directoryLocation;
fw.Filter = this.Filename;
fw.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite;
// Add event handler
fw.Changed += new FileSystemEventHandler(fw_Changed);
// Open file
System.Diagnostics.Process.Start(this.CreateAbsoluteFilePath(this.Filename));
// Begin watching.
fw.EnableRaisingEvents = true;
return fw;
}
//************************
void fw_Changed(object sender, FileSystemEventArgs e)
{
MessageBox.Show("File: " + e.FullPath + " " + e.ChangeType);
}
EDIT: The StartWatchingFile() now returns the filewatcher which is kept in a class that will not be garbage collected, just to make sure i'm holding the whole class as thought the fw_changed() function might not be able to be called. So the whole class is now not getting garbage collectioned. The class is held in a ArrayList which is a public member of a class
Regards,
Jon
I'm sorry I can't answer your question specifically. In general, I'll contribute that, if you use it enough, you'll learn that FileSystemWatcher is not reliable. Microsoft Connect shows multiple problems with it. I agree with Jason Jackson's take on FileSystemWatcher.
Is it reproduceable that it always works for the first time?
If not, the FileSystemWatcher might have been collected by the garbage collector in the meantime after StartWatchingFile has finished because it's declared locally.
If so, does the process you're starting probably lock the file so the file actually isn't modified?
I need to setup an application that watches for files being created in a directory, both locally or on a network drive.
Would the FileSystemWatcher or polling on a timer would be the best option. I have used both methods in the past, but not extensively.
What issues (performance, reliability etc.) are there with either method?
I have seen the file system watcher fail in production and test environments. I now consider it a convenience, but I do not consider it reliable. My pattern has been to watch for changes with the files system watcher, but poll occasionally to catch missing file changes.
Edit: If you have a UI, you can also give your user the ability to "refresh" for changes instead of polling. I would combine this with a file system watcher.
The biggest problem I have had is missing files when the buffer gets full. Easy as pie to fix--just increase the buffer. Remember that it contains the file names and events, so increase it to the expected amount of files (trial and error). It does use memory that cannot be paged out, so it could force other processes to page if memory gets low.
Here is the MSDN article on buffer :
FileSystemWatcher..::.InternalBufferSize Property
Per MSDN:
Increasing buffer size is expensive, as it comes from non paged memory that cannot be swapped out to disk, so keep the buffer as small as possible. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties to filter out unwanted change notifications.
We use 16MB due to a large batch expected at one time. Works fine and never misses a file.
We also read all the files before beginning to process even one...get the file names safely cached away (in our case, into a database table) then process them.
For file locking issues I spawn a process which waits around for the file to be unlocked waiting one second, then two, then four, et cetera. We never poll. This has been in production without error for about two years.
The FileSystemWatcher may also miss changes during busy times, if the number of queued changes overflows the buffer provided. This is not a limitation of the .NET class per se, but of the underlying Win32 infrastructure. In our experience, the best way to minimize this problem is to dequeue the notifications as quickly as possible and deal with them on another thread.
As mentioned by #ChillTemp above, the watcher may not work on non-Windows shares. For example, it will not work at all on mounted Novell drives.
I agree that a good compromise is to do an occasional poll to pick up any missed changes.
Also note that file system watcher is not reliable on file shares. Particularly if the file share is hosted on a non-windows server. FSW should not be used for anything critical. Or should be used with an occasional poll to verify that it hasn't missed anything.
Personally, I've used the FileSystemWatcher on a production system, and it has worked fine. In the past 6 months, it hasn't had a single hiccup running 24x7. It is monitoring a single local folder (which is shared). We have a relatively small number of file operations that it has to handle (10 events fired per day). It's not something I've ever had to worry about. I'd use it again if I had to remake the decision.
I currently use the FileSystemWatcher on an XML file being updated on average every 100 milliseconds.
I have found that as long as the FileSystemWatcher is properly configured you should never have problems with local files.
I have no experience on remote file watching and non-Windows shares.
I would consider polling the file to be redundant and not worth the overhead unless you inherently distrust the FileSystemWatcher or have directly experienced the limitations everyone else here has listed (non-Windows shares, and remote file watching).
I have run into trouble using FileSystemWatcher on network shares. If you're in a pure Windows environment, it might not be an issue, but I was watching an NFS share and since NFS is stateless, there was never a notification when the file I was watching changed.
I'd go with polling.
Network issues cause the FileSystemWatcher to be unreliable (even when overloading the error event).
Returning from the event method as quickly as possible, using another thread, solved the problem for me:
private void Watcher_Created(object sender, FileSystemEventArgs e)
{
Task.Run(() => MySubmit(e.FullPath));
}
I had some big problems with FSW on network drives: Deleting a file always threw the error event, never the deleted event. I did not find a solution, so I now avoid the FSW and use polling.
Creation events on the other hand worked fine, so if you only need to watch for file creation, you can go for the FSW.
Also, I had no problems at all on local folders, no matter if shared or not.
Using both FSW and polling is a waste of time and resources, in my opinion, and I am surprised that experienced developers suggest it. If you need to use polling to check for any "FSW misses", then you can, naturally, discard FSW altogether and use only polling.
I am, currently, trying to decide whether I will use FSW or polling for a project I develop. Reading the answers, it is obvious that there are cases where FSW covers the needs perfectly, while other times, you need polling. Unfortunately, no answer has actually dealt with the performance difference(if there is any), only with the "reliability" issues. Is there anyone that can answer that part of the question?
EDIT : nmclean's point for the validity of using both FSW and polling(you can read the discussion in the comments, if you are interested) appears to be a very rational explanation why there can be situations that using both an FSW and polling is efficient. Thank you for shedding light on that for me(and anyone else having the same opinion), nmclean.
Working solution for working with create event instead of change
Even for copy, cut, paste, move.
class Program
{
static void Main(string[] args)
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
FileSystemWatcher FileSystemWatcher = new FileSystemWatcher();
FileSystemWatcher.Path = SourceFolderPath;
FileSystemWatcher.IncludeSubdirectories = false;
FileSystemWatcher.NotifyFilter = NotifyFilters.FileName; // ON FILE NAME FILTER
FileSystemWatcher.Filter = "*.txt";
FileSystemWatcher.Created +=FileSystemWatcher_Created; // TRIGGERED ONLY FOR FILE GOT CREATED BY COPY, CUT PASTE, MOVE
FileSystemWatcher.EnableRaisingEvents = true;
Console.Read();
}
static void FileSystemWatcher_Created(object sender, FileSystemEventArgs e)
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
try
{
// DO SOMETING LIKE MOVE, COPY, ETC
File.Copy(e.FullPath, DestinationFolderPath + #"\" + e.Name);
}
catch
{
}
}
}
Solution for this file watcher while file attribute change event using static storage
class Program
{
static string IsSameFile = string.Empty; // USE STATIC FOR TRACKING
static void Main(string[] args)
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
FileSystemWatcher FileSystemWatcher = new FileSystemWatcher();
FileSystemWatcher.Path = SourceFolderPath;
FileSystemWatcher.IncludeSubdirectories = false;
FileSystemWatcher.NotifyFilter = NotifyFilters.LastWrite;
FileSystemWatcher.Filter = "*.txt";
FileSystemWatcher.Changed += FileSystemWatcher_Changed;
FileSystemWatcher.EnableRaisingEvents = true;
Console.Read();
}
static void FileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
if (e.Name == IsSameFile) //SKIPS ON MULTIPLE TRIGGERS
{
return;
}
else
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
try
{
// DO SOMETING LIKE MOVE, COPY, ETC
File.Copy(e.FullPath, DestinationFolderPath + #"\" + e.Name);
}
catch
{
}
}
IsSameFile = e.Name;
}
}
This is a workaround solution for this problem of multiple triggering event.
I would say use polling, especially in a TDD scenario, as it is much easier to mock/stub the presence of files or otherwise when the polling event is triggered than to rely on the more "uncontrolled" fsw event. + to that having worked on a number of apps which were plagued by fsw errors.