We Keep Coding

A robust solution for FileSystemWatcher firing events multiple times

FileSystemWatcher events can fire multiple times. Not good if I need predictable behaviour from my code.
This is described in the MSDN documentation:
Common file system operations might raise more than one event. For
example, when a file is moved from one directory to another, several
OnChanged and some OnCreated and OnDeleted events might be raised.
Moving a file is a complex operation that consists of multiple simple
operations, therefore raising multiple events. Likewise, some
applications (for example, antivirus software) might cause additional
file system events that are detected by FileSystemWatcher.
Good use of NotifyFilters with particular events has helped but won't give me 100% confidence in the consistency.
Here's an example, recreating a Notepad write example (but I have experienced this with other write actions too):
public ExampleAttributesChangedFiringTwice(string demoFolderPath)
{
var watcher = new FileSystemWatcher()
{
Path = #"c:\temp",
NotifyFilter = NotifyFilters.LastWrite,
Filter = "*.txt"
};
watcher.Changed += OnChanged;
watcher.EnableRaisingEvents = true;
}
private static void OnChanged(object source, FileSystemEventArgs e)
{
// This will fire twice if I edit a file in Notepad
}
Any suggestions for making this more resilient?
EDIT: meaning not repeating multiple actions when multiple events are triggered.

An approach utilising MemoryCache as a buffer that will 'throttle' additional events.
A file event (Changed in this example) is triggered
The event is handled by OnChanged but instead of completing the desired action, it stores the event in MemoryCache
with a 1 second expiration
and a CacheItemPolicy callback setup to execute on expiration.
Note that I use AddOrGetExisting as an simple way to block any additional events firing within the cache period being added to the cache.
When it expires, the callback OnRemovedFromCache completes the behaviour intended for that file event
.
class BlockAndDelayExample
{
private readonly MemoryCache _memCache;
private readonly CacheItemPolicy _cacheItemPolicy;
private const int CacheTimeMilliseconds = 1000;
public BlockAndDelayExample(string demoFolderPath)
{
_memCache = MemoryCache.Default;
var watcher = new FileSystemWatcher()
{
Path = demoFolderPath,
NotifyFilter = NotifyFilters.LastWrite,
Filter = "*.txt"
};
_cacheItemPolicy = new CacheItemPolicy()
{
RemovedCallback = OnRemovedFromCache
};
watcher.Changed += OnChanged;
watcher.EnableRaisingEvents = true;
}
// Add file event to cache for CacheTimeMilliseconds
private void OnChanged(object source, FileSystemEventArgs e)
{
_cacheItemPolicy.AbsoluteExpiration =
DateTimeOffset.Now.AddMilliseconds(CacheTimeMilliseconds);
// Only add if it is not there already (swallow others)
_memCache.AddOrGetExisting(e.Name, e, _cacheItemPolicy);
}
// Handle cache item expiring
private void OnRemovedFromCache(CacheEntryRemovedArguments args)
{
if (args.RemovedReason != CacheEntryRemovedReason.Expired) return;
// Now actually handle file event
var e = (FileSystemEventArgs) args.CacheItem.Value;
}
}
Could easily extend to:
Check file lock on expiry from cache and if not available, put it back in the cache again (sometimes events fire so fast the file isn't ready for some operations). Preferable to using try/catch loops.
Key cache on file name + event type combined

I use a FileSystemWatcher to check for MP4 files being uploaded that I ultimately have to do something with. The process doing the upload doesn't seem to establish any lock on the file, so I formerly struggled with starting the processing of them too early.
The technique I adopted in the end, which has been entirely successful for my case, was to consume the event and add the filepath to a Dictionary<string, long> of potentially interesting files, and start a timer. Periodically (60 seconds) I check the file size. The long dictionary value holds the file size from the last check, and if the current size is greater I deem it still being written to, store the new size and go back to sleep for another 60 seconds.
Upon there being a period of 60 seconds where no write activity has occurred, I can start processing.
If this isn't suitable for you, there are a few other things you could consider; hash the file every minute and store the hash instead, re-hash it periodically until the content hasn't changed. Keep tabs on the Last Modified date in the file system, perhaps
Ultimately, consider that FileSYstemWatcher might be a useful device not for notifying you which files you have to act on, but instead for files that are potentially interesting, and a separate process with more refined in-house logic can decide if a potentially interesting file should be acted on

how to wait for filesystemwatcher event to finish

Please excuse me if this is redundant, however all the questions related to this seem to point in different directions, also I am new to multithreaded programming.
I have a FileSystemWatcher class in my code, which watches the created event. It looks like a created event of file system watcher starts it's own thread. So sometimes the calling thread continues it's work before the work initiated in called thread of FileSystemWatcher created event finishes. I don't want this. My workflow needs to be single-threaded, so what I want to achieve is wait for for created event to finish it's work before the calling thread gets an opportunity to resume.
pesudo code:
main() {
FileSystemWatcher fsw = new FileSystemWatcher()
fsw.Path = ini.location;
fsw.Created += new FileSystemEventHandler(OnFileCreation);
fsw.EnableRaisingEvents = true;
main_engine.processDataToFile();
main_engine.processCreatedFile();
}
void OnFileCreation(object sender, FileSystemEventArgs e) {
// do some file processing
// takes time based on size of file and whether file is locked or not etc.
}
void processDataToFile() {
// do some data processing on received data and output to a file.
}
void processCreatedFile() {
// do not want this method to be called, unless OnFileCreation() finish it's work.
}
The reason choose to use FileSystemWatcher was because sometimes files are directly placed for processing instead of main_engine getting the data first and it works on multiple locations, so did not want to roll out a homegrown solution when FileSystemWatcher was available.

If the event fires in the separate thread you cant make it single-threaded. because this is not your code. end of story.
however it is quite simple to wait on:
...
me.WaitOne();
main_engine.processCreatedFile();
}
...
void OnFileCreation(object sender, FileSystemEventArgs e) {
// do some file processing
// takes time based on size of file and whether file is locked or not etc.
...
me.Set();
}
ManualResetEventSlim me = new ManualResetEventSlim(false);

C# - from time to time check if a file exists and read from it

I creating an windows service which will from time to time check if a certain exists and if it does, then reads from it, sends data to a server and move a file to another folder. A file's size is about 1-3 Mb.
I think I will use System.Threading.Timer here to check if a file exists. What do you think of it?
And another question. If a file is being copied then my app must not read from it. It should wait until copying is done. Only after that it must read from it and does other activities.
So the questions:
1) Is that a right decision to use System.Threading.Timer?
2) How do I check a file is being copied and wait until it's done?
3) Must I use multi-threading?

I think I will use System.Threading.Timer here to check if a file exists. What do you think of it?
I think you might take a look at the FileSystemWatcher class which will notify you when the file is created and raise an event instead of you using a Timer that will continuously poll for the existence of the file.

Timer is very much costly . You can use FileSystemWatcher Which Listens to the file system change notifications and raises events when a directory, or file in a directory, changes.
// Create a new FileSystemWatcher and set its properties.
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = /*path*/
/* Watch for changes in LastAccess and LastWrite times, and
the renaming of files or directories. */
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
// Only watch text files.
watcher.Filter = "*.txt";
// Add event handlers.
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.Deleted += new FileSystemEventHandler(OnChanged);
watcher.Renamed += new RenamedEventHandler(OnRenamed);
// Begin watching.
watcher.EnableRaisingEvents = true;
Then this would be the OnChanged method:
//This method is called when a file is created, changed, or deleted.
private static void OnChanged(object source, FileSystemEventArgs e)
{
//Show that a file has been created, changed, or deleted.
WatcherChangeTypes wct = e.ChangeType;
Console.WriteLine("File {0} {1}", e.FullPath, wct.ToString());
}
Reference: http://devproconnections.com/net-framework/how-build-folder-watcher-service-c

The usual approach I've used is to monitor the folder/s with FileSystemWatcher, (or the relevant API's if not .NET managed), and try to ensure that the only operations performed on the source/target folders are move/rename between folders on the same physical drive, and delete. If you want to add a file, open/write/flush/close it ito a temp folder on the target filesystem drive and only then move/rename it to the folder being watched. It is vital that the temp folder is on the same physical drive as the target folder so that it can be move/renamed without a data copy.
This works well on non-managed systems, not tried it on C#, but don't see any reason for to not to work OK.
Other solutions involving continual polling and/or checking file sizes are just inconvenient, inflexible, wasteful, messy and latency-ridden at best.
Multithreading - probably yes on any remote filesystem. Network file calls tend to have very long timeouts on unrechability etc. and so block the caller for what seems like forever before issuing an error/exception. If you want to get anything else done, you should probably thread off the lot unless your users can tolerate 'hourglass apps', with windows becoming unresponsive, disappearing to back, getting greyed-out and the OS offering to close them.
Oh, and another thing - best to go on a purge when starting up. Stuff can go wrong at any time, so clean any lingering rubbish from temp folders etc. when running up.

I would not use the FileSystemWatcher it's too flaky
FileSystemWatcher does not work on files created from windows service
I would use the timer set to a reasonable tick period and you should be ok.
Some sample code
Timer_TicK()
{
// remove tick event handler
// do your processing
// add back tick event handler
}
This will keep multiple tick events from happening if you have a rather large amount of processing to do. I wouldn't multi-thread until you find you need to use it due to performance issues.
In C# if you try and read a file while it's being copied by the file system you will get an exception. You will need to check for that exception or check the properties of the file (size, etc) to know when it's done.

Full timer example, where I start new threads each time the interval happens (you don't have to, but it is a good practice), which goes and checks if the file is there and reads it and deletes it if it is:
using System;
using System.ServiceProcess;
using System.Threading;
using System.Timers;
using System.IO;
namespace MyNamespace
{
public partial class Service1: ServiceBase
{
Thread syncThread = null;
System.Timers.Timer timer1;
string filePath = #"C:\myfile.txt";
int interval = 60000; // 1 min -- adjust as necessary
public Service1()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
timer1 = new System.Timers.Timer();
timer1.Interval = interval;
timer1.Enabled = true;
timer1.Elapsed += new ElapsedEventHandler(timer1_Elapsed);
timer1.Start();
}
protected override void OnStop()
{
syncThread.Abort();
timer1.Stop();
}
protected void timer1_Elapsed(object sender, ElapsedEventArgs e)
{
syncThread = new Thread(new ThreadStart(doThread));
syncThread.Start();
}
protected void doThread()
{
// whatever you put here, it will
// run for each timer interval that elapses
// in a separate thread, and each thread will
// end when the processing in this function ends
string fileContent = String.Empty;
if (File.Exists(filePath)
{
FileStream fs = new FileStream(filePath, FileMode.Open);
StreamReader sr = new StreamReader(fs);
fileContent = sr.ReadToEnd();
sr.Close();
fs.Close();
}
if (fileContent != String.Empty)
{
// File was present... process the content...
// Then I do this...
File.Delete();
}
}
}
}
I've ran it with no problems. I prefer to start new threads with each time interval so it doesn't cause issues if the previous run hasn't finished yet. Going this route, you have that control, can decide what your interval is, and your process is not always going on - just the service. The other route, with FileSystemWatcher, it is always watching and running whenever it can and you cannot adjust your time interval, like you can with a timer, to keep down the number of events/processes going on, such as when/if the file is created, then quickly modified & saved.
Only downside I see is having to check the file attributes yourself, but those are easy enough checks to make. If you do File.Delete() after processing it like I do, you only have File.Exists(filePath) to do, in order to find out if the file has been re-created. If you have to check for modification, you merely check DateTime lastModifed = File.GetLastWriteTime(filePath) (see http://www.csharp-examples.net/file-creation-modification-time/ ) against the current time + your interval (which would be DateTime lastRun = DateTime.Now.AddMilliseconds(interval)). If lastRun > lastModified, you would have to process it.

How to monitor Textfile and continuously output content in a textbox?

I'm making a program that controls a game server. One of the functions I'm making, is a live server logfile monitor.
There is a logfile (a simple textfile) that gets updated by the server as it runs.
How do I continuously check the logfile and output it's content in a RichTextBox?
I did this simple function just try and get the content of the log. It will of course just get the text row by row and output it to my textbox. Also it will lock the program for as long as the loop runs, so I know it's useless.
public void ReadLog()
{
using (StreamReader reader = new StreamReader("server.log"))
{
String line;
// Read and display lines from the file until the end of the file is reached.
while ((line = reader.ReadLine()) != null)
{
monitorTextBox.AppendText(line + "\n");
CursorDown();
}
}
}
But how would you go about solving the live monitoring as simple as possible?
*** EDIT ***
I'm using Prescots solution. great stuff.
At the moment I'm using a sstreamreader to put the text from the file to my textbox. I ran into the problem is that, whenever I tried to access any of the gui controls in my event handler the program just stopped with no error or warnings.
I found out that it has to do with threading. I solved that like this:
private void OnChanged(object source, FileSystemEventArgs e)
{
if (monitorTextField.InvokeRequired)
{
monitorTextField.Invoke((MethodInvoker)delegate { OnChanged(source, e); });
}
else
{
StreamReader reader = new StreamReader("file.txt");
monitorTextField.Text = "";
monitorTextField.Text = reader.ReadToEnd();
reader.Close();
CursorDown();
}
}
Now my only problem is that the file.txt is used by the server so I can't access it, since it's "being used by another process". I can't control that process, so maybe I'm out of luck.
But the file can be opened in notepad while the server is running, so somehow it must be possible. Perhaps I can do a temp copy of the file when it updates and read the copy. I don't know.

Check out the System.IO.FileSystemWatcher class:
public static Watch()
{
var watch = new FileSystemWatcher();
watch.Path = #"D:\tmp";
watch.Filter = "file.txt";
watch.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite; //more options
watch.Changed += new FileSystemEventHandler(OnChanged);
watch.EnableRaisingEvents = true;
}
/// Functions:
private static void OnChanged(object source, FileSystemEventArgs e)
{
if(e.FullPath == #"D:\tmp\file.txt")
{
// do stuff
}
}
Edit: if you know some details about the file, you could handle the most efficent way to get the last line. For example, maybe when you read the file, you can wipe out what you've read, so next time it's updated, you just grab whatever is there and output. Perhaps you know one line is added at a time, then your code can immediately jump to the last line of the file. Etc.

Although the FileSystemWatcher is the most simple solution I have found it to be unreliable in reality.. often a file can be updated with new contents but the FileSystemWatcher does not fire an event until seconds later and often never.
The only reliable way I have found to approach this is to check for changes to the file on a regular basis using a System.Timers.Timer object and checking the file size.
I have written a small class that demonstrates this available here:
https://gist.github.com/ant-fx/989dd86a1ace38a9ac58
Example Usage
var monitor = new LogFileMonitor("c:\temp\app.log", "\r\n");
monitor.OnLine += (s, e) =>
{
// WARNING.. this will be a different thread...
Console.WriteLine(e.Line);
};
monitor.Start();
The only real disadvantage here (apart from a slight performance delay caused by file size checking) is that because it uses a System.Timers.Timer the callback comes from a different thread.
If you are using a Windows Forms or WPF app you could easily modify the class to accept a SynchronizingObject which would ensure the event handler events are called from the same thread.

As #Prescott suggested, use a FileSystemWatcher. And make sure, you open the file with the appropriate FileShare mode (FileShare.ReadWrite seems to be appropriate), since the file might still be opened by the server. If you try to open the file exclusively while it is still used by another process, the open operation will fail.
Also in order to gain a bit of performance, you could remember the last position up to which you already have read the file and only read the new parts.

Use this answer on another post c# continuously read file.
This one is quite efficient, and it checks once per second if the file size has changed.
You can either run it on another thread (or convert to async code), but in any case you would need to marshall the text back to the main thread to append to the textbox.

Try adding a Timer and have the Timer.Tick set to an Interval of 1 second. On Timer.Tick you run the function.
private void myTimer_Tick(object sender, EventArgs e)
{
ReadLog();
}

Detect File Read in C#

I'm using FileSystemWatcher to check when a file is modified or deleted, but I'm wondering if there is any way to check when a file is read by another application.
Example:
I have the file C:\test.txt on my harddrive and am watching it using FileSystemWatcher. Another program (not under my control) goes to read that file; I would like to catch that event and, if possible, check what program is reading the file then modify the contents of the file accordingly.

It sounds like you want to write to your log file when your log file is read externally, or something to that effect. If that is the case, there is a NotifyFilters value, LastAccess. Make sure this is set as one of the flags in your FileSystemWatcher.NotifyFilter property. A change in the last access time will then fire the Changed event on FileSystemWatcher.
Currently, FileSystemWatcher does not allow you to directly differentiate between a read and a change; they both fire the Changed event based on the "change" to LastAccess. So, it would be infeasible to watch for reads to a large number of files. However, you seem to know which file you're watching, so if you had a FileInfo object for that file, and FileSystemWatcher fired its Changed event, you could get a new one and compare LastAccessTime values. If the access time changed, and LastWriteTime didn't, your file is only being read.
Now, in simplest terms, changes you make to the file while it is being read are not going to immediately show up in the other app, nor are you going to be able to "get there first", lock the file and write to it before they see it. So, you cannot use FileSystemWatcher to "intercept" a read request and show the content you want that app to see. The only way the user of another application can see what you just wrote is if the application is also watching the file and re-loads the file. That will fire another Changed event, causing an infinite loop as long as the other application continues to reload the file.
You will also get a Changed event for a read and a write. Opening a file in a text editor (virtually any will do), making some changes, then saving will fire two Changed events if you're looking for changes to Last Access Time. The first one will go off when the file is opened by the editor; at that time, you may not be able to tell that a write will happen, so if you are looking for pure read-only accesses to the file then you're SOL.

The easiest way I can think of to do this would be with a timer (System.Threading.Timer) whose callback checks and stores the last
System.IO.File.GetLastAccessTime(path)
Something like (maybe with a bit more locking...)
public class FileAccessWatcher
{
public Dictionary<string, DateTime> _trackedFiles = new Dictionary<string, DateTime>();
private Timer _timer;
public event EventHandler<EventArgs<string>> FileAccessed = delegate { };
public FileAccessWatcher()
{
_timer = new Timer(OnTimerTick, null, 500, Timeout.Infinite);
}
public void Watch(string path)
{
_trackedFiles[path] = File.GetLastAccessTime(path);
}
public void OnTimerTick(object state)
{
foreach (var pair in _trackedFiles.ToList())
{
var accessed = File.GetLastAccessTime(pair.Key);
if (pair.Value != accessed)
{
_trackedFiles[pair.Key] = accessed;
FileAccessed(this, new EventArgs<string>(pair.Key));
}
}
_timer.Change(500, Timeout.Infinite);
}
}

There is SysInternals' program FileMon... It can trace every file access in the system. If you can find its source, and understand what win32 hooks it uses, you might marshal those functions in C# and get what you want.

You could use FileInfo.LastAccessTime and FileInfo.Refresh() in a polling loop.
http://msdn.microsoft.com/en-us/library/system.io.fileinfo_members.aspx

Yes, using file system filter driver you can catch all read requests, analyze them and even substitute the data being read. Development of such driver yourself is possible, but very time-consuming and complicated. We offer a product called CallbackFilter, which includes a ready to use driver and lets you implement your filtering business logic in user-mode.

A little snippet that I found useful for detecting when another process has a lock:
static bool IsFileUsedbyAnotherProcess(string filename)
{
try
{
using(var file = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.None))
{
}
}
catch (System.IO.IOException exp)
{
return true;
}
return false;
}