We Keep Coding

C# - from time to time check if a file exists and read from it

I creating an windows service which will from time to time check if a certain exists and if it does, then reads from it, sends data to a server and move a file to another folder. A file's size is about 1-3 Mb.
I think I will use System.Threading.Timer here to check if a file exists. What do you think of it?
And another question. If a file is being copied then my app must not read from it. It should wait until copying is done. Only after that it must read from it and does other activities.
So the questions:
1) Is that a right decision to use System.Threading.Timer?
2) How do I check a file is being copied and wait until it's done?
3) Must I use multi-threading?

I think I will use System.Threading.Timer here to check if a file exists. What do you think of it?
I think you might take a look at the FileSystemWatcher class which will notify you when the file is created and raise an event instead of you using a Timer that will continuously poll for the existence of the file.

Timer is very much costly . You can use FileSystemWatcher Which Listens to the file system change notifications and raises events when a directory, or file in a directory, changes.
// Create a new FileSystemWatcher and set its properties.
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = /*path*/
/* Watch for changes in LastAccess and LastWrite times, and
the renaming of files or directories. */
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite
| NotifyFilters.FileName | NotifyFilters.DirectoryName;
// Only watch text files.
watcher.Filter = "*.txt";
// Add event handlers.
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.Deleted += new FileSystemEventHandler(OnChanged);
watcher.Renamed += new RenamedEventHandler(OnRenamed);
// Begin watching.
watcher.EnableRaisingEvents = true;
Then this would be the OnChanged method:
//This method is called when a file is created, changed, or deleted.
private static void OnChanged(object source, FileSystemEventArgs e)
{
//Show that a file has been created, changed, or deleted.
WatcherChangeTypes wct = e.ChangeType;
Console.WriteLine("File {0} {1}", e.FullPath, wct.ToString());
}
Reference: http://devproconnections.com/net-framework/how-build-folder-watcher-service-c

The usual approach I've used is to monitor the folder/s with FileSystemWatcher, (or the relevant API's if not .NET managed), and try to ensure that the only operations performed on the source/target folders are move/rename between folders on the same physical drive, and delete. If you want to add a file, open/write/flush/close it ito a temp folder on the target filesystem drive and only then move/rename it to the folder being watched. It is vital that the temp folder is on the same physical drive as the target folder so that it can be move/renamed without a data copy.
This works well on non-managed systems, not tried it on C#, but don't see any reason for to not to work OK.
Other solutions involving continual polling and/or checking file sizes are just inconvenient, inflexible, wasteful, messy and latency-ridden at best.
Multithreading - probably yes on any remote filesystem. Network file calls tend to have very long timeouts on unrechability etc. and so block the caller for what seems like forever before issuing an error/exception. If you want to get anything else done, you should probably thread off the lot unless your users can tolerate 'hourglass apps', with windows becoming unresponsive, disappearing to back, getting greyed-out and the OS offering to close them.
Oh, and another thing - best to go on a purge when starting up. Stuff can go wrong at any time, so clean any lingering rubbish from temp folders etc. when running up.

I would not use the FileSystemWatcher it's too flaky
FileSystemWatcher does not work on files created from windows service
I would use the timer set to a reasonable tick period and you should be ok.
Some sample code
Timer_TicK()
{
// remove tick event handler
// do your processing
// add back tick event handler
}
This will keep multiple tick events from happening if you have a rather large amount of processing to do. I wouldn't multi-thread until you find you need to use it due to performance issues.
In C# if you try and read a file while it's being copied by the file system you will get an exception. You will need to check for that exception or check the properties of the file (size, etc) to know when it's done.

Full timer example, where I start new threads each time the interval happens (you don't have to, but it is a good practice), which goes and checks if the file is there and reads it and deletes it if it is:
using System;
using System.ServiceProcess;
using System.Threading;
using System.Timers;
using System.IO;
namespace MyNamespace
{
public partial class Service1: ServiceBase
{
Thread syncThread = null;
System.Timers.Timer timer1;
string filePath = #"C:\myfile.txt";
int interval = 60000; // 1 min -- adjust as necessary
public Service1()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
timer1 = new System.Timers.Timer();
timer1.Interval = interval;
timer1.Enabled = true;
timer1.Elapsed += new ElapsedEventHandler(timer1_Elapsed);
timer1.Start();
}
protected override void OnStop()
{
syncThread.Abort();
timer1.Stop();
}
protected void timer1_Elapsed(object sender, ElapsedEventArgs e)
{
syncThread = new Thread(new ThreadStart(doThread));
syncThread.Start();
}
protected void doThread()
{
// whatever you put here, it will
// run for each timer interval that elapses
// in a separate thread, and each thread will
// end when the processing in this function ends
string fileContent = String.Empty;
if (File.Exists(filePath)
{
FileStream fs = new FileStream(filePath, FileMode.Open);
StreamReader sr = new StreamReader(fs);
fileContent = sr.ReadToEnd();
sr.Close();
fs.Close();
}
if (fileContent != String.Empty)
{
// File was present... process the content...
// Then I do this...
File.Delete();
}
}
}
}
I've ran it with no problems. I prefer to start new threads with each time interval so it doesn't cause issues if the previous run hasn't finished yet. Going this route, you have that control, can decide what your interval is, and your process is not always going on - just the service. The other route, with FileSystemWatcher, it is always watching and running whenever it can and you cannot adjust your time interval, like you can with a timer, to keep down the number of events/processes going on, such as when/if the file is created, then quickly modified & saved.
Only downside I see is having to check the file attributes yourself, but those are easy enough checks to make. If you do File.Delete() after processing it like I do, you only have File.Exists(filePath) to do, in order to find out if the file has been re-created. If you have to check for modification, you merely check DateTime lastModifed = File.GetLastWriteTime(filePath) (see http://www.csharp-examples.net/file-creation-modification-time/ ) against the current time + your interval (which would be DateTime lastRun = DateTime.Now.AddMilliseconds(interval)). If lastRun > lastModified, you would have to process it.

FileSystemWatcher changed event (for "LastWrite") is unreliable

I am trying to get a notification when a file is updated on disk. I am interested in getting this notifications as soon as a flush occurs, yet, it seems that the FileSystemWatcher would only send an event when the stream opens or closes.
In the code below I am writing to a file and repeatedly flush the buffer to the disk. Yet, the FileSystemWatcher only notified me once in the beginning of the writes and once when they end.
Is there another way to get these notifications? Or should I use polling for that?
The code:
class Program
{
static void Main(string[] args)
{
FileSystemWatcher watcher = new FileSystemWatcher(Environment.CurrentDirectory, "Test.txt");
watcher.Changed += watcher_Changed;
watcher.EnableRaisingEvents = true;
using(TextWriter writer = new StreamWriter("Test.txt"))
{
WriteData(writer);
WriteData(writer);
WriteData(writer);
WriteData(writer);
}
Thread.Sleep(10000);
}
private static void WriteData(TextWriter writer)
{
writer.WriteLine("Hello!");
writer.Flush();
Console.WriteLine(DateTime.Now.ToString("T") + "] Wrote data");
Thread.Sleep(3000);
}
static void watcher_Changed(object sender, FileSystemEventArgs e)
{
Console.WriteLine(DateTime.Now.ToString("T") + "] Watcher changed!");
}
}
UPDATE 1
I've corrected the ToString function of the DateTime to show seconds. Here is the output with the code above:
11:37:47 AM] Watcher changed!
11:37:47 AM] Wrote data
11:37:50 AM] Wrote data
11:37:53 AM] Wrote data
11:37:56 AM] Wrote data
11:37:59 AM] Watcher changed!
Thanks!

It has nothing to do with the FileSystemWatcher. The watcher reacts to updates on the LastWrite attribute for the filesystem.
E.g. NTFS does not update the LastWrite on every write. The value is cached and only written when the stream is closed or at some other unspecified time. This document says
Timestamps are updated at various times and for various reasons. The only guarantee about a file timestamp is that the file time is correctly reflected when the handle that makes the change is closed. [...] The NTFS file system delays updates to the last access time for a file by up to 1 hour after the last access
I assume a similar caching applies for write

I think one of your issues here is that all your writes get executed before the event gets a slice of that cpu-time.
MSDN states
The Changed event is raised when changes are made to the size, system attributes, last write time, last access time, or security permissions of a file or directory in the directory being monitored.
I did a test and inserted Sleeps after each call to WriteData(...), what i got was
09:32] Watcher changed!
09:32] Watcher changed!
09:32] -----> Wrote data
09:32] -----> Wrote data
09:32] -----> Wrote data
09:32] -----> Wrote data
09:32] Watcher changed!
09:32] Watcher changed!
I guess this kind of proves that the even is fired right after you call Flush(), it's just a question of when the event-handler gets executed (and I assume it groups events as well).
I don't know the specific needs of your project, but I wouldn't poll. Seems like a waste since FileSystemWatcher does what you want it to do in my opinion.
Edit: Ok, I guess my brain wasn't quite ready yet for thinking when I posted this.
Your conclusion that it fires when you open and close the stream seems more logical and right.
I guess I was looking for "prove" that it fires when you call flush and therefore found it - somehow.
Update
I just had a poke at the USN-Journal and it seems that wont get you what you want either as it writes the record only when the file is closed. -> http://msdn.microsoft.com/en-us/library/aa363803(VS.85).aspx
I also found a USN-Viewer in C# and http://social.msdn.microsoft.com/Forums/en/csharpgeneral/thread/c1550294-d121-4511-ac32-31551497f64e might be interesting to read as well.
I also ran DiskMon to see if it gets the changes in realtime. It doesn't, but I don't know if that's intentional or not. However, the problem with both is that they require admin-rights to run.
So I guess you are stuck with FileSystemWatcher. (What is it you need the updates for anyway, it's not like you can read the file while it's open/locked by another program.)
ps: I just noticed you are a dev of BugAid, which I was made aware of it only recently - it looks awesome :)

I know this is an old thread but the problem still exists 10 years later. I solved this by using a timer:
private Timer _checkTimer;
private DateTime _lastSaved;
Before initializing the timer, I get the last write time of the file:
_lastSaved = File.GetLastWriteTime(_watchedFile);
_checkTimer = new Timer() { Interval = 1000, AutoReset = true };
_checkTimer.Elapsed += CheckTimer_Elapsed;
_checkTimer.Start();
Then on each elapsed event I check if the last write time is newer than the one that I knew to be the last write time:
private void CheckTimer_Elapsed(object sender, ElapsedEventArgs e)
{
DateTime lastWriteTime = File.GetLastWriteTime(_watchedFile);
if (lastWriteTime > _lastSaved)
{
// Stop the timer to avoid concurrency problems
_syncTimer.Stop();
// Do here your processing
// Now, this is the new last known write time
_lastSaved = File.GetLastWriteTime(_watchedFile);
// Start again the timer
_syncTimer.Start();
}
}
This is how I personally solved this problem that I also had with FileSystemWatcher. This method is not as expensive as reading the file content or computing the file hash every second to check if there is a difference.

FileSystemWatcher Network Disconnect

I have a FileSystemWatcher monitoring a file on a network share. If an event occurs to make the share unavailable, maybe due to a network problem, the FileSystemWatcher becomes disconnected.
Obviously I can handle the "Error" event, maybe do some logging and lots of articles suggest reconnecting the FSW inside the error event handler.
But what if the network share is still unavailable inside the error event. I then need to introduce a timer to test if the network share is available and attempt to reconnect the FSW.
1) Is there a better approach?
2) Is there a property that allows me to determine that the FSW has become disconnected from the file? I notice there is a non-public member of the FSW "stopListening", which appears to be set to true when the FSW becomes disconnected. But this is not publicly exposed
Any help would be appreciated ...
Thanks
Kevin

A couple of comments and suggestions...(which grew and grew as I was typing...sorry)
The FileSystemWatcher.Error event is fired when the FileSystemWatcher gets so many events happening so quickly that it can't handle them all. It doesn't get fired when an error occurs in watching the file system (such as the network dropping out).
I believe I've had a similar situation. The problem is that when the network connection drops out, the FileSystemWatcher will never have an event triggered, because it actually can't see what it's supposed to be watching, but it doesn't seem to be aware of the fact. When the network connection comes back, the FileSystemWatcher doesn't recover - i.e. it still can't see the (restored) connection. The only solution that we came up with that seemed to work reliably was to have a timer that regularly dropped the whole FileSystemWatcher object and created a new one, setting all of the events and watch folder etc. Since dropping and creating a new FileSystemWatcher is (relatively) quick (i.e. milliseconds) you could set the timer to activate every 10 seconds or so without using up too much of the processor. Of course, if the network is still out, the FileSystemWatcher is not going to be able to see the network no matter what you do. But that's OK, it will try again in another 10 seconds.
Two things to watch out for with this solution:
When the timer activates, it needs to check that the FileSystemWatcher isn't currently processing any events, and it needs to wait if it is. So in the timer event, stop the Timer, stop the FileSystemWatcher from raising events, then wait for any FileSystemWatcher events to finish (using lock (...) {...} is a good way of doing this).
After dropping and recreating the FileSystemWatcher, you need to manually check for any events that might have occurred while the FileSystemWatcher was being refreshed (or while the network was down). For example, if you're watching for files being created, and a file gets created while refreshing the FileSystemWatcher or while the network connection is out, you won't get events for those files when you start up the new instance of the FileSystemWatcher (since the files have already been created).
I hope that helps.

Follow-up in this. At the suggestion of a Microsoft resource on the MSDN forums, I added this to Microsoft Connect.
Key points from the feedback from Microsoft:
- Error event is not only for internal buffer overflows
- They will add the possibility of exposing the stopListening property to their list of customer suggestions
Link here: http://connect.microsoft.com/VisualStudio/feedback/details/727934/filesystemwatcher-error-handling

Wouldn't something like this work? Seems to work for my simple test case.
var fsw = new FileSystemWatcher("[folder]", "*.*") { IncludeSubdirectories = true};
var fsw_processing = false;
fsw.Deleted += (s, e) =>
{
fsw_processing = true;
fsw.EnableRaisingEvents = false;
//......
fsw.EnableRaisingEvents = true;
fsw_processing = false;
};
fsw.Changed += (s, e) =>
{
fsw_processing = true;
fsw.EnableRaisingEvents = false;
//......
fsw.EnableRaisingEvents = true;
fsw_processing = false;
};
//governor thread to check FileSystemWatcher is still connected.
//It seems to disconnects on network outages etc.
Task.Run(() =>
{
while (true)
{
if (fsw.EnableRaisingEvents == false && fsw_processing == false)
{
try
{fsw.EnableRaisingEvents = true;}
catch (Exception) { fsw.EnableRaisingEvents = false; }
}
System.Threading.Thread.Sleep(1000 * 10);//sleep 10 secs
}
});

File System Watcher in C# gets fired only once

I am kicking off a File System Watcher in a system tray application. It monitor's a text file in a local folder. But the FS_changed event is fired only once after starting the application.
FileSystemWatcher fWatch = new FileSystemWatcher();
fWatch.BeginInit();
fWatch.Path = Path.GetTempPath();
fWatch.Filter = "File1.txt";
fWatch.Changed += new FileSystemEventHandler(fWatch_Changed);
fWatch.EnableRaisingEvents = true;
fWatch.EndInit();
I am not handling the Created/Deleted, kind of events. But I tried those as well(just to check) and they are not being called at all although that local file is being deleted and created during this process.
Any ideas/suggestions?

It's probably getting garbage collected. Are you keeping a reference to that object?

Are you sure that the instance of fWatch is not being disposed? Is it part of a method that keeps listening all the time otherwise you might be getting the event only while the instance lives. Can you please put the code you are using to keep the watcher alive so we can help you more in detail.

as one of the overloads coud you pass the path that you're wanting to watch into the constructor of that object
FileSystemWatcher fWatch = new FileSystemWatcher(Path.GetTempPath());
for example..?

FileSystemWatcher vs polling to watch for file changes

I need to setup an application that watches for files being created in a directory, both locally or on a network drive.
Would the FileSystemWatcher or polling on a timer would be the best option. I have used both methods in the past, but not extensively.
What issues (performance, reliability etc.) are there with either method?

I have seen the file system watcher fail in production and test environments. I now consider it a convenience, but I do not consider it reliable. My pattern has been to watch for changes with the files system watcher, but poll occasionally to catch missing file changes.
Edit: If you have a UI, you can also give your user the ability to "refresh" for changes instead of polling. I would combine this with a file system watcher.

The biggest problem I have had is missing files when the buffer gets full. Easy as pie to fix--just increase the buffer. Remember that it contains the file names and events, so increase it to the expected amount of files (trial and error). It does use memory that cannot be paged out, so it could force other processes to page if memory gets low.
Here is the MSDN article on buffer :
FileSystemWatcher..::.InternalBufferSize Property
Per MSDN:
Increasing buffer size is expensive, as it comes from non paged memory that cannot be swapped out to disk, so keep the buffer as small as possible. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties to filter out unwanted change notifications.
We use 16MB due to a large batch expected at one time. Works fine and never misses a file.
We also read all the files before beginning to process even one...get the file names safely cached away (in our case, into a database table) then process them.
For file locking issues I spawn a process which waits around for the file to be unlocked waiting one second, then two, then four, et cetera. We never poll. This has been in production without error for about two years.

The FileSystemWatcher may also miss changes during busy times, if the number of queued changes overflows the buffer provided. This is not a limitation of the .NET class per se, but of the underlying Win32 infrastructure. In our experience, the best way to minimize this problem is to dequeue the notifications as quickly as possible and deal with them on another thread.
As mentioned by #ChillTemp above, the watcher may not work on non-Windows shares. For example, it will not work at all on mounted Novell drives.
I agree that a good compromise is to do an occasional poll to pick up any missed changes.

Also note that file system watcher is not reliable on file shares. Particularly if the file share is hosted on a non-windows server. FSW should not be used for anything critical. Or should be used with an occasional poll to verify that it hasn't missed anything.

Personally, I've used the FileSystemWatcher on a production system, and it has worked fine. In the past 6 months, it hasn't had a single hiccup running 24x7. It is monitoring a single local folder (which is shared). We have a relatively small number of file operations that it has to handle (10 events fired per day). It's not something I've ever had to worry about. I'd use it again if I had to remake the decision.

I currently use the FileSystemWatcher on an XML file being updated on average every 100 milliseconds.
I have found that as long as the FileSystemWatcher is properly configured you should never have problems with local files.
I have no experience on remote file watching and non-Windows shares.
I would consider polling the file to be redundant and not worth the overhead unless you inherently distrust the FileSystemWatcher or have directly experienced the limitations everyone else here has listed (non-Windows shares, and remote file watching).

I have run into trouble using FileSystemWatcher on network shares. If you're in a pure Windows environment, it might not be an issue, but I was watching an NFS share and since NFS is stateless, there was never a notification when the file I was watching changed.

I'd go with polling.
Network issues cause the FileSystemWatcher to be unreliable (even when overloading the error event).

Returning from the event method as quickly as possible, using another thread, solved the problem for me:
private void Watcher_Created(object sender, FileSystemEventArgs e)
{
Task.Run(() => MySubmit(e.FullPath));
}

I had some big problems with FSW on network drives: Deleting a file always threw the error event, never the deleted event. I did not find a solution, so I now avoid the FSW and use polling.
Creation events on the other hand worked fine, so if you only need to watch for file creation, you can go for the FSW.
Also, I had no problems at all on local folders, no matter if shared or not.

Using both FSW and polling is a waste of time and resources, in my opinion, and I am surprised that experienced developers suggest it. If you need to use polling to check for any "FSW misses", then you can, naturally, discard FSW altogether and use only polling.
I am, currently, trying to decide whether I will use FSW or polling for a project I develop. Reading the answers, it is obvious that there are cases where FSW covers the needs perfectly, while other times, you need polling. Unfortunately, no answer has actually dealt with the performance difference(if there is any), only with the "reliability" issues. Is there anyone that can answer that part of the question?
EDIT : nmclean's point for the validity of using both FSW and polling(you can read the discussion in the comments, if you are interested) appears to be a very rational explanation why there can be situations that using both an FSW and polling is efficient. Thank you for shedding light on that for me(and anyone else having the same opinion), nmclean.

Working solution for working with create event instead of change
Even for copy, cut, paste, move.
class Program
{
static void Main(string[] args)
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
FileSystemWatcher FileSystemWatcher = new FileSystemWatcher();
FileSystemWatcher.Path = SourceFolderPath;
FileSystemWatcher.IncludeSubdirectories = false;
FileSystemWatcher.NotifyFilter = NotifyFilters.FileName; // ON FILE NAME FILTER
FileSystemWatcher.Filter = "*.txt";
FileSystemWatcher.Created +=FileSystemWatcher_Created; // TRIGGERED ONLY FOR FILE GOT CREATED BY COPY, CUT PASTE, MOVE
FileSystemWatcher.EnableRaisingEvents = true;
Console.Read();
}
static void FileSystemWatcher_Created(object sender, FileSystemEventArgs e)
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
try
{
// DO SOMETING LIKE MOVE, COPY, ETC
File.Copy(e.FullPath, DestinationFolderPath + #"\" + e.Name);
}
catch
{
}
}
}
Solution for this file watcher while file attribute change event using static storage
class Program
{
static string IsSameFile = string.Empty; // USE STATIC FOR TRACKING
static void Main(string[] args)
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
FileSystemWatcher FileSystemWatcher = new FileSystemWatcher();
FileSystemWatcher.Path = SourceFolderPath;
FileSystemWatcher.IncludeSubdirectories = false;
FileSystemWatcher.NotifyFilter = NotifyFilters.LastWrite;
FileSystemWatcher.Filter = "*.txt";
FileSystemWatcher.Changed += FileSystemWatcher_Changed;
FileSystemWatcher.EnableRaisingEvents = true;
Console.Read();
}
static void FileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
if (e.Name == IsSameFile) //SKIPS ON MULTIPLE TRIGGERS
{
return;
}
else
{
string SourceFolderPath = "D:\\SourcePath";
string DestinationFolderPath = "D:\\DestinationPath";
try
{
// DO SOMETING LIKE MOVE, COPY, ETC
File.Copy(e.FullPath, DestinationFolderPath + #"\" + e.Name);
}
catch
{
}
}
IsSameFile = e.Name;
}
}
This is a workaround solution for this problem of multiple triggering event.

I would say use polling, especially in a TDD scenario, as it is much easier to mock/stub the presence of files or otherwise when the polling event is triggered than to rely on the more "uncontrolled" fsw event. + to that having worked on a number of apps which were plagued by fsw errors.