We Keep Coding

C# and the FileSystemWatcher

i have written a service in C# which should move backup files (*.bak and *.trn) from a database server to a special backup server. This works quite well so far. The problem is that it tries to move single files twice. This fails of course. I have configured FileSystemWatcher as follows:
try
{
m_objWatcher = new FileSystemWatcher();
m_objWatcher.Filter = m_strFilter;
m_objWatcher.Path = m_strSourcepath.Substring(0, m_strSourcepath.Length - 1);
m_objWatcher.IncludeSubdirectories = m_bolIncludeSubdirectories;
m_objWatcher.NotifyFilter = NotifyFilters.LastWrite | NotifyFilters.LastAccess; // | NotifyFilters.CreationTime;
m_objWatcher.Changed += new FileSystemEventHandler(objWatcher_OnCreated);
}
catch (Exception ex)
{
m_objLogger.d(TAG, m_strWatchername + "InitFileWatcher(): " + ex.ToString());
}
Is it possible that the Watcher produces an event twice for the same file? If I set the filter to CreationTime only, it does not react at all.
How do I have to set the Watcher to fire an event only once per file?
Thanks in advance for your help

The documentation states that common file system operations might raise more than one event. Check under the Events and Buffer Sizes heading.
Common file system operations might raise more than one event. For example, when a file is moved from one directory to another, several OnChanged and some OnCreated and OnDeleted events might be raised. Moving a file is a complex operation that consists of multiple simple operations, therefore raising multiple events. Likewise, some applications (for example, antivirus software) might cause additional file system events that are detected by FileSystemWatcher.
It also offers a few guidelines, including:
Keep your event handling code as short as possible.
To that end, you could use your FileSystemWatcher.Changed event to queue files for processing, then process them later. This is a quick example of what that might look like using an instance of System.Threading.Timer to process the queue.
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.IO;
public class ServiceClass
{
public ServiceClass()
{
_processing = false;
_fileQueue = new ConcurrentQueue<string>();
_timer = new System.Threading.Timer(ProcessQueue);
// Schedule the time to run in 5 seconds, then again every 5 seconds.
_timer.Change(5000, 5000);
}
private void objWatcher_OnChanged(object sender, FileSystemEventArgs e)
{
// Just queue the file to be processed later. If the same file is added multiple
// times, we'll skip the duplicates when processing the files.
_fileQueue.Enqueue(e.FilePath);
}
private void ProcessQueue(object state)
{
if (_processing)
{
return;
}
_processing = true;
var failures = new HashSet<string>();
try
{
while (_fileQueue.TryDequeue(out string fileToProcess))
{
if (!File.Exists(fileToProcess))
{
// Probably a file that was added multiple times and it was
// already processed.
continue;
}
var file = new FileInfo(fileToProcess);
if (FileIsLocked(file))
{
// File is locked. Maybe you got the Changed event, but the file
// wasn't done being written.
failures.Add(fileToProcess);
continue;
}
try
{
fileInfo.MoveTo(/*Your destination*/);
}
catch (Exception)
{
// File failed to move. Add it to the failures so it can be tried
// again.
failutes.Add(fileToProcess);
}
}
}
finally
{
// Add any failures back to the queue to try again.
foreach (var failedFile in failures)
{
_fileQueue.Enqueue(failedFile);
}
_processing = false;
}
}
private bool IsFileLocked(FileInfo file)
{
try
{
using (FileStream stream = file.Open(FileMode.Open, FileAccess.Read,
FileShare.None))
{
stream.Close();
}
}
catch (IOException)
{
return true;
}
return false;
}
private System.Threading.Timer _timer;
private bool _processing;
private ConcurrentQueue<string> _fileQueue;
}
Credit where it's due, I took FileIsLocked from this answer.
Some other things you might need to consider:
What happens if your FileSystemWatcher misses an event? [The documentation] does state that it is possible.
Note that a FileSystemWatcher may miss an event when the buffer size is exceeded. To avoid missing events, follow these guidelines:
Increase the buffer size by setting the InternalBufferSize property.
Avoid watching files with long file names, because a long file name contributes to filling up the buffer. Consider renaming these files using shorter names.
Keep your event handling code as short as possible.
What happens if your service crashes, but the process writing backup files continues to write them? When you restart your service, will it pick those files up and move them?

I tried all sorts of ideas to stop this. The events are too close together... it can't be stopped in the FileChanged Event. Here is my working solution:
private System.Timers.Timer timer;
private FileSystemWatcher fwatcher;
static void Main(string[] args)
{
new Program();
}
private Program()
{
timer = new System.Timers.Timer(100);
timer.Elapsed += new ElapsedEventHandler(OnTimedEvent);
timer.AutoReset = false; // only once
fwatcher = new FileSystemWatcher();
fwatcher.Path = filePath;
fwatcher.Filter = fileName;
fwatcher.NotifyFilter = NotifyFilters.LastWrite;
fwatcher.Changed += new FileSystemEventHandler(FileChanged);
fwatcher.EnableRaisingEvents = true;
while (IsRunning)
{
Thread.Sleep(100);
}
Thread.Sleep(100);
}
private void FileChanged(object sender, FileSystemEventArgs e)
{
timer.Start();
}
private void OnTimedEvent(object source, ElapsedEventArgs e)
{
Console.WriteLine("file has changed!");
}
The timer will only fire once each time the file is changed.

FileSystemWatcher - Reading last line of file after update?

I'm trying to write a simple console application that waits for a change to a file then reads just the last line of that file. The file watcher works and the change event fires. But I'm struggling to work out how read from the file.
static void Main(string[] args)
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.NotifyFilter = NotifyFilters.LastWrite;
watcher.Path = "E:\\myFilePath";
watcher.Filter = "";
watcher.EnableRaisingEvents = true;
watcher.Changed += new FileSystemEventHandler(watcher_Changed);
Console.ReadKey();
}
static void watcher_Changed(object sender, FileSystemEventArgs e)
{
Console.WriteLine(File.ReadLines(e.FullPath).Last());
}
when testing (editing the file in notepad) it will work once, then after a second edit I get the error...
System.IO.IOException: 'The process cannot access the file '' because
it is being used by another process.'

I still ran into problems using lines.GetEnumerator().Dispose();
You can add a pause before opening the file, just chose your poison for doing so.
static void watcher_Changed(object sender, FileSystemEventArgs e)
{
for (int x = 0; x <= 500000; x++)
{
int t = x;
}
Console.WriteLine(File.ReadLines(e.FullPath).Last());
}
I tried using SreamReader too but still ran into the same problem.
// Does not work
using (StreamReader r = new StreamReader(e.FullPath))
{
while (r.EndOfStream == false)
{
m = r.ReadLine();
}
r.Close();
}
Console.WriteLine("{0}\n", m);

The problem is due to an already open handle to the file. File.ReadLines uses an iterator internally which is Disposable. Use this to ensure Disposal during the lifetime of your event handler.
static void watcher_Changed(object sender, FileSystemEventArgs e)
{
var lines = File.ReadLines(e.FullPath);
Console.WriteLine(lines.Last());
lines.GetEnumerator().Dispose();
}
This should guarantee disposal before the end of event handler.
Edit: In the case that a different process is holding a write lock to the file, use Thread.Sleep(ms) with a safe time as inspecting and releasing file handles is not feasible in C#.

FileSystemWatcher not firing events when file written using fprintf or printf

Yes there are many similar questions but none of them address my unique situation.
There is a separate c++ process writing the file using c++ printf and fprintf.
filename i am trying to watch is info_20160525.log
My fileSystemWatcher in winform C# application gets notification when the writer process writes to the file AND I physically access the file i-e F5 the folder or have it open in textpad and click the opened file or right click the file but I never get any event notification when I dont physically interact with the file.
Also, when I shutdown the writer application I do get the notification.
Here is my code.
public bool StartUp(string fullfilepath, int linenumber)
{
if (!File.Exists(fullfilepath))
return false;
if (!LogClass.CheckPathExists(m_applicationPath))
return false;
try
{
FileInfo info = new FileInfo(fullfilepath);
m_filename = fullfilepath;
m_fileWatcher = new FileSystemWatcher(info.DirectoryName, info.Name);
m_fileWatcher.NotifyFilter = NotifyFilters.Attributes | NotifyFilters.LastAccess
| NotifyFilters.LastWrite | NotifyFilters.Size ;
m_fileWatcher.Changed += m_fileWatcher_Changed;
m_fileWatcher.Error += m_fileWatcher_Error;
m_fileWatcher.Created += m_fileWatcher_Created;
m_fileWatcher.Deleted += m_fileWatcher_Deleted;
m_fileWatcher.Disposed += m_fileWatcher_Disposed;
m_fileWatcher.Renamed += m_fileWatcher_Renamed;
m_linesRead = linenumber;
m_fileWatcher.EnableRaisingEvents = true;
}
catch(Exception e)
{
LogClass.LogError(e, "Trouble accessing the file" + fullfilepath, m_applicationPath);
}
return true;
}
These are the handlers. I have breakpoints in each one of them but I never get a trigger unless ofcourse I physically interact with the file.
void m_fileWatcher_Renamed(object sender, RenamedEventArgs e)
{
string S = "";
}
void m_fileWatcher_Disposed(object sender, EventArgs e)
{
string S = "";
}
void m_fileWatcher_Deleted(object sender, FileSystemEventArgs e)
{
string S = "";
}
void m_fileWatcher_Created(object sender, FileSystemEventArgs e)
{
string S = "";
}
void m_fileWatcher_Error(object sender, ErrorEventArgs e)
{
string S = "";
}
void m_fileWatcher_Changed(object sender, FileSystemEventArgs args)
{
if (args.ChangeType == WatcherChangeTypes.Changed)
{
while (ParseFile(args.FullPath))
{
}
}
}

I bet this thread has your answer --> FileSystemWatcher changed event (for “LastWrite”) is unreliable
The FileSystemWatcher uses an update to the LastWrite attribute of a file to fire events, however, the LastWrite is not updated in real time and should not be relied upon as trigger for an event.
If you have enough time and resources on your hands then you probably want to look into File System Filters and the simpler approach of a Filter Manager and Minifilter Driver. It is driver type development, however, it is a sure file way to accomplish your objective.
It is dug down a little deeper by system policy but gives you a wide array of events to latch onto. If I was doing this for anything like pci compliance or similar tasks then I would not use the FileSystemWatcher.

Make sure you set IncludeSubdirectories to true.
https://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher(v=vs.110).aspx

TextFile Reader and FileSystemWatcher in Windows Service

I am trying to use FileSystemWatcher to read a textfile as soon as anything gets updated into textfile in Windows Service.Now the problem that i am facing is not getting the way where i should put my FileSystemWatcher code so that i would get called as soon as textfile gets changed.Do i need to add this into OnStart() Method of Windows Service or anywhere else.
Here is my Code Structure..
protected override void OnStart(string[] args)
{
_thread = new Thread(startReadingTextFile);
_thread.Start();
}
public void startReadingTextFile() {
_freader = new AddedContentReader(TextFileLocation);
}
private void Watcher_Changed(object sender, FileSystemEventArgs e)
{
string addedContent = _freader.GetAddedLines();
}
Please help me .Thanks ..
Updated Code..
protected override void OnStart(string[] args)
{
if (lastLineReadOffset == 0)
{
_freader = new AddedContentReader(TextFileLocation);
}
//If you have saved the last position when the application did exit then you can use that value here to start from that location like the following
//_freader = new AddedContentReader("E:\\tmp\\test.txt",lastReadPosition);
else
{
_freader = new AddedContentReader(TextFileLocation, lastLineReadOffset);
}
FileSystemWatcher Watcher = new FileSystemWatcher("C:\\temp");
Watcher.EnableRaisingEvents = true;
Watcher.Changed += new FileSystemEventHandler(Watcher_Changed);
}
private void Watcher_Changed(object sender, FileSystemEventArgs e)
{
string addedContent = _freader.GetAddedLines();
//you can do whatever you want with the lines
using (StringReader reader = new StringReader(addedContent))
{
string line;
while ((line = reader.ReadLine()) != null)
{
// Call the Processing Function
}
}
}

Do i need to add this into OnStart()
Yes.
But, there is no need to create a thread for this purpose. Once FileSystemWatcher.EnableRaisingEvents is set, then events will be fired in the thread pool: you can return from OnStart.

Richard's answer is correct. However, the Changed event fires at least twice because by default FileSystemWatcher fires it once when the file is created and then again every time the file system flushes its contents to the disk. For large files you may get multiple Change events caused by multiple disk writes. If you try to open the file the first time Change fires you may get errors if the file is locked by the writing process or get an incomplete file content.
The most reliable way I have discovered is to set a timer with a short interval (a few seconds) on the very first Change event for a new file and then reset it every subsequent time the event fires for the same file. You then open the file in the timer's own Elapsed event when it does fire a few seconds after the last Change event was fired for the file.
This requires some extra code and variables:
First, create a Dictionary<string, Timer> to keep track of timers per filename.
Inside your Change event handler you need to check if the dictionary already contains the file's name as a key (inside a lock block to take care of thread concurrency issues).
If it isn't then:
create a new Timer instance
set its state object to the name of the file so when its Elapsed event fires you'll know which file you are supposed to process (the same end result can also be achieved using a closure and a lambda function but a state object is simpler)
add the new timer instance to the dictionary using the file name as the key
If it is (i.e. this is not the first Change event for that file):
look up the Timer instance in the dictionary
reset its interval to push its Elapsed event further
Then in the handler of the timer's Elapsed event you do your actual processing and cleanup:
get the file name from the timer's state object passed in the event arguments
look up the Timer instance in the dictionary by the file name and dispose of it
remove the timer from the dictionary, i.e. Remove(key) where key is the file name (the three actions above should happen inside a lock block)
open the file and do whatever you need with it.
Here's how you might want to implement this logic inside your service:
const int DELAY = 2000; // milliseconds
const WatcherChangeTypes FILE_EVENTS = WatcherChangeTypes.Created | WatcherChangeTypes.Changed | WatcherChangeTypes.Renamed;
FileSystemWatcher _fsw;
Dictionary<string, Timer> _timers = new Dictionary<string, Timer>();
object _lock = new object();
public void Start()
{
_fsw = new FileSystemWatcher(Directory, FileFilter)
{
IncludeSubdirectories = false,
EnableRaisingEvents = true
};
_fsw.Created += OnFileChanged;
_fsw.Changed += OnFileChanged;
}
private void OnFileChanged(object sender, FileSystemEventArgs e)
{
try
{
// When a file is created in the monitored directory, set a timer to process it after a short
// delay and add the timer to the queue.
if (FILE_EVENTS.HasFlag(e.ChangeType))
{
lock (_lock)
{
// File events may fire multiple times as the file is being written to the disk and/or renamed,
// therefore the first time we create a new timer and then reset it on subsequent events so that
// the file is processed shortly after the last event fires.
if (_timers.TryGetValue(e.FullPath, out Timer timer))
{
timer.Change(DELAY, 0);
}
else
{
_timers.Add(e.FullPath, new Timer(OnTimerElapsed, e.FullPath, DELAY, 0));
}
}
}
}
catch (Exception ex)
{
// handle errors
}
}
private void OnTimerElapsed(object state)
{
var fileName = (string)state;
lock (_lock)
{
try { _timers[fileName].Dispose(); } catch { }
try { _timers.Remove(fileName); } catch { }
}
// open the file ...
}

How to end filewatcher from another method?

Below is my code for a File watcher class I wrote:
class FileWatcher
{
#region Method that begins watching
public static void watch()
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = ConfigurationManager.AppSettings["OpticusFileLoc"];
watcher.Filter = ConfigurationManager.AppSettings["OpticusFileName"];
watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName;
watcher.Changed += new FileSystemEventHandler(OnChanged);
Console.Write("\nWatcher started. Press any key to end.\n");
watcher.EnableRaisingEvents = true;
}
#endregion
#region Trigger function on change
public static void OnChanged(object source, FileSystemEventArgs e)
{
Console.WriteLine("File has been changed.\n");
//watcher.EnableRaisingEvents = false ;
//Program.Main();
}
#endregion
}
How can I, from the OnChanged method, set the watcher.EnableRaisingEvents flag to false?
I guess I could do it by moving the declaration of the FileSystemWatcher object outside the method it is in, but I'm wondering if there's another way to do it.

I guess I could do it by moving the
declaration of the FileSystemWatcher
object outside the method it is in,
but I'm wondering if there's another
way to do it.
That's exactly how you should do it (have watcher just be a member field of your FileWatcher class).
That said, you also could do it like this:
public static void OnChanged(object source, FileSystemEventArgs e)
{
Console.WriteLine("File has been changed.");
var watcher = source as FileSystemWatcher;
if (watcher != null)
{
watcher.EnableRaisingEvents = false;
}
else
{
// Hmm... some other object called this method.
// Do you really want that to be allowed?
}
}
Notice the comment I added in the code above. Generally it's a bad call to make methods such as this public, as this makes it possible for any arbitrary code to call OnChanged even when doing so makes no sense. (You want for this method to capture the event of a file being changed; if it's actually callable from anywhere, then how do you know if the file was in fact changed or if this is just some random call from elsewhere?)

source is probably the FileSystemWatcher in question, so try this:
((FileSystemWatcher)source).EnableChangingEvents = false;

The object source parameter should be your FileSystemWatcher object, so you could just use that to shut it down.
public static void OnChanged(object source, FileSystemEventArgs e)
{
Console.WriteLine("File has been changed.\n");
(source as FileSystemWatcher).EnableRaisingEvents = false;
}

Is this the kind of thing you want to do?
public static void OnChanged(object source, FileSystemEventArgs e)
{
FileSystemWatcher watcher = (FileSystemWatcher)source;
watcher.EnableRaisingEvents = false ;
}