We Keep Coding

Repeating a method call on each direcotry with timer elasped in C#

Overview:
I've created two FileSystemWatcher to detect sub-folders and its files creation. I also use a Timer to the second watcher(file watcher), which restart the timer for every file created to any of each folder before calling a zip method dynamically.
I try to make this process to every folder, but it zips all folder at once regardless if there is a file moving into those folders.
Below is part of my code to simulate the timer and zip process... I might have a problem moving the zip file to root folder when calling the zip method.
The Timer inside the file Watcher
namespace ZipperAndWatcher
{
public class Archive
{
public void CreateZip(string path)
{
// zip method code
}
}
public class FileWatcher
{
private FileSystemWatcher _fsw;
private Archive _zip;
private string _path;
private Timer _timer;
public FileWatcher()
{
_fsw = new FileSystemWatcher();
_zip = new Archive();
_path = #"D:\Documents\output";
_timer = new Timer();
}
public void StartWatching()
{
var rootWatcher = _fsw;
rootWatcher.Path = _path;
rootWatcher.Created += new FileSystemEventHandler(OnRootFolderCreated);
rootWatcher.EnableRaisingEvents = true;
rootWatcher.Filter = "*.*";
}
private void OnRootFolderCreated(object sender, FileSystemEventArgs e)
{
string watchedPath = e.FullPath; // watched path
// create another watcher for file creation and send event to timer
FileSystemWatcher subFolderWatcher = new FileSystemWatcher();
subFolderWatcher.Path = watchedPath + #"\";
// Timer setting
var aTimer = _timer;
aTimer.Interval = 20000;
// Lambda == args => expression
// send event to subFolderWatcher
aTimer.Elapsed += new ElapsedEventHandler((subfolderSender, evt) => OnTimedEvent(subfolderSender, evt, subFolderWatcher));
aTimer.AutoReset = false;
aTimer.Enabled = true;
// sub-folder sends event to timer (and wait timer to notify subfolder)
subFolderWatcher.Created += new FileSystemEventHandler((s, evt) => subFolderWatcher_Created(s, evt, aTimer));
subFolderWatcher.Filter = "*.*";
subFolderWatcher.EnableRaisingEvents = true;
}
private void OnTimedEvent(object sender, ElapsedEventArgs evt, FileSystemWatcher subFolderWatcher)
{
subFolderWatcher.EnableRaisingEvents = false;
// Explicit Casting
Timer timer = sender as Timer;
timer.Stop();
timer.Dispose();
// Once time elapsed, zip the folder here?
Console.WriteLine($"time up. zip process at {evt.SignalTime}");
Archive zip = _zip;
zip.CreateZip(subFolderWatcher.Path.Substring(0, subFolderWatcher.Path.LastIndexOf(#"\")));
subFolderWatcher.Dispose();
}
private void subFolderWatcher_Created(object sender, FileSystemEventArgs evt, Timer aTimer)
{
// if new file created, stop the timer
// then restart the timer
aTimer.AutoReset = false;
aTimer.Stop();
aTimer.Start();
Console.WriteLine($"restart the timer as {evt.Name} created on {DateTime.Now.ToString()}");
}
}
}

You wrote in your last comment: "what I want is to only zip the specific subdirectories, which has no more files being added after the time elapsed while other subdirectories might still have files adding in, and it is not ready to be zipped"
So in your timer handler (OnTimedEvent), you need to pass the path to the directory that is ready for zipping to CreateZip, not the parent directory, i.e. change
zip.CreateZip(subFolderWatcher.Path.Substring(0, subFolderWatcher.Path.LastIndexOf(#"\")));
to
zip.CreateZip(subFolderWatcher.Path);
Then in your CreateZip method just zip the directory that is passed as parameter, not all subdirectories as you do now.

C# More threads starting than expected

I'm working on a project where i have to observe a directory. The files will be deployed in the directory at the same time. This means there could be 7000 files that will be moved to the directory at once. I'm using the FileSystemWatcher to trigger a thread if a new file ist added to the directory. If I'm moving a small amount of files (1-150 Files, 20 KB each) there are the right amount of threads starting. For each file one thread. As soon as I paste in a larger amount of these files, it's showing that there were more threads started than the directory contains. As you can see, I'm printing out "test" and a counter for each thread started. In the end, the counter is a higher number than the amount of pasted files. The more files I paste, the bigger is the difference between counter and pasted files. Hope you can tell me what I'm doing wrong.
public static void Main(string[] args)
{
Queue queue = new Queue();
queue.Watch();
while (true)
{
}
}
public void Watch()
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = "directorypath\\";
watcher.NotifyFilter = NotifyFilters.LastWrite;
watcher.Filter = "*.*";
watcher.Changed += new FileSystemEventHandler(OnChanged);
watcher.EnableRaisingEvents = true;
}
public void OnChanged(object source, FileSystemEventArgs e)
{
WaitForFile(e.FullPath);
thread = new Thread(new ThreadStart(this.start));
thread.Start();
thread.Join();
}
private static void WaitForFile(string fullPath)
{
while (true)
{
try
{
using (StreamReader stream = new StreamReader(fullPath))
{
stream.Close();
break;
}
}
catch
{
Thread.Sleep(1000);
}
}
}
public void start()
{
Console.WriteLine("test" +counter);
counter++;
}

Following this article MSDN, Created event can solve your problem
The
M:System.IO.FileSystemWatcher.OnCreated(System.IO.FileSystemEventArgs)
event is raised as soon as a file is created. If a file is being
copied or transferred into a watched directory, the
M:System.IO.FileSystemWatcher.OnCreated(System.IO.FileSystemEventArgs)
event will be raised immediately
public void Watch()
{
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = "directorypath\\";
watcher.NotifyFilter = NotifyFilters.LastWrite;
watcher.Created += new FileSystemEventHandler(OnChanged);
watcher.EnableRaisingEvents = true;
}

Process items in parallel

I have a FileSystemWatcher which looks for files in a folder. Moment the file is created a Created event is fired. I am adding each created file name to a queue.
void Main(){
FileSystemWatcher fsw = new FileSystemWatcher();
fsw.Path = System.Configuration.ConfigurationManager.AppSettings["PathToDataFolder"];
//fsw.Filter = "*.docx";
fsw.EnableRaisingEvents = true;
fsw.IncludeSubdirectories = true;
fsw.Created += new FileSystemEventHandler(fsw_Created);
}
private void fsw_Created(object sender, FileSystemEventArgs e)
{
queue.Enqueue(e.FullPath);
}
Overtime the files will increase and the queue will become large. So the queue is dynamic. I want to process each file in parallel. But I dont want to process a lot of files at once because that is quite resource intensive. Once a file is processed i want to dequeue it and pick another file to process.
How can i achieve this in C#?

You can use an ActionBlock with a configurable degree of parallelism. By default, an ActionBlock uses only one Task to process incoming messages. You can make it use multiple tasks to process files in parallel. The FSW's event should post paths directly to the block :
ActionBlock<string> _block;
void Main()
{
...
var options= new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 4
};
_block=new ActionBlock<string>(path=>MyPathProcessingFunction(path), options);
//Configure the FSW as before
}
private void fsw_Created(object sender, FileSystemEventArgs e)
{
_block.Post(e.FullPath);
}

You can achieve that by using the producer/consumer pattern. In .Net the BlockingCollection class provides support for this pattern. Whenever the event handler is triggered, it will add the path to the queue and also creates a new task to process the queue. So for every monitored file a new task is created. You can change the task creation strategy if you want, and also manage how they are scheduled with the TaskScheduler.
public class Watcher
{
public Watcher()
{
_queue = new BlockingCollection<string>();
}
private BlockingCollection<string> _queue;
public void Start()
{
FileSystemWatcher fsw = new FileSystemWatcher();
fsw.Path = #"F:\a";
fsw.EnableRaisingEvents = true;
fsw.IncludeSubdirectories = true;
fsw.Created += Fsw_Created;
}
private void Fsw_Created(object sender, FileSystemEventArgs e)
{
_queue.Add(e.FullPath);
Task.Factory.StartNew(() =>
{
var path = _queue.Take();
// process the queue here
});
}
}

You are looking for producer / consumer pattern which in C# can be implemented via BlockingCollection, e.g.
private static async Task Perform() {
// Be careful with this parameter: what do you expect the system
// to do if the pipeline contains pipelineMaxLength items?
int pipelineMaxLength = 100;
int consumersCount = 10;
using (BlockingCollection<string> pipeline =
new BlockingCollection<string>(pipelineMaxLength)) {
// Producer(s)
using (FileSystemWatcher fsw = new FileSystemWatcher()) {
...
fsw.Created += (s, e) => {
// whenever new file has been created, add it to the pipeline
if (!pipeline.IsAddingCompleted)
pipeline.Add(e.FullPath);
// Whenever you have no files to add and you want quit processing call
// pipeline.CompleteAdding();
};
// Consumers (consumersCount of them are working in parallel)
var consumers = Enumerable
.Range(0, consumersCount) //
.Select(index => Task.Run(() => {
// each consumer extracts file from the pipeline and processes it
foreach (var file in pipeline.GetConsumingEnumerable()) {
//TODO: process the file here
} }))
.ToArray();
// (a)wait until all the consumers finish their work
await Task
.WhenAll(consumers);
}
}
}

How to determine whether a folder has finished copying c#

I have a question: How do I determine whether a folder has finished copying from one location to another?
At the moment my FileSystemWatcher triggers several events as soon as a file within the directory being copied. What I want though, is one single event to be triggered when all the files within that folder has been successfully copied. My code right now looks like this:
static void Main(string[] args)
{
String path = #"D:\Music";
FileSystemWatcher mWatcher = new FileSystemWatcher();
mWatcher.Path = path;
mWatcher.NotifyFilter = NotifyFilters.LastAccess;
mWatcher.NotifyFilter = mWatcher.NotifyFilter | NotifyFilters.LastWrite;
mWatcher.NotifyFilter = mWatcher.NotifyFilter | NotifyFilters.DirectoryName;
mWatcher.IncludeSubdirectories = true;
mWatcher.Created += new FileSystemEventHandler(mLastChange);
mWatcher.Changed += new FileSystemEventHandler(mLastChange);
mWatcher.EnableRaisingEvents = true;
Console.WriteLine("Watching path: " + path);
String exit;
while (true)
{
exit = Console.ReadLine();
if (exit == "exit")
break;
}
}
private static void mLastChange(Object sender, FileSystemEventArgs e)
{
Console.WriteLine(e.ChangeType + " " + e.FullPath);
}

Unfortunately FileSystemWatcher doesn't tell you when a file is finished writing. So your options are...
Set a timeout after last write when it is assumed there are no more changes coming
Have the writing application put a lock file of some variety that tells any other program that it's done.
After re-reading your question... it doesn't sound like you have any control over the other application.
So you will need some kind of timeout value that determines when all the writing is done. Basically create a timer that resets after each filesystemwatcher event... when it times out then you fire the single event that says it's done.
Here is how you could add it to your code...
static void Main(string[] args)
{
Timer.Interval = 5000; // 5 seconds - change to whatever is appropriate
Timer.AutoReset = false;
Timer.Elapsed += TimeoutDone;
String path = #"D:\Music";
FileSystemWatcher mWatcher = new FileSystemWatcher();
mWatcher.Path = path;
mWatcher.NotifyFilter = NotifyFilters.LastAccess;
mWatcher.NotifyFilter = mWatcher.NotifyFilter | NotifyFilters.LastWrite;
mWatcher.NotifyFilter = mWatcher.NotifyFilter | NotifyFilters.DirectoryName;
mWatcher.IncludeSubdirectories = true;
mWatcher.Created += new FileSystemEventHandler(mLastChange);
mWatcher.Changed += new FileSystemEventHandler(mLastChange);
mWatcher.EnableRaisingEvents = true;
Console.WriteLine("Watching path: " + path);
Timer.Start();
String exit;
while (true)
{
exit = Console.ReadLine();
if (exit == "exit")
break;
}
}
private static Timer Timer = new Timer();
private static void TimeoutDone(object source, ElapsedEventArgs e)
{
Console.WriteLine("Timer elapsed!");
}
private static void mLastChange(Object sender, FileSystemEventArgs e)
{
Console.WriteLine(e.ChangeType + " " + e.FullPath);
if (Timer != null)
{
Timer.Stop();
Timer.Start();
}
}

It's horribly cheesy, but in the past I have dealt with this problem by creating a custom decorator for the FileSystemWatcher class. Internally it creates a FileSystemWatcher and registers for the Created and Changed events, wraps them, and throws its own Created and Changed events after the files are finished, similar to this:
private void Watcher_Changed(Object sender, FileSystemEVentArgs e)
{
while (true)
{
FileStream stream;
try
{
stream = File.Open(e.FullPath, FileMode.Open, FileAccess.ReadWrite, FileShare.None);
// If this succeeds, the file is finished
Changed();
}
catch (IOException)
{
}
finally
{
if (stream != null) stream.Close();
}
}
}
Some of this is drawn from the answer here. In reality, you shouldn't use an infinite loop. You probably want to add timeouts, sleep in between checks, etc. but this is the general idea.

If the destination is a local folder, you can use the filesystem filter driver to track file create and file close events. Knowing when all files, previously created, are closed will inform you that copying is complete.

I have created a Git repo with a class that extends FileSystemWatcher to trigger the events only when copy is done.
Download FileSystemSafeWatcher and add it to your project.
Then use it as a normal FileSystemWatcher and monitor when the events are triggered.
var fsw = new FileExamSystemWatcher(file);
fsw.EnableRaisingEvents = true;
// Add event handlers here
fsw.Created += fsw_Created;

Unfortunatly there is no ready solution. but I designed a simple tricky solution to trigger (copy finished) event.
you should use timer.
FileSystemWatcher fsw = new FileSystemWatcher();
string fullPath = "";
DateTime tempTime;
fsw.Path = #"C:\temp";
private void startwatching()
{
timer1.Start();
}
fsw.EnableRaisingEvents = true;
fsw.Created += Fsw_Created;
private void Fsw_Created(object sender, FileSystemEventArgs e)
{
tempTime = DateTime.Now.AddSeconds(-4);
fullPath = e.FullPath;
}
private void timer1_Tick(object sender, EventArgs e)
{
if (fullPath!=string.Empty)
{
timer1.Stop();
if (tempTime >= Directory.GetLastAccessTime(fullPath))
{
DirectoryInfo di = new DirectoryInfo(fullPath);
listBox1.Items.Add("Folder " + di.Name + " finished copying");
fullPath = string.Empty;
}
else
{
tempTime = DateTime.Now;
}
timer1.Start();
}
}

Directory scanner Windows Service with FileSystemWatcher

I'm trying to create a windows service that will grab new files uploaded to directory, edit them and move to other directory. It seems I can COPY them but not MOVE. Why is that?
using System;
using System.ServiceProcess;
using System.Threading;
using System.IO;
namespace ImportService
{
public class ImportServer : ServiceBase
{
private System.Diagnostics.EventLog eventLog1;
private FileSystemWatcher watcher;
public ImportServer()
{
this.ServiceName = "ImportService";
this.CanHandlePowerEvent = true;
this.CanHandleSessionChangeEvent = true;
this.CanPauseAndContinue = true;
this.CanShutdown = true;
this.CanStop = true;
this.AutoLog = true;
InitializeComponent();
if (!System.Diagnostics.EventLog.SourceExists("ImportServiceLogSource"))
System.Diagnostics.EventLog.CreateEventSource("ImportServiceLogSource", "ImportServiceLog");
eventLog1.Source = "ImportServiceLogSource";
eventLog1.Log = "ImportServiceLog";
}
public static void Main()
{
ServiceBase.Run(new ImportServer());
}
protected override void OnStart(string[] args)
{
//base.OnStart(args);
eventLog1.WriteEntry("service started");
watcher = new FileSystemWatcher();
watcher.Path = "C:\\INPUT\\";
watcher.Filter = "*.jpg";
watcher.EnableRaisingEvents = true;
watcher.Created += new FileSystemEventHandler(OnCreated);
}
private void OnCreated(object sender, FileSystemEventArgs e)
{
String output_dir = "C:\\OUTPUT\\";
String output_file = Path.Combine(output_dir, e.Name);
File.Move(e.FullPath, output_file);
// File.Copy() works here
eventLog1.WriteEntry("moving file to " + output_file);
}
protected override void OnStop()
{
eventLog1.WriteEntry("service stopped");
base.OnStop();
}
protected override void OnContinue()
{
base.OnContinue();
}
protected override void OnPause()
{
base.OnPause();
}
private void InitializeComponent()
{
this.eventLog1 = new System.Diagnostics.EventLog();
((System.ComponentModel.ISupportInitialize)(this.eventLog1)).BeginInit();
((System.ComponentModel.ISupportInitialize)(this.eventLog1)).EndInit();
}
}
}
Also should I keep the base.OnStart(); etc. What does it really do?
UPDATE: How to move file that was created in the watched directory? File already in use exception problem.

You'll have to catch the IOException and make the thread sleep for a few bit then try again.
private void OnCreated(object sender, FileSystemEventArgs e)
{
String output_dir = "C:\\OUTPUT\\";
String output_file = Path.Combine(output_dir, e.Name);
while (true)
{
try
{
File.Move(e.FullPath, output_file);
break;
}
catch (IOException)
{
//sleep for 100 ms
System.Threading.Thread.Sleep(100);
}
}
eventLog1.WriteEntry("moving file to " + output_file);
}
That being said, there are a ton of problems with this. You're going to be better off having a timer event that gets called every few seconds that looks for files in the folder. If you get an IOException, you just move on. The file will still be there for processing on th next iteration (assuming the upload has finished). Can give you an example if what I'm talking about if needed.

Your implementation of OnCreated needs to handle exceptions if you are to avoid the service crashing. Lots of things might go wrong when handling the file, and your service needs to recover gracefully when they do.
One possibility here is that the OnCreated event may fire before the process writing the new file has finished writing it, and your attempt to move the file is therefore throwing an exception.

As I understand, you need to wait until the file transfer is complete and the file is closed, and only then you can move the file.
This has been discussed before on SO several times. Eg. here and here.