We Keep Coding

misunderstanding of concurrentQueue, a single consumer working from the queue on it's own thread

I'm having trouble creating a functioning SystemFileWatcher that takes the created event and stores it in a queue for a separate thread to work from. I've read countless threads here regarding this issue but I can't get my head around this particular problem.
using System;
using System.IO;
using System.Collections.Generic;
using System.Collections.Concurrent;
using System.Collections;
using System.Threading;
namespace FileSystemWatcherTest
{
class Program
{
public static BlockingCollection<string> processCollection = new BlockingCollection<string>(new ConcurrentQueue<string>());
static void Main(string[] args)
{
string path = #"C:\test\";
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = path;
watcher.EnableRaisingEvents = true;
watcher.Filter = "*.*";
watcher.Created += new FileSystemEventHandler(onCreated);
Thread Consumer = new Thread(new ThreadStart(mover));
Consumer.Start();
while (true) ;//run infinite loop so program doesn't terminate untill we force it.
}
static void onCreated(object sender, FileSystemEventArgs e)
{
processCollection.Add(e.FullPath);
}
static void mover()
{
string current;
string processed = #"C:\test\processed\";
while (true)
{
while (processCollection.IsCompleted)
{
Thread.Sleep(1000);
}
while (processCollection.TryTake(out current))
{
System.IO.File.Move(current, processed);
}
}
}
}
}
This is what I'd like to test. I'm aware this does not work. I've verified the FSW works when I'm simply writing to console when a file is placed inside the queue. My problem begins around when I try to start the mover function in it's own thread. The mover function and onCreated do not appear to communicate once I start working off the queue.
My expectation of this code is to start the mover function in it's own thread and run it alongside the SFW. My expectation is that the concurrentqueue attached to the blockingcollection auto updates (I enqueue an item through onCreated, the mover sees that it now has +1 to that queue. The mover takes one from the queue, the onCreated sees this.) I'm probably using the Thread.Sleep incorrectly. I no longer have a supporting reason for using blockingcollection (which I chose at first to handle waiting for the queue to fill up and, basically, constantly check the queue for an item to process) and am open to changing this to whatever might work. I've seen use of locks, but from what I understand this is not really necessary due to how concurrentQueue synchronizes.
The ultimate goal is to process large quantities of small files that come in at random times and can range from 1 to several hundred at any given time. These files are .EMLs.
If at all possible, I would greatly appreciate an explanation of what is happening and what a suggestion would be to get around this problem. I come humbly and expect to be told everything I understand is incorrect!
edit: I'm testing this as a console application but it will be used as a service afterwards. I added the while (true) ; before onCreated() to keep FSW running.

You have several different problems in your code example:
You are misusing the File.Move() method. It requires both parameters to be the full file name. You are passing the directory name as the second parameter, which is incorrect.
You are inspecting the IsCompleted property of the collection, as if that would be useful. It will always be false, and so that block of code does nothing. This leads to the next problem…
Your thread is running in a tight loop, consuming massive amounts of CPU time. This may or may not cause errors, but it could…FileSystemWatcher is not actually guaranteed to always report changes, and one of the reasons it might not is if it can't get enough CPU time to monitor the file system. If you starve it by using up all the CPU time, you might find it simply doesn't report a change. Note that this problem exists in your primary thread too; it also is running in a tight loop, consuming massive amounts of CPU time doing nothing. So you are completely occupying two cores of your system.
You are failing to take advantage of the producer/consumer model of execution that BlockingCollection is designed for. You should be having your worker thread enumerate the enumeration returned by GetConsumingEnumerable(), using the CompleteAdding() method to signal to that thread that there is no more work.
Here is a version of your code example that corrects the above mistakes, as well as cleans up the example a bit so that it's more self-contained:
// The default backing collection for BlockingCollection<T>
// is ConcurrentQueue<T>. There's no need to specify that
// explicitly.
public static BlockingCollection<string> processCollection = new BlockingCollection<string>();
static void Main(string[] args)
{
string testDirectory = Path.Combine(Environment.CurrentDirectory, "test");
Console.WriteLine("Creating directory: \"{0}\"", testDirectory);
Directory.CreateDirectory(testDirectory);
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.Path = testDirectory;
watcher.EnableRaisingEvents = true;
watcher.Filter = "*.*";
watcher.Created += new FileSystemEventHandler(onCreated);
Thread Consumer = new Thread(new ParameterizedThreadStart(mover));
Consumer.Start(testDirectory);
string text;
while ((text = Console.ReadLine()) != "")
{
string newFile = Path.Combine(testDirectory, text + ".txt");
File.WriteAllText(newFile, "Test file");
}
processCollection.CompleteAdding();
}
static void onCreated(object sender, FileSystemEventArgs e)
{
if (e.ChangeType == WatcherChangeTypes.Created)
{
processCollection.Add(e.FullPath);
}
}
static void mover(object testDirectory)
{
string processed = Path.Combine((string)testDirectory, "processed");
Console.WriteLine("Creating directory: \"{0}\"", processed);
Directory.CreateDirectory(processed);
foreach (string current in processCollection.GetConsumingEnumerable())
{
// Ensure that the file is in fact a file and not something else.
if (File.Exists(current))
{
System.IO.File.Move(current, Path.Combine(processed, Path.GetFileName(current)));
}
}
}

How can I reliably move files (asynchronously) from a directory as they're being created?

In my Windows Service solution have a FileSystemWatcher monitoring a directory tree for new files, and whenever it fires a Created event I am trying to move the files asynchronously to another server for further processing. Here's the code:
foreach (string fullFilePath in
Directory.EnumerateFiles(directoryToWatch, "*.*",
SearchOption.AllDirectories)
.Where(filename => fileTypes.Contains(Path.GetExtension(filename))))
{
string filename = Path.GetFileName(fullFilePath);
using (FileStream sourceStream = File.Open(filename, FileMode.Open, FileAccess.Read))
{
using (FileStream destStream = File.Create(Path.Combine(destination, filename)))
{
await sourceStream.CopyToAsync(destStream);
}
}
}
The problem is that as these files are being copied into the folder I'm watching, they're not always unlocked and available to me. I want to "retry" when I hit a locked file, but I'm not accustomed to thinking asynchronously, so I have no idea how to put the error'ed file back in the queue.

First of all you need to 'detect' the exceptions thrown in process of asynchronous execution. This can be done by something like this:
try
{
await sourceStream.CopyToAsync(destStream);
}
catch (Exception copyException)
{
}
Once an exception is detected and properly handled i.e. you decide that one particular exception is a reason for a retry, you will have to maintain your own queue of copy targets (and destinations) that are due for a retry.
Then you will have to organize a new entry point that would lead to a retry itself. Such an entry point could be triggered by a timer or a next event from the file system monitor you use (which I would not recommend). You will also have to implement a detection for an overflow of your queue for a case of multiple failures. Keep in mind that such overflow detection is also present in the file system monitor which can simply skip a notification if there are too many system events (many files appear to be copied into the monitored folders at once).
If these matters do not bother you much I would suggest you to implement a timer or to be more precise a timeout in order to retry the copy task.
If on the other hand you need a robust solution I would implement a file system monitor myself.
Concerning the timeout it could look like this:
private Queue<myCopyTask> queue;
private Timer retryTimeout;
public Program()
{
retryTimeout = new Timer(QueueProcess, null, Timeout.Infinite, Timeout.Infinite);
}
private void FileSystemMonitorEventhandler()
{
//New tasks are provided by the file system monitor.
myCopyTask newTask = new myCopyTask();
newTask.sourcePath = "...";
newTask.destinationPath = "...";
//Keep in mind that queue is touched from different threads.
lock (queue)
{
queue.Enqueue(newTask);
}
//Keep in mind that Timer is touched from different threads.
lock (retryTimeout)
{
retryTimeout.Change(1000, Timeout.Infinite);
}
}
//Start this routine only via Timer.
private void QueueProcess(object iTimeoutState)
{
myCopyTask task = null;
do
{
//Keep in mind that queue is touched from different threads.
lock (queue)
{
if (queue.Count > 0)
{
task = queue.Dequeue();
}
}
if (task != null)
{
CopyTaskProcess(task);
}
} while (task != null);
}
private async void CopyTaskProcess(myCopyTask task)
{
FileStream sourceStream = null;
FileStream destStream = null;
try
{
sourceStream = File.OpenRead(task.sourcePath);
destStream = File.OpenWrite(task.destinationPath);
await sourceStream.CopyToAsync(destStream);
}
catch (Exception copyException)
{
task.retryCount++;
//In order to avoid instant retries on several problematic tasks you probably
//should involve a mechanism to delay retries. Keep in mind that this approach
//delays worker thread that is implicitly involved by await keyword.
Thread.Sleep(100);
//Keep in mind that queue is touched from different threads.
lock (queue)
{
queue.Enqueue(task);
}
//Keep in mind that Timer is touched from different threads.
lock (retryTimeout)
{
retryTimeout.Change(1000, Timeout.Infinite);
}
}
finally
{
if (sourceStream != null)
{
sourceStream.Close();
}
if (destStream != null)
{
destStream.Close();
}
}
}
}
internal class myCopyTask
{
public string sourcePath;
public string destinationPath;
public long retryCount;
}

Threads interaction (data from one thread to another) c#

I need to pass information from thread of scanning data to recording information thread(write to xml file).
It should looks something like this:
Application.Run() - complete
Scanning thread - complete
Writing to xlm thread - ???
UI update thread - I think I did it
And what i got now:
private void StartButtonClick(object sender, EventArgs e)
{
if (FolderPathTextBox.Text == String.Empty || !Directory.Exists(FolderPathTextBox.Text)) return;
{
var nodeDrive = new TreeNode(FolderPathTextBox.Text);
FolderCatalogTreeView.Nodes.Add(nodeDrive);
nodeDrive.Expand();
var t1 = new Thread(() => AddDirectories(nodeDrive));
t1.Start();
}
}
private void AddDirectories(TreeNode node)
{
string strPath = node.FullPath;
var dirInfo = new DirectoryInfo(strPath);
DirectoryInfo[] arrayDirInfo;
FileInfo[] arrayFileInfo;
try
{
arrayDirInfo = dirInfo.GetDirectories();
arrayFileInfo = dirInfo.GetFiles();
}
catch
{
return;
}
//Write data to xml file
foreach (FileInfo fileInfo in arrayFileInfo)
{
WriteXmlFolders(null, fileInfo);
}
foreach (DirectoryInfo directoryInfo in arrayDirInfo)
{
WriteXmlFolders(directoryInfo, null);
}
foreach (TreeNode nodeFil in arrayFileInfo.Select(file => new TreeNode(file.Name)))
{
FolderCatalogTreeView.Invoke(new ThreadStart(delegate { node.Nodes.Add(nodeFil); }));
}
foreach (TreeNode nodeDir in arrayDirInfo.Select(dir => new TreeNode(dir.Name)))
{
FolderCatalogTreeView.Invoke(new ThreadStart(delegate
{node.Nodes.Add(nodeDir);
}));
StatusLabel.BeginInvoke(new MethodInvoker(delegate
{
//UI update...some code here
}));
AddDirectories(nodeDir);
}
}
private void WriteXmlFolders(DirectoryInfo dir, FileInfo file)
{//writing information into the file...some code here}
How to pass data from AddDirectories(recursive method) thread to WriteXmlFolders thread?

Here is a generic mechanism how one thread generates data that another thread consumes. No matter what approach (read: ready made classes) you would use the internal principle stays the same. The main players are (note that there are many locking classes available in System.Threading namespace that could be used but these are the most appropriate for this scenario:
AutoResetEvent - this allows a thread to go into sleep mode (without consuming resources) until another thread will wake it up. The 'auto' part means that once the thread wakes up, the class is reset so the next Wait() call will again put it in sleep, without the need to reset anything.
ReaderWriterLock or ReaderWriterLockSlim (recommended to use the second if you are using .NET 4) - this allows just one thread to lock for writing data but multiple threads can read the data. In this particular case there is only one reading thread but the approach would not be different if there were many.
// The mechanism for waking up the second thread once data is available
AutoResetEvent _dataAvailable = new AutoResetEvent();
// The mechanism for making sure that the data object is not overwritten while it is being read.
ReaderWriterLockSlim _readWriteLock = new ReaderWriterLockSlim();
// The object that contains the data (note that you might use a collection or something similar but anything works
object _data = null;
void FirstThread()
{
while (true)
{
// do something to calculate the data, but do not store it in _data
// create a lock so that the _data field can be safely updated.
_readWriteLock.EnterWriteLock();
try
{
// assign the data (add into the collection etc.)
_data = ...;
// notify the other thread that data is available
_dataAvailable.Set();
}
finally
{
// release the lock on data
_readWriteLock.ExitWriteLock();
}
}
}
void SecondThread()
{
while (true)
{
object local; // this will hold the data received from the other thread
// wait for the other thread to provide data
_dataAvailable.Wait();
// create a lock so that the _data field can be safely read
_readWriteLock.EnterReadLock();
try
{
// read the data (add into the collection etc.)
local = _data.Read();
}
finally
{
// release the lock on data
_readWriteLock.ExitReadLock();
}
// now do something with the data
}
}
In .NET 4 it is possible to avoid using ReadWriteLock and use one of the concurrency-safe collections such as ConcurrentQueue which will internally make sure that reading/writing is thread safe. The AutoResetEvent is still needed though.
.NET 4 provides a mechanism that could be used to avoid the need of even AutoResetEvent - BlockingCollection - this class provides methods for a thread to sleep until data is available. MSDN page contains example code on how to use it.

In case you use it as the answer
Take a look at a producer consumer.
BlockingCollection Class
How to: Implement Various Producer-Consumer Patterns

AutoResetEvent Reset Method

super simple question, but I just wanted some clarification. I want to be able to restart a thread using AutoResetEvent, so I call the following sequence of methods to my AutoResetEvent.
setupEvent.Reset();
setupEvent.Set();
I know it's really obvious, but MSDN doesn't state in their documentation that the Reset method restarts the thread, just that it sets the state of the event to non-signaled.
UPDATE:
Yes the other thread is waiting at WaitOne(), I'm assuming when it gets called it will resume at the exact point it left off, which is what I don't want, I want it to restart from the beginning. The following example from this valuable resource illustrates this:
static void Main()
{
new Thread (Work).Start();
_ready.WaitOne(); // First wait until worker is ready
lock (_locker) _message = "ooo";
_go.Set(); // Tell worker to go
_ready.WaitOne();
lock (_locker) _message = "ahhh"; // Give the worker another message
_go.Set();
_ready.WaitOne();
lock (_locker) _message = null; // Signal the worker to exit
_go.Set();
}
static void Work()
{
while (true)
{
_ready.Set(); // Indicate that we're ready
_go.WaitOne(); // Wait to be kicked off...
lock (_locker)
{
if (_message == null) return; // Gracefully exit
Console.WriteLine (_message);
}
}
}
If I understand this example correctly, notice how the Main thread will resume where it left off when the Work thread signals it, but in my case, I would want the Main thread to restart from the beginning.
UPDATE 2:
#Jaroslav Jandek - It's quite involved, but basically I have a CopyDetection thread that runs a FileSystemWatcher to monitor a folder for any new files that are moved or copied into it. My second thread is responsible for replicating the structure of that particular folder into another folder. So my CopyDetection thread has to block that thread from working while a copy/move operation is in progress. When the operation completes, the CopyDetection thread restarts the second thread so it can re-duplicate the folder structure with the newly added files.
UPDATE 3:
#SwDevMan81 - I actually didn't think about that and that would work save for one caveat. In my program, the source folder that is being duplicated is emptied once the duplication process is complete. That's why I have to block and restart the second thread when new items are added to the source folder, so it can have a chance to re-parse the folder's new structure properly.
To address this, I'm thinking of maybe adding a flag that signals that it is safe to delete the source folder's contents. Guess I could put the delete operation on it's own Cleanup thread.
#Jaroslav Jandek - My apologies, I thought it would be a simple matter to restart a thread on a whim. To answer your questions, I'm not deleting the source folder, only it's content, it's a requirement by my employer that unfortunately I cannot change. Files in the source folder are getting moved, but not all of them, only files that are properly validated by another process, the rest must be purged, i.e. the source folder is emptied. Also, the reason for replicating the source folder structure is that some of the files are contained within a very strict sub-folder hierarchy that must be preserved in the destination directory. Again sorry for making it complicated. All of these mechanisms are in place, have been tested and are working, which is why I didn't feel the need to elaborate on them. I only need to detect when new files are added so I may properly halt the other processes while the copy/move operation is in progress, then I can safely replicate the source folder structure and resume processing.

So thread 1 monitors and thread 2 replicates while other processes modify the monitored files.
Concurrent file access aside, you can't continue replicating after a change. So a successful replication only occurs when there is long enough delay between modifications. Replication cannot be stopped immediately since you replicate in chunks.
So the result of monitoring should be a command (file copy, file delete, file move, etc.).
The result of a successful replication should be an execution of a command.
Considering multiple operations can occur, you need a queue (or queued dictionary - to only perform 1 command on a file) of commands.
// T1:
somethingChanged(string path, CT commandType)
{
commandQueue.AddCommand(path, commandType);
}
// T2:
while (whatever)
{
var command = commandQueue.Peek();
if (command.Execute()) commandQueue.Remove();
else // operation failed, do what you like.
}
Now you may ask how to create a thread-safe query, but that probably belongs to another question (there are many implementations on the web).
EDIT (queue-less version with whole dir replication - can be used with query):
If you do not need multiple operations (eg. always replication the whole directory) and expect the replication to always finish or fail and cancel, you can do:
private volatile bool shouldStop = true;
// T1:
directoryChanged()
{
// StopReplicating
shouldStop = true;
workerReady.WaitOne(); // Wait for the worker to stop replicating.
// StartReplicating
shouldStop = false;
replicationStarter.Set();
}
// T2:
while (whatever)
{
replicationStarter.WaitOne();
... // prepare, throw some shouldStops so worker does not have to work too much.
if (!shouldStop)
{
foreach (var file in files)
{
if (shouldStop) break;
// Copy the file or whatever.
}
}
workerReady.Set();
}

I think this example clarifies (to me anyway) how reset events work:
var resetEvent = new ManualResetEvent(false);
var myclass = new MyAsyncClass();
myclass.MethodFinished += delegate
{
resetEvent.Set();
};
myclass.StartAsyncMethod();
resetEvent.WaitOne(); //We want to wait until the event fires to go on
Assume that MyAsyncClass runs the method on a another thread and fires the event when complete.
This basically turns the asynchronous "StartAsyncMethod" into a synchronous one. Many times I find a real-life example more useful.
The main difference between AutoResetEvent and ManualResetEvent, is that using AutoResetEvent doesn't require you to call Reset(), but automatically sets the state to "false". The next call to WaitOne() blocks when the state is "false" or Reset() has been called.

You just need to make it loop like the other Thread does. Is this what you are looking for?
class Program
{
static AutoResetEvent _ready = new AutoResetEvent(false);
static AutoResetEvent _go = new AutoResetEvent(false);
static Object _locker = new Object();
static string _message = "Start";
static AutoResetEvent _exitClient = new AutoResetEvent(false);
static AutoResetEvent _exitWork = new AutoResetEvent(false);
static void Main()
{
new Thread(Work).Start();
new Thread(Client).Start();
Thread.Sleep(3000); // Run for 3 seconds then finish up
_exitClient.Set();
_exitWork.Set();
_ready.Set(); // Make sure were not blocking still
_go.Set();
}
static void Client()
{
List<string> messages = new List<string>() { "ooo", "ahhh", null };
int i = 0;
while (!_exitClient.WaitOne(0)) // Gracefully exit if triggered
{
_ready.WaitOne(); // First wait until worker is ready
lock (_locker) _message = messages[i++];
_go.Set(); // Tell worker to go
if (i == 3) { i = 0; }
}
}
static void Work()
{
while (!_exitWork.WaitOne(0)) // Gracefully exit if triggered
{
_ready.Set(); // Indicate that we're ready
_go.WaitOne(); // Wait to be kicked off...
lock (_locker)
{
if (_message != null)
{
Console.WriteLine(_message);
}
}
}
}
}

How to force FileSystemWatcher to wait till the file downloaded?

I am downloading a file and want to execute the install only after the download is complete. How do I accomplish this? Seems like FileSystemWatcher onCreate event would do this but this happens in a different thread, is there a simple way to force the waiting part to happen in the same thread.
Code I have so far
FileSystemWatcher w = new FileSystemWatcher(#"C:/downloads");
w.EnableRaisingEvents = true;
w.Created += new FileSystemEventHandler(FileDownloaded);
static void FileDownloaded(object source, FileSystemEventArgs e)
{
InstallMSI(e.FullPath);
}
I looked at SynchronizingObject and WaitForChangedResult but didn't get a solid working sample.

Try:
FileInfo fInfo = new FileInfo(e.FullPath);
while(IsFileLocked(fInfo)){
Thread.Sleep(500);
}
InstallMSI(e.FullPath);
static bool IsFileLocked(FileInfo file)
{
FileStream stream = null;
try {
stream = file.Open(FileMode.Open, FileAccess.ReadWrite, FileShare.None);
}
catch (IOException) {
return true;
}
finally {
if (stream != null)
stream.Close();
}
return false;
}

If you insist on using FileSystemWatcher you would probably have to account for the fact that a file of some size isn't created (uploaded) in one single operation. The filesystem is likely to produce 1 created and x changed events before the file is ready for use.
You could catch the created events and create new (dedicated) threads (unless you already have an ongoing thread for that file) in which you loop and periodically try to open the file exclusively. If you succeed, the file is ready.

One technique would be to download to the temporary directory, and then move it into C:/downloads once it was complete.

If you are using WebClient to download, you can use set the client's DownloadFileCompleted eventhandler.
If you do it this way you can also use client.DownloadFileAsync() to make it download asynchronously.