We Keep Coding

Detect Duplicate Items in DataFlow

I've been building out a service that processes files using a Queue<string> object to manage the items.
public partial class BasicQueueService : ServiceBase
{
private readonly EventWaitHandle completeHandle =
new EventWaitHandle(false, EventResetMode.ManualReset, "ThreadCompleters");
public BasicQueueService()
{
QueueManager = new Queue<string>();
}
public bool Stopping { get; set; }
private Queue<string> QueueManager { get; }
protected override void OnStart(string[] args)
{
Stopping = false;
ProcessFiles();
}
protected override void OnStop()
{
Stopping = true;
}
private void ProcessFiles()
{
while (!Stopping)
{
var count = QueueManager.Count;
for (var i = 0; i < count; i++)
{
//Check the Stopping Variable again.
if (Stopping) break;
var fileName = QueueManager.Dequeue();
if (string.IsNullOrWhiteSpace(fileName) || !File.Exists(fileName))
continue;
Console.WriteLine($"Processing {fileName}");
Task.Run(() =>
{
DoWork(fileName);
})
.ContinueWith(ThreadComplete);
}
if (Stopping) continue;
Console.WriteLine("Waiting for thread to finish, or 1 minute.");
completeHandle.WaitOne(new TimeSpan(0, 0, 15));
completeHandle.Reset();
}
}
partial void DoWork(string fileName);
private void ThreadComplete(Task task)
{
completeHandle.Set();
}
public void AddToQueue(string file)
{
//Called by FileWatcher/Manual classes, not included for brevity.
lock (QueueManager)
{
if (QueueManager.Contains(file)) return;
QueueManager.Enqueue(file);
}
}
}
Whilst researching how to limit the number of threads on this (I've tried a manual class with an incrementing int, but there's an issue where it doesn't decrement properly in my code), I came across TPL DataFlow, which seems like its a better fit for what I'm trying to achieve - specifically, it allows me to let the framework handle threading/queueing, etc.
This is now my service:
public partial class BasicDataFlowService : ServiceBase
{
private readonly ActionBlock<string> workerBlock;
public BasicDataFlowService()
{
workerBlock = new ActionBlock<string>(file => DoWork(file), new ExecutionDataflowBlockOptions()
{
MaxDegreeOfParallelism = 32
});
}
public bool Stopping { get; set; }
protected override void OnStart(string[] args)
{
Stopping = false;
}
protected override void OnStop()
{
Stopping = true;
}
partial void DoWork(string fileName);
private void AddToDataFlow(string file)
{
workerBlock.Post(file);
}
}
This works well. However, I want to ensure that a file is only ever added to the TPL DataFlow once. With the Queue, I can check that using .Contains(). Is there a mechanism that I can use for TPL DataFlow?

Your solution with Queue works only if file goes into your service twice in a small period of time. If it came again in, say, few hours, queue will not contain it, as you Dequeue it from there.
If this solution is expected, then you may use a MemoryCache to store file paths being already handled, like this:
using System.Runtime.Caching;
private static object _lock = new object();
private void AddToDataFlow(string file)
{
lock (_lock)
{
if (MemoryCache.Default.Contains(file))
{
return;
}
// no matter what to put into the cache
MemoryCache.Default[file] = true;
// we can now exit the lock
}
workerBlock.Post(file);
}
However, if your application must run for a long time (which service is intended to do), you'll eventually run out of memory. In that case you probably need to store your file paths in database or something, so even after restarting the service your code will restore the state.

You can check it inside of DoWork.
You have to save in Hash already works items and check current filename doesn't exist in hash.

How to make a slightly modified AutoResetEvent class?

I need a synchronizing class that behaves exactly like the AutoResetEvent class, but with one minor exception:
A call to the Set() method must release all waiting threads, and not just one.
How can I construct such a class? I am simply out of ideas?
Martin.

So you have multiple threads doing a .WaitOne() and you want to release them?
Use the ManualResetEvent class and all the waiting threads should release...

Thank you very much for all your thougts and inputs which I have read with great interest. I did some more searching here on Stackoverflow, and suddenly I found this, whcih turned out to be just what I was looking for. By cutting it down to just the two methods I need, I ended up with this small class:
public sealed class Signaller
{
public void PulseAll()
{
lock (_lock)
{
Monitor.PulseAll(_lock);
}
}
public bool Wait(TimeSpan maxWaitTime)
{
lock (_lock)
{
return Monitor.Wait(_lock, maxWaitTime);
}
}
private readonly object _lock = new object();
}
and it does excactly what it should! I'm amazed that a solution could be that simple, and I love such simplicity. I'ts beautiful. Thank you, Matthew Watson!
Martin.

Two things you might try.
Using a Barrier object add conditionally adding threads too it and signaling them.
The other might be to use a publisher subscriber setup like in RX. Each thread waits on an object that it passes to a collection. When you want to call 'set' loop over a snapshot of it calling set on each member.
Or you could try bears.

If the event is being referenced by all threads in a common field or property, you could replace the common field or property with a new non-signaled event and then signal the old one. It has some cost to it since you'll be regularly creating new synchronization objects, but it would work. Here's an example of how I would do that:
public static class Example
{
private static volatile bool stopRunning;
private static ReleasingAutoResetEvent myEvent;
public static void RunExample()
{
using (Example.myEvent = new ReleasingAutoResetEvent())
{
WaitCallback work = new WaitCallback(WaitThread);
for (int i = 0; i < 5; ++i)
{
ThreadPool.QueueUserWorkItem(work, i.ToString());
}
Thread.Sleep(500);
for (int i = 0; i < 3; ++i)
{
Example.myEvent.Set();
Thread.Sleep(5000);
}
Example.stopRunning = true;
Example.myEvent.Set();
}
}
private static void WaitThread(object state)
{
while (!Example.stopRunning)
{
Example.myEvent.WaitOne();
Console.WriteLine("Thread {0} is released!", state);
}
}
}
public sealed class ReleasingAutoResetEvent : IDisposable
{
private volatile ManualResetEvent manualResetEvent = new ManualResetEvent(false);
public void Set()
{
ManualResetEvent eventToSet = this.manualResetEvent;
this.manualResetEvent = new ManualResetEvent(false);
eventToSet.Set();
eventToSet.Dispose();
}
public bool WaitOne()
{
return this.manualResetEvent.WaitOne();
}
public bool WaitOne(int millisecondsTimeout)
{
return this.manualResetEvent.WaitOne(millisecondsTimeout);
}
public bool WaitOne(TimeSpan timeout)
{
return this.manualResetEvent.WaitOne(timeout);
}
public void Dispose()
{
this.manualResetEvent.Dispose();
}
}

Another more lightweight solution you could try that uses the Monitor class to lock and unlock objects is below. However, I'm not as happy with the cleanup story for this version of ReleasingAutoResetEvent since Monitor may hold a reference to it and keep it alive indefinitely if it is not properly disposed.
There are a few limitations/gotchas with this implementation. First, the thread that creates this object will be the only one that will be able to signal it with a call to Set; other threads that attempt to do the same thing will receive a SynchronizationLockException. Second, the thread that created it will never be able to wait on it successfully since it already owns the lock. This will only be an effective solution if you have exactly one controlling thread and several other waiting threads.
public static class Example
{
private static volatile bool stopRunning;
private static ReleasingAutoResetEvent myEvent;
public static void RunExample()
{
using (Example.myEvent = new ReleasingAutoResetEvent())
{
WaitCallback work = new WaitCallback(WaitThread);
for (int i = 0; i < 5; ++i)
{
ThreadPool.QueueUserWorkItem(work, i.ToString());
}
Thread.Sleep(500);
for (int i = 0; i < 3; ++i)
{
Example.myEvent.Set();
Thread.Sleep(5000);
}
Example.stopRunning = true;
Example.myEvent.Set();
}
}
private static void WaitThread(object state)
{
while (!Example.stopRunning)
{
Example.myEvent.WaitOne();
Console.WriteLine("Thread {0} is released!", state);
}
}
}
public sealed class ReleasingAutoResetEvent : IDisposable
{
private volatile object lockObject = new object();
public ReleasingAutoResetEvent()
{
Monitor.Enter(this.lockObject);
}
public void Set()
{
object objectToSignal = this.lockObject;
object objectToLock = new object();
Monitor.Enter(objectToLock);
this.lockObject = objectToLock;
Monitor.Exit(objectToSignal);
}
public void WaitOne()
{
object objectToMonitor = this.lockObject;
Monitor.Enter(objectToMonitor);
Monitor.Exit(objectToMonitor);
}
public bool WaitOne(int millisecondsTimeout)
{
object objectToMonitor = this.lockObject;
bool succeeded = Monitor.TryEnter(objectToMonitor, millisecondsTimeout);
if (succeeded)
{
Monitor.Exit(objectToMonitor);
}
return succeeded;
}
public bool WaitOne(TimeSpan timeout)
{
object objectToMonitor = this.lockObject;
bool succeeded = Monitor.TryEnter(objectToMonitor, timeout);
if (succeeded)
{
Monitor.Exit(objectToMonitor);
}
return succeeded;
}
public void Dispose()
{
Monitor.Exit(this.lockObject);
}
}

Windows Service failed to start

This is my main:
static class Program
{
static void Main()
{
//Debugger.Launch();
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new Service1()
};
ServiceBase.Run(ServicesToRun);
}
}
And this is my Service1() code:
public partial class Service1 : ServiceBase
{
public Service1()
{
Thread messageThread = new Thread(new ThreadStart(Messaggi.Check));
messageThread.Start();
bool checkGruppoIndirizzi = true;
for (; ; )
{
SediOperative.Check();
Autisti.Check();
AutistiVeicoli.Check();
StatiVega.Check();
int res = Ordini.Check();
if (res == 0) AssegnazioniVega.Check();
Thread.Sleep(10000);
}
}
protected override void OnStart(string[] args)
{
}
protected override void OnStop()
{
}
}
First thing is I don't know if launching two threads in that way is a good thing to do, but the real problem is the program run fine inside Visual Studio but after installation (I've created a setup project using InstallShield) I try to start my service from the windows service panel and I get:
Error 1053: The service did not respond to the start or control request in a timely fashion

The problem you have is that your service will be started sucessfully after the susyem has called the Start method and it has sucessfully returned. Given that you have an infinite loop in the constructor, the system is saying to itself something like "Can't even create the this let alone call start. I'm giving up.'
Your code should be refactored along these lines:
public partial class Service1 : ServiceBase
{
public Service1()
{
}
private Thread messageThread;
private Thread otherThread;
private bool stopNow;
protected override void OnStart(string[] args)
{
this.stopNow = false;
this.messageThread = new Thread(new ThreadStart(Messaggi.Check));
this.messageThread.Start();
this.otherThread = new Thread(new ThreadStart(this.StartOtherThread));
this.otherThread.Start();
}
private void StartOtherThread()
{
bool checkGruppoIndirizzi = true;
while (this.stopNow == false)
{
SediOperative.Check();
Autisti.Check();
AutistiVeicoli.Check();
StatiVega.Check();
int res = Ordini.Check();
if (res == 0) AssegnazioniVega.Check();
for (int 1 = 0; i < 10; i++)
{
if (this.stopNow)
{
break;
}
Thread.Sleep(1000);
}
}
}
}
protected override void OnStop()
{
this.stopNow = true;
this.messageThread.Join(1000);
this.otherThread.Join(1000);
}
}
And yes, starting stuff on Threads is exactly the way to do it! You'll have to have some way of stopping them in the Stop() method. (The code above is air code so don't trust it.) for the 'otherThread' I've got it checking a bool and exiting when the bool is set. the thread.Join is just a tidy-up which isn't strictly necessary, but is good housekeeping I think.
Cheers -

C# Threading Patterns - is this a good idea?

I was playing with a project of mine today and found an interesting little snippet, given the following pattern, you can safely cleanup a thread, even if it's forced to close early. My project is a network server where it spawns a new thread for each client. I've found this useful for early termination from the remote side, but also from the local side (I can just call .Abort() from inside my processing code).
Are there any problems you can see with this, or any suggestions you'd make to anyone looking at a similar approach?
Test case follows:
using System;
using System.Threading;
class Program
{
static Thread t1 = new Thread(thread1);
static Thread t2 = new Thread(thread2);
public static void Main(string[] args)
{
t1.Start();
t2.Start();
t1.Join();
}
public static void thread1() {
try {
// Do our work here, for this test just look busy.
while(true) {
Thread.Sleep(100);
}
} finally {
Console.WriteLine("We're exiting thread1 cleanly.\n");
// Do any cleanup that might be needed here.
}
}
public static void thread2() {
Thread.Sleep(500);
t1.Abort();
}
}
For reference, without the try/finally block, the thread just dies as one would expect.

Aborting another thread at all is just a bad idea unless the whole application is coming down. It's too easy to leave your program in an unknown state. Aborting your own thread is occasionally useful - ASP.NET throws a ThreadAbortException if you want to prematurely end the response, for example - but it's not a terribly nice design.
Safe clean-up of a thread should be mutual - there should be some shared flag requesting that the thread shuts down. The thread should check that flag periodically and quit appropriately.

Whether or not this will "safely" cleanup a thread cannot be discerned from a general code sample unfortunately. It's highly dependent upon the actual code that is executed within the thread. There are multiple issues you must consider. Each represents a potential bug in the code.
If the thread is currently in native code, it will not immediately respect the Thread.Abort call. It will do all of the work it wants to do in native code and will not throw until the code returns back to managed. Until this happens thread2 will hang.
Any native resources that are not freed in a finally block will be leaked in this scenario. All native resources should be freed in a finally block but not all code does this and it's an issue to consider.
Any locks that are not freed in a finally block will remain in a lock'd state and can lead to future dead locks.
There are other issues which are slipping my mind at the moment. But hopefully this will give you some guidance with your application.

It is generally not a good idea to abort threads. What you can do is poll for a stopRequested flag which can be set from other threads. Below is a sample WorkerThread class for your reference. For more information on how to use it, please refer to http://devpinoy.org/blogs/jakelite/archive/2008/12/20/threading-patterns-the-worker-thread-pattern.aspx
public abstract class WorkerThreadBase : IDisposable
{
private Thread _workerThread;
protected internal ManualResetEvent _stopping;
protected internal ManualResetEvent _stopped;
private bool _disposed;
private bool _disposing;
private string _name;
protected WorkerThreadBase()
: this(null, ThreadPriority.Normal)
{
}
protected WorkerThreadBase(string name)
: this(name, ThreadPriority.Normal)
{
}
protected WorkerThreadBase(string name,
ThreadPriority priority)
: this(name, priority, false)
{
}
protected WorkerThreadBase(string name,
ThreadPriority priority,
bool isBackground)
{
_disposing = false;
_disposed = false;
_stopping = new ManualResetEvent(false);
_stopped = new ManualResetEvent(false);
_name = name == null ? GetType().Name : name; ;
_workerThread = new Thread(threadProc);
_workerThread.Name = _name;
_workerThread.Priority = priority;
_workerThread.IsBackground = isBackground;
}
protected bool StopRequested
{
get { return _stopping.WaitOne(1, true); }
}
protected bool Disposing
{
get { return _disposing; }
}
protected bool Disposed
{
get { return _disposed; }
}
public string Name
{
get { return _name; }
}
public void Start()
{
ThrowIfDisposedOrDisposing();
_workerThread.Start();
}
public void Stop()
{
ThrowIfDisposedOrDisposing();
_stopping.Set();
_stopped.WaitOne();
}
public void WaitForExit()
{
ThrowIfDisposedOrDisposing();
_stopped.WaitOne();
}
#region IDisposable Members
public void Dispose()
{
dispose(true);
}
#endregion
public static void WaitAll(params WorkerThreadBase[] threads)
{
WaitHandle.WaitAll(
Array.ConvertAll<WorkerThreadBase, WaitHandle>(
threads,
delegate(WorkerThreadBase workerThread)
{ return workerThread._stopped; }));
}
public static void WaitAny(params WorkerThreadBase[] threads)
{
WaitHandle.WaitAny(
Array.ConvertAll<WorkerThreadBase, WaitHandle>(
threads,
delegate(WorkerThreadBase workerThread)
{ return workerThread._stopped; }));
}
protected virtual void Dispose(bool disposing)
{
//stop the thread;
Stop();
//make sure the thread joins the main thread
_workerThread.Join(1000);
//dispose of the waithandles
DisposeWaitHandle(_stopping);
DisposeWaitHandle(_stopped);
}
protected void ThrowIfDisposedOrDisposing()
{
if (_disposing)
{
throw new InvalidOperationException(
Properties.Resources.ERROR_OBJECT_DISPOSING);
}
if (_disposed)
{
throw new ObjectDisposedException(
GetType().Name,
Properties.Resources.ERROR_OBJECT_DISPOSED);
}
}
protected void DisposeWaitHandle(WaitHandle waitHandle)
{
if (waitHandle != null)
{
waitHandle.Close();
waitHandle = null;
}
}
protected abstract void Work();
private void dispose(bool disposing)
{
//do nothing if disposed more than once
if (_disposed)
{
return;
}
if (disposing)
{
_disposing = disposing;
Dispose(disposing);
_disposing = false;
//mark as disposed
_disposed = true;
}
}
private void threadProc()
{
Work();
_stopped.Set();
}
}

Automatically terminating non essential threads in C#

I have an object in C# on which I need to execute a method on a regular basis. I would like this method to be executed only when other people are using my object, as soon as people stop using my object I would like this background operation to stop.
So here is a simple example is this (which is broken):
class Fish
{
public Fish()
{
Thread t = new Thread(new ThreadStart(BackgroundWork));
t.IsBackground = true;
t.Start();
}
public void BackgroundWork()
{
while(true)
{
this.Swim();
Thread.Sleep(1000);
}
}
public void Swim()
{
Console.WriteLine("The fish is Swimming");
}
}
The problem is that if I new a Fish object anywhere, it never gets garbage collected, cause there is a background thread referencing it. Here is an illustrated version of broken code.
public void DoStuff()
{
Fish f = new Fish();
}
// after existing from this method my Fish object keeps on swimming.
I know that the Fish object should be disposable and I should clean up the thread on dispose, but I have no control over my callers and can not ensure dispose is called.
How do I work around this problem and ensure the background threads are automatically disposed even if Dispose is not called explicitly?

Here is my proposed solution to this problem:
class Fish : IDisposable
{
class Swimmer
{
Thread t;
WeakReference fishRef;
public ManualResetEvent terminate = new ManualResetEvent(false);
public Swimmer(Fish3 fish)
{
this.fishRef = new WeakReference(fish);
t = new Thread(new ThreadStart(BackgroundWork));
t.IsBackground = true;
t.Start();
}
public void BackgroundWork()
{
bool done = false;
while(!done)
{
done = Swim();
if (!done)
{
done = terminate.WaitOne(1000, false);
}
}
}
// this is pulled out into a helper method to ensure
// the Fish object is referenced for the minimal amount of time
private bool Swim()
{
bool done;
Fish fish = Fish;
if (fish != null)
{
fish.Swim();
done = false;
}
else
{
done = true;
}
return done;
}
public Fish Fish
{
get { return fishRef.Target as Fish3; }
}
}
Swimmer swimmer;
public Fish()
{
swimmer = new Swimmer(this);
}
public void Swim()
{
Console.WriteLine("The third fish is Swimming");
}
volatile bool disposed = false;
public void Dispose()
{
if (!disposed)
{
swimmer.terminate.Set();
disposed = true;
GC.SuppressFinalize(this);
}
}
~Fish()
{
if(!disposed)
{
Dispose();
}
}
}

I think the IDisposable solution is the correct one.
If the users of your class don't follow the guidelines for using classes that implement IDisposable it's their fault - and you can make sure that the documentation explicitly mentions how the class should be used.
Another, much messier, option would be a "KeepAlive" DateTime field that each method called by your client would update. The worker thread then checks the field periodically and exits if it hasn't been updated for a certain amount of time. When a method is setting the field the thread will be restarted if it has exited.

This is how I would do it:
class Fish3 : IDisposable
{
Thread t;
private ManualResetEvent terminate = new ManualResetEvent(false);
private volatile int disposed = 0;
public Fish3()
{
t = new Thread(new ThreadStart(BackgroundWork));
t.IsBackground = true;
t.Start();
}
public void BackgroundWork()
{
while(!terminate.WaitOne(1000, false))
{
Swim();
}
}
public void Swim()
{
Console.WriteLine("The third fish is Swimming");
}
public void Dispose()
{
if(Interlocked.Exchange(ref disposed, 1) == 0)
{
terminate.Set();
t.Join();
GC.SuppressFinalize(this);
}
}
~Fish3()
{
if(Interlocked.Exchange(ref disposed, 1) == 0)
{
Dispose();
}
}
}