We Keep Coding

Detect Duplicate Items in DataFlow

I've been building out a service that processes files using a Queue<string> object to manage the items.
public partial class BasicQueueService : ServiceBase
{
private readonly EventWaitHandle completeHandle =
new EventWaitHandle(false, EventResetMode.ManualReset, "ThreadCompleters");
public BasicQueueService()
{
QueueManager = new Queue<string>();
}
public bool Stopping { get; set; }
private Queue<string> QueueManager { get; }
protected override void OnStart(string[] args)
{
Stopping = false;
ProcessFiles();
}
protected override void OnStop()
{
Stopping = true;
}
private void ProcessFiles()
{
while (!Stopping)
{
var count = QueueManager.Count;
for (var i = 0; i < count; i++)
{
//Check the Stopping Variable again.
if (Stopping) break;
var fileName = QueueManager.Dequeue();
if (string.IsNullOrWhiteSpace(fileName) || !File.Exists(fileName))
continue;
Console.WriteLine($"Processing {fileName}");
Task.Run(() =>
{
DoWork(fileName);
})
.ContinueWith(ThreadComplete);
}
if (Stopping) continue;
Console.WriteLine("Waiting for thread to finish, or 1 minute.");
completeHandle.WaitOne(new TimeSpan(0, 0, 15));
completeHandle.Reset();
}
}
partial void DoWork(string fileName);
private void ThreadComplete(Task task)
{
completeHandle.Set();
}
public void AddToQueue(string file)
{
//Called by FileWatcher/Manual classes, not included for brevity.
lock (QueueManager)
{
if (QueueManager.Contains(file)) return;
QueueManager.Enqueue(file);
}
}
}
Whilst researching how to limit the number of threads on this (I've tried a manual class with an incrementing int, but there's an issue where it doesn't decrement properly in my code), I came across TPL DataFlow, which seems like its a better fit for what I'm trying to achieve - specifically, it allows me to let the framework handle threading/queueing, etc.
This is now my service:
public partial class BasicDataFlowService : ServiceBase
{
private readonly ActionBlock<string> workerBlock;
public BasicDataFlowService()
{
workerBlock = new ActionBlock<string>(file => DoWork(file), new ExecutionDataflowBlockOptions()
{
MaxDegreeOfParallelism = 32
});
}
public bool Stopping { get; set; }
protected override void OnStart(string[] args)
{
Stopping = false;
}
protected override void OnStop()
{
Stopping = true;
}
partial void DoWork(string fileName);
private void AddToDataFlow(string file)
{
workerBlock.Post(file);
}
}
This works well. However, I want to ensure that a file is only ever added to the TPL DataFlow once. With the Queue, I can check that using .Contains(). Is there a mechanism that I can use for TPL DataFlow?

Your solution with Queue works only if file goes into your service twice in a small period of time. If it came again in, say, few hours, queue will not contain it, as you Dequeue it from there.
If this solution is expected, then you may use a MemoryCache to store file paths being already handled, like this:
using System.Runtime.Caching;
private static object _lock = new object();
private void AddToDataFlow(string file)
{
lock (_lock)
{
if (MemoryCache.Default.Contains(file))
{
return;
}
// no matter what to put into the cache
MemoryCache.Default[file] = true;
// we can now exit the lock
}
workerBlock.Post(file);
}
However, if your application must run for a long time (which service is intended to do), you'll eventually run out of memory. In that case you probably need to store your file paths in database or something, so even after restarting the service your code will restore the state.

You can check it inside of DoWork.
You have to save in Hash already works items and check current filename doesn't exist in hash.

Windows service keeps on failing

I have a windows service which is designed to continuously retrieve messages from Azure service bus queue and pass it to other queues.I have deployed this service to one of the server computer but unfortunately the service keeps failing at a random time interval.
My application handles the exceptions and writes them to a file.The main purpose of this application is to hook up to the queue and listen all the messages continuously and never move to the Application stop stage.I'm using a timer in this application and I don't think that is causing any problem.I'd like to know what would be the best approach to handle errors and make my application stable, below is the code. Thanks in advance.
public partial class Scheduler : ServiceBase
{
private Timer Scheduletimer = null;
private string servicenamespace;
private string issuesecretkey;
private string sourcequeue;
private string destinationqueue;
public Scheduler()
{
InitializeComponent();
}
protected override void OnStart(string[] args)
{
Scheduletimer = new Timer();
this.Scheduletimer.Interval = 1000;//1 sec
this.Scheduletimer.Elapsed += new System.Timers.ElapsedEventHandler(this.timer1_Tick);
Scheduletimer.Enabled = true;
WriteToFile("Application started : "+DateTime.Now.ToString());
}
protected void timer1_Tick(object sender, ElapsedEventArgs e)
{
Scheduletimer.Enabled = false;
WriteToFile("Business logic started : " + DateTime.Now.ToString());
//Business Logic code goes here
}
protected override void OnStop()
{
Scheduletimer.Enabled = false;
WriteToFile("Application stoped : "+DateTime.Now.ToString());
}
public void WriteToFile(string text)
{
string directory = AppDomain.CurrentDomain.BaseDirectory;
string logfilepath = directory + "LogFile.txt";
using (StreamWriter writer = new StreamWriter(logfilepath, true))
{
writer.WriteLine(text);
writer.Close();
}
}
public void WriteErrorsToFile(Exception ex)
{
string directory = AppDomain.CurrentDomain.BaseDirectory;
string Errorlogfilepath = directory + "ErrorLogFile.txt";
using (StreamWriter writer = new StreamWriter(Errorlogfilepath, true))
{
writer.WriteLine("Time Occured: " + DateTime.Now.ToString());
writer.WriteLine(ex.Message +" "+ DateTime.Now.ToString());
writer.Close();
}
}

Joining a thread started with StartNew()

When using the StartNew() method to kick off a process on a new thread, I need to figure out how to make another call into this object in that same thread (I assume this would be some sort of Join operation?).
The following example is dumbed down to illustrate the meat of what I am trying to do. I am well aware it is severely lacking in basic concurrency considerations. But I didn't want to cloud the code with all of that logic, so please forgive me on that.
The following console app shows what I am trying to accomplish. Assume on the StartNew() call a new thread with ID 9976 is created and the method invoked there. I would like the subsequent call to ProcessImmediate() in the file system watcher change event handler to be made on thread 9976 as well. As it stands, the call would share the same thread that is used for the file system watcher change event.
Can this be done, and if so, how?
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
var runner = new Runner();
runner.Run();
Console.ReadKey();
}
}
public class Runner
{
private Activity _activity = null;
private FileSystemWatcher _fileSystemWatcher;
public void Run()
{
_activity = new Activity();
// start activity on a new thread
Task.Factory.StartNew(() => _activity.Go());
_fileSystemWatcher = new FileSystemWatcher();
_fileSystemWatcher.Filter = "*.watcher";
_fileSystemWatcher.Path = "c:\temp";
_fileSystemWatcher.Changed += FileSystemWatcher_Changed;
_fileSystemWatcher.EnableRaisingEvents = true;
}
private void FileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
// WANT TO CALL THIS FOR ACTIVITY RUNNING ON PREVIOUSLY CALLED THREAD
_activity.ProcessImmediate();
}
}
public class Activity
{
public void Go()
{
while (!Stop)
{
// for purposes of this example, magically assume that ProcessImmediate has not been called when this is called
DoSomethingInteresting();
System.Threading.Thread.Sleep(2000);
}
}
protected virtual void DoSomethingInteresting() { }
public void ProcessImmediate()
{
// for purposes of this example, assume that Go is magically in its sleep state when ProcessImmediate is called
DoSomethingInteresting();
}
public bool Stop { get; set; }
}
}
* UPDATE *
Thanks for the excellent responses. I took Mike's suggestion and implemented it for my console app. Below is the full working code which also includes the use of a cancellation token. I post this in case someone else might find it useful.
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
var runner = new Runner();
runner.Run();
Console.ReadKey();
runner.Stop();
Console.ReadKey();
}
}
public class Runner
{
private Activity _activity = null;
private FileSystemWatcher _fileSystemWatcher;
private CancellationTokenSource _cts = new CancellationTokenSource();
public void Stop() { _cts.Cancel(); }
public void Run()
{
_activity = new Activity();
// start activity on a new thread
var task = new Task(() => _activity.Go(_cts.Token), _cts.Token, TaskCreationOptions.LongRunning);
task.Start();
_fileSystemWatcher = new FileSystemWatcher();
_fileSystemWatcher.Filter = "*.watcher";
_fileSystemWatcher.Path = "C:\\Temp\\FileSystemWatcherPath";
_fileSystemWatcher.Changed += FileSystemWatcher_Changed;
_fileSystemWatcher.EnableRaisingEvents = true;
}
private void FileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
// WANT TO CALL THIS FOR ACTIVITY RUNNING ON PREVIOUSLY CALLED THREAD
_activity.ProcessImmediate();
}
}
public class Activity : IDisposable
{
private AutoResetEvent _processing = new AutoResetEvent(false);
public void Go(CancellationToken ct)
{
Thread.CurrentThread.Name = "Go";
while (!ct.IsCancellationRequested)
{
// for purposes of this example, magically assume that ProcessImmediate has not been called when this is called
DoSomethingInteresting();
_processing.WaitOne(5000);
}
Console.WriteLine("Exiting");
}
protected virtual void DoSomethingInteresting()
{
Console.WriteLine(string.Format("Doing Something Interesting on thread {0}", Thread.CurrentThread.ManagedThreadId));
}
public void ProcessImmediate()
{
// for purposes of this example, assume that Go is magically in its sleep state when ProcessImmediate is called
_processing.Set();
}
public void Dispose()
{
if (_processing != null)
{
_processing.Dispose();
_processing = null;
}
}
}
}

First, you should use TaskCreationOptions.LongRunning if you are creating a task that will not complete quickly. Second, use an AutoResetEvent to signal the waiting thread to wake up. Note that below ProcessImmediate will return before DoSomethingInteresting has completed running on the other thread. Example:
using System.Threading;
public class Activity : IDisposable
{
private AutoResetEvent _processing = new AutoResetEvent(false);
public void Go()
{
while (!Stop)
{
// for purposes of this example, magically assume that ProcessImmediate has not been called when this is called
DoSomethingInteresting();
_processing.WaitOne(2000);
}
}
protected virtual void DoSomethingInteresting() { }
public void ProcessImmediate()
{
_processing.Set();
}
public bool Stop { get; set; }
public void Dispose()
{
if (_processing != null)
{
_processing.Dispose();
_processing = null;
}
}
}

User mike has given a better solution, which will be appropriate when you like to call the same method immediately. If you want to call a different methods immediately I'll expand mike's answer to achieve that.
using System.Threading;
public class Activity : IDisposable
{
private AutoResetEvent _processing = new AutoResetEvent(false);
private ConcurrentQueue<Action> actionsToProcess = new ConcurrentQueue<Action>();
public void Go()
{
while (!Stop)
{
// for purposes of this example, magically assume that ProcessImmediate has not been called when this is called
DoSomethingInteresting();
_processing.WaitOne(2000);
while(!actionsToProcess.IsEmpty)
{
Action action;
if(actionsToProcess.TryDeque(out action))
action();
}
}
}
protected virtual void DoSomethingInteresting() { }
public void ProcessImmediate(Action action)
{
actionsToProcess.Enqueue(action);
_processing.Set();
}
public bool Stop { get; set; }
public void Dispose()
{
if (_processing != null)
{
_processing.Dispose();
_processing = null;
}
}
}

To execute different methods on the same thread you can use a message loop that dispatches incoming requests. A simple option would be to use the event loop scheduler of the Reactive Extensions and to "recursively" schedule your Go() function - if in the mean time a different operation is scheduled it would be processed before the next Go() operation.
Here is a sample:
class Loop
: IDisposable
{
IScheduler scheduler = new EventLoopScheduler();
MultipleAssignmentDisposable stopper = new MultipleAssignmentDisposable();
public Loop()
{
Next();
}
void Next()
{
if (!stopper.IsDisposed)
stopper.Disposable = scheduler.Schedule(Handler);
}
void Handler()
{
Thread.Sleep(1000);
Console.WriteLine("Handler: {0}", Thread.CurrentThread.ManagedThreadId);
Next();
}
public void Notify()
{
scheduler.Schedule(() =>
{
Console.WriteLine("Notify: {0}", Thread.CurrentThread.ManagedThreadId);
});
}
public void Dispose()
{
stopper.Dispose();
}
}
static void Main(string[] args)
{
using (var l = new Loop())
{
Console.WriteLine("Press 'q' to quit.");
while (Console.ReadKey().Key != ConsoleKey.Q)
l.Notify();
}
}

How to make sure that threads spawned by a Windows Service Timer complete executing after stopping the Windows Service?

I'm building a Windows Service using System.Timers.Timer. The tasks computed by the Timer's delegate can take from several seconds to several minutes. I would like to make sure that, when the service is stopped, all delegated threads currently running complete before being disposed.
Here is the code, however it does not do what I expect, as currently running threads never complete if the Windows Service is stopped while they are running.
public abstract class AgentServiceBase : ServiceBase
{
private System.ComponentModel.IContainer components = null;
private System.Timers.Timer _Timer;
private string _logPath;
private const int MAXNUMBEROFTHREADS = 10;
protected int interval = 25000;
protected int numberOfAllowedThreads = 2;
public AgentServiceBase()
{
this.InitializeComponent();
this._logPath = (Path.GetDirectoryName(Assembly.GetAssembly(this.GetType()).CodeBase)).Substring(6).Replace("/", #"\");
}
protected override void OnStart(string[] args)
{
if (args.Length > 0)
{
int.TryParse(args[0], out interval);
}
if (args.Length > 1)
{
int.TryParse(args[1], out numberOfAllowedThreads);
if (numberOfAllowedThreads > MAXNUMBEROFTHREADS)
{
numberOfAllowedThreads = MAXNUMBEROFTHREADS;
}
if (numberOfAllowedThreads == 1)
{
numberOfAllowedThreads = 2;
}
}
ThreadPool.SetMaxThreads(numberOfAllowedThreads, numberOfAllowedThreads);
this._Timer = new System.Timers.Timer();
this._Timer.Elapsed += new ElapsedEventHandler(PollWrapper);
this._Timer.Interval = this.interval;
this._Timer.Enabled = true;
}
protected override void OnStop()
{
this._Timer.Enabled = false;
Process currentProcess = Process.GetCurrentProcess();
foreach (Thread t in currentProcess.Threads)
{
t.Join();
}
}
/// <summary>
protected override void Dispose(bool disposing)
{
if (disposing && (components != null))
{
components.Dispose();
}
base.Dispose(disposing);
}
private void InitializeComponent()
{
components = new System.ComponentModel.Container();
this.ServiceName = "Agent Service - Johnhenry";
}
private void PollWrapper(object sender, ElapsedEventArgs e)
{
try
{
this.Poll(sender, e);
}
catch (Exception exception)
{
string message = this.GetType().FullName + " - Windows Service Exception\n";
message += exception.GetNestedExceptionInSingleStringOutput();
FileHelper.Log(message, this._logPath, "exception", FileHelper.LogFileNameChangeFrequency.DAYLY);
}
}
protected abstract void Poll(object sender, ElapsedEventArgs e);
}
Many thanks,
Giuseppe
UPDATE:
After few different attempts with counting the current process's own threads I eventually settled with a simpler solution which is using a counter of the threads the timer had initiated and are still running. Based on that I call the Sleep on the main thread and issue a RequestAdditionalTime until all threads have ended.
Following the revised 2 methods:
protected override void OnStop()
{
this._Timer.Enabled = false;
while (numberOfRunningThreads > 0)
{
this.RequestAdditionalTime(1000);
Thread.Sleep(1000);
}
}
private void PollWrapper(object sender, ElapsedEventArgs e)
{
numberOfRunningThreads++;
try
{
this.Poll(sender, e);
}
catch (Exception exception)
{
string message = this.GetType().FullName + " - Windows Service Exception\n";
message += exception.GetNestedExceptionInSingleStringOutput();
FileHelper.Log(message, this._logPath, "exception", FileHelper.LogFileNameChangeFrequency.DAYLY);
}
finally
{
numberOfRunningThreads--;
}
}

You can achieve that by calling RequestAdditionalTime as long as your threads haven't finished the work yet in your implementation of OnStop inside the loop (before and/or after the call to Join()).
BUT BEWARE that Windows can get impatient and decide to kill your Windows Service - for example during shutdown...
For more information see the MSDN reference at http://msdn.microsoft.com/en-us/library/system.serviceprocess.servicebase.aspx

Stuck on GenerateConsoleCtrlEvent in C# with console apps

I'm having the hardest time trying to get this to work, hoping one of you has done this before.
I have a C# console app that is running a child process which inherits its console. I want a ctrl-c caught by the outer app to be passed along to the inner app so that it can have a chance to shut down nicely.
I have some very simple code. I start a Process, then poll it with WaitForExit(10). I also have a CancelKeyPress handler registered, which sets a bool to true when it fires. The polling loop also checks this, and when it's true, it calls GenerateConsoleCtrlEvent() (which I have mapped through pinvoke).
I've tried a lot of combinations of params to GenerateConsoleCtrlEvent(). 0 or 1 for the first param, and either 0 or the child process's ID for the second param. Nothing seems to work. Sometimes I get a false back and Marshal.GetLastWin32Error() returns 0, and sometimes I get true back. But none cause the child app to receive a ctrl-c.
To be absolutely sure, I wrote a test C# app to be the child app which prints out what's going on with it and verified that manually typing ctrl-c when it runs does properly cause it to quit.
I've been banging my head against this for a couple hours. Can anyone give me some pointers on where to go with this?

Not so sure this is a good approach. This only works if the child process is created with the CREATE_NEW_PROCESS_GROUP flag for CreateProcess(). The System.Diagnostics.Process class however does not support this.
Consider using the return value from the Main() method. There is already a unique value defined in the Windows SDK for Ctrl+C aborts, STATUS_CONTROL_C_EXIT or 0xC000013A. The parent process can get that return code from the Process.ExitCode property.

Did you have any luck with this? My understanding is that when you press CTRL+C in a console, by default all the processes attached to the console receive it, not just the parent one. Here's an example:
Child.cs:
using System;
public class MyClass
{
public static void CtrlCHandler(object sender, ConsoleCancelEventArgs args)
{
Console.WriteLine("Child killed by CTRL+C.");
}
public static void Main()
{
Console.WriteLine("Child start.");
Console.CancelKeyPress += CtrlCHandler;
System.Threading.Thread.Sleep(4000);
Console.WriteLine("Child finish.");
}
}
Parent.cs:
using System;
public class MyClass
{
public static void CtrlCHandler(object sender, ConsoleCancelEventArgs args)
{
Console.WriteLine("Parent killed by CTRL+C.");
}
public static void Main()
{
Console.CancelKeyPress += CtrlCHandler;
Console.WriteLine("Parent start.");
System.Diagnostics.Process child = new System.Diagnostics.Process();
child.StartInfo.UseShellExecute = false;
child.StartInfo.FileName = "child.exe";
child.Start();
child.WaitForExit();
Console.WriteLine("Parent finish.");
}
}
Output:
Y:\>parent
Parent start.
Child start.
Parent killed by CTRL+C.
Child killed by CTRL+C.
^C
Y:\>parent
Parent start.
Child start.
Child finish.
Parent finish.
So I wouldn't have thought you'd need to do anything special. However, if you really need to generate CTRL+C events yourself, things might not be so easy. I'm not sure about the problems you describe, but as far as I can tell you can only send CTRL+C events to all the processes attached to a console window. If you detach a process, you can't send it CTRL+C events. If you want to be selective in which processes to send the CTRL+C events, you seem to need to create new console windows for every one. I've no idea if there's some way to do it without visible windows or when you want to redirect I/O using pipes.

Here is my solution for sending ctrl-c to a process. FYI, I never got GenerateConsoleCtrlEvent to work.
Rather than using GenerateConsoleCtrlEvent, here is how I have found to send CTRL-C to a process. FYI, in this case, I didn't ever need to find the group process ID.
using System;
using System.Diagnostics;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
public class ConsoleAppManager
{
private readonly string appName;
private readonly Process process = new Process();
private readonly object theLock = new object();
private SynchronizationContext context;
private string pendingWriteData;
public ConsoleAppManager(string appName)
{
this.appName = appName;
this.process.StartInfo.FileName = this.appName;
this.process.StartInfo.RedirectStandardError = true;
this.process.StartInfo.StandardErrorEncoding = Encoding.UTF8;
this.process.StartInfo.RedirectStandardInput = true;
this.process.StartInfo.RedirectStandardOutput = true;
this.process.EnableRaisingEvents = true;
this.process.StartInfo.CreateNoWindow = true;
this.process.StartInfo.UseShellExecute = false;
this.process.StartInfo.StandardOutputEncoding = Encoding.UTF8;
this.process.Exited += this.ProcessOnExited;
}
public event EventHandler<string> ErrorTextReceived;
public event EventHandler ProcessExited;
public event EventHandler<string> StandartTextReceived;
public int ExitCode
{
get { return this.process.ExitCode; }
}
public bool Running
{
get; private set;
}
public void ExecuteAsync(params string[] args)
{
if (this.Running)
{
throw new InvalidOperationException(
"Process is still Running. Please wait for the process to complete.");
}
string arguments = string.Join(" ", args);
this.process.StartInfo.Arguments = arguments;
this.context = SynchronizationContext.Current;
this.process.Start();
this.Running = true;
new Task(this.ReadOutputAsync).Start();
new Task(this.WriteInputTask).Start();
new Task(this.ReadOutputErrorAsync).Start();
}
public void Write(string data)
{
if (data == null)
{
return;
}
lock (this.theLock)
{
this.pendingWriteData = data;
}
}
public void WriteLine(string data)
{
this.Write(data + Environment.NewLine);
}
protected virtual void OnErrorTextReceived(string e)
{
EventHandler<string> handler = this.ErrorTextReceived;
if (handler != null)
{
if (this.context != null)
{
this.context.Post(delegate { handler(this, e); }, null);
}
else
{
handler(this, e);
}
}
}
protected virtual void OnProcessExited()
{
EventHandler handler = this.ProcessExited;
if (handler != null)
{
handler(this, EventArgs.Empty);
}
}
protected virtual void OnStandartTextReceived(string e)
{
EventHandler<string> handler = this.StandartTextReceived;
if (handler != null)
{
if (this.context != null)
{
this.context.Post(delegate { handler(this, e); }, null);
}
else
{
handler(this, e);
}
}
}
private void ProcessOnExited(object sender, EventArgs eventArgs)
{
this.OnProcessExited();
}
private async void ReadOutputAsync()
{
var standart = new StringBuilder();
var buff = new char[1024];
int length;
while (this.process.HasExited == false)
{
standart.Clear();
length = await this.process.StandardOutput.ReadAsync(buff, 0, buff.Length);
standart.Append(buff.SubArray(0, length));
this.OnStandartTextReceived(standart.ToString());
Thread.Sleep(1);
}
this.Running = false;
}
private async void ReadOutputErrorAsync()
{
var sb = new StringBuilder();
do
{
sb.Clear();
var buff = new char[1024];
int length = await this.process.StandardError.ReadAsync(buff, 0, buff.Length);
sb.Append(buff.SubArray(0, length));
this.OnErrorTextReceived(sb.ToString());
Thread.Sleep(1);
}
while (this.process.HasExited == false);
}
private async void WriteInputTask()
{
while (this.process.HasExited == false)
{
Thread.Sleep(1);
if (this.pendingWriteData != null)
{
await this.process.StandardInput.WriteLineAsync(this.pendingWriteData);
await this.process.StandardInput.FlushAsync();
lock (this.theLock)
{
this.pendingWriteData = null;
}
}
}
}
}
Then, in actually running the process and sending the CTRL-C in my main app:
DateTime maxStartDateTime = //... some date time;
DateTime maxEndDateTime = //... some later date time
var duration = maxEndDateTime.Subtract(maxStartDateTime);
ConsoleAppManager appManager = new ConsoleAppManager("myapp.exe");
string[] args = new string[] { "args here" };
appManager.ExecuteAsync(args);
await Task.Delay(Convert.ToInt32(duration.TotalSeconds * 1000) + 20000);
if (appManager.Running)
{
// If stilll running, send CTRL-C
appManager.Write("\x3");
}
For details, please see Redirecting standard input of console application and Windows how to get the process group of a process that is already running?