I am wanting to write a service that polls a database and performs an operation depending on the data being brought back.
I am not sure what is the best way of doing this, I can find a few blogs about it and this stack overflow question Polling Service - C#. However I am wary that they are all quite old and possibly out of date.
Can anyone advise me on the current advice or best practices (if there are any) on doing something like this or point me in the direction of a more recent blog post about this. From what I can gather either using a timer or tpl tasks are two potential ways of doing this.
If timers are still suggested then how will they work when the service is stopped because the operations I intend for these services to do could potentially take 30+ minutes, this is why I say use tasks because I can use a task cancellation token but these throw exceptions when cancelled (correct me if I am wrong) and I don't think I really want that behaviour (although correct me if you think there is a reason I will want that).
Sorry that I may be asking quite a lot in a single question but I'm not entirely sure myself what I am asking.
Go with a Windows service for this. Using a scheduled task is not a bad idea per se, but since you said the polls can occur every 2 minutes then you are probably better off going with the service. The service will allow you to maintain state between polls and you would have more control over the timing of the polls as well. You said the operation might take 30+ minutes once it is kicked off so maybe you would want to defer polls until the operation complete. That is a bit easier to do when the logic is ran as a service.
In the end it does not really matter what mechanism you use to generate the polls. You could use a timer or a dedicated thread/task that sleeps or whatever. Personally, I find the dedicated thread/task easier to work with than a timer for these kinds of things because it is easier to control the polling interval. Also, you should definitely use the cooperative cancellation mechanism provided with the TPL. It does not necessary throw exceptions. It only does so if you call ThrowIfCancellationRequested. You can use IsCancellationRequested instead to just check the cancellation token's state.
Here is a very generic template you might use to get started.
public class YourService : ServiceBase
{
private CancellationTokenSource cts = new CancellationTokenSource();
private Task mainTask = null;
protected override void OnStart(string[] args)
{
mainTask = new Task(Poll, cts.Token, TaskCreationOptions.LongRunning);
mainTask.Start();
}
protected override void OnStop()
{
cts.Cancel();
mainTask.Wait();
}
private void Poll()
{
CancellationToken cancellation = cts.Token;
TimeSpan interval = TimeSpan.Zero;
while (!cancellation.WaitHandle.WaitOne(interval))
{
try
{
// Put your code to poll here.
// Occasionally check the cancellation state.
if (cancellation.IsCancellationRequested)
{
break;
}
interval = WaitAfterSuccessInterval;
}
catch (Exception caught)
{
// Log the exception.
interval = WaitAfterErrorInterval;
}
}
}
}
Like I said, I normally use a dedicated thread/task instead of a timer. I do this because my polling interval is almost never constant. I usually start slowing the polls down if a transient error is detected (like network or server availability issues) that way my log file does not fill up with the same error message over and over again in rapid succession.
You have a few options. To start with what could be essentially the easiest option, you could decide to create your app as a console application and run the executable as a task in the Windows Task Scheduler. All you would need to do is assign your executable as the program to start in the task and have the task scheduler handle the timing interval for you. This is probably the preferred way if you don't care about state and will prevent you from having to worry about creating and managing a windows service if you don't really need to. See the following link for how to use the scheduler.
Windows Task Scheduler
The next way you could do this would be to create a windows service and in that service use a timer, specifically System.Timers.Timer. Essentially you would set the timer interval to the amount of time you would like to have pass before you run your process. Then you would sign up for the timers tick event which would fire every time that interval occurred. In this event you would essentially have the process you would like to run; this could kick off addition threads if you would like. Then after that initial setup you would just call the timers Start() function or set the Enabled property to True to start the timer. A good example of what this would look like can be found in the example on MSDN page describing the object. There are plenty of tutorials out there that show how to set up a windows service so I won't bother with going into that specifically.
MSDN: System.Timers.Timer
Finally and more complex would be to set up a windows service that listens for a SqlDependency. This technique is useful if things can occur in the database outside your application yet you need to be made aware of it in your application or some other service. The following link has a good tutorial on how to set up a SqlDependency in an application.
Using SqlDependency To Monitor SQL Database Changes
Two things I would like to point out from your original post that are not specific to the question you had.
If you are writing a true windows service you don't want the service to stop. The service should be running constantly and if an exception does occur it should be handled appropriately and not stop the service.
A cancellation token doesn't have to throw an exception; simply not calling ThrowIfCancellationRequested() will cause the exception not to be thrown or if this is a CancellationTokenSource set the argument to false on the Cancel method then subsequently check the token to see if cancellation is requested in your threads and return out of the thread gracefully if so.
For example:
CancellationTokenSource cts = new CancellationTokenSource();
ParallelOptions options = new ParallelOptions
{
CancellationToken = cts.Token
};
Parallel.ForEach(data, options, i =>
{
try
{
if (cts.IsCancellationRequested) return;
//do stuff
}
catch (Exception ex)
{
cts.Cancel(false);
}
});
Related
We have a third-party method Foo which sometimes runs in a deadlock for unknown reasons.
We are executing an single-threaded tcp-server and call this method every 30 seconds to check that the external system is available.
To mitigate the problem with the deadlock in the third party code we put the ping-call in a Task.Run to so that the server does not deadlock.
Like
async Task<bool> WrappedFoo()
{
var timeout = 10000;
var task = Task.Run(() => ThirdPartyCode.Foo());
var delay = Task.Delay(timeout);
if (delay == await Task.WhenAny(delay, task ))
{
return false;
}
else
{
return await task ;
}
}
But this (in our opinion) has the potential to starve the application of free threads. Since if one call to ThirdPartyCode.Foo deadlock the thread will never recover from this deadlock and if this happens often enough we might run out of resources.
Is there a general approach how one should handle deadlocking third-party code?
A CancellationToken won't work because the third-party-api does not provide any cancellation options.
Update:
The method at hand is from the SAPNCO.dll provided by SAP to establish and test rfc-connections to a sap-system, therefore the method is not a simple network-ping. I renamed the method in the question to avoid further misunderstandings
Is there a general approach how one should handle deadlocking third-party code?
Yes, but it's not easy or simple.
The problem with misbehaving code is that it can not only leak resources (e.g., threads), but it can also indefinitely hold onto important resources (e.g., some internal "handle" or "lock").
The only way to forcefully reclaim threads and other resources is to end the process. The OS is used to cleaning up misbehaving processes and is very good at it. So, the solution here is to start a child process to do the API call. Your main application can communicate with its child process by redirected stdin/stdout, and if the child process ever times out, the main application can terminate it and restart it.
This is, unfortunately, the only reliable way to cancel uncancelable code.
Cancelling a task is a collaborative operation in that you pass a CancellationToken to the desired method and externally you use CancellationTokenSource.Cancel:
public void Caller()
{
try
{
CancellationTokenSource cts=new CancellationTokenSource();
Task longRunning= Task.Run(()=>CancellableThirdParty(cts.Token),cts.Token);
Thread.Sleep(3000); //or condition /signal
cts.Cancel();
}catch(OperationCancelledException ex)
{
//treat somehow
}
}
public void CancellableThirdParty(CancellationToken token)
{
while(true)
{
// token.ThrowIfCancellationRequested() -- if you don't treat the cancellation here
if(token.IsCancellationRequested)
{
// code to treat the cancellation signal
//throw new OperationCancelledException($"[Reason]");
}
}
}
As you can see in the code above , in order to cancel an ongoing task , the method running inside it must be structured around the CancellationToken.IsCancellationRequested flag or simply CancellationToken.ThrowIfCancellationRequested method ,
so that the caller just issues the CancellationTokenSource.Cancel.
Unfortunately if the third party code is not designed around CancellationToken ( it does not accept a CancellationToken parameter ), then there is not much you can do.
Your code isn't cancelling the blocked operation. Use a CancellationTokenSource and pass a cancellation token to Task.Run instead :
var cts=new CancellationTokenSource(timeout);
try
{
await Task.Run(() => ThirdPartyCode.Ping(),cts.Token);
return true;
}
catch(TaskCancelledException)
{
return false;
}
It's quite possible that blocking is caused due to networking or DNS issues, not actual deadlock.
That still wastes a thread waiting for a network operation to complete. You could use .NET's own Ping.SendPingAsync to ping asynchronously and specify a timeout:
var ping=new Ping();
var reply=await ping.SendPingAsync(ip,timeout);
return reply.Status==IPStatus.Success;
The PingReply class contains far more detailed information than a simple success/failure. The Status property alone differentiates between routing problems, unreachable destinations, time outs etc
Windows service: Generating a set of FileWatcher objects from a list of directories to watch in a config file, have the following requirements:
File processing can be time consuming - events must be handled on their own task threads
Keep handles to the event handler tasks to wait for completion in an OnStop() event.
Track the hashes of uploaded files; don't reprocess if not different
Persist the file hashes to allow OnStart() to process files uploaded while the service was down.
Never process a file more than once.
(Regarding #3, we do get events when there are no changes... most notably because of the duplicate-event issue with FileWatchers)
To do these things, I have two dictionaries - one for the files uploaded, and one for the tasks themselves. Both objects are static, and I need to lock them when adding/removing/updating files and tasks. Simplified code:
public sealed class TrackingFileSystemWatcher : FileSystemWatcher {
private static readonly object fileWatcherDictionaryLock = new object();
private static readonly object runningTaskDictionaryLock = new object();
private readonly Dictionary<int, Task> runningTaskDictionary = new Dictionary<int, Task>(15);
private readonly Dictionary<string, FileSystemWatcherProperties> fileWatcherDictionary = new Dictionary<string, FileSystemWatcherProperties>();
// Wired up elsewhere
private void OnChanged(object sender, FileSystemEventArgs eventArgs) {
this.ProcessModifiedDatafeed(eventArgs);
}
private void ProcessModifiedDatafeed(FileSystemEventArgs eventArgs) {
lock (TrackingFileSystemWatcher.fileWatcherDictionaryLock) {
// Read the file and generate hash here
// Properties if the file has been processed before
// ContainsNonNullKey is an extension method
if (this.fileWatcherDictionary.ContainsNonNullKey(eventArgs.FullPath)) {
try {
fileProperties = this.fileWatcherDictionary[eventArgs.FullPath];
}
catch (KeyNotFoundException keyNotFoundException) {}
catch (ArgumentNullException argumentNullException) {}
}
else {
// Create a new properties object
}
fileProperties.ChangeType = eventArgs.ChangeType;
fileProperties.FileContentsHash = md5Hash;
fileProperties.LastEventTimestamp = DateTime.Now;
Task task;
try {
task = new Task(() => new DatafeedUploadHandler().UploadDatafeed(this.legalOrg, datafeedFileData), TaskCreationOptions.LongRunning);
}
catch {
..
}
// Only lock long enough to add the task to the dictionary
lock (TrackingFileSystemWatcher.runningTaskDictionaryLock) {
try {
this.runningTaskDictionary.Add(task.Id, task);
}
catch {
..
}
}
try {
task.ContinueWith(t => {
try {
lock (TrackingFileSystemWatcher.runningTaskDictionaryLock) {
this.runningTaskDictionary.Remove(t.Id);
}
// Will this lock burn me?
lock (TrackingFileSystemWatcher.fileWatcherDictionaryLock) {
// Persist the file watcher properties to
// disk for recovery at OnStart()
}
}
catch {
..
}
});
task.Start();
}
catch {
..
}
}
}
}
What's the effect of requesting a lock on the FileSystemWatcher collection in the ContinueWith() delegate when the delegate is defined within a lock on the same object? I would expect it to be fine, that even if the task starts, completes, and enters the ContinueWith() before ProcessModifiedDatafeed() releases the lock, the task thread would simply be suspended until the creating thread has released the lock. But I want to make sure I'm not stepping on any delayed execution landmines.
Looking at the code, I may be able to release the lock sooner, avoiding the issue, but I'm not certain yet... need to review the full code to be sure.
UPDATE
To stem the rising "this code is terrible" comments, there are very good reasons why I catch the exceptions I do, and am catching so many of them. This is a Windows service with multi-threaded handlers, and it may not crash. Ever. Which it will do if any of those threads have an unhandled exception.
Also, those exceptions are written to future bulletproofing. The example I've given in comments below would be adding a factory for the handlers... as the code is written today, there will never be a null task, but if the factory is not implemented correctly, the code could throw an exception. Yes, that should be caught in testing. However, I have junior developers on my team... "May. Not. Crash." (also, it must shut down gracefully if there is an unhandled exception, allowing currently-running threads to complete - which we do with an unhandled exception handler set in main()). We have enterprise-level monitors configured to send alerts when application errors appear on the event log – those exceptions will log and flag us. The approach was a deliberate and discussed decision.
Each possible exception has each been carefully considered and chosen to fall into one of two categories - those that apply to a single datafeed and will not shut down the service (the majority), and those that indicate clear programming or other errors that fundamentally render the code useless for all datafeeds. For example, we've chosen to shut down the service down if we can't write to the event log, as that's our primary mechanism for indicating datafeeds are not getting processed. The exceptions are caught locally, because the local context is the only place where the decision to continue can be made. Furthermore, allowing exceptions to bubble up to higher levels (1) violates the concept of abstraction, and (2) makes no sense in a worker thread.
I'm surprised at the number of people who argue against handling exceptions. If I had a dime for every try..catch(Exception){do nothing} I see, you'd get your change in nickels for the rest of eternity. I would argue to the death1 that if a call into the .NET framework or your own code throws an exception, you need to consider the scenario that would cause that exception to occur and explicitly decide how it should be handled. My code catches UnauthorizedExceptions in IO operations, because when I considered how that could happen, I realized that adding a new datafeed directory requires permissions to be granted to the service account (it won't have them by default).
I appreciate the constructive input... just please don't criticize simplified example code with a broad "this sucks" brush. The code does not suck - it is bulletproof, and necessarily so.
1 I would only argue a really long time if Jon Skeet disagrees
First, your question: it's not a problem in itself to request lock inside ContinueWith. If you bother you do that inside another lock block - just don't. Your continuation will execute asynchronously, in different time, different thread.
Now, code itself is questionable. Why do you use many try-catch blocks around statements that almost cannot throw exceptions? For example here:
try {
task = new Task(() => new DatafeedUploadHandler().UploadDatafeed(this.legalOrg, datafeedFileData), TaskCreationOptions.LongRunning);
}
catch {}
You just create task - I cannot imagine when this can throw. Same story with ContinueWith. Here:
this.runningTaskDictionary.Add(task.Id, task);
you can just check if such key already exists. But even that is not necessary because task.Id is unique id for given task instance which you just created. This:
try {
fileProperties = this.fileWatcherDictionary[eventArgs.FullPath];
}
catch (KeyNotFoundException keyNotFoundException) {}
catch (ArgumentNullException argumentNullException) {}
is even worse. You should not use exceptions lile this - don't catch KeyNotFoundException but use appropriate methods on Dictionary (like TryGetValue).
So to start with, remove all try catch blocks and either use one for the whole method, or use them on statements that can really throw exceptions and you cannot handle that situation otherwise (and you know what to do with exception thrown).
Then, your approach to handle filesystem events is not quite scaleable and reliable. Many programs will generate multiple change events in short intervals when they are saving changes to a file (there are also other cases of multiple events for the same file going in sequence). If you just start processing file on every event, this might lead to different kind of troubles. So you might need to throttle events coming for a given file and only start processing after certain delay after last detected change. That might be a bit advanced stuff, though.
Don't forget to grab a read lock on the file as soon as possible, so that other processes cannot change file while you are working with it (for example, you might calculate md5 of a file, then someone changes file, then you start uploading - now your md5 is invalid). Other approach is to record last write time and when it comes to uploading - grab read lock and check if file was not changed in between.
What is more important is that there can be a lot of changes at once. Say I copied 1000 files very fast - you do not want to start uploading them all at once with 1000 threads. You need a queue of files to process, and take items from that queue with several threads. This way thousands of events might happen at once and your upload will still work reliably. Right now you create new thread for each change event, where you immediatly start upload (according to method names) - this will fail under serious load of events (and in cases described above).
No it will not burn you. Even if the ContinueWith is inlined into to the current thread that was running the new Task(() => new DatafeedUploadHandler().. it will get the lock e.g. no dead lock.
The lock statement is using the Monitor class internally, and it is reentrant. e.g. a thread can aquire a lock multiple times if it already got/owns the lock. Multithreading and Locking (Thread-Safe operations)
And the other case where the task.ContinueWith starts before the ProcessModifiedDatafeed finished is like you said. The thread that is running the ContinueWith simply would have to wait to get the lock.
I would really consider to do the task.ContinueWith and the task.Start() outside of the lock if you reviewed it. And it is possible based on your posted code.
You should also take a look at the ConcurrentDictionary in the System.Collections.Concurrent namespace. It would make the code easier and you dont have to manage the locking yourself. You are doing some kind of compare exchange/update here if (this.fileWatcherDictionary.ContainsNonNullKey(eventArgs.FullPath)). e.g. only add if not already in the dictionary. This is one atomic operation. There is no function to do this with a ConcurrentDictionary but there is an AddOrUpdate method. Maybe you can rewrite it by using this method. And based on your code you could safely use the ConcurrentDictionary at least for the runningTaskDictionary
Oh and TaskCreationOptions.LongRunning is literally creating a new thread for every task which is kind of an expensive operation. The windows internal thread pool is intelligent in new windows versions and is adapting dynamically. It will "see" that you are doing lots of IO stuff and will spawn new threads as needed and practical.
Greetings
I have not fully followed the logic of this code but are you aware that task continuations and calls to Wait/Result can be inlined onto the current thread? This can cause reentrancy.
This is very dangerous and has burned many.
Also I don't quite see why you are starting task delayed. This is a code smell. Also why are you wrapping the task creation with try? This can never throw.
This clearly is a partial answer. But the code looks very tangled to me. If it's this hard to audit it you probably should write it differently in the first place.
I'm writing a Windows Store app that will rely on a JSON api. The provider of the API asks that no more than 1 api request is made per second.
So I created a class that would allow me to queue requests on a blocking queue, and on a background thread it would run a loop that resembles the following:
Loop
{
// this will block until a request is added to the queue
MyRequest = Queue.Take()
// Create task to make the api request here.
Thread.Sleep(1000)
}
This way, it would wait at least one second before trying to Dequeue another request.
I've found that Thread.Sleep is not available for windows store apps. Task.Delay() seems unnecessarily wasteful, since it will create a new task each time it is called.
I feel like there is probably a known way to do this that I'm not aware of?
Thanks,
I know Task.Delay() seems wasteful, but it's recommended by a Microsoft Employee and moderator on MSDN here
Using .Sleep() or an infinite loop like TGH suggests would cause the program to become unresponsive while it waits. If you want it to be responsive while processing your queue, you'd use something like this:
await Task.Delay(1000);
If you want the task to execute in the background, I would suggest a System.Threading.Timer. Set it up as a one-shot rather than a periodic timer, and refresh it after every request. Something like:
System.Threading.Timer myTimer;
// initialize timer
myTimer = new Timer(TimerProc, null, 1000, Timeout.Infinite);
void TimerProc(object state)
{
// do request here
// then reset the timer
myTimer.Change(1000, Timeout.Infinite);
}
That will have the effect of waiting one second between when one request ends and the next one starts.
If you want ticks to execute on one-second intervals, you have to make sure that they can't overlap. That's easy enough with Monitor. You'll need a lock object, and to make your timer an periodic timer:
object timerLock = new object();
// initialize periodic timer
myTimer = new Timer(TimerProc, null, 1000, 1000);
And your timer proc:
void TimerProc(object state)
{
if (!Monitor.TryEnter(timerLock))
{
// a request is currently being processed
return;
}
try
{
// do request here
}
catch (expected exceptions)
{
}
Monitor.Exit(timerLock);
}
Note that I don't do the Monitor.Exit in a finally. See Eric Lippert's Locks and exceptions do not mix for the reasons why.
I have created a timeout function based on things I have seen in various places but am pretty sure I am not doing it a great way! (But it does seem to work.)
I am connecting to a piece of hardware that if working connects in a few seconds but if not takes around 1 minute to timeout. So if I can create my own timeout function I can set it at 20 seconds and save lots of time and waiting.
I have tried to make it so my timeout returns a string:
static string CallWithTimeout(Action action, int timeoutMilliseconds)
{
string reply = "";
Thread threadToKill = null;
Action wrappedAction = () =>
{
threadToKill = Thread.CurrentThread;
action();
};
IAsyncResult result = wrappedAction.BeginInvoke(null, null);
if (result.AsyncWaitHandle.WaitOne(timeoutMilliseconds))
{
reply = "Connected";
wrappedAction.EndInvoke(result);
return reply;
}
else
{
threadToKill.Abort();
reply = "Error";
return reply;
}
}
then I call it with something like :
string replyfromreader = CallWithTimeout(connectToHardware, 20000);
the connectToHardware is just a one liner so no need to post.
It's okayish as far as .NET state is concerned. You won't call EndInvoke(), that leaks resources for 10 minutes, the default lifetime of remoted objects.
In a case like this, calling Thread.Abort() has a very small chance of succeeding. A managed thread needs to be in an alertable wait state to be abortable, it just never is when the thread is buried deep inside native code that ultimately waits for some device driver call to complete.
Leaving the CLR in a state where it keeps trying to abort a thread and never succeeds is not particularly pleasant, not something I've ever tried on purpose so no real idea what the side-effects are. It does however mean that your code will block on the Abort() method call so you still haven't fixed the problem. The best thing to do is therefore to not abort the thread but just abandon it. Setting a flag that marks the device dead so you don't try to do this ever again.
If you want to continue running your program, even without the device being in a usable state, and you want to provide a way to recover from the problem then you'll need an entirely different approach. You'll need to put the device related code in a separate process. Which you can then Kill() when the device is unresponsive, relying on Windows to clean up the shrapnel. Interop with that process using a low-level mechanism like named pipes or sockets is best so you can recover from the disconnect fairly easily.
Avoiding Thread.Abort is always a good idea. Avoiding it on a thread you did not create is even better.
Assuming if the hardware is not working, and you want the timeout, it does not matter if connectToHardware is left to timeout on its own and no error/exception details are wanted, then you can use the Task Parallel Library (TPL): System.Threading.Tasks.Task:
// True => worked, False => timeout
public static bool CallWithTimeout(Action method, TimeSpan timeout) {
Exception e;
Task worker = Task.Factory.StartNew(method)
.ContineueWith(t => {
// Ensure any exception is observed, is no-op if no exception.
// Using closure to help avoid this being optimised out.
e = t.Exception;
});
return worker.Wait(timeout);
}
(If the passed Action could interact with a passed CancellationToken this could be made cleaner, allowing the underlying method to fail quickly on timeout.)
I've inherited a multi-threaded Windows Service (C#, .NET) with no support for unhanded exceptions. The service is using multiple threads to collect data from telemetry devices for analysis. A thread in this service could run for 24 hours or more. I'm struggling with how to tell the main thread that a thread is experiencing problems and needs to recycle. Here's a simplified view of the code:
class Program
{
static void Main(string[] args)
{
var workerObject = new WorkerObject();
var workerThread = new Thread(workerObject.DoWork);
workerThread.Start()
}
}
class WorkerObject
{
//todo: My fields and properties go here
public void DoWork()
{
const int timeout = 100;
//todo: setup wait handles here
try
{
//Start monitoring channels for this device
while (true)
{
// Wait on any waithandle or timeout once per decisecond.
int channelIndex = WaitHandle.WaitAny(waitHandles, timeout, false);
//todo: process incoming data for this channel
}
}
//Catch all exceptions so we can notify mommy that we're in a bad way
catch (Exception ex)
{
//todo: log anything that can help us figure this problem out
//How do I tell the main thread I've failed
}
finally
{
//But of course we want to clean up after ourselves
}
}
}
Is this even the best .NET threading mechanism to use? Should I be using Async Delegates or Task Parallel Library instead?
I had a similar requirement recently. I don't think that you can rely on the child threads telling the parent when they are bad. What if they are caught in an infinite loop or deadlocked for some reason? I settled in on a 'watch dog' approach. Each of the child thread needed to send a 'heartbeat' back to the main thread. If this signal was not received, then the thread was assumed 'dead' and an action could be taken. Here is a post about the architecture I settled in on:
http://blog.bobcravens.com/2009/08/monitored-watchdog-asynchronous-process-in-c/
Hope this helps.
Bob
Daniel Moth has blogged a lot about multithreading and more recently the Parallel Tasks support in .net 4.0. It's a great resource, http://www.danielmoth.com and very interesting. I'd definitely check that out.
I'm not sure if this helps but I worked on a "Report Runner" Windows Service which takes jobs from a queue and then schedules the work on the ThreadPool; this worked well because if a "Report" fails, I just log the error and exit the function and since it was scheduled in the ThreadPool it just gets returned to the pool to run more reports. Using the ThreadPool avoids the code needed to manage spinning up a new thread if the code you're running fails.