We Keep Coding

How to implement delay that can be cancelled?

I have a windows service that is responsible for listening JMS messages. I am giving a simplified version of implementation details. As messages arrive they are handed over for processing to a different Task (thread) and limit a max number of tasks with the help of BlockingCollection. There is a retry mechanism in place to retry until the processing is successful with some amount of delay between each retry or max retry attempts are exhausted. The reason for retry mechanism is to cope with issues in Legacy applications that consume these messages. Legacy systems are built using Pessimistic locking and sometimes the processing of message runs into errors, which eventually goes thru after few retry attempts. Due to cost benefit analysis, it was decided not to address the issues in Legacy systems as those applications will be replaced in 2 to 3 years.
This retry mechanism runs on the same task thread that is responsible for handling the processing of message. Initially I used Thread.Sleep to introduce delay between each retry attempt. It worked, but when I try to shutdown the windows service, it is taking longer if there are messages currently being processed and waiting to be retried.
I then went on an adventure of implementing a way to cancel the waiting mechanism if a shutdown event was triggered.
I used two different approaches.
Option #1
One using ManualResetEvent and when I have to wait I have following code in place (posting only relevant code blocks)
private readonly ManualResetEvent _lockEvent = new ManualResetEvent(false);
if (_lockEvent.WaitOne(TimeSpan.FromMilliseconds(120000)))
{
Log.Info($"Thread interrupted. Retrying will resume after windows service restarts for message id {messageId}");
return;
}
When a shutdown event occurs, I cancel the cancellationTokenSource and set the ManualResetEvent. Everything appears to do what I want. Its just that I have to do two operations so that any code that depends on CancellationToken know to gracefully cancel and also gracefully break the retry waiting.
_subscriberCancellationTokenSource.Cancel();
_lockEvent.Set();
Option #2
After upgrading to .Net 4.6, I started using Task type wherever I can. I realized, I could use Task to implement a delay as well, so here is a simplified version of code that I tried
private void WaitBeforeRetrying(CancellationToken cancellationToken)
{
var waitingTask = Task.Delay(120000, cancellationToken);
waitingTask.Wait(cancellationToken);
}
Where ever I need delay, I just invoke the method by passing a CancellationToken
WaitBeforeRetrying(SubscriberCancellationToken);
When a shutdown event occurs, I simply invoke cancel on CancellationTokenSource and everything shuts down gracefully.
_subscriberCancellationTokenSource.Cancel();
Both Option 1 and Option 2 appears to be doing the job.
Are there any drawbacks for Option 2 over Option 1? Any other better option than what I have so far? Really appreciate any input.
UPDATE
After reading the comments from #EricLippert, I understood what I was doing wrong. Most of my threads were going into a waiting state instead of actually doing any productive work. It was the result of sprinkling few asynchronous calls in the synchronous workflow.
I now modified my delay method as follows
private async Task WaitBeforeRetrying(CancellationToken cancellationToken)
{
await Task.Delay(120000, cancellationToken);
}
And I invoke it as
await WaitBeforeRetrying(SubscriberCancellationToken);
And then refactored rest of the code to propagate async mechanism all the way to the top layer. It not only helped to easily cancel the delay if I don't have to wait, but also prevented the threads to be in a blocked state unnecessarily. Really appreciate every ones feedback.

I don't think they're any different. The effect of both is that the thread is blocked until the time runs out.
If you're using this in ASP.NET, then blocking threads is not a good thing. In that case, you can make your method async and use await Task.Delay. That'll resume the code after the delay, but allow the thread to work on other things in the mean time.
private async Task WaitBeforeRetrying(CancellationToken cancellationToken)
{
await Task.Delay(120000, cancellationToken);
}

Continuous processing when thread is waiting

I'm executing a web request to get a message, then await the processing of that message, then repeat the whole process again.
The processing of the message will be long running and the thread may be in a waiting state that may allow it to be used elsewhere. What I'd like is to continue the while loop, get more messages and process them when threads become free.
Current synchronous code:
while(!cancellationToken.IsCancelled) {
var message = await GetMessage();
await ProcessMessage(message); // I'll need it to continue from here if thread is released.
}
The scenario this is used in is a message queue Consumer service.

Given the use of async / await, your current code isn't necessarily synchronous (in thread terms - the continuations can be invoked on different threads), although the dependency between getting a message and processing it obviously must be upheld.
Re: the thread may be in a waiting state that may allow it to be used elsewhere
Awaiting on well coded I/O-bound work doesn't need to consume a thread at all - see Stephen Cleary's There is no thread. Assuming the two awaited tasks are IO-bound, your code will likely consume no threads at all while it is awaiting IO bound work, i.e. the rest of your application will have the use of the Threadpool. So if your only concern was wasting threads, then nothing more is needed.
If however your concern is about performance and additional throughput, if there is downstream capacity to do concurrent calls to ProcessMessage (e.g. multiple downstream web servers or additional database capacity), then you could look at parallelizing the IO bound work (again, without requiring more Threadpool threads)
For instance, if you are able to re-write the GetMessages call to retrieve a batch at a time, you could try this:
var messages = await GetMessages(10);
var processTasks = messages
.Select(message => ProcessMessage(message));
await Task.WhenAll(processTasks);
(and if you can't touch code, you could just loop GetMessages to retrieve 10 individual messages before the Task.WhenAll)
However, if you do NOT have any further capacity to do concurrent ProcessMessage calls, then you should instead look at addressing the bottleneck - e.g. adding more servers, optimizing code, or parallelizing the work done in ProcessMessage work, etc.
The rationale is that, as you say, GetMessages retrieves data off a queue. If you have no capacity to process the messages you've retrieved, all you could do is queue messages somewhere else, which seems rather pointless - rather leave the messages on the Queue until you are ready to process them. The queue depth will also create visibility of the backlog of work building up, which you can monitor.
Edit, Re : Occasionally one ProcessMessage() call takes much longer than others
As per the comments, OP has additional information that an occasional ProcessMessage call takes much longer than others, and would like to continue processing other messages in the interim.
One approach could be to apply a timeout to the Parallel tasks using this clever pattern here, which, if reached, will leave any long running ProcessTasks running, and will continue with the next batch of messages.
The below is potentially dangerous, in that it will require careful balancing of the timeout (1000ms below) against the observed frequency of the misbehaving ProcessMessage calls - if the timeout is too low vs the frequency of 'slow' ProcessMessages, the downstream resources can become overwhelmed.
A safer (yet more complicated) addition would be to track the concurrent number of incomplete ProcessMessage tasks via Task.IsCompleted, and if this hits a threshold, then to await completion of enough of these tasks to bring the backlog to a safe level.
while(!cancellationToken.IsCancelled)
{
// Ideally, the async operations should all accept cancellationTokens too
var message = await GetMessages(10, cancellationToken);
var processTasks = messages
.Select(message => ProcessMessage(message, cancellationToken));
await Task.WhenAny(Task.WhenAll(processTasks),
Task.Delay(1000, cancellationToken));
}
Re : Throttling for safe levels of downstream load - TPL DataFlow more than likely would be of use here.

Take a look at https://msdn.microsoft.com/library/hh191443(vs.110).aspx should get you going. Also, seems like ProcessMessage aught to end with 'Async' according to the C#/.NET style guide.
You'll want to set up a Task<ReturnTypeOfProcessMessage> procMessageTask = ProcessMessageAsync(message);
then you can do your business while its running,
SomeBusiness(...)
then
await procMessageTask;
Seems like you may also want some type of await-with-timeout functionality so that you can poll, here's a question related to that:
Asynchronously wait for Task<T> to complete with timeout
HTH

Event handler timing and threading

I am still learning C# so please be easy on me. I am thinking about my application I am working on and I can't seem to figure out the best approach. This is not a forms application but rather a console. I am listening to a UDP port. I get UDP messages as fast as 10 times per second. I then look for a trigger in the UDP message. I am using an event handler that is raised each time i get a new UDP packet which will then call methods to parse the packet and look for my trigger. So, i have these questions.
With regard to threading, I assume a thread like my thread that listens to the UDP data should be a permanent thread?
Also on threading, when I get my trigger and decide to do something, in this case send a message out, i gather that I should use a thread pool each time I want to perform this task?
On thread pools, I am reading that they are not very high priority, is that true? If the message I need to send out is critical, can i rely on thread pools?
With the event handler which is raised when i get a UDP packet and then calls methods, what is the best way to ensure my methods all complete before the next packet/event is raised? At times I see event queue problems because if any of the methods take a bit longer than they should (for exampe writing to a DB) and the next packet comes in 100ms later, you get event queue growth because you cannot consume events in a timely manner. Is there a good way to address this?

With regard to threading, I assume a thread like my thread that listens to the UDP data should be a permanent thread?
There are no permanent threads. However there should be a thread that is responsible for receiving. Once you start it, let it run until you no longer need to receive any messages.
Also on threading, when I get my trigger and decide to do something, in this case send a message out, i gather that I should use a thread pool each time I want to perform this task?
That depends on how often would you send out messages. If your situation is more like consumer/producer than a separate thread for sending is a good idea. But if you send out a message only rarely, you can use thread pool. I can't define how often rare means in this case, you should watch your app and decide.
On thread pools, I am reading that they are not very high priority, is that true? If the message I need to send out is critical, can i rely on thread pools?
You can, it's more like your message will be delayed because of slow message processing or slow network rather than the thread pool.
With the event handler which is raised when i get a UDP packet and then calls methods, what is the best way to ensure my methods all complete before the next packet/event is raised? At times I see event queue problems because if any of the methods take a bit longer than they should (for exampe writing to a DB) and the next packet comes in 100ms later, you get event queue growth because you cannot consume events in a timely manner. Is there a good way to address this?
Queue is a perfect solution. You can have more queues if some messages are independent of others and their execution won't collide and then execute them in parallel.

I'll adress your points:
your listeting thread must be a 'permanent' thread that gets messages and distribute them.
(2+3) - Look at the TPL libarary you should use it instead of working with threads and thread pools (unless you need some fine control over the operations which, from your question, seems like you dont need) - as MSDN states:
The Task Parallel Library (TPL) is based on the concept of a task, which represents an asynchronous operation. In some ways, a task resembles a thread or ThreadPool work item, but at a higher level of abstraction
Look into using MessageQueues since what you need is a place to receive messages, store them for some time (in memory in your case)and handle them at your own pace.
You could implement this yourself but you'll find it gets complicated quickly,
I recommend looking into NetMQ - it's easy to use, especially for what you describe, and it's in c#.

How to create a "Spool" service for a class in C#

I am looking into a C# programming fairly scrub to the language. I would like to think I have a good understanding of object oriented programming in general, and what running multiple threads means, at a high level, but actual implementation I am as said scrub.
What I am looking to do is to create a tool that will have many threads running and interacting with each other independent, each will serve their own task and may call others.
My strategy to ensure communication (without losing anything with multiple updates occurring same time from different threads) is on each class to create a spool like task that can be called external, and add tasks to a given thread, or spool service for these. I am not sure if I should place this on the class or external and have the class itself call the spool for new tasks and keeping track of the spool. Here I am in particular considering how to signal the class if an empty spool gets a task (listener approach, so tasks can subscribe to pools if they want to be awoken if new stuff arrive), or make a "check every X seconds if out of tasks and next task is not scheduled" approach
What would a good strategy be to create this, should I create this in the actual class, or external? What are the critical regions in the implementation, as the "busy wait check" allows it to only be on adding new jobs, and removing jobs on the actual spool, while the signaling will require both adding/removing jobs, but also the goto sleep on signaling to be critical, and that suddenly add a high requirement for the spool of what to do if the critical region has entered, as this could result in blocks, causing other blocks, and possible unforeseen deadlocks.

I use such a model often, on various systems. I define a class for the agents, say 'AgentClass' and one for the requests, say 'RequestClass'. The agent has two abstract methods, 'submit(RequestClass *message)' and 'signal()'. Typically, a thread in the agent constructs a producer-consumer queue and waits on it for RequestClass instances, the submit() method queueing the passed RequestClass instances to the queue. The RequestClass usually contains a 'command' enumeration that tells the agent what needs doing, together with all data required to perform the request and the 'sender' agent instance. When an agent gets a request, it switches on the enumeration to call the correct function to do the request. The agent acts only on the data in the RequestClass - results, error messages etc. are placed in data members of the RequestClass. When the agent has performed the request, (or failed and generated error data), it can either submit() the request back to the sender, (ie. the request has been performed asynchronously), or call the senders signal() function, whch signals an event upon which the sender was waiting, (ie. the request was performed synchronously).
I usually construct a fixed number of RequestClass instances at startup and store them in a global 'pool' P-C queue. Any agent/thread/whatever than needs to send a request can dequeue a RequestClass instance, fill in data, submit() it to the agent and then wait asynchronously or synchronously for the request to be performed. When done with, the RequestClass is returned to the pool. I do this to avoid continual malloc/free/new/dispose, ease debugging, (I dump the pool level to a status bar using a timer, so I always notice if a request leaks or gets double-freed), and to eliminate the need for explicit thread termination on app close, (if multiple threads are only ever reading/writing to data areas that outlive the application forms etc, the app will close easily and the OS can deal with all the threads - there are hundreds of posts about 'cleanly shutting down threads upon app close' - I never bother!).
Such message-passing designs are quite resistant to deadlocks since the only locks, (if any), are in the P-C queues, though you can certainly achieve it if you try hard enough:)
Is this the sort of system that you seem to need , or have I got it wrong?
Rgds,
Martin

how to create singleton which always running in separate thread?

I would like to rephrase my previous question How to create Singleton with async method?
Imagine messaging application (like icq) - something that should be always connected to server and can post messages.
I need to implment class Connection. It should be singleton, because it contains "socket" inside and that socket should persist during entirely application lifetime.
Then I want to implement async method Connection.postMessage
Because postMessage can take significant ammount of time:
postMessage should be async
postMessage should queue messages if neccesary
Note my application posts dozens messages per second, so it is not appropiate to create new Thread for each postMessage call.
I diffenetely need to create exactly one extra thread for messages posting but I don't know where and how.
upd: good example http://msdn.microsoft.com/en-us/library/yy12yx1f(v=vs.80).aspx

No, Postmessage (itself) should not be async .
It should
be Thread-safe
ensure the Processing thread is running
queue the message (ConcurrentQueue)
return
And the Processing Thread should
Wait on the Queue
Process the messages
maybe Terminate itself when idle for xx milliseconds
What you have is a classic Producer/Consumer situation with 1 Consumer and multiple Producers.
PostMessage is the entry-point for all producers.

jp,
You're looking at a classic producer/consumer problem here... During initialisation the Connection should create a MessageQueue start a Sender in it's own background thread.
Then the connection posts just messages to the queue, for the Sender to pickup and forward when ready.
The tricky bit is managing the maximum queue size... If the producer consistently outruns the consumer then queue can grow to an unmanagable size. The simplest approach is to block the producer thread until the queue is no longer full. This can be done with a back-off-ARQ. ie: while(queue.isFull) sleep(100, "milliseconds"); queue.add(message); If you don't require 100% transmission (like a chat-app, for instance) then you can simply throw a MessageQueueFullException, and the poor client will just have to get over it... just allways allow them to resubmit later... allowing the user manage the retrys for you.
That's how I'd tackle it anyway. I'll be interested to see what others suggestions are muted.
Hope things work out for you. Cheers. Keith.