We Keep Coding

How to synchronize the recurrent execution of three tasks that depend on each other?

I would like to ask expert developers in C#. I have three recurrent tasks that my program needs to do. Task 2 depends on task 1 and task 3 depends on task 2, but task 1 doesn't need to wait for the other two tasks to finish in order to start again (the program is continuously running). Since each task takes some time, I would like to run each task in one thread or a C# Task. Once task 1 finishes task 2 starts and task 1 starts again ... etc.
I'm not sure what is the best way to implement this. I hope someone can guide me on this.

One way to achieve this is using something called the the Task Parallel Library. This provides a set of classes that allow you to arrange your tasks into "blocks". You create a method that does A, B and C sequentially, then TPL will take care of running multiple invocations of that method simultaneously. Here's a small example:
async Task Main()
{
var actionBlock = new ActionBlock<int>(DoTasksAsync, new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 2 // This is the number of simultaneous executions of DoTasksAsync that will be run
};
await actionBlock.SendAsync(1);
await actionBlock.SendAsync(2);
actionBlock.Complete();
await actionBlock.Completion;
}
async Task DoTasksAsync(int input)
{
await DoTaskAAsync();
await DoTaskBAsync();
await DoTaskCAsync();
}

I would probably use some kind of queue pattern.
I am not sure what the requirements for if task 1 is threadsafe or not, so I will keep it simple:
Task 1 is always executing. As soon as it finished, it posts a message on some queue and starts over.
Task 2 is listening to the queue. Whenever a message is available, it starts working on it.
Whenever task 2 finishes working, it calls task 3, so that it can do it's work.
As one of the comments mentioned, you should probably be able to use async/await successfully in your code. Especially between task 2 and 3. Note that task 1 can be run in parallel to task 2 and 3, since it is not dependent on any of the other task.

You could use the ParallelLoop method below. This method starts an asynchronous workflow, where the three tasks are invoked in parallel to each other, but sequentially to themselves. So you don't need to add synchronization inside each task, unless some task produces global side-effects that are visible from some other task.
The tasks are invoked on the ThreadPool, with the Task.Run method.
/// <summary>
/// Invokes three actions repeatedly in parallel on the ThreadPool, with the
/// action2 depending on the action1, and the action3 depending on the action2.
/// Each action is invoked sequentially to itself.
/// </summary>
public static async Task ParallelLoop<TResult1, TResult2>(
Func<TResult1> action1,
Func<TResult1, TResult2> action2,
Action<TResult2> action3,
CancellationToken cancellationToken = default)
{
// Arguments validation omitted
var task1 = Task.FromResult<TResult1>(default);
var task2 = Task.FromResult<TResult2>(default);
var task3 = Task.CompletedTask;
try
{
int counter = 0;
while (true)
{
counter++;
var result1 = await task1.ConfigureAwait(false);
cancellationToken.ThrowIfCancellationRequested();
task1 = Task.Run(action1); // Restart the task1
if (counter <= 1) continue; // In the first loop result1 is undefined
var result2 = await task2.ConfigureAwait(false);
cancellationToken.ThrowIfCancellationRequested();
task2 = Task.Run(() => action2(result1)); // Restart the task2
if (counter <= 2) continue; // In the second loop result2 is undefined
await task3.ConfigureAwait(false);
cancellationToken.ThrowIfCancellationRequested();
task3 = Task.Run(() => action3(result2)); // Restart the task3
}
}
finally
{
// Prevent fire-and-forget
Task allTasks = Task.WhenAll(task1, task2, task3);
try { await allTasks.ConfigureAwait(false); } catch { allTasks.Wait(); }
// Propagate all errors in an AggregateException
}
}
There is an obvious pattern in the implementation, that makes it trivial to add overloads having more than three actions. Each added action will require its own generic type parameter (TResult3, TResult4 etc).
Usage example:
var cts = new CancellationTokenSource();
Task loopTask = ParallelLoop(() =>
{
// First task
Thread.Sleep(1000); // Simulates synchronous work
return "OK"; // The result that is passed to the second task
}, result =>
{
// Second task
Thread.Sleep(1000); // Simulates synchronous work
return result + "!"; // The result that is passed to the third task
}, result =>
{
// Third task
Thread.Sleep(1000); // Simulates synchronous work
}, cts.Token);
In case any of the tasks fails, the whole loop will stop (with the loopTask.Exception containing the error). Since the tasks depend on each other, recovering from a single failed task is not possible¹. What you could do is to execute the whole loop through a Polly Retry policy, to make sure that the loop will be reincarnated in case of failure. If you are unfamiliar with the Polly library, you could use the simple and featureless RetryUntilCanceled method below:
public static async Task RetryUntilCanceled(Func<Task> action,
CancellationToken cancellationToken)
{
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
try { await action().ConfigureAwait(false); }
catch { if (cancellationToken.IsCancellationRequested) throw; }
}
}
Usage:
Task loopTask = RetryUntilCanceled(() => ParallelLoop(() =>
{
//...
}, cts.Token), cts.Token);
Before exiting the process you are advised to Cancel() the CancellationTokenSource and Wait() (or await) the loopTask, in order for the loop to terminate gracefully. Otherwise some tasks may be aborted in the middle of their work.
¹ It is actually possible, and probably preferable, to execute each individual task through a Polly Retry policy. The parallel loop will be suspended until the failed task is retried successfully.

Propagation time of cancellation request to all tasks (TPL)

With TPL we have CancellationTokenSource which provides tokens, useful to cooperatively cancellation of current task (or its start).
Question:
How long it take to propagate cancellation request to all hooked running tasks?
Is there any place, where code could look to check that: "from now" every interested Task, will find that cancellation has been requested?
Why there is need for it?
I would like to have stable unit test, to show that cancellation works in our code.
Problem details:
We have "Executor" which produces tasks, these task wrap some long running actions. Main job of executor is to limit how many concurrent actions were started. All of these tasks can be cancelled individually, and also these actions will respect CancellationToken internally.
I would like to provide unit test, which shows that when cancellation occurred while task is waiting for slot to start given action, that task will cancel itself (eventually) and does not start execution of given action.
So, idea was to prepare LimitingExecutor with single slot. Then start blocking action, which would request cancellation when unblocked. Then "enqueue" test action, which should fail when executed. With that setup, tests would call unblock and then assert that task of test action will throw TaskCanceledException when awaited.
[Test]
public void RequestPropagationTest()
{
using (var setupEvent = new ManualResetEvent(initialState: false))
using (var cancellation = new CancellationTokenSource())
using (var executor = new LimitingExecutor())
{
// System-state setup action:
var cancellingTask = executor.Do(() =>
{
setupEvent.WaitOne();
cancellation.Cancel();
}, CancellationToken.None);
// Main work action:
var actionTask = executor.Do(() =>
{
throw new InvalidOperationException(
"This action should be cancelled!");
}, cancellation.Token);
// Let's wait until this `Task` starts, so it will got opportunity
// to cancel itself, and expected later exception will not come
// from just starting that action by `Task.Run` with token:
while (actionTask.Status < TaskStatus.Running)
Thread.Sleep(millisecondsTimeout: 1);
// Let's unblock slot in Executor for the 'main work action'
// by finalizing the 'system-state setup action' which will
// finally request "global" cancellation:
setupEvent.Set();
Assert.DoesNotThrowAsync(
async () => await cancellingTask);
Assert.ThrowsAsync<TaskCanceledException>(
async () => await actionTask);
}
}
public class LimitingExecutor : IDisposable
{
private const int UpperLimit = 1;
private readonly Semaphore _semaphore
= new Semaphore(UpperLimit, UpperLimit);
public Task Do(Action work, CancellationToken token)
=> Task.Run(() =>
{
_semaphore.WaitOne();
try
{
token.ThrowIfCancellationRequested();
work();
}
finally
{
_semaphore.Release();
}
}, token);
public void Dispose()
=> _semaphore.Dispose();
}
Executable demo (via NUnit) of this problem could be found at GitHub.
However, implementation of that test sometimes fails (no expected TaskCanceledException), on my machin maybe 1 in 10 runs. Kind of "solution" to this problem is to insert Thread.Sleep right after request of cancellation. Even with sleep for 3 seconds this test sometimes fails (found after 20-ish runs), and when it passes, that long waiting is usually unnecessary (I guess). For reference, please see diff.
"Other problem", was to ensure that cancellation comes from "waiting time" and not from Task.Run, because ThreadPool could be busy (other executing tests), and it cold postpone start of second task after request of cancellation - that would render this test "falsy-green". The "easy fix by hack" was to actively wait until second task starts - its Status becomes TaskStatus.Running. Please check version under this branch and see that test without this hack will be sometimes "green" - so exampled bug could pass through it.

Your test method assumes that cancellingTask always takes the slot (enters the semaphore) in LimitingExecutor before the actionTask. Unfortunatelly, this assumption is wrong, LimitingExecutor does not guarantee this and it's just a matter of luck, which of the two task takes the slot (actually on my computer it only happens in something like 5% of runs).
To resolve this problem, you need another ManualResetEvent, that will allow main thread to wait until cancellingTask actually occupies the slot:
using (var slotTaken = new ManualResetEvent(initialState: false))
using (var setupEvent = new ManualResetEvent(initialState: false))
using (var cancellation = new CancellationTokenSource())
using (var executor = new LimitingExecutor())
{
// System-state setup action:
var cancellingTask = executor.Do(() =>
{
// This is called from inside the semaphore, so it's
// certain that this task occupies the only available slot.
slotTaken.Set();
setupEvent.WaitOne();
cancellation.Cancel();
}, CancellationToken.None);
// Wait until cancellingTask takes the slot
slotTaken.WaitOne();
// Now it's guaranteed that cancellingTask takes the slot, not the actionTask
// ...
}
.NET Framework doesn't provide API to detect task transition to the Running state, so if you don't like polling the State property + Thread.Sleep() in a loop, you'll need to modify LimitingExecutor.Do() to provide this information, probably using another ManualResetEvent, e.g.:
public Task Do(Action work, CancellationToken token, ManualResetEvent taskRunEvent = null)
=> Task.Run(() =>
{
// Optional notification to the caller that task is now running
taskRunEvent?.Set();
// ...
}, token);

Handle exception with CancellationToken.Register properly

I am trying to cancel a time consuming task after a certain milliseconds and for my case, I think CancellationToken.Register method best fits compared to other approaches with constant polling or WaitHandle. CancellationToken.Register method will help me define a delegate where I plan to take the task to cancelled state and stop further execution of the task; this delegate will be invoked when the task is cancelled (certain milliseconds later as per my goal). Here is the test code I have which I intend to extend for multiple tasks and with nested tasks later:
List<Task> tasks = new List<Task>();
CancellationTokenSource tokenSource = new CancellationTokenSource();
CancellationToken cancellationToken = tokenSource.Token;
Task t1 = Task.Factory.StartNew(() =>
{
// check cancellation token before task begins
if (cancellationToken.IsCancellationRequested) cancellationToken.ThrowIfCancellationRequested();
// register a callback to handle cancellation token anytime it occurs
cancellationToken.Register(() =>
{
Console.WriteLine("Task t1 cancelled");
cancellationToken.ThrowIfCancellationRequested();
});
// simulating a massive task; note, it is not a repeating task
Thread.Sleep(12000);
}, cancellationToken);
tasks.Add(t1);
try
{
// cancel token after 3000 ms + wait for all tasks
tokenSource.CancelAfter(3000);
Task.WaitAll(tasks.ToArray());
// OR wait for all tasks for 3000 ms and then cancel token immediately
//Task.WaitAll(tasks.ToArray(), 3000);
//tokenSource.Cancel();
}
catch (AggregateException e)
{
Console.WriteLine("\nAggregateException thrown with the following inner exceptions:");
// Display information about each exception.
foreach (var v in e.InnerExceptions)
{
if (v is TaskCanceledException)
Console.WriteLine(" TaskCanceledException: Task {0}", ((TaskCanceledException)v).Task.Id);
else
Console.WriteLine(" Exception: {0}", v.GetType().Name);
}
Console.WriteLine();
}
finally
{
tokenSource.Dispose();
}
I am, however, facing an issue with exception handling during execution of the callback method inside cancellationToken.Register. The call to cancellationToken.ThrowIfCancellationRequested() gives me exception "OperationCanceledException was unhandled by user code" followed by "AggregateException was unhandled". I have read about VS settings with unchecking User-unhandled exception for the first OperationCanceledException exception but my application terminates after second AggregateException exception; the try..catch block for Task.WaitAll does not seem to handle this.
I tried to enclose cancellationToken.ThrowIfCancellationRequested() in a try..catch block but the problem with this approach was that the task continued with the remaining steps which I do not desire. I do not see this behaviour with polling approach.
// poll continuously to check for cancellation instead of Register
// but I do not want my massive task inside this repeating block
while (true)
{
if (cancellationToken.IsCancellationRequested)
{
Console.WriteLine("Task t1 Canceled.");
cancellationToken.ThrowIfCancellationRequested();
}
}
What am I doing wrong with the CancellationToken.Register approach?

The error you see is exactly because you didn't wrap the ThrowIfCancellationRequested inside a try-catch block.
In my opinion it really depends on what you're doing in place of that Sleep().
Ending the task in a cooperative way like
while(!cancellationToken.IsCancellationRequested)
{
// Do stuff
// Also check before doing something that may block or take a while
if(!cancellationToken.IsCancellationRequested)
{
Stream.Read(buffer, 0, n);
}
}
should be the best way to stop it.
If you really need to stop it no matter what, I would do wrap it in another task
Task.Run(() =>
{
// Simulate a long running task
Thread.Sleep(12*1000);
}, cancellationToken);
(tested and working)
This way you should not see any Exception Unhandled thrown.
Also, you might want to take a look at this: How do I abort/cancel TPL Tasks?

Task being marked as RanToCompletion at await, when still Running

I'm still getting up to speed with async & multi threading. I'm trying to monitor when the Task I Start is still running (to show in a UI). However it's indicating that it is RanToCompletion earlier than I want, when it hits an await, even when I consider its Status as still Running.
Here is the sample I'm doing. It all seems to be centred around the await's. When it hits an await, it is then marked as RanToCompletion.
I want to keep track of the main Task which starts it all, in a way which indicates to me that it is still running all the way to the end and only RanToCompletion when it is all done, including the repo call and the WhenAll.
How can I change this to get the feedback I want about the tskProdSeeding task status?
My Console application Main method calls this:
Task tskProdSeeding;
tskProdSeeding = Task.Factory.StartNew(SeedingProd, _cts.Token);
Which the runs this:
private async void SeedingProd(object state)
{
var token = (CancellationToken)state;
while (!token.IsCancellationRequested)
{
int totalSeeded = 0;
var codesToSeed = await _myRepository.All().ToListAsync(token);
await Task.WhenAll(Task.Run(async () =>
{
foreach (var code in codesToSeed)
{
if (!token.IsCancellationRequested)
{
try
{
int seedCountByCode = await _myManager.SeedDataFromLive(code);
totalSeeded += seedCountByCode;
}
catch (Exception ex)
{
_logger.InfoFormat(ex.ToString());
}
}
}
}, token));
Thread.Sleep(30000);
}
}

If you use async void the outer task can't tell when the task is finished, you need to use async Task instead.
Second, once you do switch to async Task, Task.Factory.StartNew can't handle functions that return a Task, you need to switch to Task.Run(
tskProdSeeding = Task.Run(() => SeedingProd(_cts.Token), _cts.Token);
Once you do both of those changes you will be able to await or do a .Wait() on tskProdSeeding and it will properly wait till all the work is done before continuing.
Please read "Async/Await - Best Practices in Asynchronous Programming" to learn more about not doing async void.
Please read "StartNew is Dangerous" to learn more about why you should not be using StartNew the way you are using it.
P.S. In SeedingProd you should switch it to use await Task.Delay(30000); insetad of Thread.Sleep(30000);, you will then not tie up a thread while it waits. If you do this you likely could drop the
tskProdSeeding = Task.Run(() => SeedingProd(_cts.Token), _cts.Token);
and just make it
tskProdSeeding = SeedingProd(_cts.Token);
because the function no-longer has a blocking call inside of it.

I'm not convinced that you need a second thread (Task.Run or StartNew) at all. It looks like the bulk of the work is I/O-bound and if you're doing it asynchronously and using Task.Delay instead of Thread.Sleep, then there is no thread consumed by those operations and your UI shouldn't freeze. The first thing anyone new to async needs to understand is that it's not the same thing as multithreading. The latter is all about consuming more threads, the former is all about consuming fewer. Focus on eliminating the blocking and you shouldn't need a second thread.
As others have noted, SeedingProd needs to return a Task, not void, so you can observe its completion. I believe your method can be reduced to this:
private async Task SeedingProd(CancellationToken token)
{
while (!token.IsCancellationRequested)
{
int totalSeeded = 0;
var codesToSeed = await _myRepository.All().ToListAsync(token);
foreach (var code in codesToSeed)
{
if (token.IsCancellationRequested)
return;
try
{
int seedCountByCode = await _myManager.SeedDataFromLive(code);
totalSeeded += seedCountByCode;
}
catch (Exception ex)
{
_logger.InfoFormat(ex.ToString());
}
}
await Task.Dealy(30000);
}
}
Then simply call the method, without awaiting it, and you'll have your task.
Task mainTask = SeedingProd(token);

When you specify async on a method, it compiles into a state machine with a Task, so SeedingProd does not run synchronously, but acts as a Task even if returns void. So when you call Task.Factory.StartNew(SeedingProd) you start a task that kick off another task - that's why the first one finishes immediately before the second one. All you have to do is add the Task return parameter instead of void:
private async Task SeedingProdAsync(CancellationToken ct)
{
...
}
and call it as simply as this:
Task tskProdSeeding = SeedingProdAsync(_cts.Token);

Aborting a long running task in TPL

Our application uses the TPL to serialize (potentially) long running units of work. The creation of work (tasks) is user-driven and may be cancelled at any time. In order to have a responsive user interface, if the current piece of work is no longer required we would like to abandon what we were doing, and immediately start a different task.
Tasks are queued up something like this:
private Task workQueue;
private void DoWorkAsync
(Action<WorkCompletedEventArgs> callback, CancellationToken token)
{
if (workQueue == null)
{
workQueue = Task.Factory.StartWork
(() => DoWork(callback, token), token);
}
else
{
workQueue.ContinueWork(t => DoWork(callback, token), token);
}
}
The DoWork method contains a long running call, so it is not as simple as constantly checking the status of token.IsCancellationRequested and bailing if/when a cancel is detected. The long running work will block the Task continuations until it finishes, even if the task is cancelled.
I have come up with two sample methods to work around this issue, but am not convinced that either are proper. I created simple console applications to demonstrate how they work.
The important point to note is that the continuation fires before the original task completes.
Attempt #1: An inner task
static void Main(string[] args)
{
CancellationTokenSource cts = new CancellationTokenSource();
var token = cts.Token;
token.Register(() => Console.WriteLine("Token cancelled"));
// Initial work
var t = Task.Factory.StartNew(() =>
{
Console.WriteLine("Doing work");
// Wrap the long running work in a task, and then wait for it to complete
// or the token to be cancelled.
var innerT = Task.Factory.StartNew(() => Thread.Sleep(3000), token);
innerT.Wait(token);
token.ThrowIfCancellationRequested();
Console.WriteLine("Completed.");
}
, token);
// Second chunk of work which, in the real world, would be identical to the
// first chunk of work.
t.ContinueWith((lastTask) =>
{
Console.WriteLine("Continuation started");
});
// Give the user 3s to cancel the first batch of work
Console.ReadKey();
if (t.Status == TaskStatus.Running)
{
Console.WriteLine("Cancel requested");
cts.Cancel();
Console.ReadKey();
}
}
This works, but the "innerT" Task feels extremely kludgey to me. It also has the drawback of forcing me to refactor all parts of my code that queue up work in this manner, by necessitating the wrapping up of all long running calls in a new Task.
Attempt #2: TaskCompletionSource tinkering
static void Main(string[] args)
{ var tcs = new TaskCompletionSource<object>();
//Wire up the token's cancellation to trigger the TaskCompletionSource's cancellation
CancellationTokenSource cts = new CancellationTokenSource();
var token = cts.Token;
token.Register(() =>
{ Console.WriteLine("Token cancelled");
tcs.SetCanceled();
});
var innerT = Task.Factory.StartNew(() =>
{
Console.WriteLine("Doing work");
Thread.Sleep(3000);
Console.WriteLine("Completed.");
// When the work has complete, set the TaskCompletionSource so that the
// continuation will fire.
tcs.SetResult(null);
});
// Second chunk of work which, in the real world, would be identical to the
// first chunk of work.
// Note that we continue when the TaskCompletionSource's task finishes,
// not the above innerT task.
tcs.Task.ContinueWith((lastTask) =>
{
Console.WriteLine("Continuation started");
});
// Give the user 3s to cancel the first batch of work
Console.ReadKey();
if (innerT.Status == TaskStatus.Running)
{
Console.WriteLine("Cancel requested");
cts.Cancel();
Console.ReadKey();
}
}
Again this works, but now I have two problems:
a) It feels like I'm abusing TaskCompletionSource by never using it's result, and just setting null when I've finished my work.
b) In order to properly wire up continuations I need to keep a handle on the previous unit of work's unique TaskCompletionSource, and not the task that was created for it. This is technically possible, but again feels clunky and strange.
Where to go from here?
To reiterate, my question is: are either of these methods the "correct" way to tackle this problem, or is there a more correct/elegant solution that will allow me to prematurely abort a long running task and immediately starting a continuation? My preference is for a low-impact solution, but I'd be willing to undertake some huge refactoring if it's the right thing to do.
Alternately, is the TPL even the correct tool for the job, or am I missing a better task queuing mechanism. My target framework is .NET 4.0.

The real issue here is that the long-running call in DoWork is not cancellation-aware. If I understand correctly, what you're doing here is not really cancelling the long-running work, but merely allowing the continuation to execute and, when the work completes on the cancelled task, ignoring the result. For example, if you used the inner task pattern to call CrunchNumbers(), which takes several minutes, cancelling the outer task will allow continuation to occur, but CrunchNumbers() will continue to execute in the background until completion.
I don't think there's any real way around this other than making your long-running calls support cancellation. Often this isn't possible (they may be blocking API calls, with no API support for cancellation.) When this is the case, it's really a flaw in the API; you may check to see if there are alternate API calls that could be used to perform the operation in a way that can be cancelled. One hack approach to this is to capture a reference to the underlying Thread being used by the Task when the Task is started and then call Thread.Interrupt. This will wake up the thread from various sleep states and allow it to terminate, but in a potentially nasty way. Worst case, you can even call Thread.Abort, but that's even more problematic and not recommended.
Here is a stab at a delegate-based wrapper. It's untested, but I think it will do the trick; feel free to edit the answer if you make it work and have fixes/improvements.
public sealed class AbandonableTask
{
private readonly CancellationToken _token;
private readonly Action _beginWork;
private readonly Action _blockingWork;
private readonly Action<Task> _afterComplete;
private AbandonableTask(CancellationToken token,
Action beginWork,
Action blockingWork,
Action<Task> afterComplete)
{
if (blockingWork == null) throw new ArgumentNullException("blockingWork");
_token = token;
_beginWork = beginWork;
_blockingWork = blockingWork;
_afterComplete = afterComplete;
}
private void RunTask()
{
if (_beginWork != null)
_beginWork();
var innerTask = new Task(_blockingWork,
_token,
TaskCreationOptions.LongRunning);
innerTask.Start();
innerTask.Wait(_token);
if (innerTask.IsCompleted && _afterComplete != null)
{
_afterComplete(innerTask);
}
}
public static Task Start(CancellationToken token,
Action blockingWork,
Action beginWork = null,
Action<Task> afterComplete = null)
{
if (blockingWork == null) throw new ArgumentNullException("blockingWork");
var worker = new AbandonableTask(token, beginWork, blockingWork, afterComplete);
var outerTask = new Task(worker.RunTask, token);
outerTask.Start();
return outerTask;
}
}