We Keep Coding

Is there a proper pattern for multiple ContinueWith methods

In the docs for TPL I found this line:
Invoke multiple continuations from the same antecedent
But this isn't explained any further. I naively assumed you could chain ContinueWiths in a pattern matching like manner until you hit the right TaskContinuationOptions.
TaskThatReturnsString()
.ContinueWith((s) => Console.Out.WriteLine(s.Result), TaskContinuationOptions.OnlyOnRanToCompletion)
.ContinueWith((f) => Console.Out.WriteLine(f.Exception.Message), TaskContinuationOptions.OnlyOnFaulted)
.ContinueWith((f) => Console.Out.WriteLine("Cancelled"), TaskContinuationOptions.OnlyOnCanceled)
.Wait();
But this doesn't work like I hoped for at least two reasons.
The continuations are properly chained so the 2nd ContinueWith gets the result form the 1st, that is implemented as new Task, basically the ContinueWith task itself. I realize that the String could be returned onwards, but won't that be a new task with other info lost?
Since the first option is not met, the Task is just cancelled. Meaning that the second set will never be met and the exceptions are lost.
So what do they mean in the docs when they say multiple continuations from the same antecedent?
Is there a proper patter for this or do we just have to wrap the calls in try catch blocks?
EDIT
So I guess this was what I was hoping I could do, note this is a simplified example.
public void ProccessAllTheThings()
{
var theThings = util.GetAllTheThings();
var tasks = new List<Task>();
foreach (var thing in theThings)
{
var task = util.Process(thing)
.ContinueWith((t) => Console.Out.WriteLine($"Finished processing {thing.ThingId} with result {t.Result}"), TaskContinuationOptions.OnlyOnRanToCompletion)
.ContinueWith((t) => Console.Out.WriteLine($"Error on processing {thing.ThingId} with error {t.Exception.Message}"), TaskContinuationOptions.OnlyOnFaulted);
tasks.Add(task);
}
Task.WaitAll(tasks.ToArray());
}
Since this wasn't possible I was thinking I would have to wrap each task call in a try catch inside the loop so I wouldn't stop the process but not wait on it there. I wasn't sure what the correct way.
Sometimes a solution is just staring you in the face, this would work wouldn't it?
public void ProccessAllTheThings()
{
var theThings = util.GetAllTheThings();
var tasks = new List<Task>();
foreach (var thing in theThings)
{
var task = util.Process(thing)
.ContinueWith((t) =>
{
if (t.Status == TaskStatus.RanToCompletion)
{
Console.Out.WriteLine($"Finished processing {thing.ThingId} with result {t.Result}");
}
else
{
Console.Out.WriteLine($"Error on processing {thing.ThingId} - {t.Exception.Message}");
}
});
tasks.Add(task);
}
Task.WaitAll(tasks.ToArray());
}

What you did is to create a sequential chain of multiple tasks.
What you need to do is attach all your continuation tasks to the first one:
var firstTask = TaskThatReturnsString();
var t1 = firstTask.ContinueWith (…);
var t2 = firstTask.ContinueWith (…);
var t3 = firstTask.ContinueWith (…);
Then you need to wait for all the continuation tasks:
Task.WaitAll (t1, t2, t3);

How to properly execute a List of Tasks async in C#

I have a list of objects that I need to run a long running process on and I would like to kick them off asynchronously, then when they are all finished return them as a list to the calling method. I've been trying different methods that I have found, however it appears that the processes are still running synchronously in the order that they are in the list. So I am sure that I am missing something in the process of how to execute a list of tasks.
Here is my code:
public async Task<List<ShipmentOverview>> GetShipmentByStatus(ShipmentFilterModel filter)
{
if (string.IsNullOrEmpty(filter.Status))
{
throw new InvalidShipmentStatusException(filter.Status);
}
var lookups = GetLookups(false, Brownells.ConsolidatedShipping.Constants.ShipmentStatusType);
var lookup = lookups.SingleOrDefault(sd => sd.Name.ToLower() == filter.Status.ToLower());
if (lookup != null)
{
filter.StatusId = lookup.Id;
var shipments = Shipments.GetShipments(filter);
var tasks = shipments.Select(async model => await GetOverview(model)).ToList();
ShipmentOverview[] finishedTask = await Task.WhenAll(tasks);
return finishedTask.ToList();
}
else
{
throw new InvalidShipmentStatusException(filter.Status);
}
}
private async Task<ShipmentOverview> GetOverview(ShipmentModel model)
{
String version;
var user = AuthContext.GetUserSecurityModel(Identity.Token, out version) as UserSecurityModel;
var profile = AuthContext.GetProfileSecurityModel(user.Profiles.First());
var overview = new ShipmentOverview
{
Id = model.Id,
CanView = true,
CanClose = profile.HasFeatureAction("Shipments", "Close", "POST"),
CanClear = profile.HasFeatureAction("Shipments", "Clear", "POST"),
CanEdit = profile.HasFeatureAction("Shipments", "Get", "PUT"),
ShipmentNumber = model.ShipmentNumber.ToString(),
ShipmentName = model.Name,
};
var parcels = Shipments.GetParcelsInShipment(model.Id);
overview.NumberParcels = parcels.Count;
var orders = parcels.Select(s => WareHouseClient.GetOrderNumberFromParcelId(s.ParcelNumber)).ToList();
overview.NumberOrders = orders.Distinct().Count();
//check validations
var vals = Shipments.GetShipmentValidations(model.Id);
if (model.ValidationTypeId == Constants.OrderValidationType)
{
if (vals.Count > 0)
{
overview.NumberOrdersTotal = vals.Count();
overview.NumberParcelsTotal = vals.Sum(s => WareHouseClient.GetParcelsPerOrder(s.ValidateReference));
}
}
return overview;
}

It looks like you're using asynchronous methods while you really want threads.
Asynchronous methods yield control back to the calling method when an async method is called, then wait until the methods has completed on the await. You can see how it works here.
Basically, the only usefulness of async/await methods is not to lock the UI, so that it stays responsive.
If you want to fire multiple processings in parallel, you will want to use threads, like such:
using System.Threading.Tasks;
public void MainMethod() {
// Parallel.ForEach will automagically run the "right" number of threads in parallel
Parallel.ForEach(shipments, shipment => ProcessShipment(shipment));
// do something when all shipments have been processed
}
public void ProcessShipment(Shipment shipment) { ... }

Marking the method as async doesn't auto-magically make it execute in parallel. Since you're not using await at all, it will in fact execute completely synchronously as if it wasn't async. You might have read somewhere that async makes functions execute asynchronously, but this simply isn't true - forget it. The only thing it does is build a state machine to handle task continuations for you when you use await and actually build all the code to manage those tasks and their error handling.
If your code is mostly I/O bound, use the asynchronous APIs with await to make sure the methods actually execute in parallel. If they are CPU bound, a Task.Run (or Parallel.ForEach) will work best.
Also, there's no point in doing .Select(async model => await GetOverview(model). It's almost equivalent to .Select(model => GetOverview(model). In any case, since the method actually doesn't return an asynchronous task, it will be executed while doing the Select, long before you get to the Task.WhenAll.
Given this, even the GetShipmentByStatus's async is pretty much useless - you only use await to await the Task.WhenAll, but since all the tasks are already completed by that point, it will simply complete synchronously.

If your tasks are CPU bound and not I/O bound, then here is the pattern I believe you're looking for:
static void Main(string[] args) {
Task firstStepTask = Task.Run(() => firstStep());
Task secondStepTask = Task.Run(() => secondStep());
//...
Task finalStepTask = Task.Factory.ContinueWhenAll(
new Task[] { step1Task, step2Task }, //more if more than two steps...
(previousTasks) => finalStep());
finalStepTask.Wait();
}

Correct way to link Tasks together when return values are needed at different times #2

I asked a question yesterday and, unfortunately, even with the answers provided, I'm still hitting up on stumbling blocks about how to do things correctly... My issue is that my code actually works, but I'm a complete novice at concurrency programming and it feels like I'm not programming the correct way and, most importantly, I'm afraid of developing bad habits.
To make up a simplistic example to elaborate on yesterday's question, suppose I had the following methods:
static Task<IEnumerable<MyClass>> Task1(CancellationToken ct)
static Task<IEnumerable<int>> Task2(CancellationToken ct, List<string> StringList)
static Task<IEnumerable<String>> Task3(CancellationToken ct)
static Task<IEnumerable<Double>> Task4(CancellationToken ct)
static Task Task5(CancellationToken ct, IEnumerable<int> Task2Info, IEnumerable<string> Task3Info, IEnumerable<double> Task4Info)
static Task Task6(CancellationToken ct, IEnumerable<int> Task2Info, IEnumerable<MyClass> Task1Info)
And the code I've written that utilizes them looks as follows:
static Task execute(CancellationToken ct)
{
IEnumerable<MyClass> Task1Info = null;
List<string> StringList = null;
IEnumerable<int> Task2Info = null;
IEnumerable<string> Task3Info = null;
IEnumerable<double> Task4Info = null;
var TaskN = Task.Run(() =>
{
Task1Info = Task1(ct).Result;
}
, ct)
.ContinueWith(res =>
{
StringList = Task1Info.Select(k=> k.StringVal).ToList();
Task2Info = Task2(ct, StringList).Result;
}
, ct);
return Task.Run(() =>
{
return Task.WhenAll
(
TaskN,
Task.Run(() => { Task3Info = Task3(ct).Result; }, ct),
Task.Run(() => { Task4Info = Task4(ct).Result; }, ct)
)
.ContinueWith(res =>
{
Task5(ct, Task2Info, Task3Info, Task4Info).Wait();
}
, ct)
.ContinueWith(res =>
{
Task6(ct, Task2Info, Task1Info).Wait();
}
, ct);
});
}
In other words:
I need the results of Task1 to calculate StringList and to run Task2
Task2, Task3 and Task4 can all run concurrently
I need the return values from all of the above for later method calls
Once these are run, I use their results to run Task5
Once Task5 is run, I use all the results in running Task6
As a simple explanation, imagine the first portion is data gathering, the second is data cleansing and the third data reporting
Like I said, my challenge is that this actually runs, but I simply feel that it's more of a "hack" that the right way to program - Concurrency programming is very new to me and I definitely want to learn the best ways this should be done...

I would feel better about my answer if I could see how your TaskN methods were implemented. Specifically, I would want to validate the need for your TaskN method calls to be wrapped inside calls to Task.Run() if they are already returning a Task return value.
But personally, I would just use the async-await style of programming. I find it fairly easy to read/write. Documentation can be found here.
I would then envision your code looking something like this:
static async Task execute(CancellationToken ct)
{
// execute and wait for task1 to complete.
IEnumerable<MyClass> Task1Info = await Task1(ct);
List<string> StringList = Task1Info.Select(k=> k.StringVal).ToList();
// start tasks 2 through 4
Task<IEnumerable<int>> t2 = Task2(ct, StringList);
Task<IEnumerable<string>> t3 = Task3(ct);
Task<IEnmerable<Double>> t4 = Task4(ct);
// now that tasks 2 to 4 have been started,
// wait for all 3 of them to complete before continuing.
IEnumerable<int> Task2Info = await t2;
IEnumerable<string> Task3Info = await t3;
IEnumerable<Double> Task4Info = await t4;
// execute and wait for task 5 to complete
await Task5(ct, Task2Info, Task3Info, Task4Info);
// finally, execute and wait for task 6 to complete
await Task6(ct, Task2Info, Task1Info);
}

I don't fully understand the code in question but it is very easy to link tasks by dependency. Example:
var t1 = ...;
var t2 = ...;
var t3 = Task.WhenAll(t1, t2).ContinueWith(_ => RunT3(t1.Result, t2.Result));
You can express an arbitrary dependency DAG like that. This also makes it unnecessary to store into local variables, although you can do that if you need.
That way you also can get rid of the awkward Task.Run(() => { Task3Info = Task3(ct).Result; }, ct) pattern. This is equivalent to Task3(ct) except for the store to the local variable which probably should not exist in the first place.
.ContinueWith(res =>
{
Task5(ct, Task2Info, Task3Info, Task4Info).Wait();
}
This probably should be
.ContinueWith(_ => Task5(ct, Task2Info, Task3Info, Task4Info)).Unwrap()
Does that help? Leave a comment.

it feels like I'm not programming the correct way and, most importantly, I'm afraid of developing bad habits.
I definitely want to learn the best ways this should be done...
First, distinguish between asynchronous and parallel, which are two different forms of concurrency. An operation is a good match for "asynchronous" if it doesn't need a thread to do its work - e.g., I/O and other logical operations where you wait for something like timers. "Parallelism" is about splitting up CPU-bound work across multiple cores - you need parallelism if your code is maxing out one CPU at 100% and you want it to run faster by using other CPUs.
Next, follow a few guidelines for which APIs are used in which situations. Task.Run is for pushing CPU-bound work to other threads. If your work isn't CPU-bound, you shouldn't be using Task.Run. Task<T>.Result and Task.Wait are even more on the parallel side; with few exceptions, they should really only be used for dynamic task parallelism. Dynamic task parallelism is extremely powerful (as #usr pointed out, you can represent any DAG), but it's also low-level and awkward to work with. There are almost always better approaches. ContinueWith is another example of an API that is for dynamic task parallelism.
Since your method signatures return Task, I'm going to assume that they're naturally asynchronous (meaning: they don't use Task.Run, and are probably implemented with I/O). In that case, Task.Run is the incorrect tool to use. The proper tools for asynchronous work are the async and await keywords.
So, taking your desired behavior one at a time:
// I need the results of Task1 to calculate StringList and to run Task2
var task1Result = await Task1(ct);
var stringList = CalculateStringList(task1Result);
// Task2, Task3 and Task4 can all run concurrently
var task2 = Task2(ct, stringList);
var task3 = Task3(ct);
var task4 = Task4(ct);
await Task.WhenAll(task2, task3, task4);
// I need the return values from all of the above for later method calls
var task2Result = await task2;
var task3Result = await task3;
var task4Result = await task4;
// Once these are run, I use their results to run Task5
await Task5(ct, task2Result, task3Result, task4Result);
// Once Task5 is run, I use all the results in running Task6
await Task6(ct, task2Result, task1Result);
For more about what APIs are appropriate in which situations, I have a Tour of Task on my blog, and I cover a lot of concurrency best practices in my book.

Running multiple async tasks and waiting for them all to complete

I need to run multiple async tasks in a console application, and wait for them all to complete before further processing.
There's many articles out there, but I seem to get more confused the more I read. I've read and understand the basic principles of the Task library, but I'm clearly missing a link somewhere.
I understand that it's possible to chain tasks so that they start after another completes (which is pretty much the scenario for all the articles I've read), but I want all my Tasks running at the same time, and I want to know once they're all completed.
What's the simplest implementation for a scenario like this?

Both answers didn't mention the awaitable Task.WhenAll:
var task1 = DoWorkAsync();
var task2 = DoMoreWorkAsync();
await Task.WhenAll(task1, task2);
The main difference between Task.WaitAll and Task.WhenAll is that the former will block (similar to using Wait on a single task) while the latter will not and can be awaited, yielding control back to the caller until all tasks finish.
More so, exception handling differs:
Task.WaitAll:
At least one of the Task instances was canceled -or- an exception was thrown during the execution of at least one of the Task instances. If a task was canceled, the AggregateException contains an OperationCanceledException in its InnerExceptions collection.
Task.WhenAll:
If any of the supplied tasks completes in a faulted state, the returned task will also complete in a Faulted state, where its exceptions will contain the aggregation of the set of unwrapped exceptions from each of the supplied tasks.
If none of the supplied tasks faulted but at least one of them was canceled, the returned task will end in the Canceled state.
If none of the tasks faulted and none of the tasks were canceled, the resulting task will end in the RanToCompletion state.
If the supplied array/enumerable contains no tasks, the returned task will immediately transition to a RanToCompletion state before it's returned to the caller.

You could create many tasks like:
List<Task> TaskList = new List<Task>();
foreach(...)
{
var LastTask = new Task(SomeFunction);
LastTask.Start();
TaskList.Add(LastTask);
}
Task.WaitAll(TaskList.ToArray());

You can use WhenAll which will return an awaitable Task or WaitAll which has no return type and will block further code execution simular to Thread.Sleep until all tasks are completed, canceled or faulted.
WhenAll
WaitAll
Any of the supplied tasks completes in a faulted state
A task with the faulted state will be returned. The exceptions will contain the aggregation of the set of unwrapped exceptions from each of the supplied tasks.
An AggregateException will be thrown.
None of the supplied tasks faulted but at least one of them was canceled
The returned task will end in the TaskStatus.Canceled state
An AggregateException will be thrown which contains an OperationCanceledException in its InnerExceptions collection
An empty list was given
An ArgumentException will be thrown
The returned task will immediately transition to a TaskStatus.RanToCompletion State before it's returned to the caller.
Doesn't block the current thread
Blocks the current thread
Example
var tasks = new Task[] {
TaskOperationOne(),
TaskOperationTwo()
};
Task.WaitAll(tasks);
// or
await Task.WhenAll(tasks);
If you want to run the tasks in a particular/specific order you can get inspiration from this answer.

The best option I've seen is the following extension method:
public static Task ForEachAsync<T>(this IEnumerable<T> sequence, Func<T, Task> action) {
return Task.WhenAll(sequence.Select(action));
}
Call it like this:
await sequence.ForEachAsync(item => item.SomethingAsync(blah));
Or with an async lambda:
await sequence.ForEachAsync(async item => {
var more = await GetMoreAsync(item);
await more.FrobbleAsync();
});

Yet another answer...but I usually find myself in a case, when I need to load data simultaneously and put it into variables, like:
var cats = new List<Cat>();
var dog = new Dog();
var loadDataTasks = new Task[]
{
Task.Run(async () => cats = await LoadCatsAsync()),
Task.Run(async () => dog = await LoadDogAsync())
};
try
{
await Task.WhenAll(loadDataTasks);
}
catch (Exception ex)
{
// handle exception
}

Do you want to chain the Tasks, or can they be invoked in a parallel manner?
For chaining
Just do something like
Task.Run(...).ContinueWith(...).ContinueWith(...).ContinueWith(...);
Task.Factory.StartNew(...).ContinueWith(...).ContinueWith(...).ContinueWith(...);
and don't forget to check the previous Task instance in each ContinueWith as it might be faulted.
For the parallel manner
The most simple method I came across: Parallel.Invoke
Otherwise there's Task.WaitAll or you can even use WaitHandles for doing a countdown to zero actions left (wait, there's a new class: CountdownEvent), or ...

This is how I do it with an array Func<>:
var tasks = new Func<Task>[]
{
() => myAsyncWork1(),
() => myAsyncWork2(),
() => myAsyncWork3()
};
await Task.WhenAll(tasks.Select(task => task()).ToArray()); //Async
Task.WaitAll(tasks.Select(task => task()).ToArray()); //Or use WaitAll for Sync

I prepared a piece of code to show you how to use the task for some of these scenarios.
// method to run tasks in a parallel
public async Task RunMultipleTaskParallel(Task[] tasks) {
await Task.WhenAll(tasks);
}
// methode to run task one by one
public async Task RunMultipleTaskOneByOne(Task[] tasks)
{
for (int i = 0; i < tasks.Length - 1; i++)
await tasks[i];
}
// method to run i task in parallel
public async Task RunMultipleTaskParallel(Task[] tasks, int i)
{
var countTask = tasks.Length;
var remainTasks = 0;
do
{
int toTake = (countTask < i) ? countTask : i;
var limitedTasks = tasks.Skip(remainTasks)
.Take(toTake);
remainTasks += toTake;
await RunMultipleTaskParallel(limitedTasks.ToArray());
} while (remainTasks < countTask);
}

There should be a more succinct solution than the accepted answer. It shouldn't take three steps to run multiple tasks simultaneously and get their results.
Create tasks
await Task.WhenAll(tasks)
Get task results (e.g., task1.Result)
Here's a method that cuts this down to two steps:
public async Task<Tuple<T1, T2>> WhenAllGeneric<T1, T2>(Task<T1> task1, Task<T2> task2)
{
await Task.WhenAll(task1, task2);
return Tuple.Create(task1.Result, task2.Result);
}
You can use it like this:
var taskResults = await Task.WhenAll(DoWorkAsync(), DoMoreWorkAsync());
var DoWorkResult = taskResults.Result.Item1;
var DoMoreWorkResult = taskResults.Result.Item2;
This removes the need for the temporary task variables. The problem with using this is that while it works for two tasks, you'd need to update it for three tasks, or any other number of tasks. Also it doesn't work well if one of the tasks doesn't return anything. Really, the .Net library should provide something that can do this

If you're using the async/await pattern, you can run several tasks in parallel like this:
public async Task DoSeveralThings()
{
// Start all the tasks
Task first = DoFirstThingAsync();
Task second = DoSecondThingAsync();
// Then wait for them to complete
var firstResult = await first;
var secondResult = await second;
}

Continuation tasks not executing in correct order

Been trying to execute tasks sequentially but they are executed in a random order instead.
Appending .Unwrap after .ContinueWith doesn't help
Returning a Task of T from these methods instead of Task and assigning their result in the caller doesn't work either
Not sure about signature of my methods, whether they should contain async/await or not.
Sequencing tasks :
Task biographies = LoadArtistBiographies(apiKey);
Task blogs = LoadArtistBlogs(apiKey);
Task familiarity = LoadArtistFamiliarity(apiKey);
Task hottness = LoadArtistHottness(apiKey);
Task images = LoadArtistImages(apiKey);
await biographies.ContinueWith(b => blogs);
await blogs.ContinueWith(f => familiarity);
await familiarity.ContinueWith(h => hottness);
await hottness.ContinueWith(i => images);
await images;
Sample of executed methods :
private async Task LoadArtistBiographies(string apiKey)
{
var parameters = new ArtistBiographiesParameters();
parameters.SetDefaultValues();
parameters.ApiKey = apiKey;
parameters.Id = _artistId;
ArtistBiographies biographies = await Queries.ArtistBiographies(parameters);
ItemsControlBiographies.ItemsSource = biographies.Biographies;
}
The Queries.* methods are also asynchronous :
public static async Task<ArtistBlogs> ArtistBlogs(ArtistBlogsParameters parameters)
What is the correct syntax for chaining tasks that themselves are executing asynchronous tasks ?

If you want to execute the tasks in a specific order, you should await them directly:
await LoadArtistBiographies(apiKey);
await LoadArtistBlogs(apiKey);
await LoadArtistFamiliarity(apiKey);
await LoadArtistHottness(apiKey);
await LoadArtistImages(apiKey);
This will cause the second task (LoadArtistBlogs) to be scheduled after the first task completes.
Right now, the tasks are executing "in random order" because you've assigned them to Task instances, which allows each to be executed simultaneously.
That being said, I would actually recommend changing your methods around to returning the values, instead of assigning them to the datasource within the method:
private async Task<Biographies> LoadArtistBiographiesAsync(string apiKey)
{
var parameters = new ArtistBiographiesParameters();
parameters.SetDefaultValues();
parameters.ApiKey = apiKey;
parameters.Id = _artistId;
var bio = await Queries.ArtistBiographies(parameters);
return bio.Biographies;
}
You could then write these as:
ItemsControlBiographies.ItemsSource = await LoadArtistBiographiesAsync(apiKey);
// Other methods below, with await as this example
This makes the intent as the logic flows through the async methods a bit more clear, in my opinion.

Your example code will start executing all the tasks without waiting for each one to complete. It then waits for them to complete in order.
The key is that an async method starts when you call it. So if you don't want to start it yet, don't call the method yet:
await LoadArtistBiographies(apiKey);
await LoadArtistBlogs(apiKey);
await LoadArtistFamiliarity(apiKey);
await LoadArtistHottness(apiKey);
await LoadArtistImages(apiKey);

await will wait for the given task to complete, it will not start the task. Your Load*-methods all most likely start a task. All five tasks are running in an arbitrary order.
At the point when you get to await, your task may already has finished or not. It does not matter. You call ContinueWith on it, telling your task it should continue with this method once finished. This will return a new Task, on which you finally await.

Actually I've just found a way but without ContinueWith :
ArtistBiographies biographies = await LoadArtistBiographies(apiKey);
ItemsControlBiographies.ItemsSource = biographies.Biographies;
ArtistBlogs blogs = await LoadArtistBlogs(apiKey);
ItemsControlBlogs.ItemsSource = blogs.Blogs;
ArtistFamiliarity familiarity = await LoadArtistFamiliarity(apiKey);
ContentControlFamiliarity.Content = familiarity.artist;
ArtistHotttnesss hottness = await LoadArtistHottness(apiKey);
ContentControlHottness.Content = hottness.Artist;
ArtistImages images = await LoadArtistImages(apiKey);
ItemsControlImages.ItemsSource = images.Images;
Curious if someone could provide the answer using ContinueWith.