We Keep Coding

How to write async methods with await, but make them as short as with Task.Run?

After browsing different await vs Task.Run questions on SO my takeaway is that await is better for I/O operations and Task.Run for CPU-intensive operations. However the code with await seems to always be longer than Task.Run. Below is an example method of how I am creating a context object with await in my app:
public async AppContext CreateAppContext()
{
var context = new AppContext();
var customersTask = _dataAccess.GetCustomers();
var usersTask = _dataAccess.GetUsers();
var reportsTask = _dataAccess.GetReports();
context.Customers = await customersTask;
context.Users = await usersTask;
context.Reports = await reportsTask;
return context;
}
If I was to rewrite this with Task.Run I could do
public async AppContext CreateAppContext()
{
var context = new AppContext();
await Task.WhenAll(new[]
{
Task.Run(async () => { context.Customers = await _dataAccess.GetCustomers(); }),
Task.Run(async () => { context.Users = await _dataAccess.GetUsers(); }),
Task.Run(async () => { context.Reports = await _dataAccess.GetReports(); });
})
return context;
}
The difference is not major when I create an object with 3 properties but I have objects where I need to initialize 20+ properties in this manner which makes the await code a lot longer (nearly double) than Task.Run. Is there a way for me to initialize the object using await with code that is not a lot longer than what I can do with Task.Run?

Personally, I prefer an asynchronous factory pattern over that kind of code; but if you need to do concurrent asynchronous work like that multiple times, then you'll probably want to write your own helper method.
The BCL-provided WhenAll works best when either all tasks have no results, or when all tasks have the same type of result. One fairly common approach to help WhenAll work with different types of tasks is to return a tuple of results, which can then be deconstructed into different variables if desired. E.g.,
public static class TaskEx
{
public static async Task<(T1, T2, T3)> WhenAll<T1, T2, T3>(Task<T1> task1, Task<T2> task2, Task<T3> task3)
{
await Task.WhenAll(task1, task2, task3);
return (await task1, await task2, await task3);
}
}
// Usage
public async AppContext CreateAppContext()
{
var context = new AppContext();
(context.Customers, context.Users, conext.Reports) =
await TaskEx.WhenAll(
_dataAccess.GetCustomers(),
_dataAccess.GetUsers(),
_dataAccess.GetReports());
return context;
}
You can even define a tuple GetAwaiter extension method if you want, to make it more implicit:
// Usage
public async AppContext CreateAppContext()
{
var context = new AppContext();
(context.Customers, context.Users, conext.Reports) =
await (
_dataAccess.GetCustomers(),
_dataAccess.GetUsers(),
_dataAccess.GetReports());
return context;
}
There are a couple of disadvantages to these approaches, though. First, you have to define as many overloads as you need. Second, the multiple-assignment code is not very nice; it's fine for 2 or 3 properties, but would get ugly (IMO) if done for much more than that.
So I think what you really want is a custom delegate form of WhenAll. Something like this should work:
public static class TaskEx
{
public static async Task WhenAll(params Func<Task>[] tasks)
{
return Task.WhenAll(tasks.Select(action => action()));
}
}
// Usage
public async AppContext CreateAppContext()
{
var context = new AppContext();
await TaskEx.WhenAll(
async () => context.Customers = await _dataAccess.GetCustomers(),
async () => context.Users = await _dataAccess.GetUsers(),
async () => conext.Reports = await _dataAccess.GetReports());
return context;
}
Since this solution avoids dealing with the different result types entirely, multiple overloads aren't needed.

If you really want to keep your general pattern (which I'd avoid - it would be much better to do all the work and then assign all the results at the same time; look into the return value of Task.WhenAll), all you need is a simple helper method:
static async Task Assign<T>(Action<T> assignment, Func<Task<T>> getValue)
=> assignment(await getValue());
Then you can use it like this:
await Task.WhenAll
(
Assign(i => context.Customers = i, _dataAccess.GetCustomers),
Assign(i => context.Users = i, _dataAccess.GetUsers),
Assign(i => context.Reports = i, _dataAccess.GetReports)
);
There's many other ways to make this even simpler, but this is the most equivalent to your Task.Run code without having to involve another thread indirection just to do an assignment. It also avoids the very common mistake when you happen to use the wrong Task.Run overload and get a race condition (as Task.Run returns immediately instead of waiting for the result).
Also, you misunderstood the "await vs. Task.Run" thing. There's actually not that much difference between await and Task.Run in your code - mainly, it forces a thread switch (and a few other subtle things). The argument is against using Task.Run to run synchronous code; that wastes a thread waiting for a thing to complete, your code doesn't.
Do keep in mind that WhenAll comes with its own complications, though. While it does mean you don't have to worry about some of the tasks ending up unobserved (and not waited on!), it also means you have to completely rethink your exception handling, since you're going to get an AggregateException rather than anything more specific. If you're relying on error handling based on identifying exceptions, you need to be very careful. Usually, you don't want AggregateException to leak out of methods - it's very difficult to handle in a global manner; the only method that knows the possibilities of what can happen is the method that calls the WhenAll. Hopefully.
It's definitely a good idea to run parallel operations like this in a way that cannot produce dangerous and confusing side-effects. In your code, you either get a consistent object returned, or you get nothing - that's exactly the right approach. Be wary of this approach leaking into other contexts - it can get really hard to debug issues where randomly half of the operations succeed and other half fails :)

Is there a way for me to initialize the object using await with code that is not a lot longer than what I can do with Task.Run?
If you want to run all tasks in parallel - in short no, you cant shorten number of lines. Also note that those two snippets are not fully functionally equivalent - see Why should I prefer single await Task.WhenAll over multiple awaits.
You can simplify (and maybe even improve performance a bit) your Task.WhenAll approach by introducing a method which will await and assign. Something along these lines:
public async AppContext CreateAppContext()
{
var context = new AppContext();
await Task.WhenAll(
AwaitAndAssign(val => context.Customers = val, _dataAccess.GetCustomers()),
AwaitAndAssign(val => context.Users = val, _dataAccess.Users()),
AwaitAndAssign(val => context.Reports = val, _dataAccess.GetReports())
);
return context;
async Task AwaitAndAssign<T>(Action<T> assign, Task<T> valueTask) =>
assign(await valueTask);
}

C# LanguageExt - combine multiple async calls into one grouped call

I have a method that looks up an item asynchronously from a datastore;
class MyThing {}
Task<Try<MyThing>> GetThing(int thingId) {...}
I want to look up multiple items from the datastore, and wrote a new method to do this. I also wrote a helper method that will take multiple Try<T> and combine their results into a single Try<IEnumerable<T>>.
public static class TryExtensions
{
Try<IEnumerable<T>> Collapse<T>(this IEnumerable<Try<T>> items)
{
var failures = items.Fails().ToArray();
return failures.Any() ?
Try<IEnumerable<T>>(new AggregateException(failures)) :
Try(items.Select(i => i.Succ(a => a).Fail(Enumerable.Empty<T>())));
}
}
async Task<Try<MyThing[]>> GetThings(IEnumerable<string> ids)
{
var results = new List<Try<Things>>();
foreach (var id in ids)
{
var thing = await GetThing(id);
results.Add(thing);
}
return results.Collapse().Map(p => p.ToArray());
}
Another way to do it would be like this;
async Task<Try<MyThing[]>> GetThings(IEnumerable<string> ids)
{
var tasks = ids.Select(async id => await GetThing(id)).ToArray();
await Task.WhenAll(tasks);
return tasks.Select(t => t.Result).Collapse().Map(p => p.ToArray());
}
The problem with this is that all the tasks will run in parallel and I don't want to hammer my datastore with lots of parallel requests. What I really want is to make my code functional, using monadic principles and features of LanguageExt. Does anyone know how to achieve this?
Update
Thanks for the suggestion #MatthewWatson, this is what it looks like with the SemaphoreSlim;
async Task<Try<MyThing[]>> GetThings(IEnumerable<string> ids)
{
var mutex = new SemaphoreSlim(1);
var results = ids.Select(async id =>
{
await mutex.WaitAsync();
try { return await GetThing(id); }
finally { mutex.Release(); }
}).ToArray();
await Task.WhenAll(tasks);
return tasks.Select(t => t.Result).Collapse().Map(Enumerable.ToArray);
return results.Collapse().Map(p => p.ToArray());
}
Problem is, this is still not very monadic / functional, and ends up with more lines of code than my original code with a foreach block.

In the "Another way" you almost achieved your goal when you called:
var tasks = ids.Select(async id => await GetThing(id)).ToArray();
Except that Tasks doesn't run sequentially so you will end up with many queries hitting your datastore, which is caused by .ToArray() and Task.WhenAll. Once you called .ToArray() it allocated and started the Tasks already, so if you can "tolerate" one foreach to achieve the sequential tasks running, like this:
public static class TaskExtensions
{
public static async Task RunSequentially<T>(this IEnumerable<Task<T>> tasks)
{
foreach (var task in tasks) await task;
}
}
Despite that running a "loop" of queries is not a quite good practice
in general, unless you have in some background service and some
special scenario, leveraging this to the Database engine through
WHERE thingId IN (...) in general is a better option. Even you
have big amount of thingIds we can slice it into small 10s, 100s.. to
narrow the WHERE IN footprint.
Back to our RunSequentially, I would like to make it more functional like this for example:
tasks.ToList().ForEach(async task => await task);
But sadly this will still run kinda "Parallel" tasks.
So the final usage should be:
async Task<Try<MyThing[]>> GetThings(IEnumerable<string> ids)
{
var tasks = ids.Select(id => GetThing(id));// remember don't use .ToArray or ToList...
await tasks.RunSequentially();
return tasks.Select(t => t.Result).Collapse().Map(p => p.ToArray());
}
Another overkill functional solution is to get Lazy in a Queue recursively !!
Instead GetThing, get a Lazy one GetLazyThing that returns Lazy<Task<Try<MyThing>>> simply by wrapping GetThing:
new Lazy<Task<Try<MyThing>>>(() => GetThing(id))
Now using couple extensions/functions:
public static async Task RecRunSequentially<T>(this IEnumerable<Lazy<Task<T>>> tasks)
{
var queue = tasks.EnqueueAll();
await RunQueue(queue);
}
public static Queue<T> EnqueueAll<T>(this IEnumerable<T> list)
{
var queue = new Queue<T>();
list.ToList().ForEach(m => queue.Enqueue(m));
return queue;
}
public static async Task RunQueue<T>(Queue<Lazy<Task<T>>> queue)
{
if (queue.Count > 0)
{
var task = queue.Dequeue();
await task.Value; // this unwraps the Lazy object content
await RunQueue(queue);
}
}
Finally:
var lazyTasks = ids.Select(id => GetLazyThing(id));
await lazyTasks.RecRunSequentially();
// Now collapse and map as you like
Update
However if you don't like the fact that EnqueueAll and RunQueue are not "pure", we can take the following approach with the same Lazy trick
public static async Task AwaitSequentially<T>(this Lazy<Task<T>>[] array, int index = 0)
{
if (array == null || index < 0 || index >= array.Length - 1) return;
await array[index].Value;
await AwaitSequentially(array, index + 1); // ++index is not pure :)
}
Now:
var lazyTasks = ids.Select(id => GetLazyThing(id));
await tasks.ToArray().AwaitSequentially();
// Now collapse and map as you like

How does parallelization work on async/await?

I have the following code, that I intend to run asynchronously. My goal is that GetPictureForEmployeeAsync() is called in parallel as many times as needed. I'd like to make sure that 'await' on CreatePicture does not prevent me from doing so.
public Task<Picture[]> GetPictures(IDictionary<string, string> tags)
{
var query = documentRepository.GetRepositoryQuery();
var employees = query.Where(doc => doc.Gender == tags["gender"]);
return Task.WhenAll(employees.Select(employee => GetPictureForEmployeeAsync(employee, tags)));
}
private Task<Picture> GetPictureForEmployeeAsync(Employee employee, IDictionary<string, string> tags)
{
var base64PictureTask = blobRepository.GetBase64PictureAsync(employee.ID.ToString());
var documentTask = documentRepository.GetItemAsync(employee.ID.ToString());
return CreatePicture(tags, base64PictureTask, documentTask);
}
private static async Task<Picture> CreatePicture(IDictionary<string, string> tags, Task<string> base64PictureTask, Task<Employee> documentTask)
{
var document = await documentTask;
return new Picture
{
EmployeeID = document.ID,
Data = await base64PictureTask,
ID = document.ID.ToString(),
Tags = tags,
};
}
If I understand it correctly, Task.WhenAll() is not affected by the two awaited tasks inside CreatePicture() because GetPictureForEmployeeAsync() is not awaited. Am I right about this? If not, how should I restructure the code to achieve what I want?

I'd like to make sure that 'await' on CreatePicture does not prevent me from doing so.
It doesn't.
If I understand it correctly, Task.WhenAll() is not affected by the two awaited tasks inside CreatePicture() because GetPictureForEmployeeAsync() is not awaited. Am I right about this?
Yes and no. The WhenAll isn't limited in any way by the awaited tasks in CreatePicture, but that has nothing to do with whether GetPictureForEmployeeAsync is awaited or not. These two lines of code are equivalent in terms of behavior:
return Task.WhenAll(employees.Select(employee => GetPictureForEmployeeAsync(employee, tags)));
return Task.WhenAll(employees.Select(async employee => await GetPictureForEmployeeAsync(employee, tags)));
I recommend reading my async intro to get a good understanding of how async and await work with tasks.
Also, since GetPictures has non-trivial logic (GetRepositoryQuery and evaluating tags["gender"]), I recommend using async and await for GetPictures, as such:
public async Task<Picture[]> GetPictures(IDictionary<string, string> tags)
{
var query = documentRepository.GetRepositoryQuery();
var employees = query.Where(doc => doc.Gender == tags["gender"]);
var tasks = employees.Select(employee => GetPictureForEmployeeAsync(employee, tags)).ToList();
return await Task.WhenAll(tasks);
}
As a final note, you may find your code cleaner if you don't pass around "tasks meant to be awaited" - instead, await them first and pass their result values:
async Task<Picture> GetPictureForEmployeeAsync(Employee employee, IDictionary<string, string> tags)
{
var base64PictureTask = blobRepository.GetBase64PictureAsync(employee.ID.ToString());
var documentTask = documentRepository.GetItemAsync(employee.ID.ToString());
await Task.WhenAll(base64PictureTask, documentTask);
return CreatePicture(tags, await base64PictureTask, await documentTask);
}
static Picture CreatePicture(IDictionary<string, string> tags, string base64Picture, Employee document)
{
return new Picture
{
EmployeeID = document.ID,
Data = base64Picture,
ID = document.ID.ToString(),
Tags = tags,
};
}

The thing to keep in mind about calling an async method is that, as soon as an await statement is reached inside that method, control immediately goes back to the code that invoked the async method -- no matter where the await statement happens to be in the method. With a 'normal' method, control doesn't go back to the code that invokes that method until the end of that method is reached.
So in your case, you can do the following:
private async Task<Picture> GetPictureForEmployeeAsync(Employee employee, IDictionary<string, string> tags)
{
// As soon as we get here, control immediately goes back to the GetPictures
// method -- no need to store the task in a variable and await it within
// CreatePicture as you were doing
var picture = await blobRepository.GetBase64PictureAsync(employee.ID.ToString());
var document = await documentRepository.GetItemAsync(employee.ID.ToString());
return CreatePicture(tags, picture, document);
}
Because the first line of code in GetPictureForEmployeeAsync has an await, control will immediately go right back to this line...
return Task.WhenAll(employees.Select(employee => GetPictureForEmployeeAsync(employee, tags)));
...as soon as it is invoked. This will have the effect of all of the employee items getting processed in parallel (well, sort of -- the number of threads that will be allotted to your application will be limited).
As an additional word of advice, if this application is hitting a database or web service to get the pictures or documents, this code will likely cause you issues with running out of available connections. If this is the case, consider using System.Threading.Tasks.Parallel and setting the maximum degree of parallelism, or use SemaphoreSlim to control the number of connections used simultaneously.

Getting "The connection does not support MultipleActiveResultSets" in a ForEach with async-await

I have the following code using Dapper.SimpleCRUD :
var test = new FallEnvironmentalCondition[] {
new FallEnvironmentalCondition {Id=40,FallId=3,EnvironmentalConditionId=1},
new FallEnvironmentalCondition {Id=41,FallId=3,EnvironmentalConditionId=2},
new FallEnvironmentalCondition {Id=42,FallId=3,EnvironmentalConditionId=3}
};
test.ToList().ForEach(async x => await conn.UpdateAsync(x));
With this code, I am getting following exception:
InvalidOperationException: The connection does not support MultipleActiveResultSets
I don't understand I am awaiting each update so why am I getting this error.
Note: I have no control on the connection string so I can't turn MARS on.

You need to add attribute MultipleActiveResultSets in connection string and set it to true to allow multiple active result sets.
"Data Source=MSSQL1;" & _
"Initial Catalog=AdventureWorks;Integrated Security=SSPI;" & _
"MultipleActiveResultSets=True"
Read more at: https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql/enabling-multiple-active-result-sets

That code starts a Task for each item in the list, but does not wait for the each task to complete before starting the next one. Inside each Task it waits for the update to complete. Try
Enumerable.Range(1, 10).ToList().ForEach(async i => await Task.Delay(1000).ContinueWith(t => Console.WriteLine(DateTime.Now)));
Which is equivalent to
foreach (var i in Enumerable.Range(1, 10).ToList() )
{
var task = Task.Delay(1000).ContinueWith(t => Console.WriteLine(DateTime.Now));
}
If you're in a non-async method you will have to Wait(), not await each task. EG
foreach (var i in Enumerable.Range(1, 10).ToList() )
{
var task = Task.Delay(1000).ContinueWith(t => Console.WriteLine(DateTime.Now));
//possibly do other stuff on this thread
task.Wait(); //wait for this task to complete
}

The problem is the ForEach method is not an asynchronous method. It will not await the Task returned by your lambda. Running that code will fire every task and not wait for completion of any of them.
General point: marking a lambda as async does not make a synchronous method you pass it into behave asynchronously.
Solution: you will need to use a foreach loop which awaits the tasks' completion.
eg: foreach (var x in xs) await f(x);
You can wrap that in a helper method if you prefer.
(I know it's an old question, but I don't think it was clearly answered)

MARS has some limitations and also a non-zero overhead. You can use the following helpers to make the updates sequential:
public static async Task WhenAllOneByOne<T>(this IEnumerable<T> source, Func<T, Task> process)
{
foreach (var item in source)
await process(item);
}
public static async Task<List<U>> WhenAllOneByOne<T, U>(this IEnumerable<T> source, Func<T, Task<U>> transform)
{
var results = new List<U>();
foreach (var item in source)
results.Add(await transform(item));
return results;
// I would use yield return but unfortunately it is not supported in async methods
}
So your example would turn into
await test.WhenAllOneByOne(conn.UpdateAsync);
I usually call the second helper instead of Task.WhenAll, as follows:
await Task.WhenAll(source.Select(transform)); // not MARS-safe
await source.WhenAllOneByOne(transform); // MARS-safe

How to Subscribe with async method in Rx?

I have following code:
IObservable<Data> _source;
...
_source.Subscribe(StoreToDatabase);
private async Task StoreToDatabase(Data data) {
await dbstuff(data);
}
However, this does not compile. Is there any way how to observe data asynchronously? I tried async void, it works, but I feel that given solution is not feasible.
I also checked Reactive Extensions Subscribe calling await, but it does not answer my question (I do not care about the SelectMany result.)

You don't have to care about the SelectMany result. The answer is still the same... though you need your task to have a return type (i.e. Task<T>, not Task).
Unit is essentially equivalent to void, so you can use that:
_source.SelectMany(StoreToDatabase).Subscribe();
private async Task<Unit> StoreToDatabase(Data data)
{
await dbstuff(data);
return Unit.Default;
}
This SelectMany overload accepts a Func<TSource, Task<TResult> meaning the resulting sequence will not complete until the task is completed.

Late answer, but I think that the following extension methods correctly encapsulate what Charles Mager proposed in his answer:
public static IDisposable SubscribeAsync<T>(this IObservable<T> source,
Func<Task> asyncAction, Action<Exception> handler = null)
{
Func<T,Task<Unit>> wrapped = async t =>
{
await asyncAction();
return Unit.Default;
};
if(handler == null)
return source.SelectMany(wrapped).Subscribe(_ => { });
else
return source.SelectMany(wrapped).Subscribe(_ => { }, handler);
}
public static IDisposable SubscribeAsync<T>(this IObservable<T> source,
Func<T,Task> asyncAction, Action<Exception> handler = null)
{
Func<T, Task<Unit>> wrapped = async t =>
{
await asyncAction(t);
return Unit.Default;
};
if(handler == null)
return source.SelectMany(wrapped).Subscribe(_ => { });
else
return source.SelectMany(wrapped).Subscribe(_ => { }, handler);
}

I've been using TPL DataFlow to control back pressure and have used it to solve this problem.
The key part is ITargetBlock<TInput>.AsObserver() - source.
// Set a block to handle each element
ITargetBlock<long> targetBlock = new ActionBlock<long>(async p =>
{
Console.WriteLine($"Received {p}");
await Task.Delay(1000);
Console.WriteLine($"Finished handling {p}");
},
new ExecutionDataflowBlockOptions { BoundedCapacity = 1 });
// Generate an item each second for 10 seconds
var sequence = Observable.Interval(TimeSpan.FromSeconds(1)).Take(10);
// Subscribe with an observer created from the target block.
sequence.Subscribe(targetBlock.AsObserver());
// Await completion of the block
await targetBlock.Completion;
The important part here is that the ActionBlock's bounded capacity is set to 1. This prevents the block from receiving more than one item at a time and will block OnNext if an item is already being processed!
My big surprise here was that it can be safe to call Task.Wait and Task.Result inside your subscription. Obviously, if you have called ObserverOnDispatcher() or similar you will probably hit deadlocks. Be careful!

So you want to run the Store Data Procedure, possibly some other procedure and asynchronously await the completion or partial result. How about Create constructor shown here:
IObservable<Int32> NotifyOfStoringProgress =
Observable.Create(
(Func<IObserver<Int32>, Task>)
(async (ObserverToFeed) =>
{
ObserverToFeed.OnNext(-1);
Task StoreToDataBase = Task.Run(()=> {;});
ObserverToFeed.OnNext(0);
;;
await StoreToDataBase;
ObserverToFeed.OnNext(1);
;;
}));
NotifyOfStoringProgress.Subscribe(onNext: Notification => {;});