I have a method below in my WCF service:
public List<string> ProcessTask(IEnumerable<string> data)
{
var contentTasks = ..........
List<string> contentWeb = new List<string>();
Task.Factory.ContinueWhenAll(contentTasks, tasks =>
{
foreach (var task in tasks)
{
if (task.IsFaulted)
{
Trace.TraceError(task.Exception.GetBaseException().Message);
continue;
}
if (task.Result == null || String.IsNullOrEmpty(task.Result.Content))
{
continue;
}
contentWeb.Add(task.Result.Content);
}
});
}
How do I return the List of strings that have Result.Content from all
the tasks? These tasks are asynchronous tasks, so basically I have to wait until all tasks are done before I return the result.
You should return a Task<List<string>>:
public Task<List<string>> ProcessTasksAsync(IEnumerable<string> data)
{
var contentTasks = ..........
return Task.Factory.ContinueWhenAll(contentTasks, tasks =>
{
var contentWeb = new List<string>(); // Build this in the continuation
foreach (var task in tasks)
{
// ...same code...
contentWeb.Add(task.Result.Content);
}
return contentWeb; // Set the task's result here
});
}
As this is a WCF service, you can use the Task<T> method to implement an asynchronous method pair by returning the Task<T> in the Begin*** method, and unwrapping the Task<T> in the End*** method.
This makes this method asynchronous in a proper manner.
Note that this is far easier in C# 5 using async/await:
public async Task<List<string>> ProcessTasksAsync(IEnumerable<string> data)
{
var contentTasks = ..........
await Task.WhenAll(contentTasks);
var contentWeb = new List<string>(); // Build this in the continuation
foreach (var task in contentTasks)
{
// ...same code...
contentWeb.Add(task.Result.Content);
}
return contentWeb;
}
instead of continuewhenall i had to use Task.WaitAll(contentTasks);
Related
Is it possible to use Async when using ForEach? Below is the code I am trying:
using (DataContext db = new DataLayer.DataContext())
{
db.Groups.ToList().ForEach(i => async {
await GetAdminsFromGroup(i.Gid);
});
}
I am getting the error:
The name 'Async' does not exist in the current context
The method the using statement is enclosed in is set to async.
List<T>.ForEach doesn't play particularly well with async (neither does LINQ-to-objects, for the same reasons).
In this case, I recommend projecting each element into an asynchronous operation, and you can then (asynchronously) wait for them all to complete.
using (DataContext db = new DataLayer.DataContext())
{
var tasks = db.Groups.ToList().Select(i => GetAdminsFromGroupAsync(i.Gid));
var results = await Task.WhenAll(tasks);
}
The benefits of this approach over giving an async delegate to ForEach are:
Error handling is more proper. Exceptions from async void cannot be caught with catch; this approach will propagate exceptions at the await Task.WhenAll line, allowing natural exception handling.
You know that the tasks are complete at the end of this method, since it does an await Task.WhenAll. If you use async void, you cannot easily tell when the operations have completed.
This approach has a natural syntax for retrieving the results. GetAdminsFromGroupAsync sounds like it's an operation that produces a result (the admins), and such code is more natural if such operations can return their results rather than setting a value as a side effect.
This little extension method should give you exception-safe async iteration:
public static async Task ForEachAsync<T>(this List<T> list, Func<T, Task> func)
{
foreach (var value in list)
{
await func(value);
}
}
Since we're changing the return type of the lambda from void to Task, exceptions will propagate up correctly. This will allow you to write something like this in practice:
await db.Groups.ToList().ForEachAsync(async i => {
await GetAdminsFromGroup(i.Gid);
});
Starting with C# 8.0, you can create and consume streams asynchronously.
private async void button1_Click(object sender, EventArgs e)
{
IAsyncEnumerable<int> enumerable = GenerateSequence();
await foreach (var i in enumerable)
{
Debug.WriteLine(i);
}
}
public static async IAsyncEnumerable<int> GenerateSequence()
{
for (int i = 0; i < 20; i++)
{
await Task.Delay(100);
yield return i;
}
}
More
The simple answer is to use the foreach keyword instead of the ForEach() method of List().
using (DataContext db = new DataLayer.DataContext())
{
foreach(var i in db.Groups)
{
await GetAdminsFromGroup(i.Gid);
}
}
Here is an actual working version of the above async foreach variants with sequential processing:
public static async Task ForEachAsync<T>(this List<T> enumerable, Action<T> action)
{
foreach (var item in enumerable)
await Task.Run(() => { action(item); }).ConfigureAwait(false);
}
Here is the implementation:
public async void SequentialAsync()
{
var list = new List<Action>();
Action action1 = () => {
//do stuff 1
};
Action action2 = () => {
//do stuff 2
};
list.Add(action1);
list.Add(action2);
await list.ForEachAsync();
}
What's the key difference? .ConfigureAwait(false); which keeps the context of main thread while async sequential processing of each task.
This is not an old question, but .Net 6 introduced Parallel.ForeachAsync:
var collectionToIterate = db.Groups.ToList();
await Parallel.ForEachAsync(collectionToIterate, async (i, token) =>
{
await GetAdminsFromGroup(i);
});
ForeachAsync also accepts a ParallelOptions object, but usually you don't want to mess with the MaxDegreeOfParallelism property:
ParallelOptions parallelOptions = new ParallelOptions { MaxDegreeOfParallelism = 4 };
var collectionToIterate = db.Groups.ToList();
await Parallel.ForEachAsync(collectionToIterate, parallelOptions , async (i, token) =>
{
await GetAdminsFromGroup(i);
});
From Microsoft Docs: https://learn.microsoft.com/en-us/dotnet/api/system.threading.tasks.paralleloptions.maxdegreeofparallelism?view=net-6.0
By default, For and ForEach will utilize however many threads the underlying scheduler provides, so changing MaxDegreeOfParallelism from the default only limits how many concurrent tasks will be used.
Generally, you do not need to modify this setting....
Add this extension method
public static class ForEachAsyncExtension
{
public static Task ForEachAsync<T>(this IEnumerable<T> source, int dop, Func<T, Task> body)
{
return Task.WhenAll(from partition in Partitioner.Create(source).GetPartitions(dop)
select Task.Run(async delegate
{
using (partition)
while (partition.MoveNext())
await body(partition.Current).ConfigureAwait(false);
}));
}
}
And then use like so:
Task.Run(async () =>
{
var s3 = new AmazonS3Client(Config.Instance.Aws.Credentials, Config.Instance.Aws.RegionEndpoint);
var buckets = await s3.ListBucketsAsync();
foreach (var s3Bucket in buckets.Buckets)
{
if (s3Bucket.BucketName.StartsWith("mybucket-"))
{
log.Information("Bucket => {BucketName}", s3Bucket.BucketName);
ListObjectsResponse objects;
try
{
objects = await s3.ListObjectsAsync(s3Bucket.BucketName);
}
catch
{
log.Error("Error getting objects. Bucket => {BucketName}", s3Bucket.BucketName);
continue;
}
// ForEachAsync (4 is how many tasks you want to run in parallel)
await objects.S3Objects.ForEachAsync(4, async s3Object =>
{
try
{
log.Information("Bucket => {BucketName} => {Key}", s3Bucket.BucketName, s3Object.Key);
await s3.DeleteObjectAsync(s3Bucket.BucketName, s3Object.Key);
}
catch
{
log.Error("Error deleting bucket {BucketName} object {Key}", s3Bucket.BucketName, s3Object.Key);
}
});
try
{
await s3.DeleteBucketAsync(s3Bucket.BucketName);
}
catch
{
log.Error("Error deleting bucket {BucketName}", s3Bucket.BucketName);
}
}
}
}).Wait();
If you are using EntityFramework.Core there is an extension method ForEachAsync.
The example usage looks like this:
using Microsoft.EntityFrameworkCore;
using System.Threading.Tasks;
public class Example
{
private readonly DbContext _dbContext;
public Example(DbContext dbContext)
{
_dbContext = dbContext;
}
public async void LogicMethod()
{
await _dbContext.Set<dbTable>().ForEachAsync(async x =>
{
//logic
await AsyncTask(x);
});
}
public async Task<bool> AsyncTask(object x)
{
//other logic
return await Task.FromResult<bool>(true);
}
}
I would like to add that there is a Parallel class with ForEach function built in that can be used for this purpose.
The problem was that the async keyword needs to appear before the lambda, not before the body:
db.Groups.ToList().ForEach(async (i) => {
await GetAdminsFromGroup(i.Gid);
});
This is method I created to handle async scenarios with ForEach.
If one of tasks fails then other tasks will continue their execution.
You have ability to add function that will be executed on every exception.
Exceptions are being collected as aggregateException at the end and are available for you.
Can handle CancellationToken
public static class ParallelExecutor
{
/// <summary>
/// Executes asynchronously given function on all elements of given enumerable with task count restriction.
/// Executor will continue starting new tasks even if one of the tasks throws. If at least one of the tasks throwed exception then <see cref="AggregateException"/> is throwed at the end of the method run.
/// </summary>
/// <typeparam name="T">Type of elements in enumerable</typeparam>
/// <param name="maxTaskCount">The maximum task count.</param>
/// <param name="enumerable">The enumerable.</param>
/// <param name="asyncFunc">asynchronous function that will be executed on every element of the enumerable. MUST be thread safe.</param>
/// <param name="onException">Acton that will be executed on every exception that would be thrown by asyncFunc. CAN be thread unsafe.</param>
/// <param name="cancellationToken">The cancellation token.</param>
public static async Task ForEachAsync<T>(int maxTaskCount, IEnumerable<T> enumerable, Func<T, Task> asyncFunc, Action<Exception> onException = null, CancellationToken cancellationToken = default)
{
using var semaphore = new SemaphoreSlim(initialCount: maxTaskCount, maxCount: maxTaskCount);
// This `lockObject` is used only in `catch { }` block.
object lockObject = new object();
var exceptions = new List<Exception>();
var tasks = new Task[enumerable.Count()];
int i = 0;
try
{
foreach (var t in enumerable)
{
await semaphore.WaitAsync(cancellationToken);
tasks[i++] = Task.Run(
async () =>
{
try
{
await asyncFunc(t);
}
catch (Exception e)
{
if (onException != null)
{
lock (lockObject)
{
onException.Invoke(e);
}
}
// This exception will be swallowed here but it will be collected at the end of ForEachAsync method in order to generate AggregateException.
throw;
}
finally
{
semaphore.Release();
}
}, cancellationToken);
if (cancellationToken.IsCancellationRequested)
{
break;
}
}
}
catch (OperationCanceledException e)
{
exceptions.Add(e);
}
foreach (var t in tasks)
{
if (cancellationToken.IsCancellationRequested)
{
break;
}
// Exception handling in this case is actually pretty fast.
// https://gist.github.com/shoter/d943500eda37c7d99461ce3dace42141
try
{
await t;
}
#pragma warning disable CA1031 // Do not catch general exception types - we want to throw that exception later as aggregate exception. Nothing wrong here.
catch (Exception e)
#pragma warning restore CA1031 // Do not catch general exception types
{
exceptions.Add(e);
}
}
if (exceptions.Any())
{
throw new AggregateException(exceptions);
}
}
}
Consider the following ideal code (which doesn't work). I'm essentially trying to create a list of Tasks that return a specific object, associate them with a string identifier, then execute all of them in bulk with Task.WhenAll. At the end of the execution, I need to have the results of those Tasks still associated with the string identifiers they were originally created with:
public async Task<SomeObject> DoSomethingAsync(string thing)
{
// implementation elided
}
public async Task<SomeObject> DoSomethingElseAsync(string thing)
{
// different implementation elided
}
public async Task<IEnumerable<(string, SomeObject)>>
DoManyThingsAsync(IEnumerable<string> someListOfStrings, bool condition)
{
var tasks = new List<(string, Task<SomeObject>)>();
foreach (var item in someListOfStrings)
{
if (condition)
{
tasks.Add((item, DoSomethingAsync(item)));
}
else
{
tasks.Add((item, DoSomethingElseAsync(item)));
}
}
// this doesn't compile, i'm just demonstrating what i want to achieve
var results = await Task.WhenAll(tasks);
return results;
}
This can be rewritten to the following:
public async Task<(string, SomeObject)> DoSomethingWrapperAsync(string thing)
=> (thing, await DoSomethingAsync(thing));
public async Task<(string, SomeObject)> DoSomethingElseWrapperAsync(string thing)
=> (thing, await DoSomethingElseAsync(thing));
public async Task<IEnumerable<(string, SomeObject)>>
DoManyThingsAsync(IEnumerable<string> someListOfStrings, bool condition)
{
var tasks = new List<Task<(string, SomeObject)>>();
foreach (var thing in someListOfStrings)
{
if (condition)
{
tasks.Add(DoSomethingWrapperAsync(thing));
}
else
{
tasks.Add(DoSomethingElseWrapperAsync(thing));
}
}
// this does compile
var results = await Task.WhenAll(tasks);
return results;
}
The problem is that I need an extra wrapper method for every possible discrete async function I'm going to call, which feels unnecessary and wasteful and is a lot of code (because there will be MANY of these methods). Is there a simpler way of achieving what I need?
I looked into implementing the awaitable/awaiter pattern, but can't see how I could get it to work with Task.WhenAll which requires a collection of Task or Task<TResult>, since the guidance seems to be "don't extend those classes".
You can either do the zipping as you go:
public async Task<IEnumerable<(string, SomeObject)>>
DoManyThingsAsync(IEnumerable<string> someListOfStrings, bool condition)
{
var tasks = someListOfStrings
.Select(async item =>
condition ?
(item, await DoSomethingAsync(item)) :
(item, await DoSomethingElseAsync(item)))
.ToList();
return await Task.WhenAll(tasks);
}
Or, you can keep the input as a separate collection and zip it later:
public async Task<IEnumerable<(string, SomeObject)>>
DoManyThingsAsync(IEnumerable<string> someListOfStrings, bool condition)
{
// Reify input so we don't enumerate twice.
var input = someListOfStrings.ToList();
var tasks = input
.Select(item =>
condition ?
DoSomethingAsync(item) :
DoSomethingElseAsync(item))
.ToList();
var taskResults = await Task.WhenAll(tasks);
return input.Zip(taskResults, (item, taskResult) => ((item, taskResult)));
}
From what I can gather you are using condition to determine which method (of the exact same signature) is being called. Why not pass a callback that is called for each item instead of doing the logic inside the foreach loop?
public async Task<SomeObject> DoSomethingAsync(string thing)
{
// ...
}
public async Task<SomeObject> DoSomethingElseAsync(string thing)
{
// ...
}
public async Task<IEnumerable<(string, SomeObject)>> DoManyThingsAsync(IEnumerable<string> someListOfStrings, bool condition)
{
Func<string, Task<SomeObject>> callback = condition ? DoSomethingAsync : DoSomethingElseAsync;
var results = await Task.WhenAll(
someListOfStrings.Select(async thing => (thing, await callback(thing)))
);
return results;
}
Furthermore, you can extract this as an extension method.
public static class AsyncExtensions
{
public static async Task<IEnumerable<(T, TResult)>> WhenAllAsync(this IEnumerable<T> collection, Func<T, Task<TResult>> selector)
{
var results = await Task.WhenAll(
collection.Select(async item => (item, await selector(item)))
);
return results;
}
}
public async Task MyMethodAsync()
{
// ...
var results = await myListOfStrings.WhenAllAsync(condition ? DoSomethingAsync : DoSomethingElseAsync);
// ...
}
Or do the mapping in the callback.
public static class AsyncExtensions
{
public static Task<IEnumerable<TResult>> WhenAllAsync(this IEnumerable<T> collection, Func<T, Task<TResult>> selector)
=> Task.WhenAll(collection.Select(selector)));
}
public async Task MyMethodAsync()
{
// ...
var results = await myListOfStrings.WhenAllAsync(async thing => (thing, await (condition ? DoSomethingAsync(thing) : DoSomethingElseAsync(thing))));
// ...
}
I have a method that returns some value based on an API call, this API limits the amount of calls that you can do per period of time. I need to access the results of this call from multiple threads. Right now i have the following code:
class ReturningSemaphoreLocker<TOutput>
{
private readonly SemaphoreSlim _semaphore = new SemaphoreSlim(1, 1);
public async Task<T> LockAsync<T>(Func<Task<T>> worker)
{
await _semaphore.WaitAsync();
try
{
return await worker();
}
finally
{
_semaphore.Release();
}
}
}
Usage example:
...
private static readonly ReturningSemaphoreLocker<List<int>> LockingSemaphore = new ReturningSemaphoreLocker<List<int>>();
...
public async Task<List<int>> GetStuff()
{
return await LockingSemaphore.LockAsync(async () =>
{
var client = _clientFactory.CreateClient("SomeName");
using (var cts = GetDefaultRequestCts())
{
var resp = await client.GetAsync("API TO QUERY URL", cts.Token);
var jsonString = await resp.Content.ReadAsStringAsync();
var items = JsonConvert.DeserializeObject<List<int>>(jsonString);
return items;
}
});
}
So the question is: how do i get the same result from GetStuff() if it's already running WITHOUT querying the API again and query the API again if the method is not running at this very moment?
The trick here is to hold onto the Task<T> that is the incomplete result; consider the following completely untested approach - the _inProgress field is the key here:
private static readonly ReturningSemaphoreLocker<List<int>> LockingSemaphore = new ReturningSemaphoreLocker<List<int>>();
...
private Task<List<int>> _inProgress;
public Task<List<int>> GetStuffAsync()
{
if (_inProgress != null) return _inProgress;
return _inProgress = GetStuffImplAsync();
}
private async Task<List<int>> GetStuffImplAsync()
{
var result = await LockingSemaphore.LockAsync(async () =>
{
var client = _clientFactory.CreateClient("SomeName");
using (var cts = GetDefaultRequestCts())
{
var resp = await client.GetAsync("API TO QUERY URL", cts.Token);
var jsonString = await resp.Content.ReadAsStringAsync();
var items = JsonConvert.DeserializeObject<List<int>>(jsonString);
return items;
}
});
// this is important so that if everything turns
// out to be synchronous, we don't nuke the _inProgress field *before*
// it has actually been set
await Task.Yield();
// and now wipe the field since we know it is no longer in progress;
// the next caller should actually try to do something interesting
_inProgress = null;
return result;
}
Here is a class that you could use for time-based throttling, instead of the ReturningSemaphoreLocker:
class ThrottledOperation
{
private readonly object _locker = new object();
private readonly Stopwatch _stopwatch = Stopwatch.StartNew();
private Task _task;
public Task<T> GetValueAsync<T>(Func<Task<T>> taskFactory, TimeSpan interval)
{
lock (_locker)
{
if (_task != null && (_stopwatch.Elapsed < interval || !_task.IsCompleted))
{
return (Task<T>)_task;
}
_task = taskFactory();
_stopwatch.Restart();
return (Task<T>)_task;
}
}
}
The GetValueAsync method returns the same task, until the throttling interval has been elapsed and the task has been completed. At that point it creates and returns a new task, using the supplied task-factory method.
Usage example:
private static readonly ThrottledOperation _throttledStuff = new ThrottledOperation();
public Task<List<int>> GetStuffAsync()
{
return _throttledStuff.GetValueAsync(async () =>
{
var client = _clientFactory.CreateClient("SomeName");
using (var cts = GetDefaultRequestCts())
{
var resp = await client.GetAsync("API TO QUERY URL", cts.Token);
var jsonString = await resp.Content.ReadAsStringAsync();
var items = JsonConvert.DeserializeObject<List<int>>(jsonString);
return items;
}
}, TimeSpan.FromSeconds(30));
}
I have an async method, say:
public async Task<T> GetAsync()
{
}
and would be called from:
public async Task<IEnumerable<T>> GetAllAsync()
{
foreach (var item in something)
{
var result = await GetAsync();
yield return result;
}
}
The above syntax is not valid but basically I am after asynchronous generators. I know it can be handled via Observables. I did experiment with Rx.NET and it worked to some extent. But I am trying to avoid the complexity it brings to codebase, and more importantly the above requirement is still essentially not a reactive system (ours is still pull based). For e.g. I would only listen to the incoming async streams for a certain time and I have to stop the producer (not just unsubscribe the consumer) from the consumer side.
I can invert the method signature like this:
public IEnumerable<Task<T>> GetAllAsync()
But this makes doing LINQ operations bit tricky without blocking. I want it to be non-blocking as well as without loading the entire thing into memory. This library: AsyncEnumerable does exactly what I am looking for but how can the same be done with Ix.NET? They are meant for the same thing I believe.
In other words, how can I make use of Ix.NET to generate an IAsyncEnumerable when dealing with await? Like,
public async IAsyncEnumerable GetAllAsync()
{
foreach (var item in something)
{
var result = await GetAsync();
return // what?
}
}
(Edited)
Using System.Linq.Async 4.0.0 from NuGet, now you can use SelectAwait.
class Program
{
static void Main(string[] args)
{
Task.Run(async () =>
await GetAllAsync().ForEachAsync((x) => Console.WriteLine(x)));
Thread.Sleep(4000);
}
static IAsyncEnumerable<string> GetAllAsync()
{
var something = new[] { 1, 2, 3 };
return something
.ToAsyncEnumerable()
.SelectAwait(async (x) => await GetAsync(x));
}
static async Task<string> GetAsync(int item)
{
await Task.Delay(1000); // heavy
return "got " + item;
}
}
(Obsolete)
Using System.Interactive.Async 3.2.0 from NuGet, how about this? Currently Select() does not support async lambda, you have to implement it by yourself.
Better support for async - Task based overloads for AsyncEnumerable
class Program
{
static void Main(string[] args)
{
Task.Run(async () =>
await GetAllAsync().ForEachAsync((x) => Console.WriteLine(x)));
Thread.Sleep(4000);
}
static IAsyncEnumerable<string> GetAllAsync()
{
var something = new[] { 1, 2, 3 };
return something.SelectAsync(async (x) => await GetAsync(x));
}
static async Task<string> GetAsync(int item)
{
await Task.Delay(1000); // heavy
return "got " + item;
}
}
static class AsyncEnumerableExtensions
{
public static IAsyncEnumerable<TResult> SelectAsync<T, TResult>(this IEnumerable<T> enumerable, Func<T, Task<TResult>> selector)
{
return AsyncEnumerable.CreateEnumerable(() =>
{
var enumerator = enumerable.GetEnumerator();
var current = default(TResult);
return AsyncEnumerable.CreateEnumerator(async c =>
{
var moveNext = enumerator.MoveNext();
current = moveNext
? await selector(enumerator.Current).ConfigureAwait(false)
: default(TResult);
return moveNext;
},
() => current,
() => enumerator.Dispose());
});
}
}
The extension method is quoted from this sample. https://github.com/maca88/AsyncGenerator/issues/94#issuecomment-385286972
i have below code, looks like the await statement in ApiClass will cause function "AControllerMethodInAspMVC" to return earlier before each api.GetResultFromAnotherService is finished.
the main thread return before all children thread finished. is there a way to fix this issue?
private ApiClass api = new ApiClass();
[HttpPost]
public Task<JsonResult> AControllerMethodInAspMVC()
{
var arrayOfItem = …;
List<object> resultObjs = new List<object>();
var resultLock = new SemaphoreSlim(1);
Parallel.ForEach(
arrayOfItem,
async item =>
{
var result = await api.GetResultFromAnotherService(item.id);
var resultObj = new {
// prepare resultObj from result
};
await resultLock.WaitAsync();
resultObjs.add(resultObj);
resultLock.Release();
});
return Task.FromResult(this.Json(resultObjs));
}
Public class ApiClass
{
Public async Task<string> GetResultFromAnotherService(string id)
{
….
…
await Call AnAsyncOperationToGetResult
…
…
}
}
Parallel.ForEach() does not understand async, so your lambda is compiled as async void. What this means is that as soon as you hit the await, the ForEach() thinks the iteration is complete and continues with another iteration.
One way to fix that would be to first start all of the service calls at the same time and then wait for all of them to complete using Task.WhenAll():
public async Task<JsonResult> AControllerMethodInAspMVC()
{
var arrayOfItem = …;
var tasks = arrayOfItem.Select(
item => api.GetResultFromAnotherService(item.id));
return await Task.WhenAll(tasks);
}
If you want to limit how many times is the service call executed in parallel, you could use SemaphoreSlim's WaitAsync():
var semaphore = new SemaphoreSlim(degreeOfParallelism);
var tasks = arrayOfItem.Select(
async item =>
{
await semaphore.WaitAsync();
try
{
return await api.GetResultFromAnotherService(item.id);
}
finally
{
sempahore.Release();
}
});