We Keep Coding

Azure functions - Parallel tasks seems not to run simultaneously

I have a question referencing the usage of concurrently running tasks in Azure Functions, on the consumption plan.
One part of our application allows users to connect their mail accounts, then downloads messages every 15 minutes. We have azure function to do so, one for all users. The thing is, as users count increases, the function need's more time to execute.
In order to mitigate a timeout case, I've changed our function logic. You can find some code below. Now it creates a separate task for each user and then waits for all of them to finish. There is also some exception handling implemented, but that's not the topic for today.
The problem is, that when I check some logs, I see executions as the functions weren't executed simultaneously, but rather one after one. Now I wonder if I made some mistake in my code, or is it a thing with azure functions that they cannot run in such a scenario (I haven't found anything suggesting it on the Microsoft sites, quite the opposite actually)
PS - I do know about durable functions, however, for some reason I'd like to resolve this issue without them.
My code:
List<Task<List<MailMessage>>> tasks = new List<Task<List<MailMessage>>>();
foreach (var account in accounts)
{
using (var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromMinutes(6)))
{
try
{
tasks.Add(GetMailsForUser(account, cancellationTokenSource.Token, log));
}
catch (TaskCanceledException)
{
log.LogInformation("Task was cancelled");
}
}
}
try
{
await Task.WhenAll(tasks.ToArray());
}
catch(AggregateException aex)
{
aex.Handle(ex =>
{
TaskCanceledException tcex = ex as TaskCanceledException;
if (tcex != null)
{
log.LogInformation("Handling cancellation of task {0}", tcex.Task.Id);
return true;
}
return false;
});
}
log.LogInformation($"Zakończono pobieranie wiadomości.");
private async Task<List<MailMessage>> GetMailsForUser(MailAccount account, CancellationToken cancellationToken, ILogger log)
{
log.LogInformation($"[{account.UserID}] Rozpoczęto pobieranie danych dla konta {account.EmailAddress}");
IEnumerable<MailMessage> mails;
try
{
using (var client = _mailClientFactory.GetIncomingMailClient(account))
{
mails = client.GetNewest(false);
}
log.LogInformation($"[{account.UserID}] Pobrano {mails.Count()} wiadomości dla konta {account.EmailAddress}.");
return mails.ToList();
}
catch (Exception ex)
{
log.LogWarning($"[{account.UserID}] Nie udało się pobrać wiadomości dla konta {account.EmailAddress}");
log.LogError($"[{account.UserID}] {ex.Message} {ex.StackTrace}");
return new List<MailMessage>();
}
}
Output:

Azure functions in a consumption plan scales out automatically. Problem is that the load needs to be high enough to trigger the scale out.
What is probably happening is that the scaling is not being triggered, therefore everything runs on the same instance, therefore the calls run sequentially.
There is a discussion on this with some code to test it here: https://learn.microsoft.com/en-us/answers/questions/51368/http-triggered-azure-function-not-scaling-to-extra.html

The compiler will give you a warning for GetMailsForUser:
CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(…)' to do CPU-bound work on a background thread.
It's telling you it will run synchronously, which is the behaviour you're seeing. In the warning message there's a couple of recommendations:
Use await. This would be the most ideal solution, since it will reduce the resources your Azure Function uses. However, this means your _mailClientFactory will need to support asynchronous APIs, which may be too much work to take on right now (many SMTP libraries still do not support async).
Use thread pool threads. Task.Run is one option, or you could use PLINQ or Parallel. This solution will consume one thread per account, and you'll eventually hit scaling issues there.

If you want to identify which Task is running in which Function instance etc. use invocation id ctx.InvocationId.ToString(). May be prefix all your logs with this id.
Your code isn't written such that it can be run in parallel by the runtime. See this: Executing tasks in parallel
You can also get more info about the trigger using trigger meta-data. Depends on trigger. This is just to get more insight into what function is handling what message etc.

Worker Service mysteriously stops doing work

My good ladies and gentlemen, I recently had my first go at Worker services in .Net Core 3.1, and only the second go at Windows services in general (first one was made in .Net Framework and works fine to this day). If anyone could maybe shed some light at what I'm missing in the example that I will provide that would be great.
So, to keep it simple, my problem is this:
My supposed long (forever) running Worker service unexpectedly stops doing work at an arbitrary time of day, but still is shown as "Running" in service manager (that's probably how Windows deals with services). It doesn't necessarily have to be every day, but it stops doing work every now and then until I manually stop it and then restart it in Service Manager.
I have also stumbled upon this question which seemed to deal with my problem, but even after completely wrapping all of my service's code blocks in try-catchs, even on top-level, I still get nothing registered in my Log table, or even in the file I set up to write in if my DB connection fails. Service seems to just stop calling ExecuteAsync() method.
Ok here's how my code's logically structured, I have excluded implementation and I'm just showing what happens until DoWork is called:
public class Worker : BackgroundService
{
private readonly IConfiguration _configuration;
public Worker(IConfiguration configuration)
{
_configuration = configuration;
}
public override Task StartAsync(CancellationToken cancellationToken)
{
return base.StartAsync(cancellationToken);
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
while (true)
{
try //paranoid try-catch
{
await DoWork();
await Task.Delay(TimeSpan.FromSeconds(45), stoppingToken);
}
catch (Exception e)
{
await Log(e, customMessage: "Proccess failed at top level.");
}
}
}
catch (Exception e)
{
await Log(e, customMessage: "Proccess failed at topmost level.");
}
}
private async Task DoWork()
{
try
{
}
catch (Exception e)
{
await Log(e);
}
}
public async Task Log(Exception e, string user = null, string emailID = null, string customMessage = null)
{
}
}
As you can see, I am not handling cancellation, as in the question I linked above. Now that I think about it maybe I should, and something is inadvertently sending cancellation? The reason I didn't is because I'm not sure what events exactly signal the cancellation. Only the manual stopping of service, or something else maybe? And if it is the cancellation that was sent that caused my service to stop doing work, shouldn't it also stop my service from running?
Btw I just tested cancellation on dummy service which implements my logic with while(true) and it catches the stopping exception, even though it's a bit awkward, as it catches it and logs it multiple times before stopping, so I presume it may not be the cancellation token that is causing my DoWork not to fire.

Ok guys I'd fixed it. See comment below.
I'd guessed that what was causing deadlock was probably too many concurrent calls from different threads to database over the same connection.
Not that that I knew that would be the cause (and I still don't know and can only guess why this happens so if someone can clarify why this happens and why don't the calls get queued please do), but as I tried to fix it that seemed like a good starting point.
What I did was just limit possible concurrent calls to 1:
Instantiate SemaphoreSlim on a class level:
private static SemaphoreSlim Semaphore = new SemaphoreSlim(1);
Insert a SemaphoreSlim.WaitAsync before each of my DB calls and its respective SemaphoreSlim.Release in a finally block after the call:
try
{
await Semaphore.WaitAsync();
var id = await sqlCommand.ExecuteScalarAsync().ToString();
}
finally
{
Semaphore.Release();
}
I thought this would decrease the performance but to my pleasant surprise I felt no noticeable difference.
Also, I was tempted to set Semaphore's initial count to more than 1 thread but I figured if deadlock happens for many threads, then it might happen for 2-10 threads. Does anyone perhaps know anything more about this number? Is it processor related, SQL related, or perhaps C# related?

Have you implemented a dispose method to close the database connection after finishing the DoWork method? I had a deadlock problem using worker service and realized the database connection wasn’t disposed. After implementing a dispose method, it works for me to solve the problem.

Old question, and I don't know how or even if Manus's issue eventually resolved, but in my experience, exceptions in threads that aren't the main thread caused this problem for us in a Windows Service. And we didn't see this when running the same code as a Windows Forms. Try/catch wrapped around the main processing line does not catch them. We had to add it in the threaded method.

Why async controller method is executing just like a sync method?

I've created sample .NET Core WebApi application to test how async methods can increase the throughput. App is hosted on IIS 10.
Here is a code of my controller:
[HttpGet("sync")]
public IEnumerable<string> Get()
{
return this.GetValues().Result;
}
[HttpGet("async")]
public async Task<IEnumerable<string>> GetAsync()
{
return await this.GetValues();
}
[HttpGet("isresponding")]
public Task<bool> IsResponding()
{
return Task.FromResult(true);
}
private async Task<IEnumerable<string>> GetValues()
{
await Task.Delay(TimeSpan.FromSeconds(10)).ConfigureAwait(false);
return new string[] { "value1", "value2" };
}
there are methods:
Get() - to get result synchronously
GetAsync() - to get result asynchronously.
IsResponding() - to check that server can serve requests
Then I created sample console app, which creates 100 requests to sync and async method (no waiting for result) of the controller. Then I call method IsResponding() to check whether server is available.
Console app code is:
using (var httpClient = new HttpClient())
{
var methodUrl = $"http://localhost:50001/api/values/{apiMethod}";
Console.WriteLine($"Executing {methodUrl}");
//var result1 = httpClient.GetAsync($"http://localhost:50001/api/values/{apiMethod}").Result.Content.ReadAsStringAsync().Result;
Parallel.For(0, 100, ((i, state) =>
{
httpClient.GetAsync(methodUrl);
}));
var sw = Stopwatch.StartNew();
var isAlive = httpClient.GetAsync($"http://localhost:50001/api/values/isresponding").Result.Content;
Console.WriteLine($"{sw.Elapsed.TotalSeconds} sec.");
Console.ReadKey();
}
where {apiMethod} is "sync" or "async", depending on user input.
In both cases server is not responding for a long time (about 40 sec).
I expexted that in async case server should continue serving requests fast, but it doesn't.
UPDATE 1:
I've changed client code like this:
Parallel.For(0, 10000, ((i, state) =>
{
var httpClient = new HttpClient();
httpClient.GetAsync($"http://localhost:50001/api/values/{apiMethod}");
}));
using (var httpClient = new HttpClient())
{
var sw = Stopwatch.StartNew();
// this method should evaluate fast when we called async version and should evaluate slowly when we called sync method (due to busy threads ThreadPool)
var isAlive = httpClient.GetAsync($"http://localhost:50001/api/values/isresponding").Result.Content;
Console.WriteLine($"{sw.Elapsed.TotalSeconds} sec.");
}
and calling IsResponding() method executing for a very long time.
UPDATE 2
Yes, I know how async methods work. Yes, I know how to use HttpClient. It's just a sample to prove theory.
UPDATE 3
As it mentioned by StuartLC in one of the comments, IIS somehow throtling or blocking requests. When I started my WebApi as SelfHosted it started working as expected:
Executing time of "isresponsible" method after bunch of requests to ASYNC method is very fast, at about 0.02 sec.
Executing time of "isresponsible" method after bunch of requests to SYNC method is very slow, at about 35 sec.

You don't seem to understand async. It doesn't make the response return faster. The response cannot be returned until everything the action is doing is complete, async or not. If anything, async is actually slower, if only slightly, because there's additional overhead involved in asynchronous processing not necessary for synchronous processing.
What async does do is potentially allow the active thread servicing the request to be returned to the pool to service other requests. In other words, async is about scale, not performance. You'll only see benefits when your server is slammed with requests. Then, when incoming requests would normally have been queued sync, you'll process additional requests from some of the async tasks forfeiting their threads to the cause. Additionally, there is no guarantee that the thread will be freed at all. If the async task completes immediately or near immediately, the thread will be held, just as with sync.
EDIT
You should also realize that IIS Express is single-threaded. As such, it's not a good guage for performance tuning. If you're running 1000 simultaneous requests, 999 are instantly queued. Then, you're not doing any asynchronous work - just returning a completed task. As such, the thread will never be released, so there is literally no difference between sync and async in this scenario. Therefore, you're down to just how long it takes to process through the queue of 999 requests (plus your status check at the end). You might have better luck at teasing out a difference if you do something like:
await Task.Delay(500);
Instead of just return Task.FromResult. That way, there's actual idle time on the thread that may allow it to be returned to the pool.

IIS somehow throtling or blocking requests (as mentioned in one of the comments). When I started my WebApi as SelfHosted it started working as expected:
Executing time of isresponsible method after bunch of requests to ASYNC method is very fast, at about 0.02 sec.
Executing time of isresponsible method after bunch of requests to SYNC method is very slow, at about 35 sec.

I'm not sure this will yield any major improvement, but you should call ConfigureAwait(false) on every awaitable in the server, including in GetAsync.
It should produce better use of the threads.

Why an additional async operation is making my code faster than when the operation is not taking place at all?

I'm working on a SMS-based game (Value Added Service), in which a question must be sent to each subscriber on a daily basis. There are over 500,000 subscribers and therefore performance is a key factor. Since each subscriber can be a difference state of the competition with different variables, database must be queried separately for each subscriber before sending a text message. To achieve the best performance I'm using .Net Task Parallel Library (TPL) to spawn parallel threadpool threads and do as much async operations as possible in each thread to finally send texts asap.
Before describing the actual problem there are some more information necessary to give about the code.
At first there was no async operation in the code. I just scheduled some 500,000 tasks with the default task scheduler into the Threadpool and each task would work through the routines, blocking on all EF (Entity Framework) queries and sequentially finishing its job. It was good, but not fast enough. Then I changed all EF queries to Async, the outcome was superb in speed but there has been so many deadlocks and timeouts in SQL server that about a third of the subscribers never received a text! After trying different solutions, I decided not to do too many Async Database operations while I have over 500,000 tasks running on a 24 core server (with at least 24 concurrent threadpool threads)!
I rolled back all the changes (the Asycn ones) expect for one web service call in each task which remained Async.
Now the weird case:
In my code, I have a boolean variable named "isCrossSellActive". When the variable is set some more DB operations take place and an asycn webservice call will happen on which the thread awaits. When this variable is false, none of these operations will happen including the async webservice call. Awkwardly when the variable is set the code runs so much faster than when it's not! It seems like for some reason the awaited async code (the cooperative thread) is making the code faster.
Here is the code:
public async Task AutoSendMessages(...)
{
//Get list of subscriptions plus some initialization
LimitedConcurrencyLevelTaskScheduler lcts = new LimitedConcurrencyLevelTaskScheduler(numberOfThreads);
TaskFactory taskFactory = new TaskFactory(lcts);
List<Task> tasks = new List<Task>();
//....
foreach (var sub in subscriptions)
{
AutoSendData data = new AutoSendData
{
ServiceId = serviceId,
MSISDN = sub.subscriber,
IsCrossSellActive = bolCrossSellHeader
};
tasks.Add(await taskFactory.StartNew(async (x) =>
{
await SendQuestion(x);
}, data));
}
GC.Collect();
try
{
Task.WaitAll(tasks.ToArray());
}
catch (AggregateException ae)
{
ae.Handle((ex) =>
{
_logRepo.LogException(1, "", ex);
return true;
});
}
await _autoSendRepo.SetAutoSendingStatusEnd(statusId);
}
public async Task SendQuestion(object data)
{
//extract variables from input parameter
try
{
if (isCrossSellActive)
{
int pieceCount = subscriptionRepo.GetSubscriberCarPieces(curSubscription.service, curSubscription.subscriber).Count(c => c.isConfirmed);
foreach (var rule in csRules)
{
if (rule.Applies)
{
if (await HttpClientHelper.GetJsonAsync<bool>(url, rule.TargetServiceBaseAddress))
{
int noOfAddedPieces = SomeCalculations();
if (noOfAddedPieces > 0)
{
crossSellRepo.SetPromissedPieces(curSubscription.subscriber, curSubscription.service,
rule.TargetShortCode, noOfAddedPieces, 0, rule.ExpirationLimitDays);
}
}
}
}
}
// The rest of the code. (Some db CRUD)
await SmsClient.SendSoapMessage(subscriber, smsBody);
}
catch (Exception ex){//...}
}

Ok, thanks to #usr and the clue he gave me, the problem is finally solved!
His comment drew my attention to the awaited taskFactory.StartNew(...) line which sequentially adds new tasks to the "tasks" list which is then awaited on by Task.WaitAll(tasks);
At first I removed the await keyword before the taskFactory.StartNew() and it led the code towards a horrible state of malfunction! I then returned the await keyword to before taskFactory.StartNew() and debugged the code using breakpoints and amazingly saw that the threads are ran one after another and sequentially before the first thread reaches the first await inside the "SendQuestion" routine. When the "isCrossSellActive" flag was set despite the more jobs a thread should do the first await keyword is reached earlier thus enabling the next scheduled task to run. But when its not set the only await keyword is the last line of the routine so its most likely to run sequentially to the end.
usr's suggestion to remove the await keyword in the for loop seemed to be correct but the problem was the Task.WaitAll() line would wait on the wrong list of Task<Task<void>> instead of Task<void>. I finally used Task.Run instead of TaskFactory.StartNew and everything changed. Now the service is working well. The final code inside the for loop is:
tasks.Add(Task.Run(async () =>
{
await SendQuestion(data);
}));
and the problem was solved.
Thank you all.
P.S. Read this article on Task.Run and why TaskFactory.StartNew is dangerous: http://blog.stephencleary.com/2013/08/startnew-is-dangerous.html

It's extremly hard to tell unless you add some profiling that tell you which code is taking longer now.
Without seeing more numbers my best guess would be that the SMS service doesn't like when you send too many requests in a short time and chokes. When you add the extra DB calls the extra delay make the sms service work better.
A few other small details:
await Task.WhenAll is usually a bit better than Task.WaitAll. WaitAll means the thread will sit around waiting. Making a deadlock slightly more likely.
Instead of:
tasks.Add(await taskFactory.StartNew(async (x) =>
{
await SendQuestion(x);
}, data));
You should be able to do
tasks.Add(SendQuestion(data));

async Task<HttpResponseMessage> Get VS HttpResponseMessage Get

I would need your help in the following. For nearly a month, I have been reading regarding Tasks and async .
I wanted to try to implement my new acquired knowledege, in a simple wep api project. I have the following methods and both of them working as expected:
public HttpResponseMessage Get()
{
var data = _userServices.GetUsers();
return Request.CreateResponse(HttpStatusCode.OK, data);
}
public async Task<HttpResponseMessage> Get()
{
var data = _userServices.GetUsers();
return await Task<HttpResponseMessage>.Factory.StartNew(() =>
{
return Request.CreateResponse(HttpStatusCode.OK, data);
});
}
So the question. I have tried to use fiddler and see what is the difference between these two. The async one is little faster, but apart from that, what is the real benefit in implementing something like that in a web api?

As others have pointed out, the point of async on ASP.NET is that it frees up one of the ASP.NET thread pool threads. This works great for naturally-asynchronous operations such as I/O-bound operations because that's one less thread on the server (there is no thread that is "processing" the async operation, as I explain on my blog). Thus, the primary benefit of async on the server side is scalability.
However, you want to avoid Task.Run (and, even worse, Task.Factory.StartNew) on ASP.NET. I call this "fake asynchrony" because they're just doing synchronous/blocking work on a thread pool thread. They're useful in UI apps where you want to push work off the UI thread so the UI remains responsive, but they should (almost) never be used on ASP.NET or other server apps.
Using Task.Run or Task.Factory.StartNew on ASP.NET will actually decrease your scalability. They will cause some unnecessary thread switches. For longer-running operations, you could end up throwing off the ASP.NET thread pool heuristics, causing additional threads to be created and later destroyed needlessly. I explore these performance problems step-by-step in another blog post.
So, you need to think about what each action is doing, and whether any of that should be async. If it should, then that action should be async. In your case:
public HttpResponseMessage Get()
{
var data = _userServices.GetUsers();
return Request.CreateResponse(HttpStatusCode.OK, data);
}
What exactly is Request.CreateResponse doing? It's just creating response object. That's it - just a fancy new. There's no I/O going on there, and it certainly isn't something that needs to be pushed off to a background thread.
However, GetUsers is much more interesting. That sounds more like a data read, which is I/O-based. If your backend can scale (e.g., Azure SQL / Tables / etc), then you should look at making that async first, and once your service is exposing a GetUsersAsync, then this action could become async too:
public async Task<HttpResponseMessage> Get()
{
var data = await _userServices.GetUsersAsync();
return Request.CreateResponse(HttpStatusCode.OK, data);
}

Using async on your server can dramatically improve scalability as it frees up the thread serving the request to handle other requests while the async operation is in progress. For example in a synchronous IO operaton, the thread would be suspended and doing nothing until the operation completes and would not be available to serve another request.
That being said, using Task.Factory.StartNew starts another thread so you don't get the scalability benefits at all. Your original thread can be reused, but you have offloaded the work to another thread so there is no net benefit at all. in fact there is a cost of switching to another thread, but that is minimal.
Truly asynchronous operations do not start a thread and I would look to see if such an operation exists, or if one can be written for Request.CreateResponse. Then your code would be much more scalable. If not, you are better off sticking with the synchronous approach.

It makes more sense where the call is happening with major IO operations.
Yes, Async is faster because it frees up the request thread for the time that the operations is being performed. Thus, from Web server point of view, you are giving a thread back to the pool that can be used by the server for any future calls coming through.
So for e.g. when you are performing a search operation on SQL server, you might want to do async and see the performance benefit.
It is good for scalability that involves multiple servers.
So, for e.g. when the SearchRecordAsync sends its SQL to the database, it returns an incomplete task, and when the request hits the await, it returns the request thread to the thread pool. Later, when the DB operation completes, a request thread is taken from the thread pool and used to continue the request.
Even if you are not using, SQL operations, let say you want to send an email to 10 people. In this case also async makes more sense.
Async is also very handy to show the progress of long event. So user will still get the active GUI, while the task is running at the background.
To understand, please have a look at this sample.
Here I am trying to initiate task called send mail. Interim I want to update database, while the background is performing send mail task.
Once the database update has happened, it is waiting for the send mail task to be completed. However, with this approach it is quite clear that I can run task at the background and still proceed with original (main) thread.
using System;
using System.Threading;
using System.Threading.Tasks;
public class Program
{
public static void Main()
{
Console.WriteLine("Starting Send Mail Async Task");
Task task = new Task(SendMessage);
task.Start();
Console.WriteLine("Update Database");
UpdateDatabase();
while (true)
{
// dummy wait for background send mail.
if (task.Status == TaskStatus.RanToCompletion)
{
break;
}
}
}
public static async void SendMessage()
{
// Calls to TaskOfTResult_MethodAsync
Task<bool> returnedTaskTResult = MailSenderAsync();
bool result = await returnedTaskTResult;
if (result)
{
UpdateDatabase();
}
Console.WriteLine("Mail Sent!");
}
private static void UpdateDatabase()
{
for (var i = 1; i < 1000; i++) ;
Console.WriteLine("Database Updated!");
}
private static async Task<bool> MailSenderAsync()
{
Console.WriteLine("Send Mail Start.");
for (var i = 1; i < 1000000000; i++) ;
return true;
}
}