I have a question referencing the usage of concurrently running tasks in Azure Functions, on the consumption plan.
One part of our application allows users to connect their mail accounts, then downloads messages every 15 minutes. We have azure function to do so, one for all users. The thing is, as users count increases, the function need's more time to execute.
In order to mitigate a timeout case, I've changed our function logic. You can find some code below. Now it creates a separate task for each user and then waits for all of them to finish. There is also some exception handling implemented, but that's not the topic for today.
The problem is, that when I check some logs, I see executions as the functions weren't executed simultaneously, but rather one after one. Now I wonder if I made some mistake in my code, or is it a thing with azure functions that they cannot run in such a scenario (I haven't found anything suggesting it on the Microsoft sites, quite the opposite actually)
PS - I do know about durable functions, however, for some reason I'd like to resolve this issue without them.
My code:
List<Task<List<MailMessage>>> tasks = new List<Task<List<MailMessage>>>();
foreach (var account in accounts)
{
using (var cancellationTokenSource = new CancellationTokenSource(TimeSpan.FromMinutes(6)))
{
try
{
tasks.Add(GetMailsForUser(account, cancellationTokenSource.Token, log));
}
catch (TaskCanceledException)
{
log.LogInformation("Task was cancelled");
}
}
}
try
{
await Task.WhenAll(tasks.ToArray());
}
catch(AggregateException aex)
{
aex.Handle(ex =>
{
TaskCanceledException tcex = ex as TaskCanceledException;
if (tcex != null)
{
log.LogInformation("Handling cancellation of task {0}", tcex.Task.Id);
return true;
}
return false;
});
}
log.LogInformation($"Zakończono pobieranie wiadomości.");
private async Task<List<MailMessage>> GetMailsForUser(MailAccount account, CancellationToken cancellationToken, ILogger log)
{
log.LogInformation($"[{account.UserID}] Rozpoczęto pobieranie danych dla konta {account.EmailAddress}");
IEnumerable<MailMessage> mails;
try
{
using (var client = _mailClientFactory.GetIncomingMailClient(account))
{
mails = client.GetNewest(false);
}
log.LogInformation($"[{account.UserID}] Pobrano {mails.Count()} wiadomości dla konta {account.EmailAddress}.");
return mails.ToList();
}
catch (Exception ex)
{
log.LogWarning($"[{account.UserID}] Nie udało się pobrać wiadomości dla konta {account.EmailAddress}");
log.LogError($"[{account.UserID}] {ex.Message} {ex.StackTrace}");
return new List<MailMessage>();
}
}
Output:
Azure functions in a consumption plan scales out automatically. Problem is that the load needs to be high enough to trigger the scale out.
What is probably happening is that the scaling is not being triggered, therefore everything runs on the same instance, therefore the calls run sequentially.
There is a discussion on this with some code to test it here: https://learn.microsoft.com/en-us/answers/questions/51368/http-triggered-azure-function-not-scaling-to-extra.html
The compiler will give you a warning for GetMailsForUser:
CS1998: This async method lacks 'await' operators and will run synchronously. Consider using the 'await' operator to await non-blocking API calls, or 'await Task.Run(…)' to do CPU-bound work on a background thread.
It's telling you it will run synchronously, which is the behaviour you're seeing. In the warning message there's a couple of recommendations:
Use await. This would be the most ideal solution, since it will reduce the resources your Azure Function uses. However, this means your _mailClientFactory will need to support asynchronous APIs, which may be too much work to take on right now (many SMTP libraries still do not support async).
Use thread pool threads. Task.Run is one option, or you could use PLINQ or Parallel. This solution will consume one thread per account, and you'll eventually hit scaling issues there.
If you want to identify which Task is running in which Function instance etc. use invocation id ctx.InvocationId.ToString(). May be prefix all your logs with this id.
Your code isn't written such that it can be run in parallel by the runtime. See this: Executing tasks in parallel
You can also get more info about the trigger using trigger meta-data. Depends on trigger. This is just to get more insight into what function is handling what message etc.
Related
We have a third-party method Foo which sometimes runs in a deadlock for unknown reasons.
We are executing an single-threaded tcp-server and call this method every 30 seconds to check that the external system is available.
To mitigate the problem with the deadlock in the third party code we put the ping-call in a Task.Run to so that the server does not deadlock.
Like
async Task<bool> WrappedFoo()
{
var timeout = 10000;
var task = Task.Run(() => ThirdPartyCode.Foo());
var delay = Task.Delay(timeout);
if (delay == await Task.WhenAny(delay, task ))
{
return false;
}
else
{
return await task ;
}
}
But this (in our opinion) has the potential to starve the application of free threads. Since if one call to ThirdPartyCode.Foo deadlock the thread will never recover from this deadlock and if this happens often enough we might run out of resources.
Is there a general approach how one should handle deadlocking third-party code?
A CancellationToken won't work because the third-party-api does not provide any cancellation options.
Update:
The method at hand is from the SAPNCO.dll provided by SAP to establish and test rfc-connections to a sap-system, therefore the method is not a simple network-ping. I renamed the method in the question to avoid further misunderstandings
Is there a general approach how one should handle deadlocking third-party code?
Yes, but it's not easy or simple.
The problem with misbehaving code is that it can not only leak resources (e.g., threads), but it can also indefinitely hold onto important resources (e.g., some internal "handle" or "lock").
The only way to forcefully reclaim threads and other resources is to end the process. The OS is used to cleaning up misbehaving processes and is very good at it. So, the solution here is to start a child process to do the API call. Your main application can communicate with its child process by redirected stdin/stdout, and if the child process ever times out, the main application can terminate it and restart it.
This is, unfortunately, the only reliable way to cancel uncancelable code.
Cancelling a task is a collaborative operation in that you pass a CancellationToken to the desired method and externally you use CancellationTokenSource.Cancel:
public void Caller()
{
try
{
CancellationTokenSource cts=new CancellationTokenSource();
Task longRunning= Task.Run(()=>CancellableThirdParty(cts.Token),cts.Token);
Thread.Sleep(3000); //or condition /signal
cts.Cancel();
}catch(OperationCancelledException ex)
{
//treat somehow
}
}
public void CancellableThirdParty(CancellationToken token)
{
while(true)
{
// token.ThrowIfCancellationRequested() -- if you don't treat the cancellation here
if(token.IsCancellationRequested)
{
// code to treat the cancellation signal
//throw new OperationCancelledException($"[Reason]");
}
}
}
As you can see in the code above , in order to cancel an ongoing task , the method running inside it must be structured around the CancellationToken.IsCancellationRequested flag or simply CancellationToken.ThrowIfCancellationRequested method ,
so that the caller just issues the CancellationTokenSource.Cancel.
Unfortunately if the third party code is not designed around CancellationToken ( it does not accept a CancellationToken parameter ), then there is not much you can do.
Your code isn't cancelling the blocked operation. Use a CancellationTokenSource and pass a cancellation token to Task.Run instead :
var cts=new CancellationTokenSource(timeout);
try
{
await Task.Run(() => ThirdPartyCode.Ping(),cts.Token);
return true;
}
catch(TaskCancelledException)
{
return false;
}
It's quite possible that blocking is caused due to networking or DNS issues, not actual deadlock.
That still wastes a thread waiting for a network operation to complete. You could use .NET's own Ping.SendPingAsync to ping asynchronously and specify a timeout:
var ping=new Ping();
var reply=await ping.SendPingAsync(ip,timeout);
return reply.Status==IPStatus.Success;
The PingReply class contains far more detailed information than a simple success/failure. The Status property alone differentiates between routing problems, unreachable destinations, time outs etc
I've run into an issue with tasks I can't seem to figure out. This application makes repeated HTTP calls via WebClient to several servers. It maintains a dictionary of tasks that are running the HTTP calls, and every five seconds it checks for results, then once the results are in it makes an HTTP call again. This goes on for the lifetime of the application.
Recently, it has started having a problem where the tasks are randomly getting stuck in WaitingForActivation. In the debugger the task shows as "[Scheduled and waiting to run]", but it never runs.
This is the function that it's running, when I click on the "Scheduled" task in the debugger, it points to the DownloadStringTaskAsync() line:
private static async Task<string> DownloadString(string url)
{
using (var client = new WebClient()) {
try {
var result = await client.DownloadStringTaskAsync(url).ConfigureAwait(false);
return result;
} catch (WebException) {
return null;
}
}
}
The code that is actually creating the task that runs the above function is this. It only hits this line once the existing task is completed, Task.IsCompleted never returns true since it's stuck in scheduled status. Task.Status gets stuck in WaitingForActivation.
tasks[resource] = Task.Run(() => DownloadString("http://" + resources[resource] + ":8181/busy"));
The odd thing about this is that, as far as I can tell, this code ran perfectly fine for two years, until we recently did a server migration which included an upgraded OS and spawned a network packet loss issue. That's when we started noticing this particular problem, though I don't see how either of those would be related.
Also, this tends to only happen after the application has been running for several thousand seconds. It runs perfectly fine for a while until tasks, one-by-one, start getting stuck. After about a day, there's usually four or five tasks stuck in scheduled. Since it usually takes time for the first task to get stuck, that seems to me like there would be a race condition of some sort, but I don't see how that could be the case.
Is there a reason a task would get stuck in scheduled and never actually run?
I'm not familiar with ancient WebClient (maybe it contains bugs) but can suggest the recommended by Microsoft way to get a response from a server using System.Net.Http.HttpClient. Also HttpClient is rather faster works with multiple requests per endpoint, especially in .NET Core/.NET 5.
// HttpClient is intended to be instantiated once per application, rather than per-use
private static readonly HttpClient client = new HttpClient();
private static async Task<string> DownloadString(string url)
{
try
{
return await client.GetStringAsync(url).ConfigureAwait(false);
}
catch (HttpRequestException ex)
{
Debug.WriteLine(ex.Message);
return null;
}
}
Also remove Task.Run, it's a kind of redundancy.
tasks[resource] = DownloadString($"http://{resources[resource]}:8181/busy");
Asynchronous programming - read the article. You have to get a difference between I/O-bound and CPU-bound work, and don't spawn Threads without special need for concurrency. You need no Thread here.
We have a logic for a background job to keep running, either every 20 minuts, or when a task if finished.
A simplified version of what I want to do is as followed:
Task to control if we need to exit:
private static TaskCompletionSource<bool> forceSyncTask = new TaskCompletionSource<bool>();
Background job:
Task.Factory.StartNew(
async () =>
{
do
{
await Dosomething();
await Task.WhenAny(Task.Delay(TimeSpan.FromMinutes(20)), forceSyncTask.Task);
// Always reset the force sync property
forceSyncTask = new TaskCompletionSource<bool>();
}
while (true);
});
Then everytime there is a notification comes, I run the following to force to exit the Task.WhenAny
if (!forceSyncTask.Task.IsCompleted)
{
forceSyncTask.TrySetResult(true);
}
I tested it in dev box and it works. However after I deployed it to our webervice in prod environment,
even if I successfully SetResult (I have logging to know if TrySetResult returns true or not), the Task.WhenAny does not exit as expected.
Anyone has any idea why?
First, I recommend you use an established solution for pausing asynchronous methods, such as Stephen Toub's PauseToken or the PauseToken from my AsyncEx library. There's some red flags in the code as it currently stands: StartNew with an async delegate and without a TaskScheduler, and TaskCompletionSource<T> being used without the RunContinuationsAsynchronously option. It's better to stick to higher-level constructs (Task.Run and PauseToken, respectively) because there are lots of sharp corners on the low-level constructs.
As far as what exactly the problem is, that's difficult to tell, especially since you (and we) cannot reproduce it locally. Here's my top guesses:
You're running into a problem caused by the fact that continuations run synchronously if possible - i.e., TrySetResult ends up directly invoking some code within Task.WhenAny.
You're experiencing thread exhaustion on your production server.
This might not be specific to SemaphoreSlim exclusively, but basically my question is about whether there is a difference between the below two methods of throttling a collection of long running tasks, and if so, what that difference is (and when if ever to use either).
In the example below, let's say that each tracked task involves loading data from a Url (totally made up example, but is a common one that I've found for SemaphoreSlim examples).
The main difference comes down to how the individual tasks are added to the list of tracked tasks. In the first example, we call Task.Run() with a lambda, whereas in the second, we new up a Func(<Task<Result>>()) with a lambda and then immediately call that func and add the result to the tracked task list.
Examples:
Using Task.Run():
SemaphoreSlim ss = new SemaphoreSlim(_concurrentTasks);
List<string> urls = ImportUrlsFromSource();
List<Task<Result>> trackedTasks = new List<Task<Result>>();
foreach (var item in urls)
{
await ss.WaitAsync().ConfigureAwait(false);
trackedTasks.Add(Task.Run(async () =>
{
try
{
return await ProcessUrl(item);
}
catch (Exception e)
{
_log.Error($"logging some stuff");
throw;
}
finally
{
ss.Release();
}
}));
}
var results = await Task.WhenAll(trackedTasks);
Using a new Func:
SemaphoreSlim ss = new SemaphoreSlim(_concurrentTasks);
List<string> urls = ImportUrlsFromSource();
List<Task<Result>> trackedTasks = new List<Task<Result>>();
foreach (var item in urls)
{
trackedTasks.Add(new Func<Task<Result>>(async () =>
{
await ss.WaitAsync().ConfigureAwait(false);
try
{
return await ProcessUrl(item);
}
catch (Exception e)
{
_log.Error($"logging some stuff");
throw;
}
finally
{
ss.Release();
}
})());
}
var results = await Task.WhenAll(trackedTasks);
There are two differences:
Task.Run does error handling
First off all, when you call the lambda, it runs. On the other hand, Task.Run would call it. This is relevant because Task.Run does a bit of work behind the scenes. The main work it does is handling a faulted task...
If you call a lambda, and the lambda throws, it would throw before you add the Task to the list...
However, in your case, because your lambda is async, the compiler would create the Task for it (you are not making it by hand), and it will correctly handle the exception and make it available via the returned Task. Therefore this point is moot.
Task.Run prevents task attachment
Task.Run sets DenyChildAttach. This means that the tasks created inside the Task.Run run independently from (are not synchronized with) the returned Task.
For example, this code:
List<Task<int>> trackedTasks = new List<Task<int>>();
var numbers = new int[]{0, 1, 2, 3, 4};
foreach (var item in numbers)
{
trackedTasks.Add(Task.Run(async () =>
{
var x = 0;
(new Func<Task<int>>(async () =>{x = item; return x;}))().Wait();
Console.WriteLine(x);
return x;
}));
}
var results = await Task.WhenAll(trackedTasks);
Will output the numbers from 0 to 4, in unknown order. However the following code:
List<Task<int>> trackedTasks = new List<Task<int>>();
var numbers = new int[]{0, 1, 2, 3, 4};
foreach (var item in numbers)
{
trackedTasks.Add(new Func<Task<int>>(async () =>
{
var x = 0;
(new Func<Task<int>>(async () =>{x = item; return x;}))().Wait();
Console.WriteLine(x);
return x;
})());
}
var results = await Task.WhenAll(trackedTasks);
Will output the numbers from 0 to 4, in order, every time. This is odd, right? What happens is that the inner task is attached to outer one, and executed right away in the same thread. But if you use Task.Run, the inner task is not attached and scheduled independently.
This remain true even if you use await, as long as the task you await does not go to an external system...
What happens with external system? Well, for example, if your task is reading from an URL - as in your example - the system would create a TaskCompletionSource, get the Task from it, set a response handler that writes the result to the TaskCompletionSource, make the request, and return the Task. This Task is not scheduled, it running on the same thread as a parent task makes no sense. And thus, it can break the order.
Since, you are using await to wait on an external system, this point is moot too.
Conclusion
I must conclude that these are equivalent.
If you want to be safe, and make sure it works as expected, even if - in a future version - some of the above points stops being moot, then keep Task.Run. On the other hand, if you really want to optimize, use the lambda and avoid the Task.Run (very small) overhead. However, that probably won't be a bottleneck.
Addendum
When I talk about a task that goes to an external system, I refer to something that runs outside of .NET. There a bit of code that will run in .NET to interface with the external system, but the bulk of the code will not run in .NET, and thus will not be in a managed thread at all.
The consumer of the API specify nothing for this to happen. The task would be a promise task, but that is not exposed, for the consumer there is nothing special about it.
In fact, a task that goes to an external system may barely run in the CPU at all. Futhermore, it might just be waiting on something exterior to the computer (it could be the network or user input).
The pattern is as follows:
The library creates a TaskCompletionSource.
The library sets a means to recieve a notification. It can be a callback, event, message loop, hook, listening to a socket, a pipe line, waiting on a global mutex... whatever is necesary.
The library sets code to react to the notification that will call SetResult, or SetException on the TaskCompletionSource as appropiate for the notification recieved.
The library does the actual call to the external system.
The library returns TaskCompletionSource.Task.
Note: with extra care of optimization not reordering things where it should not, and with care of handling errors during the setup phase. Also, if a CancellationToken is involved, it has to be taken into account (and call SetCancelled on the TaskCompletionSource when appropiate). Also, there could be tear down necesary in the reaction to the notification (or on cancellation). Ah, do not forget to validate your parameters.
Then the external system goes and does whatever it does. Then when it finishes, or something goes wrong, gives the library the notification, and your Task is sudendtly completed, faulted... (or if cancellation happened, your Task is now cancelled) and .NET will schedule the continuations of the task as needed.
Note: async/await uses continuations behind the scenes, that is how execution resumes.
Incidentally, if you wanted to implement SempahoreSlim yourself, you would have to do something very similar to what I describe above. You can see it in my backport of SemaphoreSlim.
Let us see a couple of examples of promise tasks...
Task.Delay: when we are waiting with Task.Delay, the CPU is not spinning. This is not running in a thread. In this case the notification mechanism will be an OS timer. When the OS sees that the time of the timer has elapsed, it will call into the CLR, and then the CLR will mark the task as completed. What thread was waiting? none.
FileStream.ReadSync: when we are reading from storage with FileStream.ReadSync the actual work is done by the device. The CRL has to declare a custom event, then pass the event, the file handle and the buffer to the OS... the OS calls the device driver, the device driver interfaces with the device. As the storage device recovers the information, it will write to memory (directly on the specified buffer) via DMA technology. And when it is done, it will set an interruption, that is handled by the driver, that notifies the OS, that calls the custom event, that marks the task as completed. What thread did read the data from storage? none.
A similar pattern will be used to download from a web page, except, this time the device goes to the network. How to make an HTTP request and how the system waits for a response is beyond the scope of this answer.
It is also possible that the external system is another program, in which case it would run on a thread. But it won't be a managed thread on your process.
Your take away is that these task do not run on any of your threads. And their timing might depend on external factors. Thus, it makes no sense to think of them as running in the same thread, or that we can predict their timing (well, except of course, in the case of the timer).
Both are not very good because they create the tasks immediately. The func version is a little less overhead since it saves the Task.Run route over the thread pool just to immediately end the thread pool work and suspend on the semaphore. You don't need an async Func, you could simplify this by using an async method (possibly a local function).
But you should not do this at all. Instead, use a helper method that implements a parallel async foreach.
public static Task ForEachAsync<T>(this IEnumerable<T> source, int dop, Func<T, Task> body)
{
return Task.WhenAll(
from partition in Partitioner.Create(source).GetPartitions(dop)
select Task.Run(async delegate {
using (partition)
while (partition.MoveNext())
await body(partition.Current);
}));
}
Then you just go urls.ForEachAsync(myDop, async input => await ProcessAsync(input));
Here, the tasks are created on demand. You can even make the input stream lazy.
I'm working on a SMS-based game (Value Added Service), in which a question must be sent to each subscriber on a daily basis. There are over 500,000 subscribers and therefore performance is a key factor. Since each subscriber can be a difference state of the competition with different variables, database must be queried separately for each subscriber before sending a text message. To achieve the best performance I'm using .Net Task Parallel Library (TPL) to spawn parallel threadpool threads and do as much async operations as possible in each thread to finally send texts asap.
Before describing the actual problem there are some more information necessary to give about the code.
At first there was no async operation in the code. I just scheduled some 500,000 tasks with the default task scheduler into the Threadpool and each task would work through the routines, blocking on all EF (Entity Framework) queries and sequentially finishing its job. It was good, but not fast enough. Then I changed all EF queries to Async, the outcome was superb in speed but there has been so many deadlocks and timeouts in SQL server that about a third of the subscribers never received a text! After trying different solutions, I decided not to do too many Async Database operations while I have over 500,000 tasks running on a 24 core server (with at least 24 concurrent threadpool threads)!
I rolled back all the changes (the Asycn ones) expect for one web service call in each task which remained Async.
Now the weird case:
In my code, I have a boolean variable named "isCrossSellActive". When the variable is set some more DB operations take place and an asycn webservice call will happen on which the thread awaits. When this variable is false, none of these operations will happen including the async webservice call. Awkwardly when the variable is set the code runs so much faster than when it's not! It seems like for some reason the awaited async code (the cooperative thread) is making the code faster.
Here is the code:
public async Task AutoSendMessages(...)
{
//Get list of subscriptions plus some initialization
LimitedConcurrencyLevelTaskScheduler lcts = new LimitedConcurrencyLevelTaskScheduler(numberOfThreads);
TaskFactory taskFactory = new TaskFactory(lcts);
List<Task> tasks = new List<Task>();
//....
foreach (var sub in subscriptions)
{
AutoSendData data = new AutoSendData
{
ServiceId = serviceId,
MSISDN = sub.subscriber,
IsCrossSellActive = bolCrossSellHeader
};
tasks.Add(await taskFactory.StartNew(async (x) =>
{
await SendQuestion(x);
}, data));
}
GC.Collect();
try
{
Task.WaitAll(tasks.ToArray());
}
catch (AggregateException ae)
{
ae.Handle((ex) =>
{
_logRepo.LogException(1, "", ex);
return true;
});
}
await _autoSendRepo.SetAutoSendingStatusEnd(statusId);
}
public async Task SendQuestion(object data)
{
//extract variables from input parameter
try
{
if (isCrossSellActive)
{
int pieceCount = subscriptionRepo.GetSubscriberCarPieces(curSubscription.service, curSubscription.subscriber).Count(c => c.isConfirmed);
foreach (var rule in csRules)
{
if (rule.Applies)
{
if (await HttpClientHelper.GetJsonAsync<bool>(url, rule.TargetServiceBaseAddress))
{
int noOfAddedPieces = SomeCalculations();
if (noOfAddedPieces > 0)
{
crossSellRepo.SetPromissedPieces(curSubscription.subscriber, curSubscription.service,
rule.TargetShortCode, noOfAddedPieces, 0, rule.ExpirationLimitDays);
}
}
}
}
}
// The rest of the code. (Some db CRUD)
await SmsClient.SendSoapMessage(subscriber, smsBody);
}
catch (Exception ex){//...}
}
Ok, thanks to #usr and the clue he gave me, the problem is finally solved!
His comment drew my attention to the awaited taskFactory.StartNew(...) line which sequentially adds new tasks to the "tasks" list which is then awaited on by Task.WaitAll(tasks);
At first I removed the await keyword before the taskFactory.StartNew() and it led the code towards a horrible state of malfunction! I then returned the await keyword to before taskFactory.StartNew() and debugged the code using breakpoints and amazingly saw that the threads are ran one after another and sequentially before the first thread reaches the first await inside the "SendQuestion" routine. When the "isCrossSellActive" flag was set despite the more jobs a thread should do the first await keyword is reached earlier thus enabling the next scheduled task to run. But when its not set the only await keyword is the last line of the routine so its most likely to run sequentially to the end.
usr's suggestion to remove the await keyword in the for loop seemed to be correct but the problem was the Task.WaitAll() line would wait on the wrong list of Task<Task<void>> instead of Task<void>. I finally used Task.Run instead of TaskFactory.StartNew and everything changed. Now the service is working well. The final code inside the for loop is:
tasks.Add(Task.Run(async () =>
{
await SendQuestion(data);
}));
and the problem was solved.
Thank you all.
P.S. Read this article on Task.Run and why TaskFactory.StartNew is dangerous: http://blog.stephencleary.com/2013/08/startnew-is-dangerous.html
It's extremly hard to tell unless you add some profiling that tell you which code is taking longer now.
Without seeing more numbers my best guess would be that the SMS service doesn't like when you send too many requests in a short time and chokes. When you add the extra DB calls the extra delay make the sms service work better.
A few other small details:
await Task.WhenAll is usually a bit better than Task.WaitAll. WaitAll means the thread will sit around waiting. Making a deadlock slightly more likely.
Instead of:
tasks.Add(await taskFactory.StartNew(async (x) =>
{
await SendQuestion(x);
}, data));
You should be able to do
tasks.Add(SendQuestion(data));