I've set up an Azure function and I want it to run asynchronously because I expect to have hundreds/thousands/more messages in my queue that will all get dequeued at the same time, so I've implemented it as such below (maybe there's a better way). Or do I need to worry about running code in the functions asynchronously?
Will Azure handle thousands of these functions run at the same time, if thousands of messages in the queue are all dequeued at once? This Azure function article says only a few hundred can run at once
Where is the best place to put the try catch statement? Inside the asynchronous call around my logic or outside the asynchronous call like my code? Or does it matter?
public static class CancelEvent
{
[FunctionName("CancelEvent")]
public static async void RunAsync([ServiceBusTrigger("canceleventqueue", AccessRights.Manage, Connection = "service_bus_key")]string myQueueItem, TraceWriter log, ExecutionContext context)
{
try
{
await Task.Run(() => Processor.ProcessAsync());
}
catch(Exception ex)
{
}
}
}
public class Processor
{
public static void ProcessAsync()
{
// do the work
}
}
TLDR; don't build your Azure Functions to be async.
In my recent experience with azure functions, I found it best NOT to build my functions as asynchronous.
In my situation, I was using a ServiceBusTrigger to receive a message, and then was awaiting a processing method, and was baffled that when bubbling an exception out it would not show up in the azure portal as failed, nor would it capture the exception detail in the Application Insights account, nor would it properly abandon or deadletter the message. I would only discover the exception detail by going to the files\eventlog.xml section in the storage account where I would find a complaint about an unhandled exception bringing the function down to its knees.
After a day spent going down the road of doing a full try/catch and handling message.Abandon() and message.Deadletter() logic myself as well as logging all the AppInsights telemetry manually, I discovered that simply NOT doing async and bubbling out he exception (after first performing error reporting or handling logic) resulted in all the behavior I expected out of the platform.
The 'run history' of the function in the portal correctly showed passed and failed executions, and all the exception detail was captured.
This is a cautionary tale to keep your Azure functions static, synchronous, and simple. The message-handling nature of their design should already be async enough.
Adding await Task.Run around the call to a synchronous method doesn't make much sense. Strive to make ProcessAsync really async (returning Task and non-blocking).
So the better option (note that both methods return Task):
[FunctionName("CancelEvent")]
public static async Task RunAsync([ServiceBusTrigger("canceleventqueue", AccessRights.Manage, Connection = "service_bus_key")]string myQueueItem, TraceWriter log, ExecutionContext context)
{
try
{
await Processor.ProcessAsync();
}
catch (Exception ex)
{
// Try catch will work fine
}
}
public class Processor
{
public static async Task ProcessAsync()
{
// do the work
}
}
and the worse option, but also viable:
[FunctionName("CancelEvent")]
public static void RunAsync([ServiceBusTrigger("canceleventqueue", AccessRights.Manage, Connection = "service_bus_key")]string myQueueItem, TraceWriter log, ExecutionContext context)
{
try
{
Processor.Process();
}
catch (Exception ex)
{
// Try catch will work fine
}
}
public class Processor
{
public static void Process()
{
// do the work
}
}
Azure will run multiple executions in parallel in both cases. It's just that the first option will be more lean in resource usage.
Not all messages will be processed at once, but Azure will scale the amount of parallel executions based on some internal scaling logic.
Related
What happens if in an Azure Function an async Task is started but orphaned before it finishes (it is never awaited), e.g.:
[Function("FuncAsync")]
public static async Task<HttpResponseData> FuncAsync(
[HttpTrigger(AuthorizationLevel.Function, "post", Route = "FuncAsync")]
HttpRequestData req,
FunctionContext context)
{
var obj = FactoryClass.GetObject(); // return some object with async LongTaskAsync method
obj.LongTaskAsync(); // async Task LongTaskAsync()
return req.CreateResponse(HttpStatusCode.OK);
}
The intention here was (I guess) to initiate a long running process and instantly return from the function.
I assume it is a bad practice but seems to work in a legacy code. I suspect there's no guarantee for the life of that async task and the azure process can be randomly winded up, when no function entry point is running/triggered.
For a console application if a running async Task is orphaned (and keeps running while the process terminates) it is abandoned/killed (and no exception is thrown).
class Program
{
public static async Task RunFuncAsync()
{
try
{
Console.WriteLine("Task started.");
await Task.Delay(10 * 1000);
Console.WriteLine("Task finished."); // this is never executed
}
catch (Exception e)
{
Console.WriteLine("Exception: " + e.Message); // this is never executed
}
}
static async Task MainAsync(string[] args)
{
var t = RunFuncAsync(); // no awaiting - purposefuly
await Task.Delay(5 * 1000);
Console.WriteLine("Exiting.");
}
static void Main(string[] args)
{
MainAsync(args).GetAwaiter().GetResult();
}
}
Output:
Task started.
Exiting.
C:\TestTasks_Console.exe (process 6528) exited with code 0.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .
What happens if in an Azure Function an async Task is started but
orphaned before it finishes ?
The answer for the above question which you have is correct that it will get terminated. Azure Functions have many features that make our work a lot easier. However, stateless Azure Functions are not suitable for long running operations which would require to store a state of progress.
For such cases the Durable Functions are the best option. With this the restrictions on execution times no longer apply.
The task-based programming model and async/await are well suited for mapping workflows of Durable Functions.
If you check the Azure Function best practices, you will also find that we should avoid long running function and our function should be stateless.
Another option is to go for the WebJobs. You can use Azure WebJobs to execute custom jobs as background tasks within an Azure Web App
In conclusion, we can say that stateless Azure Functions not recommended for long running operations as you may get timeout issues or like in your case may get terminated.
My good ladies and gentlemen, I recently had my first go at Worker services in .Net Core 3.1, and only the second go at Windows services in general (first one was made in .Net Framework and works fine to this day). If anyone could maybe shed some light at what I'm missing in the example that I will provide that would be great.
So, to keep it simple, my problem is this:
My supposed long (forever) running Worker service unexpectedly stops doing work at an arbitrary time of day, but still is shown as "Running" in service manager (that's probably how Windows deals with services). It doesn't necessarily have to be every day, but it stops doing work every now and then until I manually stop it and then restart it in Service Manager.
I have also stumbled upon this question which seemed to deal with my problem, but even after completely wrapping all of my service's code blocks in try-catchs, even on top-level, I still get nothing registered in my Log table, or even in the file I set up to write in if my DB connection fails. Service seems to just stop calling ExecuteAsync() method.
Ok here's how my code's logically structured, I have excluded implementation and I'm just showing what happens until DoWork is called:
public class Worker : BackgroundService
{
private readonly IConfiguration _configuration;
public Worker(IConfiguration configuration)
{
_configuration = configuration;
}
public override Task StartAsync(CancellationToken cancellationToken)
{
return base.StartAsync(cancellationToken);
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
while (true)
{
try //paranoid try-catch
{
await DoWork();
await Task.Delay(TimeSpan.FromSeconds(45), stoppingToken);
}
catch (Exception e)
{
await Log(e, customMessage: "Proccess failed at top level.");
}
}
}
catch (Exception e)
{
await Log(e, customMessage: "Proccess failed at topmost level.");
}
}
private async Task DoWork()
{
try
{
}
catch (Exception e)
{
await Log(e);
}
}
public async Task Log(Exception e, string user = null, string emailID = null, string customMessage = null)
{
}
}
As you can see, I am not handling cancellation, as in the question I linked above. Now that I think about it maybe I should, and something is inadvertently sending cancellation? The reason I didn't is because I'm not sure what events exactly signal the cancellation. Only the manual stopping of service, or something else maybe? And if it is the cancellation that was sent that caused my service to stop doing work, shouldn't it also stop my service from running?
Btw I just tested cancellation on dummy service which implements my logic with while(true) and it catches the stopping exception, even though it's a bit awkward, as it catches it and logs it multiple times before stopping, so I presume it may not be the cancellation token that is causing my DoWork not to fire.
Ok guys I'd fixed it. See comment below.
I'd guessed that what was causing deadlock was probably too many concurrent calls from different threads to database over the same connection.
Not that that I knew that would be the cause (and I still don't know and can only guess why this happens so if someone can clarify why this happens and why don't the calls get queued please do), but as I tried to fix it that seemed like a good starting point.
What I did was just limit possible concurrent calls to 1:
Instantiate SemaphoreSlim on a class level:
private static SemaphoreSlim Semaphore = new SemaphoreSlim(1);
Insert a SemaphoreSlim.WaitAsync before each of my DB calls and its respective SemaphoreSlim.Release in a finally block after the call:
try
{
await Semaphore.WaitAsync();
var id = await sqlCommand.ExecuteScalarAsync().ToString();
}
finally
{
Semaphore.Release();
}
I thought this would decrease the performance but to my pleasant surprise I felt no noticeable difference.
Also, I was tempted to set Semaphore's initial count to more than 1 thread but I figured if deadlock happens for many threads, then it might happen for 2-10 threads. Does anyone perhaps know anything more about this number? Is it processor related, SQL related, or perhaps C# related?
Have you implemented a dispose method to close the database connection after finishing the DoWork method? I had a deadlock problem using worker service and realized the database connection wasn’t disposed. After implementing a dispose method, it works for me to solve the problem.
Old question, and I don't know how or even if Manus's issue eventually resolved, but in my experience, exceptions in threads that aren't the main thread caused this problem for us in a Windows Service. And we didn't see this when running the same code as a Windows Forms. Try/catch wrapped around the main processing line does not catch them. We had to add it in the threaded method.
I am using a BackgroundService object in an aspnet core application.
Regarding the way the operations that run in the ExecuteAsync method are implemented, the Aspnet core fails to initialize or stop correctly. Here is what I tried:
I implemented the abstract ExecuteAsync method the way it is explained in the documentation.
the pipeline variable is an IEnumerable<IPipeline> that is injected in the constructor.
public interface IPipeline {
Task Start();
Task Cycle();
Task Stop();
}
...
protected override async Task ExecuteAsync(CancellationToken stoppingToken) {
log.LogInformation($"Starting subsystems");
foreach(var engine in pipeLine) {
try {
await engine.Start();
}
catch(Exception ex) {
log.LogError(ex, $"{nameof(engine)} failed to start");
}
}
log.LogInformation($"Runnning main loop");
while(!stoppingToken.IsCancellationRequested) {
foreach(var engine in pipeLine) {
try {
await engine.Cycle();
}
catch(Exception ex) {
log.LogError(ex, $"{engine.GetType().Name} error in Cycle");
}
}
}
log.LogInformation($"Stopping subsystems");
foreach(var engine in pipeLine) {
try {
await engine.Stop();
}
catch(Exception ex) {
log.LogError(ex, $"{nameof(engine)} failed to stop");
}
}
}
Because of the current development state of the project, there are many "nop" Pipeline that contains an empty Cycle() operation that is implemented this way:
public async Task Cycle() {
await Task.CompletedTask;
}
What I noticed is:
If at least one IPipeline object contains an actual asynchronous method (await Task.Delay(1)), then everything runs smoothly and I can stop the service gracefully using CTRL+C.
If all IPipeline objects contains await Task.CompletedTask;,
Then on one hand, aspnetcore fails to initialize correctly. I mean, there is no "Now listening on: http://[::]:10001 Application started. Press Ctrl+C to shut down." on the console.
On the other, when I hit CTRL+C, the console shows "Application is shutting down..." but the cycle loop continues to run as if the CancellationToken was never requested to stop.
So basically, if I change a single Pipeline object to this:
public async Task Cycle() {
await Task.Delay(1);
}
Then everything is fine, and I dont understand why. Can someone explain me what I did not understood regarding Task processing ?
The simplest workaround is to add await Task.Yield(); as line one of ExecuteAsync.
I am not an expert... but the "problem" is that all the code inside this ExecuteAsync actually running synchronously.
If all the "cycles" return a Task that has completed synchronously (as Task.CompletedTask will be) then the while and therefore the ExecuteAsync method never "yield"s.
The framework essentially does a foreach over the registered IHostedServices and calls StartAsync. If your service does not yield then the foreach gets blocked. So any other services (including the AspNetCore host) will not be started. As bootstrapping cannot finish, things like ctrl-C handling etc also never get setup.
Putting await Task.Delay(1) in one of the cycles "releases" or "yields" the Task. This allows the host to "capture" the task and continue. Which allows the wiring up of Cancellation etc to happen.
Putting Task.Yield() at the top of ExecuteAsync is just the more direct way of achieving this and means the cycles do not need to be aware of this "issue" at all.
note: this is usually only really an issue in testing... 'cos why would you have a no-op cycle?
note2: If you are likely to have "compute" cycles (ie they don't do any IO, database, queues etc) then switching to ValueTask will help with perf/allocations.
I've looked numerous posts on task exception handling and I'm not clear how to resolve this issue. My application crashes anytime there is an exception in one of my tasks. I am unable to catch the exception and my application is left in an inoperable state.
UPDATE: This only seems to happen when calling the ExecuteAsync method in the DataStax Cassandra C# driver. This leads me to believe it's an issue in the driver itself. When I create my own task and throw an exception it works fine.
Most use cases seem to await all asynchronous calls, but in my case I need to fire off a group of asynchronous commands and then use WhenAll to await them together (rather than awaiting each one individually).
This is based off of this post which shows how to batch up tasks to send to a Cassandra database:
https://lostechies.com/ryansvihla/2014/08/28/cassandra-batch-loading-without-the-batch-keyword/
This is the same practice recommended by Microsoft when you want to perform multiple async requests without having to chain them:
https://social.msdn.microsoft.com/Forums/en-US/6ab8c611-6b0c-4390-933c-351e56b62526/await-multiple?forum=async
My application entry point:
public void Go()
{
dbTest().Wait();
My async method...
private async Task dbTest() {
List<Task> tasks = new List<Task>();
Task<RowSet> resultSetFuture = session.ExecuteAsync(bind); // spawn a db exception
Task<RowSet> resultSetFuture2 = session.ExecuteAsync(bind);
Task<RowSet> resultSetFuture3 = session.ExecuteAsync(bind);
tasks.Add(resultSetFuture);
tasks.Add(resultSetFuture2);
tasks.Add(resultSetFuture3);
try
{
await Task.WhenAll(tasks.ToArray());
}
catch (Exception ex)
{
...
}
All indications are that WhenAll should properly catch any exceptions from async methods, but it just locks up my application in this case.
I have a situation (right or wrong) where I might need to abandon a running instance of a Task and start a new one. Cancellation is not an option as this is pretty much legacy code which would take a huge effort in building in support for passing cancellation tokens. I mean something like this
public Task TaskUnderObservation{get; private set;}
public async Task WatchTaskInternal(Func<Task<TResult>> task)
{
try
{
if (TaskUnderObservation != null) //basically if we are re-using this instance of class
ResetCommonTaskState();
TaskUnderObservation = task(); //Task is defined as a class level variable of type Task
LogManager.Instance.Info(this, "Started Task");
await Task;
}
catch (Exception ex)
{
LogManager.Instance.Error(this, "WatchTaskInternal Errored", ex);
}
}
The question is what happens to the running TaskUnderObservation is the method WatchTaskInternal is called repeatedly something like this.
var x = new TaskWatcherClass();
x.WatchTaskInternal (//some Func);
x.WatchTaskInternal (//some other FUNC);
Would my running task be abandoned or ignored? If I had a WPF application which was binding to the result of my TaskUnderObservation object would it bind it to the result of the first invocation or the most recent one?
You can't abandon running code. The running code must be complicit, and so support some kind of cancellation.
You can abandon the task by simply dropping the reference to it, but the code that is running will run to completion.
Don't be tempted to use Thread.Abort. It is pure evil.