I have added http triggered azure function and deployed it in function app. function app contains only one this http trigger on demand azure function. function app has app service plan, not consumption plan.
also, function app version is ~1. so that timeout is unlimited.
In the azure function code, I am reading one file having thousands of historical records and processing those records. this task is taking more than hour of time. this is one time task.
when I invoke this azure function after deployment, it gets invoked and after some time I noticed that it is getting invoked again and processing already processed records again.
Can anyone help me to understand invoking strategy of azure function, if azure function running for a long time without any status, will it callback itself?
if yes, how to stop this to call back again till it completes its processing.
Functions are supposed to be short-lived, they shouldn't run long time.The strength of Functions is in short-lived executions with small- or variable throughput.
Whenever possible, refactor large functions into smaller function sets that work together and return responses fast. For example, a webhook or HTTP trigger function might require an acknowledgment response within a certain time limit; it's common for webhooks to require an immediate response. You can pass the HTTP trigger payload into a queue to be processed by a queue trigger function. This approach lets you defer the actual work and return an immediate response.
Have a look of this:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-best-practices#avoid-long-running-functions
With Durable Functions you can easily support long-running processes, applying the Async HTTP APIs. When in case you are dealing with functions that require some time to process the payload or request, running under an 'App Service Plan, WebJob, or Durable Functions' is the right way.
As suggested by #Thiago Custodio, you also need to split the large files into smaller ones, and pass them to activities in your durable functions workflow.
Related
We have a pretty memory heavy OData/Web API written in C# and running on an Azure App Service instance. Some specific endpoints which get used rarely can take a long time to execute (20s) or they can take an extremely long time (10min) depending on the query. We would like in that case to have some of sort of background task mechanism for executing these kind of tasks, ideally a header we can attach to any call to tell the API to run it in the background.
We have tried:
Returning the request immediately with a tracking id and then running the SendAsync task in a HostingEnvironment.QueueBackgroundWorkItem, however when the work is completed it throws an exception as its trying to do something with the initial web request which is already disposed
Returning the request immediately with a tracking id and then passing the SendAsync to HostingEnvironment.QueueBackgroundWorkItem as a function, however that also throws an error upon activating
Returning the request immediately with a tracking id and then enqueuing a job in the Hangfire which will make a direct call to the app service and can wait as long as it takes. However even using the internal dns, the azure app service has a timeout of 230 seconds, and we can not change that in any way
We've also tried running the business code itself in a different thread, but we use autofac DI and if we try and get a new scope it still closes when the original request returns.
Similar SO questions have gotten answers that I don't think apply to us:
Running the work itself in Hangfire, that would mean spinning up the whole API again, just for these rare calls, also we can't do that because our Hangfire is on AKS on Linux, and the API is .net framework.
Using a webjob - similar issue as the api is very complex and deploying it to a webjob seems very dangerous, also we will then have to pay for 24/7 webjob.
Using a queue, the problem is that I don't really know how we can take a message from a queue and then execute it against the odata controllers, which we will need to do to populate various contexts and then get data from the DB that we need for these request.
Using IHostedService - Our API is .net framework and not .net core
Basically any thoughts on how we can achieve this, running an odata web request for as long as needed after we return a tracking id to the client (we understand the risks), without having to do another deployment of our API?
I have an Azure Function (~4) running in a Linux Consumption plan. It targets .Net 6. It uses a ServiceBusTrigger. The ServiceBus has two queues, qprocessing and qcomplete. The first queue, qprocessing, has several messages, which are scheduled for delivery to this function. The ServiceBusTrigger is not firing and the messages stay on the queue until I investigate why they didn't execute.
I use the explorer to peek at the messages. Then they fire. When the function executes the message is moved to the queue, qcomplete. The following examples show what I received in the complete queue.
"DeliveryDateTime":"2022-01-15T12:00:00","SendRequested":"2022-01-16T10:12:40.3301147Z"
"DeliveryDateTime":"2022-01-15T12:00:00","SendRequested":"2022-01-16T10:12:40.3285614Z"
DeliveryDateTime is EST. SendRequested is UTC as set by the function when it executes. These messages remained on the queue for 17 hours. And they didn't fire until I used the explorer to peek at them.
I've been noticing this issue of unreliable delivery when scheduling a message to be enqueued.
I have Application Insights enabled, and I see no errors or exceptions when I execute the following traces for the last three days.
traces
| where message contains '"state": "Error"'
traces
| where message contains "Exception while executing function"
The function executes, but I have to peek at the ServiceBus queue first.
Or I have to access the Azure function app's web site. Just showing the Azure function app's web site generates a result.
For now, I have a monitor running every 15 minutes, which accesses the function app's web site. It's the page that says, "Your Functions 4.0 App is up and running."
UPDATED
The problem is in the Scale Controller not becoming aware of your trigger or having problems with it.
Add the SCALE_CONTROLLER_LOGGING_ENABLED setting to your configuration as per this doc: Configure scale controller logs
This will add in the traces table logging about the Scale Controller and you might see something like this
"Function app does not contain any active triggers", which indicates that when your app will go idle, the Scale Controller will not wake it up, not being aware of any trigger.
After the function is deployed there must be a sync of triggers sometimes is automatic, sometimes is manual, sometimes it fails.
In my case altering the host.json file was the issue (like this) and also "leftovers" from previous deploys inside the storage account used by the function, both in the Blobs and in the File Shares that gave different kind of problems but still they invalidated my trigger
In other cases is a mixture of deployment method not triggering stuff, by design or by failure.
I have an existing Azure Function that is on a consumption plan
I am writing another function which will call this
Once the existing function is running, it processes files in storage account.
In order the files in my storage account to be processed, we have to manually go into the portal and "wake up" the function by navigating
Is there a way to do this via C# code?
This function is hosted on a consumption based plan
May be this is exact solution you are looking for. I came across this article "An Azure Function to keep other Functions/URLs warmed up" while looking for such solution, haven't tried it yet but I will. If you try it first do post the result.
https://www.sharepointnutsandbolts.com/2018/09/Azure-Function-Warmup-Cold-Start.html
The other approach that I came across is "pinging a health endpoint within your Azure Functions through Azure Monitor." Create a URL ping test. https://learn.microsoft.com/en-us/azure/azure-monitor/app/monitor-web-app-availability
I am in the process of trying these out. Hope this helps.
In serverless computing, Azure functions get executed/spin up whenever you make an invoke, if you want to call another function inside a function you can do via HTTP call.
Durable Functions lets you write stateful functions in a serverless environment. There's nothing built-in in Function Apps to call one HTTP function from other functions without actually making the HTTP call paticularly Function Chaining.
I am using an azure web job to pick up messages off the queue and process them as shown below:
namespace TicketProcessor
{
public class Functions
{
public static void ProcessQueueMessage([QueueTrigger("ticketprocessorqueue")] string message, TextWriter log)
{
//DO STUFF HERE...
}
}
}
And this web job exists within an app service in my azure account. My volume has been steadily increasing, and I now need to "scale this out". I need this same web job to run on multiple instances simultaneously. There are 2 reasons for this:
More processing power
My service targets a web api that throttles according to IP
I see the option within my app service to "Scale Out". But I'm not sure of the internal workings of the message queue as it relates to the scale out. My particular concern is this: If I scale this app service out to 2 instances, and my web job runs on both (as shown above), will each message in the queue get processed twice (once on each instance)? Or does the Azure queue mechanism handle it in such a way that each message is processed only once by one of the two instances?
One more thing to consider - in reading up on this, I found that there are two types of queues in Azure (storage queues and service bus queues). One thing I found interesting in the docs (in the "Foundational Capabilities" section) is that the Delivery guarantee for storage queues is "At-Least-Once". But the Delivery guarantee for service bus queues is both "At-Least-Once" and "At-Most-Once". This seems to indicate that if my process runs in a service bus queue, then it is guaranteed to run only once. But if it runs in a storage queue, it could possibly run more than once. In my case, I am running it with a storage queue. Any clarity on this will be helpful.
If I scale this app service out to 2 instances, and my web job runs on
both (as shown above), will each message in the queue get processed
twice (once on each instance)?
The answer to your question is no. Each message will be processed only once. From this link ("Multiple Instances" section):
If your web app runs on multiple instances, a continuous WebJob runs
on each machine, and each machine will wait for triggers and attempt
to run functions. The WebJobs SDK queue trigger automatically prevents
a function from processing a queue message multiple times; functions
do not have to be written to be idempotent. However, if you want to
ensure that only one instance of a function runs even when there are
multiple instances of the host web app, you can use the Singleton
attribute.
So here is the problem statement. I have a service which services mobile devices. Each user on the trigger of an event sends message to the service to register itself. After which the service performs a particular set of tasks at regular intervals(say 5 min) from the time of registration. So the execution time will be different for each user based on registration time.
I Implemented this using threads and Timers, it worked to an extent but as the users increased, the threads are killed and the tasks are not completed. Also this service is hosted on azure. I have created a WCF service with WebHtpp binding which accepts and returns JSON data.
Web jobs are a suggestion given to me. But since the execution times vary I am unable to use that as well. Is it even possible to perform such a task using C# and asp.net or am i going i the wrong direction entirely.
You need to identify where's the bottleneck that stops your threads before completion.
I would solve this using another approach: every new user, put a message in a queue, and create one Azure Function to dequeue the message and perform the logic of your service. This way your application will scale better, and you'll save money with the serverless approach.