I have an Azure function running on a consumption service plan (Y1:0) on a Windows OS that is triggered by an Azure Service Bus queue item. However, the Azure function idles for extended periods of time and will not pull in queue items from the Service Bus. The following represent some of the idling issues:
Items being added onto the Service Bus and waiting 30-minutes to 1 hour or more to be dequeued and processed by the Azure function.
The Azure function processes a few items from the Service Bus and stops, leaving all other queued items to sit idle for similar time spans to the above point.
Items in the Service Bus dead lettering after 3 delivery attempts, but there are no logs or execution counts on Azure that indicate the function attempted to execute
The problem here isn't that a cold start takes a few minutes to get going, it's that the function seems to sit idle for extended periods of time and when or how long these idle periods happen seem to be random. During these idle periods, sometimes the function's memory working set will be above 0 and after long periods of time, it will drop to 0.
If items were left long enough in the Service Bus queue and did not dead letter, the Azure function would eventually process them. And occasionally, the function pulls from the Service Bus quickly. However, I need the function to run more consistently. From what I have read, the idle times I am experiencing are uncharacteristic for a consumption-based Azure function.
Here is more information on the function and Service Bus:
Service Bus
lockDuration: 'PT5M'
requiresDuplicateDetection: false
requiresSession: false
defaultMessageTimeToLive: 'P14D'
deadLetteringOnMessageExpiration: false
enableBatchedOperations: true
duplicateDetectionHistoryTimeWindow: 'PT10M'
maxDeliveryCount: 3
status: 'Active'
enablePartitioning: false
enableExpress: false
Function
sku: {
name: Y1
}
kind: 'windows'
netFrameworkVersion: 'v6.0'
The following is the host.json file. However, I am using Azure Infrastructure as Code so some shared properties are defined in a bicep file.
{
"version": "2.0",
"logging": {
"applicationInsights": {
"samplingSettings": {
"isEnabled": true,
"excludedTypes": "Request"
}
}
}
}
Based on other posts, I have tried switching the managed identity permissions of the function to be an owner instead of a receiver (since I am not using connection strings) but that has not solved any of the issues outlined previously.
Related
I'm getting a MessageLockLostException when performing a complete operation on Azure Service Bus after performing a long operation of 30 minutes to over an hour. I want this process to scale and be resilient to failures so I keep hold of the Message lock and renew it well within the default lock duration of 1 minute. However when I try to complete the message at the end, even though I can see all the lock renewals have occurred at the correct time I get a MessageLockLostException. I want to scale this up in the future however there is currently only one instance of the application and I can confirm that the message still exists on the Service Bus Subscription after it errors so the problem is definitely around the lock.
Here are the steps I take.
Obtain a message and configure a lock
messages = await Receiver.ReceiveAsync(1, TimeSpan.FromSeconds(10)).ConfigureAwait(false);
var message = messages[0];
var messageBody = GetTypedMessageContent(message);
Messages.TryAdd(messageBody, message);
LockTimers.TryAdd(
messageBody,
new Timer(
async _ =>
{
if (Messages.TryGetValue(messageBody, out var msg))
{
await Receiver.RenewLockAsync(msg.SystemProperties.LockToken).ConfigureAwait(false);
}
},
null,
TimeSpan.FromSeconds(Config.ReceiverInfo.LockRenewalTimeThreshold),
TimeSpan.FromSeconds(Config.ReceiverInfo.LockRenewalTimeThreshold)));
Perform the long running process
Complete the message
internal async Task Complete(T message)
{
if (Messages.TryGetValue(message, out var msg))
{
await Receiver.RenewLockAsync(msg.SystemProperties.LockToken);
await Receiver.CompleteAsync(msg.SystemProperties.LockToken).ConfigureAwait(false);
}
}
The code above is a stripped down version of what's there, I removed some try catch error handling and logging we have but I can confirm that when debugging the issue I can see the timer execute on time. It's just the "CompleteAsync" that fails.
Additional Info;
Service Bus Topic has Partitioning Enabled
I have tried renewing it at 80% of the threshold (48 seconds), 30% of the Threshold (18 seconds) and 10% of the Threshold (6 seconds)
I've searched around for an answer and the closest thing I found was this article but it's from 2016.
I couldn't get it to fail in a standalone Console Application so I don't know if it's something I'm doing in my Application but I can confirm that the lock renewal occurs for the duration of the processing and returns the correct DateTime for the updated lock, I'd expect if the lock was truely lost that the CompleteAsync would fail
I'm using the Microsoft.Azure.ServiceBus nuget package Version="4.1.3"
My Application is Dotnet Core 3.1 and uses a Service Bus Wrapper Package which is written in Dotnet Standard 2.1
The message completes if you don't hold onto it for a long time and occasionally completes even when you do.
Any help or advice on how I could complete my Service Bus message successfully after an hour would be great
The issue here wasn't with my code. It was with Partitioning on the Service Bus topic. If you search around there are some issues on the Microsoft GitHub around completion of messages. That's not important anyway because the fix I used here was to use the Subscription forwarding feature to move the message to a new Topic with partitioning disabled and then read the message from that new topic and I was able to use the exact same code to keep the message locked for a long time and still complete it successfully
Scenario: A Azure WebJob that will get all the Vendor record from NetSuite via WSDL.
Problem: The dataset is too large. Even with service set to 12 minutes time out. It still time out and the code failed.
NetSuite have a async process that basically run whatever you want on the server and it will return a JobId that allowed you to check the process on the server.
What I did currently is by making a search call first asking for all the Vendor records and it is to be process on the server. After I got the JobId, i wrote a void Recursion that check if the job is finish on the server with Thread Sleep set to 10 minutes.
private static bool ChkProcess(VendorsService vendorService, string jobId)
{
var isJobDone = false;
//Recursion
void ChkAsyncProgress(bool isFinish)
{
if (isFinish) return;
var chkJobProgress = vendorService.NsCheckProcessStatus(jobId);
if (chkJobProgress.OperationResult.IsFinish) isJobDone = true;
Thread.Sleep(TimeSpan.FromMinutes(10));
ChkAsyncProgress(isJobDone);
}
ChkAsyncProgress(false);
return isJobDone;
}
It work but is there a better approach?
Thanks
I think that since you're working with Azure already, with Service BUS you can implement a really low cost solution for this (if not free, depending on how much frequent is your job running)
Basically it's a queue where you enqueue messages (which can be objects with properties too, so they could also contain your result of the elaboration potentially).
A service bus is used to enqueue.
An azure function of type ServiceBusTrigger listens automatically if any new message on service bus has arrived and gets triggered if so (or, you can set messages to be enqueued, but be available after a certain future time only).
So, in the webjob code, at the end you could add code to enqueue a message which will mark the webjob has finished elaboration.
The azure function will get immediately noticed as soon as the message gets in the queue and you can retrieve the data without polling constantly for job completion, as azure will take care of all of that for you for a ridiculous price and without any effort by you.
Also, these task aren't priced timely based, but execution based, so you will pay only when it effectively put a message in queue.
They have a certain number of executions free, so it might be that you don't even need to pay anything.
Here some microsoft code sample for doing so.
Looking for some self triggering function which starts on a time given.
eg. A user has a conference to start at 25/04/2019 05:30
Here the user should get a notification at 25/04/2019 05:25 | 05:29 That the conference is about to start.
Have created a azure function (Time triggred) which triggers every minute and checks if current time is the Conference Time - 4 minutes, Then send a notification regarding conference to be started.
In future will have multiple users and so I do not want the function to run every minute, Is there a way in which at 05:25 or at the conference time the function will execute itself.
So there can be 100 users and they will have different. Just looking for options about how to implement in a better way.
.net core site,
hosted on azure,
Have azure functions running every minute to check the remainder
When user registered on conference you can queue message(with user details) to queue with visibility delay:
queue.AddMessage(message, initialVisibilityDelay: TimeSpanDelay);
For example, user registered at 6 PM, and conference will be next day at 8 PM, so delay time will be 25 hours and and 55 minutes(supposed, that user want to be notified 5 minutes before conference). Then instead of time triggered function you will use queue triggered function, which will send notifications, when messages from queue become visible:
public static void Run([QueueTrigger("notifications")]QueueTrigger message, TraceWriter log)
{
Notifier.Send(message.UserName, message.UserPhoneNumber, message.Email);
}
Moreover, if by some reason your notification handler will be failed, queue messages will not be lost, so you can retry to process them several times.
I recommend you to write a windows service using quartz library to schedule a job. You can pickup the job schedule time from db or somewhere.
Refer: https://www.quartz-scheduler.net
From your description, you already have a TimerTrigger Function however your Timer set the function running every minutes. Actually you could set more precise Timer trigger with CRON expressions.
This is CRON expressions in Azure Function, it includes six fields:{second} {minute} {hour} {day} {month} {day-of-week}. So you could set your Function 0 25,29 5 22 March *.
And after you deploy the function to Azure, it will keep running however it won't execute the Run method until it triggers the Timer.
public static void Run(TimerInfo myTimer, ILogger log)
{
log.LogInformation($"C# Timer trigger function executed at: {DateTime.Now}");
}
I observed this strange issue with azure function which is triggered by a service bus queue message.
Azure function is continuous running which is trigger when service bus queue gets new message.
but when there is no message into queue for long time like suppose
1-2 days and when new message come into queue after 2 days ,Its
strange that azure function does not get triggered , why any clue?
public static class TestController
{
[FunctionName("TestController")]
public static async Task Run([ServiceBusTrigger("%TestController.Topic%", "%TestController.Subscription%", AccessRights.Listen,
Connection = "ConnServiceBus")]BrokeredMessage currentMessage, TraceWriter log, ExecutionContext context)
{
log.Info("TestControllerprocessing start " + DateTime.Now);
Try force resyncing your triggers by hitting the Refresh button. This is a known issue for Azure Functions that are idle for a nondeterministic period of time. Also, verify if you're seeing the messages in the dead letter queue.
Prevent your Azure Function from going idle to begin by enabling "Always On": Open your Function App in the Azure portal. Click on Function App Settings at the top of the Function App blade. Scroll to the bottom of the page and click on Go to App Service Settings. In the Settings blade, scroll down and click on Application Settings. In the Application Settings blade, make sure that the Always On setting is set to On.
I want to trigger an Azure Webjob 24Hours after I have added a record to a database using .NET . Obviously there will be multiple tasks for the Webjob to handle, all at their designated time. Is there a way ( in the Azure Library for .NET) in which i can schedule this tasks ?
I am free to use Message Queues , but I want to try and avoid the unnecessary polling of the WebJob for new messages.
If you want to trigger the execution of a WebJob 24 hours after a record insertion in a SQL database I would definitely use Azure Queues for this. So after you insert the record, just add a message to the queue.
In order to do this you can easily leverage the initialVisibilityDelay property that can be passed to the CloudQueue.AddMessage() method. This will make the message invisible for 24 hours in your case, and then it will appear to be processed by your Webjob. You don't have to schedule anything, just have a Continuous WebJob listening to a queue running.
Here's some sample code:
public void AddMessage(T message, TimeSpan visibilityDelay)
{
var serializedMessage = JsonConvert.SerializeObject(message);
var queue = GetQueueReference(message);
queue.AddMessage(new CloudQueueMessage(serializedMessage), null, visibilityDelay);
}
private static CloudQueue GetQueueReference(T message)
{
var storageAccount = CloudStorageAccount.Parse("Insert connection string");
var queueClient = storageAccount.CreateCloudQueueClient();
var queueReference = queueClient.GetQueueReference("Insert Queue Name");
queueReference.CreateIfNotExists();
return queueReference;
}
Hope this helps
Since the event of adding a record to the database is the trigger here, You can use Azure Management Libraries to create a Azure Scheduler Job to execute after 24hrs from the time the db record is inserted. Azure Scheduler Jobs can do only 3 things : make HTTP/HTTPS requests or Put Message in Queue. Since you do not want to poll queues, here are two options
Deploy the existing Web Job as Wep API where each task is reachable by unique URLs, so that the scheduler task can execute the right HTTP/HTTPS request
Create a new WebAPI/Wep API which takes accepts request (like a man in the middle) and pro-grammatically run the existing web job on demand, again using Azure management libraries.
Please let me know if any of these strategies help.
To invoke a WebJob from your Website,is not good idea rather than you can add the WebJob code inside your Website and simply call that code. you can still easily use the WebJob SDK from inside your Website.
https://github.com/Azure/azure-webjobs-sdk-samples
we wouldn't recommend to invoke the WebJob from your Website is that the invocation contains a secret you rather not store on your Website (deployment credentials).
Recommendation:
To separate WebJob and Website code, the best thing to do is to communicate using a queue, the WebJob listens on the queue and the Website pushes the request to the queue.