Apologies for the bad title, I'm not sure how to succinctly describe this.
I'm building a service in .net core to process files. The general idea would be to (in a loop) check for new tasks to run on a file by using a DB call, and if found, fire them off async. There would be a throttler to limit the number of running tasks. If it hit the limit, it would just wait until a task is finished before firing off a new one. If the DB call returns nothing, then it would just sleep for a bit and try again later.
Fairly basic I think, but I'm running into issues because everything I've read says you should always await a task and not "fire and forget". But if I do that, then I'm left either running files synchronously or batching them. I'm missing something I'm sure.
Currently it's implemented as a IHostedService (BackgroundService specifically). Here us the main ExecuteAsync method that does the work. Implemented as fire and forget. (Visual studio gives the "call not awaited" warning on Task.Run)
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var throttleTimeout = 10000;
//set up semaphore
var taskThrottler = new SemaphoreSlim(10, 10);
while (!stoppingToken.IsCancellationRequested)
{
//wait for free slot
await taskThrottler.WaitAsync(throttleTimeout,stoppingToken);
//get next file to process
var currentFile = await _fileService.GetNextFileForProcessing();
// create task
Task.Run( async
() =>
{
try
{
//do work with file tracker
var result = await _fileProcessorController.ProcessFile(currentFile);
}
finally
{
taskThrottler.Release();
}
}
, stoppingToken);
}
}
Now if I change that and tack on an await before Task.Run, then I'm (essentially) running synchronously because each task will be awaited before the next is fired..
If I add the tasks to a list I can use "await Task.WhenAll(tasks);". But then I'm batching.
I need to be able to gracefully handle failed processing jobs, so an await in some form is a must I think.
All the examples I've found that tackle a similar problem usually have the list of tasks (or files) to do in advance, and then just iterate through. But I'm not sure what the best method is when I need to add tasks to the list while processing others.
Perhaps something that uses a task list and WhenAny? And then can I add new tasks to the task list on each loop if any are found?
I have an existing Function App with 2 Functions and a storage queue. F1 is triggered by a message in a service bus topic. For each msg received, F1 calculates some sub-tasks (T1,T2,...) which have to be executed with varying amount of delay. Ex - T1 to be fired after 3 mins, T2 after 5min etc. F1 posts messages to a storage queue with appropriate visibility timeouts (to simulate the delay) and F2 is triggered whenever a message is visible in the queue. All works fine.
I now want to migrate this app to use 'Durable Functions'. F1 now only starts the orchestrator. The orchestrator code is something as follows -
public static async Task Orchestrator([OrchestrationTrigger] DurableOrchestrationContext context, TraceWriter log)
{
var results = await context.CallActivityAsync<List<TaskInfo>>("CalculateTasks", "someinput");
List<Task> tasks = new List<Task>();
foreach (var value in results)
{
var pnTask = context.CallActivityAsync("PerformSubTask", value);
tasks.Add(pnTask);
}
//dont't await as we want to fire and forget. No fan-in!
//await Task.WhenAll(tasks);
}
[FunctionName("PerformSubTask")]
public async static Task Run([ActivityTrigger]TaskInfo info, TraceWriter log)
{
TimeSpan timeDifference = DateTime.UtcNow - info.Origin.ToUniversalTime();
TimeSpan delay = TimeSpan.FromSeconds(info.DelayInSeconds);
var actualDelay = timeDifference > delay ? TimeSpan.Zero : delay - timeDifference;
//will still keep the activity function running and incur costs??
await Task.Delay(actualDelay);
//perform subtask work after delay!
}
I would only like to fan-out (no fan-in to collect the results) and start the subtasks. The orchestrator starts all the tasks and avoids call 'await Task.WhenAll'. The activity function calls 'Task.Delay' to wait for the specified amount of time and then does its work.
My questions
Does it make sense to use Durable Functions for this workflow?
Is this the right approach to orchestrate 'Fan-out' workflow?
I do not like the fact that the activity function is running for specified amount of time (3 or 5 mins) doing nothing. It will incurs costs,or?
Also if a delay of more than 10 minutes is required there is no way for an activity function to succeed with this approach!
My earlier attempt to avoid this was to use 'CreateTimer' in the orchestrator and then add the activity as a continuation, but I see only timer entries in the 'History' table. The continuation does not fire! Am I violating the constraint for orchestrator code - 'Orchestrator code must never initiate any async operation' ?
foreach (var value in results)
{
//calculate time to start
var timeToStart = ;
var pnTask = context.CreateTimer(timeToStart , CancellationToken.None).ContinueWith(t => context.CallActivityAsync("PerformSubTask", value));
tasks.Add(pnTask);
}
UPDATE : using approach suggested by Chris
Activity that calculates subtasks and delays
[FunctionName("CalculateTasks")]
public static List<TaskInfo> CalculateTasks([ActivityTrigger]string input,TraceWriter log)
{
//in reality time is obtained by calling an endpoint
DateTime currentTime = DateTime.UtcNow;
return new List<TaskInfo> {
new TaskInfo{ DelayInSeconds = 10, Origin = currentTime },
new TaskInfo{ DelayInSeconds = 20, Origin = currentTime },
new TaskInfo{ DelayInSeconds = 30, Origin = currentTime },
};
}
public static async Task Orchestrator([OrchestrationTrigger] DurableOrchestrationContext context, TraceWriter log)
{
var results = await context.CallActivityAsync<List<TaskInfo>>("CalculateTasks", "someinput");
var currentTime = context.CurrentUtcDateTime;
List<Task> tasks = new List<Task>();
foreach (var value in results)
{
TimeSpan timeDifference = currentTime - value.Origin;
TimeSpan delay = TimeSpan.FromSeconds(value.DelayInSeconds);
var actualDelay = timeDifference > delay ? TimeSpan.Zero : delay - timeDifference;
var timeToStart = currentTime.Add(actualDelay);
Task delayedActivityCall = context
.CreateTimer(timeToStart, CancellationToken.None)
.ContinueWith(t => context.CallActivityAsync("PerformSubtask", value));
tasks.Add(delayedActivityCall);
}
await Task.WhenAll(tasks);
}
Simply scheduling tasks from within the orchestrator seems to work.In my case I am calculating the tasks and the delays in another activity (CalculateTasks) before the loop. I want the delays to be calculated using the 'current time' when the activity was run. I am using DateTime.UtcNow in the activity. This somehow does not play well when used in the orchestrator. The activities specified by 'ContinueWith' just don't run and the orchestrator is always in 'Running' state.
Can I not use the time recorded by an activity from within the orchestrator?
UPDATE 2
So the workaround suggested by Chris works!
Since I do not want to collect the results of the activities I avoid calling 'await Tasks.WhenAll(tasks)' after scheduling all activities. I do this in order to reduce the contention on the control queue i.e. be able to start another orchestration if reqd. Nevertheless the status of the 'orchestrator' is still 'Running' till all the activities finish running. I guess it moves to 'Complete' only after the last activity posts a 'done' message to the control queue.
Am I right? Is there any way to free the orchestrator earlier i.e right after scheduling all activities?
The ContinueWith approach worked fine for me. I was able to simulate a version of your scenario using the following orchestrator code:
[FunctionName("Orchestrator")]
public static async Task Orchestrator(
[OrchestrationTrigger] DurableOrchestrationContext context,
TraceWriter log)
{
var tasks = new List<Task>(10);
for (int i = 0; i < 10; i++)
{
int j = i;
DateTime timeToStart = context.CurrentUtcDateTime.AddSeconds(10 * j);
Task delayedActivityCall = context
.CreateTimer(timeToStart, CancellationToken.None)
.ContinueWith(t => context.CallActivityAsync("PerformSubtask", j));
tasks.Add(delayedActivityCall);
}
await Task.WhenAll(tasks);
}
And for what it's worth, here is the activity function code.
[FunctionName("PerformSubtask")]
public static void Activity([ActivityTrigger] int j, TraceWriter log)
{
log.Warning($"{DateTime.Now:o}: {j:00}");
}
From the log output, I saw that all activity invocations ran 10 seconds apart from each other.
Another approach would be to fan out to multiple sub-orchestrations (like #jeffhollan suggested) which are simple a short sequence of a durable timer delay and your activity call.
UPDATE
I tried using your updated sample and was able to reproduce your problem! If you run locally in Visual Studio and configure the exception settings to always break on exceptions, then you should see the following:
System.InvalidOperationException: 'Multithreaded execution was detected. This can happen if the orchestrator function code awaits on a task that was not created by a DurableOrchestrationContext method. More details can be found in this article https://learn.microsoft.com/en-us/azure/azure-functions/durable-functions-checkpointing-and-replay#orchestrator-code-constraints.'
This means the thread which called context.CallActivityAsync("PerformSubtask", j) was not the same as the thread which called the orchestrator function. I don't know why my initial example didn't hit this, or why your version did. It has something to do with how the TPL decides which thread to use to run your ContinueWith delegate - something I need to look more into.
The good news is that there is a simple workaround, which is to specify TaskContinuationOptions.ExecuteSynchronously, like this:
Task delayedActivityCall = context
.CreateTimer(timeToStart, CancellationToken.None)
.ContinueWith(
t => context.CallActivityAsync("PerformSubtask", j),
TaskContinuationOptions.ExecuteSynchronously);
Please try that and let me know if that fixes the issue you're observing.
Ideally you wouldn't need to do this workaround when using Task.ContinueWith. I've opened an issue in GitHub to track this: https://github.com/Azure/azure-functions-durable-extension/issues/317
Since I do not want to collect the results of the activities I avoid calling await Tasks.WhenAll(tasks) after scheduling all activities. I do this in order to reduce the contention on the control queue i.e. be able to start another orchestration if reqd. Nevertheless the status of the 'orchestrator' is still 'Running' till all the activities finish running. I guess it moves to 'Complete' only after the last activity posts a 'done' message to the control queue.
This is expected. Orchestrator functions never actually complete until all outstanding durable tasks have completed. There isn't any way to work around this. Note that you can still start other orchestrator instances, there just might be some contention if they happen to land on the same partition (there are 4 partitions by default).
await Task.Delay is definitely not the best option: you will pay for this time while your function won't do anything useful. The max delay is also bound to 10 minutes on Consumption plan.
In my opinion, raw Queue messages are the best option for fire-and-forget scenarios. Set the proper visibility timeouts, and your scenario will work reliably and efficiently.
The killer feature of Durable Functions are awaits, which do their magic of pausing and resuming while keeping the scope. Thus, it's a great way to implement fan-in, but you don't need that.
I think durable definitely makes sense for this workflow. I do think the best option would be to leverage the delay / timer feature as you said, but based on the synchronous nature of execution I don't think I would add everything to a task list which is really expecting a .WhenAll() or .WhenAny() which you aren't aiming for. I think I personally would just do a sequential foreach loop with timer delays for each task. So pseudocode of:
for(int x = 0; x < results.Length; x++) {
await context.CreateTimer(TimeSpan.FromMinutes(1), ...);
await context.CallActivityAsync("PerformTaskAsync", results[x]);
}
You need those awaits in there regardless, so just avoiding the await Task.WhenAll(...) is likely causing some issues in code sample above. Hope that helps
You should be able to use the IDurableOrchestrationContext.StartNewOrchestration() method that's been added in 2019 to suport this scenario. See https://github.com/Azure/azure-functions-durable-extension/issues/715 for context
I have extensively tried to search for this matter and could not reach any conclusion. I am using Xamarin with Visual Studio to design an app for Android in C# language. The app runs an async task which reads samples from the device microphone. So far there is no control of timing for this task. The code is as follows:
button.Click += async delegate
{
//do some other stuff
await read_mic_task();
}
The read_mic_task() is an async Task in which I read the samples from the microphone. I am measuring time between one and other execution using a Stopwatch and I can see the task runs with random periods. I would like to perform this task periodically and, despite searching a lot, I got nothing. Could you please give a help?
Thanks in advance!
There are multiple possibilities if you also want to share it across platforms i would go with system threading timer. This timer can also be used in a portable class.
You can use it for example like:
private System.Threading.Timer _timer;
...
// setup timer with callback
_timer = new System.Threading.Timer(OnTimerTick);
// start timer: duetime = 0 when to first execute callback (0 = right now) and intervall 5s
_timer.Change(0, 5000);
...
// stop timer
_timer.Change(Timeout.Infinite, Timeout.Infinite);
...
private void OnTimerTick(object state)
{
// do the intervall stuff here
}
I have a UWP application where I have added background task support for doing certain things while my application is in background. I am doing exactly as it is mentioned here: https://msdn.microsoft.com/en-us/windows/uwp/launch-resume/create-and-register-a-background-task
I have one separate project for Background Tasks and in my package manifest file I have declared that my app uses background tasks (but "timer" task since I am using TimerTrigger). Code:
BackgroundTaskBuilder backgroundTaskBuilder = new BackgroundTaskBuilder { Name = "NotificationUpdater", TaskEntryPoint = "NamespaceOfMyBackgroundTaskInterfaceImplementation.BackgroundTask"};
backgroundTaskBuilder.SetTrigger(new TimeTrigger(15, false));
BackgroundTaskRegistration backgroundTaskRegistration = backgroundTaskBuilder.Register();
Code inside BackgroundTask class:
namespace NamespaceOfMyBackgroundTaskInterfaceImplementation
{
public sealed class BackgroundTask : IBackgroundTask
{
public async void Run(IBackgroundTaskInstance taskInstance)
{
//Code to run in the background
taskInstance.Canceled += OnTaskInstanceCanceled;
}
private void OnTaskInstanceCanceled(IBackgroundTaskInstance sender, BackgroundTaskCancellationReason reason)
{
_logger.LogInfo("Background Agent: The Operating System requested cancellation. Reason: {0}.", reason);
}
}
}
When I close my application, after 20-25 minutes the background task gets triggered but when it starts executing the Run() method, OS cancels the execution with BackgroundTaskCancellationReason as executiontimeexceeded. Sometimes the cancellation happens right after the background task starts executing.
Note: When I use lifecycle events dropdown in VS to trigger the background task, the OS never cancels the execution and it keeps running without any issues.
Edit: I added some logging to know the timings and here is what I found:
BG Task started at 5/4/2016 5:58:25 PM
BG Task Cancelled at 5/4/2016 5:58:50 PM
So it's not even waiting for 30 seconds (which is supposed to be the time limit for the background task execution) and it terminates it in approximately 25 seconds (in some cases this time difference was as low as 20 seconds).
My service will run daily tasks at multiple times. Here is my service code:
private DateTime m_scheduleTime;
private List<Task> m_tasksList; // Task includes id and time ("23:00")
protected override void OnStart(string[] args) {
SetSchedule();
}
private void SetSchedule()
{
Timer[] timers = new Timer[m_tasksList.Count];
int iHours = 0; int iMinutes = 0;
for (int i = 0; i < timers.Length; i++)
{
iHours = Int16.Parse(m_tasksList[i].Time.Split(':')[0]);
iMinutes = Int16.Parse(m_tasksList[i].Time.Split(':')[1]);
m_scheduleTime = DateTime.Today.AddDays(1).AddHours(iHours).AddMinutes(iMinutes);
timers[i] = new Timer();
timers[i].Enabled = true;
timers[i].Interval = m_scheduleTime.Subtract(DateTime.Now).TotalSeconds * 1000;
timers[i].Elapsed += new System.Timers.ElapsedEventHandler(Timer_Elapsed);
}
}
protected void Timer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
DoWork(int taskId);
}
The code above is not tested, I built it from relevant code samples. So my problems are I don't know whether I am on the right track and if I'm right, I don't know which timer fires the _Elapsed event so that I can pass respective parameter. Any better solution would be appreciated!
You'd likely have better results using a single System.Threading.Timer that expires periodically at some minimum resolution, say every 60 seconds.
When your service starts, calculate the next scheduled run time for all of your tasks, inserting them into a priority queue as you go. The priority queue should be ordered by the scheduled run time.
Each time your timer expires, evaluate the scheduled run time of the task at the head of your queue. If the task's run time has been reached, pop it from the queue and push it to queue where it will be executed by your worker threads. Repeat this process until you either run out of tasks in the queue or find a task that is not yet ready to run.
Once a task has completed, calculate it's next scheduled run time and insert it into the scheduled priority queue.
If your service is purely for the purpose of running these scheduled tasks, I would suggest that you consider using the Windows Task Scheduler instead. Honestly, even if your service provides other functionality, using the Windows Task Scheduler to run periodic tasks will still be a better option than trying to roll your own scheduler.
You can find a managed wrapper for the Windows Task Scheduler, here http://taskscheduler.codeplex.com/. I have no knowledge of this wrapper, I found it referenced in this question: C# API for Task Scheduler 2.0.