I have a simple Azure Worker role running that performs a task every few seconds. Below is the code that accomplishes this.
public override void Run()
{
try
{
while (true)
{
DoSomething();
System.Threading.Thread.Sleep(3000);
}
}
catch (Exception ex)
{
Log.Add(ex, true);
}
}
What I'd like to do now is add a second task DoSomethingElse() that fires once and only once per day. I've thought of a couple of ways to accomplish this:
Add a counter that calls the new task every nth loop
Add conditional logic to the new task that compares the current time to a prescribed time of day
Use some TBD scheduler library (such as Quartz.NET)
The first two solutions strike me as very brittle without additional code to deal with situations where the service is stopped and restarted. The third solution strikes me as potentially overkill.
My question is, what is the best practice for scheduling tasks at different intervals within an Azure Worker Role? I have a slight preference for sticking with straight .NET and not using a third-party library (though I'm not ruling it out).
Note, #3 above comes from this older question Recommend a C# Task Scheduling Library
The first two options are the simplest but they are brittle - especially in the cloud where roles can be recycled/load balanced etc... If the persistence is in memory or even disk based in the cloud, then it will be brittle.
Outside of other third party options, you could look at persisting the schedule and execution data into external storage (table services, sql azure, etc...). On a periodic timer, the worker role can query for the jobs that are due to be performed, record starting and then run the job. That also allows you to potentially scale out the number of worker roles since it's persistence is external.
This can get complicated in a hurry but if you keep it simple with frequency and recording run times, it can be fairly straight forward.
Steve Marx wrote a nice couple of blog entries on how to build a task scheduler on Windows Azure using blob leases, I think you will find this very useful.
Related
I'm Working with ASP.NET Core 2.2 Web API multi-tenant application. The application uses Hangfire to run background tasks. We are trying to improve the performance of the application.
It stores all the jobs in a separate DB (Hangfire DB), but this impact on API performance. I have traced the API request in order to check the request time, here is the result:
Here is the code
public async Task<string> AddUser(UserModel user)
{
CreateUserInBackgorund(user);
// removed code
return "some status";
}
[Queue(Constants.Critical)]
public void CreateUserInBackgorund(UserModel user)
{
BackgroundJob.Enqueue(() => CreateUser(user));
}
public async Task CreateUser(UserModel user)
{
try
{
//Other code
}
catch (Exception ex)
{
_logger.Error(ex.Message, ex);
}
}
The trace logs seem that background calls impacted on the performance of the request. Is there a way to reduce this time or use another approach?
Because of some lack of information, here are suggestions on what to check:
There are issues might be issues with SQL Server THREADPOOL Waits rather than with C# code.
https://www.sqlskills.com/help/waits/threadpool/
Also, you may get more information about long running queries by check SQL Server tools
https://www.sqlshack.com/how-to-identify-slow-running-queries-in-sql-server/
Please, be aware each transaction locks the table, and if there are a lot of transactions in queue they may significantly reduce performance as well.
In [Queue(Constants.Critical)] there is definitely logic that the main request thread needs to execute before to start execute method. It could be also reasonable to try something like
Task.Run(()=> CreateUserInBackgorund(user)); in AddUser(UserModel user)
and to check how it would influence.
It's weird practice to use background jobs in such way. Usually background jobs are utilized as schedulers for the business logic parts that are not executed by users and for those cases when some logic needs to be executed in specific time or period. E.g. calculate something each day, or each week, month and etc. If you are using background jobs just only for the purpose to pass response to the user faster, it would be better to use some Queue / Service Bus e.g. RabbitMQ for messaging events and than to add listeners that would be executing the required tasks on_message_received event on demand. So it would not be necessary to use hangfire for those things that it has not been really designed and for those massive amount of user requests. It would gain possibility to significantly reduce the amount of hangfire related requests to its database and the overall pressure on SQL Server as well.
I would like to build a job scheduler.
So this job scheduler allows the user to configure:
Job start time. (datetime value)
Job frequency: minutes, hours, days, months, years (any integer value)
I have 2 options on how to build this scheduler:
Use the C# Timer class. But this also means that I have to create a new Timer object for every scheduled job. I am planning to expose an API endpoint for the user to call and POST the starttime and frequency info. So when the endpoint is called, I will need to create a new Timer object.
Will this even scale? How do I manage the Timer objects? I need to allow user to create, update and delete their jobs.
Use the Azure Scheduler. However, I have a very large user database. I had a look at the pricing, the maximum total jobs that can run on 1 instance is 5 million jobs only. Plus, it is difficult for me to manage the running instances if I have more than 1 instance running. How do I decide to load balance multiple instances of schedulers?
You could use Azure Worker roles, one worker role per job type, and different schedules for each job, users can create their own separate schedule for each job, picking from a list of predefined "job types".
For this you can use RabbitMQ. Basically, every scheduler is just producer-consumer concept over standard messenger. I recommend you to use some messenger for this task, because they can guarantee that your task executed if it was scheduled, even if your system was shutted down. It is self-balanced, stable and pretty much everything you need already implemented here.
More here: https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html
For your task you simply specify producer side which if running will enqueue tasks into RabbitMQ with some schedule without care about how they execute in messenger. This way you can add, delete, edit, read schedules in your producer. Also, a little advice, instead of frequency terms use cron-expressions. It is widely support, easy to understand concept for specifying human-readable schedules.
More here: https://crontab.guru/
There are existing libraries that you can use, e.g. Quartz or Hangfire. The first one is rather simple to use library that I have used successfully, the latter has a UI in addition of running tasks, etc. that you can serve.
I am sure there are plenty of other libraries if those are not good enough.
Derek here from Azure Scheduler.We have enterprise customers using Scheduler similar to the scenarios you described, depending on what your user/job profile looks like, it should be easy to come up with a good way to partition them into different job collections.
How many jobs do you expect tot have? Sounds like you need much more than the 5 million we support in a single job collection with P20 plan. I'd like to better understand your scenario and help you decided whether Azure Scheduler is the right solution.
You can reach me at Derek.Li (at) microsoft dot com.
I am working on an application where I have multiple servers on different machines doing long operations for me. There is a windows service running on those machines written with hangfire/topshelf. Only one operation can run at a time per machine. Additionally I want to do some status check and cleaning jobs periodically on each server, so I can't just queue them as jobs.
Is there a way to do that in hangfire? Also, is there a way to send a follow-up job to the same server as an earlier job?
ADD-ON: I know one possibility would be to add another hangfire layer: Make each of the services a hangfire client with own DB and serve themselves, and then schedule recurring jobs for them, but that seems awfully complicated - especially when scaling out and adding servers.
If your task is to run some scheduled task on each server, I think, the best option is to implement it yourself, Hangfire don't support events handling, only command handling. I think, you reached the point of Hangfire possibilities and need to switch to more powerful and general tool.
For events and their handling you can use other systems, for example RabbitMQ. You just specify event generator and subscribe all your machines for this event.
I know this is a bit late, but the way we handle this sort of thing is just to write a simple console application and schedule it with Windows Task Scheduler.
You've probably resolved this by now, but
1 - one job per server - as you have it - worker count - probably the best as you can have multiple queues per server and the filters won't help you there.
2 - should the cleanup run after each processing job?
if yes, you can create the cleanup job from within your process job execution (ok maybe not perfect design but it works just fine) and assign to a queue on the same server, just add some logic in filters to ensure processing job is followed by a cleanup job and you're sorted.
alternatively you can use Continuation jobs (as on the site https://www.hangfire.io/) - Haven't used these but sounds like it might do the trick.
if you just want to periodically run the cleanup code then just schedule the job as recurring on each of the servers
I have a simple Azure Worker role running that performs a task every day at 12 PM. Below is the code that accomplishes this.
public override void Run()
{
try
{
while (true)
{
int time = Convert.ToInt32(DateTime.Now.TimeOfDay);
if (time == 12)
{
DoSomethingElse();
}
}
}
catch (Exception ex)
{
Log.Add(ex, true);
}
}
Here DoSomethingElse() is a method to send an email at every day 12 PM, and also fires once and only once per day.
How can I implement a scheduler that fire when the time is 12PM and execute DoSomethingElse().
My question is: Is this (above code) is the best method or use any 3rd party tool.
There are several other questions here that deal with this (and I've marked one above). Having said that, and at the risk of repeating what other answers already state:
In your case, a simple message on a Windows Azure Queue, time-delayed to not show up until noon, would work. This also helps deal with multi-instance scenarios: If you're running two instances of your role, you don't want the same scheduled task running twice, so you need a way to have only one of those instances execute this code. This is easily handled via queue message, or you could run scheduler code on a single instance by using something like a blob lease (which may only have one write-lock against it) as a mutex. This is covered in #smarx's blog post, here.
You could also use Quartz.Net http://quartznet.sourceforge.net/
using the blob lease mentioned ab ove is a great way to make sure only one of your instances is hosting tasks.
Using Quartz.Net to Schedule Jobs in Windows Azure Worker Roles
Cloud Scheduler specifically deals with task scheduling in the cloud.
I just came across this so I Haven't tried it.
http://getcloudscheduler.com/
Update: Forget cloud scheduler! I ran my daily schedule 600 times consecutively resulting on 600 email being sent to my clients. Don't use!!!
Use the Azure Task Scheduler. Nice tutorial by Scott Gu here.
Specifically, I would look at the Storage Queue action type - just register for queue events in your Worker Role.
(Note that this service may cost money if you want to schedule tasks more frequently than every hour.)
I often find when I'm writing a service class (within an ASP.NET app) that in addition to the usual method calls, I'd like scheduled method calls which invoke, say, every day at 1am.
My idea is to create a custom attribute which would be used against any scheduled task methods:
[ScheduledTask(Every=TimeSpans.Day, At=6)]
public static void Cleanup()
{
// some clean up code here
}
In order to implement this, I would have to scan the assemblies loaded in the AppDomain at the web app's Application_Start method. Look for static methods with the ScheduledTask attribute and then invoke the scheduler to run these methods according to the attribute.
I've looked around the net and found that no-one else seems to have done this and wondered whether this is generally a bad idea for some reason!? Would it be possible do you think?
thanks!
Some problems I see:
Separation of concerns. Your class provides the Cleanup implementation and the schedule to run it. An administrator is likely to want to control the schedule (e.g. run Cleanup at times of low activity), so it's more common to use configuration to provide a schedule.
Difficult for an administrator to get an overview of what tasks are scheduled.
Interpretation of the schedule. Is your example 6 UTC or local time?
Ease of modification of the schedule - you need to recompile.
ASP.NET applications can shut down, e.g. after a period of inactivity, or on a predefined schedule, and won't start up again until another request arrives. Which makes IIS a poor host for running scheduled tasks.