We Keep Coding

Build a job scheduler in C#

I would like to build a job scheduler.
So this job scheduler allows the user to configure:
Job start time. (datetime value)
Job frequency: minutes, hours, days, months, years (any integer value)
I have 2 options on how to build this scheduler:
Use the C# Timer class. But this also means that I have to create a new Timer object for every scheduled job. I am planning to expose an API endpoint for the user to call and POST the starttime and frequency info. So when the endpoint is called, I will need to create a new Timer object.
Will this even scale? How do I manage the Timer objects? I need to allow user to create, update and delete their jobs.
Use the Azure Scheduler. However, I have a very large user database. I had a look at the pricing, the maximum total jobs that can run on 1 instance is 5 million jobs only. Plus, it is difficult for me to manage the running instances if I have more than 1 instance running. How do I decide to load balance multiple instances of schedulers?

You could use Azure Worker roles, one worker role per job type, and different schedules for each job, users can create their own separate schedule for each job, picking from a list of predefined "job types".

For this you can use RabbitMQ. Basically, every scheduler is just producer-consumer concept over standard messenger. I recommend you to use some messenger for this task, because they can guarantee that your task executed if it was scheduled, even if your system was shutted down. It is self-balanced, stable and pretty much everything you need already implemented here.
More here: https://www.rabbitmq.com/tutorials/tutorial-one-dotnet.html
For your task you simply specify producer side which if running will enqueue tasks into RabbitMQ with some schedule without care about how they execute in messenger. This way you can add, delete, edit, read schedules in your producer. Also, a little advice, instead of frequency terms use cron-expressions. It is widely support, easy to understand concept for specifying human-readable schedules.
More here: https://crontab.guru/

There are existing libraries that you can use, e.g. Quartz or Hangfire. The first one is rather simple to use library that I have used successfully, the latter has a UI in addition of running tasks, etc. that you can serve.
I am sure there are plenty of other libraries if those are not good enough.

Derek here from Azure Scheduler.We have enterprise customers using Scheduler similar to the scenarios you described, depending on what your user/job profile looks like, it should be easy to come up with a good way to partition them into different job collections.
How many jobs do you expect tot have? Sounds like you need much more than the 5 million we support in a single job collection with P20 plan. I'd like to better understand your scenario and help you decided whether Azure Scheduler is the right solution.
You can reach me at Derek.Li (at) microsoft dot com.

Custom Command Windows Services on HIGH Priority

I have an Work Tracker WPF application which deployed in Windows Server 2008 and this Tracker application is communicating with (Tracker)windows service VIA WCF Service.
User can create any work entry/edit/add/delete/Cancel any work entry from Worker Tracker GUI application. Internally it will send a request to the Windows service. Windows Service will get the work request and process it in multithreading. Each workrequest entry will actually create n number of work files (based on work priority) in a output folder location.
So each work request will take to complete the work addition process.
Now my question is If I cancel the currently creating work entry. I want to to stop the current windows service work in RUNTIME. The current thread which is creating output files for the work should get STOPPED. All the thread should killed. All the thread resources should get removed once the user requested for CANCEL.
My workaround:
I use Windows Service On Custom Command method to send custom values to the windows service on runtime. What I am achieving here is it is processing the current work or current thread (ie creating output files for the work item recieved).and then it is coming to custom command for cancelling the request.
Is there any way so that the Work item request should get stopped once we get the custom command.
Any work around is much appreciated.

Summary
You are essentially talking about running a task host for long running tasks, and being able to cancel those tasks. Your specific question seems to want to know the best way to implement this in .NET. Your architecture is good, although you are brave to roll your own rather than using existing frameworks, and you haven't mentioned scaling your architecture later.
My preference is for using the TPL Task object. It supports cancellation, and is easy to poll for progress, etc. You can only use this in .NET 4 onwards.
It is hard to provide code without basically designing a whole job hosting engine for you and knowing your .NET version. I have described the steps in detail below, with references to example code.
Your approach of using the Windows Service OnCustomCommand is fine, you could also use a messaging service (see below) if you have that option for client-service comms. This would be more appropriate for a scenario where you have many clients talking to a central job service, and the job service is not on the same machine as the client.
Running and cancelling tasks on threads
Before we look at your exact context, it would be good to review MSDN - Asynchronous Programming Patterns. There are three main .NET patterns to run and cancel jobs on threads, and I list them in order of preference for use:
TAP: Task-based Asynchronous Pattern
Based on Task, which has been available only since .NET 4
The prefered way to run and control any thread-based activity from .NET 4 onwards
Much simpler to implement that EAP
EAP: Event-based Asynchronous Pattern
Your only option if you don't have .NET 4 or later.
Hard to implement, but once you have understood it you can roll it out and it is very reliable to use
APM: Asynchronous Programming Model
No longer relevant unless you maintain legacy code or use old APIs.
Even with .NET 1.1 you can implement a version of EAP, so I will not cover this as you say you are implementing your own solution
The architecture
Imagine this like a REST based service.
The client submits a job, and gets returned an identifier for the job
A job engine then picks up the job when it is ready, and starts running it
If the client doesn't want the job any more, then they delete the job, using it's identifier
This way the client is completely isolated from the workings of the job engine, and the job engine can be improved over time.
The job engine
The approach is as follows:
For a submitted task, generate a universal identifier (UID) so that you can:
Identify a running task
Poll for results
Cancel the task if required
return that UID to the client
queue the job using that identifier
when you have resources
run the job by creating a Task
store the Task in a dictionary against the UID as a key
When the client wants results, they send the request with the UID and you return progress by checking against the Task that you retrieve from the dictionary. If the task is complete they can then send a request for the completed data, or in your case just go and read the completed files.
When they want to cancel they send the request with the UID, and you cancel the Task by finding it in the dictionary and telling it to cancel.
Cancelling inside a job
Inside your code you will need to regularly check your cancellation token to see if you should stop running code (see How do I abort/cancel TPL Tasks? if you are using the TAP pattern, or Albahari if you are using EAP). At that point you will exit your job processing, and your code, if designed well, should dispose of IDiposables where required, remove big strings from memory etc.
The basic premise of cancellation is that you check your cancellation token:
After a block of work that takes a long time (e.g. a call to an external API)
Inside a loop (for, foreach, do or while) that you control, you check on each iteration
Within a long block of sequential code, that might take "some time", you insert points to check on a regular basis
You need to define how quickly you need to react to a cancellation - for a windows service it should be within milliseconds, preferably, to make sure that windows doesn't have problems restarting or stopping the service.
Some people do this whole process with threads, and by terminating the thread - this is ugly and not recommended any more.
Reliability
You need to ask: what happens if your server restarts, the windows service crashes, or any other exception happens causing you to lose incomplete jobs? In this case you may want a queue architecture that is reliable in order to be able to restart jobs, or rebuild the queue of jobs you haven't started yet.
If you don't want to scale, this is simple - use a local database that the windows service stored job information in.
On submission of a job, record its details in the database
When you start a job, record that against the job record in the database
When the client collects the job, mark it for delayed garbage collection in the database, and then delete it after a set amount of time (1 hour, 1 day ...)
If your service restarts and there are "in progress jobs" then requeue them and then start your job engine again.
If you do want to scale, or your clients are on many computers, and you have a job engine "farm" of 1 or more servers, then look at using a message queue instead of directly communicating using OnCustomCommand.
Message Queues have multiple benefits. They will allow you to reliably submit jobs to a central queue that many workers can then pick up and process, and to decouple your clients and servers so you can scale out your job running services. They are used to ensure jobs are reliably submitted and processed in a highly decoupled fashion, and this can work locally or globally, but always reliably, you can even then combine it with running your windows service on cloud workers which you can dynamically scale.
Examples of technologies are MSMQ (if you want to maintain your own, or must stay inside your own firewall), or Windows Azure Service Bus (WASB) - which is cheap, and already done for you. In either case you will want to use Patterns and Best Practices for Enterprise Integration. In the case of WASB then there are many (MSDN), many (MSDN samples for BrokeredMessaging etc.), many (new Task-based API) developer resources, and NuGet packages for you to use

Need to schedule a method excecution in asp .net

I am creating a web application in which I need to allow the user to schedule the excecution of a task.
I have gone through the various threads for scheduling the task but all of them are using windows service that I am not aware of. Moreover I cannot install visual studio in the server systems due to budget constraints.
Is there a way to create a method that runs a scheduler in a background thread in the asp .net application.Any code sample will be of great help.

That's not the way to go. If you need a scheduled task you should create a console application and run it using the Windows task scheduler.
You could create an application that sends an email to the user with a link to the page where the task is supposed to be done.

One thing to understand is that ASP.NET is intended to service requests from the network.
Everything in it is geared towards that. So, yes, you can run background tasks, but there are a number of caveats and gotcha's.
For instance, IIS will shut down your ASP.NET application if it does not receive any requests for some period. Also, your Application may be restarted without warning, if it detects changes to the file system. So, running code will be aborted.
That's not to say you can't work around these, but it's really not the best fit for scheduled task execution.
Your best bet would be to store the details of the task in a database, and then using either a single always-running Windows Service (really not that difficult to do, there are plenty of examples), or a console application (as suggested) scheduled manually to run regularly, to execute these tasks.
You may find that a library such as Quartz.NET may be of help scheduling/running these tasks.

Questions about a Windows Service that is mostly idle

I have a Windows Service that is meant to run a 2-minute job once a day. The remaining 23:58 of the day, it just sits around looking pretty.
My question relates firstly to the idle time: is it better to use a Timer.Tick, or a Thread.Sleep? Is there any difference between the resources either of these use?
Secondly, during that idle time, if someone shuts down the Windows Service, do I need to interrupt those idle threads for the program to shut down cleanly, or will the Shutdown handle it for me?
Thirdly, am I barking up the wrong tree by using a Windows Service? It would seem to make sense to put a record into the Windows Task Scheduler, but I couldn't find any way to do that using InstallShield, which is what we're using to deploy our product. Any better ideas?

If you start a background thread with ThreadPool.QueueUserWorkItem I think you could have a Thread.Sleep in there and when you shut the service down you would not have to do anything with it. I think Timer Tick would automatically do the thread creation for you when it ticks, so you would have to do even less if you used that (out of the two the timer tick would, I think match what you want to achieve better so would be the better choice).
This definately feels like something that would be better done by a scheduler like you say though. I don't know if you can do it directly within InstallShield but perhaps you could create a little console app that you run from the installer that based on a command line argument either talks to the windows task schedular API - http://msdn.microsoft.com/enus/library/windows/desktop/aa383614(v=vs.85).aspx to register itself or does the job you want to acheive (i.e -install - set it up on the schedular, no args do whatever it is you need to do once a day).
I think it is a C++ API so you could do a bit of p/invoke or, better, just have some managed C++ in a class libaray and reference it from a c# based console application.

FWIW, InstallShield doesn't have native capabilities to define scheduled tasks but it does support VBScript, C++ and InstallScript custom actions. All of these options support calling COM Automation Interfaces and Microsoft exposes the task scheduler through a series of Task Scheduler objects.
Task Scheduler Reference
CPU activity aside, a Windows service running all the time also consumes memory. In fact, every version of Windows has less services.
Why Windows 8 Uses So Much Less Memory Than Windows 7

I would use Timer.Tick. Thread.Sleep blocks the thread and you can not abort the thread nicely when you shut down your service.

I would recommend the Windows Service as you are just duplicating its functionality otherwise.
If InstallShield doesn't support adding to the task scheduler, then can you get InstallShield to run a batch script or a small .net app after install to add it to the task scheduler?

As long as your requirements for scheduling are very simple (as they currently are), I think the service approach is fine.
Personally, I don't really like the windows scheduler because I find it unstable and annoying to maintain (admittedly, I haven't tried the new version in Windows 2008).
I would probably go with a scheduling library like Quartz.Net if I were to build an application with more advanced scheduling requirements.

If you dont have scheduling app like autosys, windows service should be better option, also I dnt see any harm in thread.sleep, it is meant to pause app and utilize
0% cpu

Windows Service Idea?

Let me explain the scenario and what I am trying to accomplish.
Scenario:
I have a web application that collects a date (ex 07/12/2011) and a time (ex 07:45PM) and store them into database (SQL).
What I am trying to do:
At 07:45PM on 07/12/2011, I want to call a web service to run another job.
I am thinking about building a windows service that runs every 15 minutes everyday, gathers all the "pending" requests (dates and times), queues them up, and executes the requests in that order.
Please feel free to provide any other approach for this.

In the past when I have done this I use the Windows Task Scheduler to run the exe that does what I want.
For what you are wanting to do a windows service seems like overkillm, I typically just create a basic console app that does what I need. With the Task Scheduler you can specify exactly when you want to run it and you are done.

Windows Services add a (sometimes) unnecessary level of complexity to a problem like this.
I would recommend starting with a simple console application and using Windows Scheduler to run it every x minutes.
If you decide to convert to a "real" service at a later time almost all of your code should be reusable.

You could evaluate the following solutions before writing out a windows service.
http://www.firedaemon.com/ - FireDeamon provides a free version for scheduling jobs.
http://quartznet.sourceforge.net/ - An open source scheduling library, good to go for if your windows service need to support more features.
If you are working on .NET Framework 4, this link should shed some light on this issue.

I've used Quartz.Net with some success. It's a bit more flexible than you've described. Scheduling a new task is as easy as:
public static void Schedule(DateTime when, string applicationId)
{
ISchedulerFactory factory = new StdSchedulerFactory();
IScheduler scheduler = factory.GetScheduler();
JobDetail jobDetail = new JobDetail("Realization Job", null, typeof(CustomTask));
jobDetail.JobDataMap["applicationId"] = applicationId;
Trigger trigger = new SimpleTrigger("Custom Task Trigger", DateTime.UtcNow, null, 0, TimeSpan.Zero);
scheduler.ScheduleJob(jobDetail, trigger);
}
Note that I have a wrapper around the JobScheduler that allows it to act as a Windows service. Creating this as a Windows service allows me to have more robust error handling and does not force me to rely on the OS like the Task Scheduler does.

I've used both Windows Task Scheduler and Windows Services in different web projects, both have their merits.
Personally, I prefer using scheduled tasks. Usually, I'll have a small generic tool calling a URL in the main web application. Sort of like a web service call. The output is appended to a log file.
The benefit of this setup is that if you deploy a new version of the web application, the service is updated as well.
I'd recommend a Windows Service only if you have to perform long-running tasks or tasks that require access to unsafe resources since these don't work well with web applications.
Then again, the same sort of tasks could also be performed from a command line tool.
In practice I've found that main problem with Windows Services is the fact they run indefinitely. In a perfect world that's not a problem. In the real world however, I've seen services leaking memory (yes, .NET based services). Over time these services will start to suck up more and more resources.
A scheduled task will start a new process for each invocation, limiting the amount of damage a leaky task can do.