We Keep Coding

Dynamically Creating Scheduled Tasks In An ASP.NET Website

I am in need of some advice on the best approach to dynamically creating some form of scheduled task at web application level. The scheduled task will be pulling information from an external API endpoint to store within a database at regular intervals until a specific end time.
Normally, I would create a Windows Service that will carry out this job for me. Unfortunately, the site administrator could create multiple tasks to query different criteria from the external API. So there could be multiple tasks running.
I have been working on implementing a background worker thread that runs on a timer. Would this be the best approach? Or would using something like Hangfire or Quartz.net be a better solution to ensure my background process is constantly running?
I don't know if it's feasible inside a web application to be ensure a task can constantly be run when required for a specific duration of time.

I tried to use Hangfire and Quartz.Net and in my opinion both of them are very good for scheduling task. Hangfire has a good interface for managing scheduled tasks, but as I encountered some problems after publishing the application on IIS, I switched to use Quartz.Net which works perfectly with Keep Alive Service For IIS 6.0/7.5. So, for using Quartz.Net you might have a look at Scheduled Tasks In ASP.NET With Quartz.Net. On the other hand, in order to make your published application to be alive after application pool recycling, IIS/Application restarting, etc. just install Keep Alive Service For IIS 6.0/7.5 on the server to which you publish your application. For detailed information regarding to these problems have a look at my answers on the pages below:
Quartz.net scheduler doesn't fire jobs/triggers once deployed
execute a schedule with quartz in visual start now with interval 24 hours
Hope this helps...

ASP.NET global background worker thread

I would like to create a background thread to pull data from another server periodically and send out real time email alerts to users.
Requirement:
The worker should never time out.
The worker should be running on a separate thread which means the server should still be able to handle user's requests while it is running
The worker should not be created per request but rather as a global thread which will be running when the server starts
I know it is a bad idea to create a background thread per request, but what is the best practice for a global backgroundworker thread?
Thanks in advance
Edited:
I am using Windows Azure to host the site, so I dont think I can create a windows service to run the task

Usually, you don't run such tasks in the web application itself as the application pool will be shut down after some time of inactivity depending on the configuration of the environment.
To make this work in a reliable way, create a separate application that periodically retrieves the data and sends the alerts. There are several ways to achieve this:
A very lightweight approach would be to create a console application and have a scheduler (e.g. Windows task scheduler) run it periodically.
A more sophisticated way is to create a Windows service that is started when the system starts and periodically executes the task.
If your application is integrated into a specific environment, there might already be a scheduling component available, e.g. in SharePoint you can implement jobs and let the Timer service run these.

I needed something similar to build scrapers. What I did is use the .Net ThreadPool class to send async http requests. I built a wrapper for building async requests with state object and then call:
ThreadPool.QueueUserWorkItem(new WaitCallback(asyncWrapper.BeginGetMethod), asyncStateObject);

Most people would recommend you to work with a Windows Service to accomplish it. However, a reasonable way to do this would be using a scheduling framework like Quartz .NET:
http://quartznet.sourceforge.net/
That way, if you think about it, your application deployment would even be easier - no win services or EXEs to deploy.
So If you decide to do it and run it embedded in your ASP.NET application, then you can start utilize it in the Global.Asaxfile, at Application_Start(), like this:
NameValueCollection properties = new NameValueCollection();
properties["quartz.scheduler.instanceName"] = "RemoteServer";
// set thread pool info
properties["quartz.threadPool.type"] = "Quartz.Simpl.SimpleThreadPool, Quartz";
properties["quartz.threadPool.threadCount"] = "5";
properties["quartz.threadPool.threadPriority"] = "Normal";
ISchedulerFactory sf = new StdSchedulerFactory(properties);
IScheduler sched = sf.GetScheduler();
sched.Start();

Typically web servers are designed for serving up web pages. Requests come in, HTML pages (among other things) go out. Unless you have full control over the server, what you are trying to do will be hard to achieve.

Polling Service in ASP.NET Webservice with Backgroundworker is not continueing

I need to run a polling service in a ASP.NET-WebService. Now, I have implemented it with a BackgroundWorker which I am starting in Application_Start of the HttpApplication. This is working. But suddenly, it will not continue working anymore.
Now, I have the fallowing Question:
What is the Lifetime for a HttpApplication, after it has started and executed Application_Start?
Have I to implement and run my BackgroundWorker at another place?

BackgroundWorker is not designed for your use case. As quoted from MSDN:
The BackgroundWorker class allows you to run an operation on a
separate, dedicated thread.
Normally asp.net app pool is configured (default IIS setting) to shut down after 20 minutes of idle time. Which is what I think is happening in your case.
You should develop a Windows Service. Following is brief about the same:
Windows services enable you to create long-running executable
applications that run in their own Windows sessions. These services
can be automatically started when the computer boots, can be paused
and restarted, and do not show any user interface. These features make
services ideal for use on a server or whenever you need long-running
functionality that does not interfere with other users who are working
on the same computer.
You can reuse all of your existing code. Just call you web service periodically from this windows service as per your requirement. That's it.

Is there a way to have code run on a time schedule in an ASP.NET web app?

I have an ASP.NET web app providing access to a database on the server. What I need is a way to run code in the background on a given schedule that auto-updates the server database from another source.
I know how to do this in a windows app by adding a timer, linking up a function to the timer tick event and starting the timer. I do not know how to do this in a web app.
Is there a start-up event for a web app or somewhere where I can start this background process regardless of whatever any users are doing on the site?

You should not do this in an ASP.NET website - this is a major no-no. You are correct in thinking to use a timer on a background .exe. You should look into creating either a Windows Task (a console .exe executed by the server task timer), or a Windows Service. I would suggest the Windows Service as that is standard practice.

If you have access to the computer hosting your site I would write a little app that was run from the Task Scheduler.
The web server is not meant to handle long-running background tasks. It's the wrong tool.
If you dont have access to the hosting computer then I would suggest building some kind of interface whereby another computer rebuilt the database and uploaded it. I'm using the terms "interface" and "upload" in the loosest, broadest sense - apply your own definition.

I was searching for a solution myself couple of months ago, and even though I haven't found enough time to try it so far, I guess I can share the link with you. Maybe you'll find it helpful.
If yes, please, let me know here.
http://quartznet.sourceforge.net/
How to use Quartz.net with ASP.NET

you can use Windows Service or use Timer Control (In the Ajax Category)
Or

As other answers have stated, doing this full function - updating a database and scheduling it as an ASP.NET app is using the wrong tool for the job.
ASP.NET can be used to update a database - that's perfectly valid. Where it breaks down is in the timer. ASP.NET apps aren't meant to be long-running, which is necessary for the timer to work.
If you can do it, I'd strongly suggest using the approach others have suggested - a Windows Service or a Scheduled Task.
However, if you have no access to the actual server, other than to post ASP.NET code - you can't install a service and you can't set up a Windows app to run on a scheduled basis, here's an out-of-the box idea.
Set up a web service or ASPX page that does the update, and then call that page from a scheduled task on a machine you DO control.
So if this was at http://www.someserver.net/updatedb.aspx, there's no reason you can't set a scheduled task on your own PC to call that URL.
I'd consider this a last-ditch solution to be used only if you can't do one of the other options.

The global.asax.cs file has a method that is fired when your application starts: Application_Start. You can hook up your timer method in that event. Just beware, depending on how IIS configured, your app pool may shutdown. For example, if no one hits the site in 20 minutes for example. Just make sure if you HAVE to have this run every X minutes that you have IIS configured to ALWAYS be running and start your app. This is harder than it sounds. In the end, you may want to go with a regular windows scheduled task.

Spawning and executing a Worker process in Azure

The google has really failed me on this one. I am new to Azure and am only intermediate at .NET
I have an Azure solution going and I've written some code in a Web Role which runs great. What I would like to do now is move some of this code into an Azure Worker, which will be initialized by a controller function in the Web Role
What on earth do I need to do to get this going locally? I have created the Worker project within the SLN. I just need to know how to fire it up and run it.
I think part of my problem is I am assuming these workers behave like Heroku workers... is this the case? Because what I need is something like a queue system (a bunch of "worker tasks" in one big queue).
A lot of the links I've found for tutorials seem to tap dance around how to actually initialize the process from a Web Role.

Workers in Windows Azure are not tasks; they're entire VMs. To make your life easier, memorize this little detail: Web Role instances are Windows Server 2008 with IIS running, and Worker Roles are the same thing but with IIS disabled.
When you added that worker role to your project, you actually now have a new set of virtual machines running (at least one, depending on the instance count you set). These VMs have their own OnStart() and Run() methods you can put code into, for bootstrapping purposes.
If you grab the Windows Azure training kit, you'll see a few labs that show how to communicate between your various role instances (a common pattern being the use of Windows Azure queues). There's a good example of background processes with the Guestbook hands-on lab (the very first lab).

More info on this, as I've gotten it going now..
If you're coming from a Heroku background, then an Azure Worker is more or less the function in Rails that you'd actually execute with the queue. Unlike Heroku queued operations, an Azure Worker just runs endlessly and keeps polling for new stuff to do... hence the templated sleep(10000) in the Run() function.
The most conventional way I've found to make a Web and Worker talk to each other is by queue messages via Azure ServiceBus which is currently NOT emulated, meaning you need a functioning Azure account to make this work, and it will work even if you are running locally. You just need internet access.
A ServiceBus message can pass an entire object over to the Worker (so long as the Worker proj has the right dependencies in it), so that's kind of nice.

I think you're just having trouble starting the azure emulator along with your worker/web roles? Just set the azure configuration project as the start up project and run that. It'll boot up the emulator along with all your roles.