I'm writing a .NET MVC application that needs two background processes to be running and I'm not sure if these processes should be threads within the web app, or if they should be separate windows services. I'd prefer them to be within the web app so that it's easier to deploy, configure and manage.
The first process is a timed event that needs to check a datasource every 15 minutes to determine if certain products have been shipped. If any have been shipped, this process would take the info about the product and create a "job" record in a database table (a queue) so that it could be processed later.
The second process is another timed event that needs to check the database table (the queue) to see if there is any work to do. If there are unprocessed records, it would read them into a thread safe list, and then use several .NET Tasks to process them in parallel.
The reason I'm building this as a web app is that I'm going to give my customers the ability to view historical info on this process, the ability to manually submit jobs, and the ability to configure which products the application should "look" for.
Is there a good way to build all of this into the web app or should I be looking at splitting it up into multiple applications? The demand on the web views will be pretty low.
Phil Haack recently blogged about the challenges you will have to face if you ever decide to implement recurring background tasks directly in your web application in contrast to externalizing them in a separate service.
I would advise a dedicated service for the background actions. The reason for this is that IIS can tear down the whole app at any point in time.
Related
I am trying to develop an application that will scan a website, get data from the website and save that data into database 3x per day at given hour that can be set in xml configuration file.
As an addition group of users can trigger the start of the application manually max few times per day.
I am looking for pros and cons of using a windows service for this solution or should I set a 3 scheduled tasks that will run the console application?
If I will decide to use a windows service then what is the best way to trigger a manual start of the service while the service is already running? The group of users will have some kind of web interface to trigger manual start.
This could be easily done using a scheduled task. I would just set a 3x scheduled tasks that will run the application at given time and the group of users could just start the .exe file from the web interface. However how to only allow the user to run a manual trigger only if application is not already running?
Since the UI is ultimately in web, and thus the service itself won't need any UI, I would go with Windows service which can be triggered to start by the user through web or automatically as the time comes (by its internal code).
Then, either:
In the service it has something to indicate its status as running which can be captured by the web app to see it, or,
In the web, there is mechanism to request/monitor the service status
Is quite flexible I think. I would go with whichever is easier.
I picked the task scheduler option for my case. It was easier to implement the manual triggering by users mechanism.
Pros - Triggers can be added easly using the taks scheduler library.
I have an c# asp.net management system with a button that calls a SQL Server Query to get 90,000 strings of text in multiple languages and categorized into sections. This in turn is sorted and 150 Binary files made before saving as a .ZIP and emailing the user with the results. The total time to process this and email the results is about 6 minutes. In this time the Web Page is sat waiting for the whole process to complete. I would like to be able to press the start process button and then allow this to work away in the background while I continue using the web management system, but I am unsure what is the most efficient method for doing this. I initially created an asmx file thinking this would work but the result is the same and so I am now looking at async and await. Can anyone give me any pointers on this and let me know if I am on the right track. I am currently not getting anything back to let me know the process has completed successfully as I can handle this by emailing the user to say something went wrong. The reason for this is the user could be on any number of pages.
There are probably a few ways to go about tackling this problem. Your options will vary based on what version of .NET you are using, so I'll not post code directly; however, the you can implement the concept I describe using ASMX web services, WCF, MVC, and so on.
Start-and-poll Approach
The classic response for this kind of problem is to implement a StartSomething() method and a GetProgress() method. A very-simple example of this approach using ASMX-based web services is presented here.
In the example, one service method is used to start a process on a background thread. Myself, I would change the example by having the start method return a value to the client to identify which background process was started, as you could feasibly have several going on at a time.
The client then can call a separate method to get progress updates, and continue polling until the process is complete.
There are a number of reasons why you should prefer to do lengthy background processing in a non-IIS service. I recommend using a Windows service to protect yourself from IIS somewhat-randomly restarting your application pool in the middle of a big job.
WebSockets
Another option worth some exploration on your part is to use WebSockets, which allow the server to contact a modern browser when the process is complete. The main advantage of this approach is that the client does not need to busily poll the service for updates. Its primary disadvantage is that WebSockets are new enough that there are still plenty of browsers that could not be clients for such a service.
Good luck!
In my web application, there is a web page displaying certain fields and its value. The values can be edited by the users. Once the values are edited, they are persisted into the database. Now comes the problem where i need suggestions. Once the values are updated into DB, i need to run few modules which are time consuming. So this time consuming part may or may not be done instantaneously depending on the configuration. The options in the configuration can be Instantaneous, Hourly, Daily, Weekly or Monthly. So if it instantaneous i have to run the modules or else i should trigger the task at the specified time based on the configuration.
Is job scheduler, a good way of solving this problem? Heard about Quartz.NET. Are there any .NET libraries available to achieve this. Pls provide suggesions.
It much recommended to store these jobs in a queue and create another asynchronize service to execute the queue jobs. So you will not load all the effort on the web-server (IIS) and the application will not lost the jobs in case of the web-server restart or the web-application restarts.
To create this solution, you will have 3 points to think about.
Queue design and management
Sure you can use one of the Queuing servers (e.g. MSMQ, ActiveMQ, ZeroQ). But I recommend to use a new table on your database as a queue for simplicity. Find in the following link how to use database table as a queue.
http://rusanu.com/2010/03/26/using-tables-as-queues/
Job Scheduling
You have two options here Windows Task Scheduler and QuatrZ.NET. I recommend to use QuatrZ.Net for this part as it is more customizable and can fit for all your needs.
Hosting Service
You can use QuartZ.NET server deployment or you can build your own hosting service using .NET 4 Managed Extensibility Framework (MEF) and based on QuartZ IJob contract and its Corn expressions.
I am developing a asp.net site that needs hit a few social media sites daily for blanket friend/follower data. I have chosen arvixe business class as my hosting. In the future if we grow, I'd love to get onto a dedicated server and run a windows service, however since that is not in the cards at this point I need another reliable way of running scheduled tasks. I am familiar with running a thread timer from the app_code(global.aspx). However the app pool recycling will cause some problems with the timer. I have never used task scheduling like quartz but have read a lot about it on stackoverflow. I was looking for some advise as to how to approach my goal. One big problem I have using either method is that I will need the crawler threads to sleep for up to an hour regularly due to api call limits. My first thoughts were to use the db to save the starting and ending of a job. When the app pool recycles I would clear out any parts not completed and only start parts that do not have a record of running on that day. What do the experts here think? any good links to sample architecture of this type of scheduling?
It doesn't really matter what method you use, whether you roll your own or use Quartz. You are at the mercy of ASP.NET/IIS because that's where you want to host it.
Do you have a spare computer laying around that can just run a scheduled task and upload data to a hosted database? To be honest, it's possibly safer (depending on your use case) to just do it that way then try to run a scheduler in ASP.NET.
Somewhat along the lines of Bryan's post;
Find a spare computer.
Instead of allowing DB access have it call up a web service on your site. This service call should be the initiator of the process you are trying to do. Don't try to put params into it, just something like "StartProcess()" should work fine.
As far as going to sleep and resuming later take a look at Workflow Foundation. There are some nice built in features to persist state.
Don't expose your DB to the outside world, instead expose that page or web service and wraps some security around that. WCF has some nice built in security features for that.
The best part is when you decide to move off, you can keep your web service and have it called from a Windows Service in the same manner.
As long as you use a persistent job store (like a database) and you write and schedule your jobs so that they can handle things like being killed half way through, having IIS recycle your process is not that big a deal.
The bigger issue is that IIS shuts your site down if it doesn't have traffic. If you can keep your site up, then just make sure you set the misfire policy appropriately and that your jobs store any state data needed to pick up where they left off, you should be able to pull it off.
If you are language-agnostic and don't mind writing your "job-activation-script" in your favourite, Linux-supported language...
One solution that has worked very well for me is:
Getting relatively cheap, stable Linux hosting(from reputable
companies),
Creating a WCF service on your .Net hosted platform that will contain the logic you want to run regularly (RESTfully or SOAP or XMLRPC... whichever suits you),
Handling the calls through your Linux hosted cron jobs, written in your language of choice(I use PHP).
Working very well, like I said. No VPS expense,configurable and externally activated. I have one central place where my jobs are activated, with 99 to 100% uptime(never had any failures).
There doesn't seem to be many Windows Workflow Foundation gurus out there :(
Here are couple of challenges that I face:
How many workflow runtimes should there be running for in an Asp.Net MVC application? One per application, per session or per request?
How frequently should the workflow runtime be started and stopped? Once per application instance, once per session or once per request?
What are the pros and cons of doing one or another in the above options?
Any comments or suggestions are welcome,
Thanks,
Cullen
You would normally only run one workflow runtime per application. It is possible to define more than one and there may be some complex scenarios where that is desirable but its highly unlikely. I can't see any scenario where multiple runtimes for the same configuration would be run in the same process.
For a web hosted workflow you really need the SqlWorkflowPersistenceService. IIS expects to be able to recycle an application pool with minimul impact on the application. Hence you need idled workflows to be persisted so that they survive such recycles.
On a similar note you should use the ManualWorkflowSchedulerService which plays nice with ASP.NET use of threads, its also really handy in being able to perform end-to-end processing of a request to a response through workflow on a single thread. Just be sure to include the useActiveTimers="true" attribute so that delay activities work.
In line with the above you need to be sure that any active workflow does not take longer to complete or go idle than the application pool's shutdown time limit. Otherwise on recycle IIS may force the process to terminate before a workflow has persisted.
As to starting and stopping the workflow, its again difficult to see a scenario where you wouldn't just want it to start on application start and remain running. I guess if you have a workflow which never idles but just runs from beginning to end and you only run such workflows very occasionally then it might be simpler to start the runtime and the end it afterward. However even that can get messy, I wouldn't bother just start it on app start and be done with it.
How many workflow runtimes should there be running for in an Asp.Net MVC application?
one per application, unless you need more for scalability purposes (too many requests)
How frequently should the workflow runtime be started and stopped?
typically, once per application instance
The pro's and con's are trivial, you can scale better with more session requests and instances, but it takes more overhead to manage them all.
Your best bet is to use just enough of what you need and grow later if necessary.