I am developing a asp.net site that needs hit a few social media sites daily for blanket friend/follower data. I have chosen arvixe business class as my hosting. In the future if we grow, I'd love to get onto a dedicated server and run a windows service, however since that is not in the cards at this point I need another reliable way of running scheduled tasks. I am familiar with running a thread timer from the app_code(global.aspx). However the app pool recycling will cause some problems with the timer. I have never used task scheduling like quartz but have read a lot about it on stackoverflow. I was looking for some advise as to how to approach my goal. One big problem I have using either method is that I will need the crawler threads to sleep for up to an hour regularly due to api call limits. My first thoughts were to use the db to save the starting and ending of a job. When the app pool recycles I would clear out any parts not completed and only start parts that do not have a record of running on that day. What do the experts here think? any good links to sample architecture of this type of scheduling?
It doesn't really matter what method you use, whether you roll your own or use Quartz. You are at the mercy of ASP.NET/IIS because that's where you want to host it.
Do you have a spare computer laying around that can just run a scheduled task and upload data to a hosted database? To be honest, it's possibly safer (depending on your use case) to just do it that way then try to run a scheduler in ASP.NET.
Somewhat along the lines of Bryan's post;
Find a spare computer.
Instead of allowing DB access have it call up a web service on your site. This service call should be the initiator of the process you are trying to do. Don't try to put params into it, just something like "StartProcess()" should work fine.
As far as going to sleep and resuming later take a look at Workflow Foundation. There are some nice built in features to persist state.
Don't expose your DB to the outside world, instead expose that page or web service and wraps some security around that. WCF has some nice built in security features for that.
The best part is when you decide to move off, you can keep your web service and have it called from a Windows Service in the same manner.
As long as you use a persistent job store (like a database) and you write and schedule your jobs so that they can handle things like being killed half way through, having IIS recycle your process is not that big a deal.
The bigger issue is that IIS shuts your site down if it doesn't have traffic. If you can keep your site up, then just make sure you set the misfire policy appropriately and that your jobs store any state data needed to pick up where they left off, you should be able to pull it off.
If you are language-agnostic and don't mind writing your "job-activation-script" in your favourite, Linux-supported language...
One solution that has worked very well for me is:
Getting relatively cheap, stable Linux hosting(from reputable
companies),
Creating a WCF service on your .Net hosted platform that will contain the logic you want to run regularly (RESTfully or SOAP or XMLRPC... whichever suits you),
Handling the calls through your Linux hosted cron jobs, written in your language of choice(I use PHP).
Working very well, like I said. No VPS expense,configurable and externally activated. I have one central place where my jobs are activated, with 99 to 100% uptime(never had any failures).
Related
I have an c# asp.net management system with a button that calls a SQL Server Query to get 90,000 strings of text in multiple languages and categorized into sections. This in turn is sorted and 150 Binary files made before saving as a .ZIP and emailing the user with the results. The total time to process this and email the results is about 6 minutes. In this time the Web Page is sat waiting for the whole process to complete. I would like to be able to press the start process button and then allow this to work away in the background while I continue using the web management system, but I am unsure what is the most efficient method for doing this. I initially created an asmx file thinking this would work but the result is the same and so I am now looking at async and await. Can anyone give me any pointers on this and let me know if I am on the right track. I am currently not getting anything back to let me know the process has completed successfully as I can handle this by emailing the user to say something went wrong. The reason for this is the user could be on any number of pages.
There are probably a few ways to go about tackling this problem. Your options will vary based on what version of .NET you are using, so I'll not post code directly; however, the you can implement the concept I describe using ASMX web services, WCF, MVC, and so on.
Start-and-poll Approach
The classic response for this kind of problem is to implement a StartSomething() method and a GetProgress() method. A very-simple example of this approach using ASMX-based web services is presented here.
In the example, one service method is used to start a process on a background thread. Myself, I would change the example by having the start method return a value to the client to identify which background process was started, as you could feasibly have several going on at a time.
The client then can call a separate method to get progress updates, and continue polling until the process is complete.
There are a number of reasons why you should prefer to do lengthy background processing in a non-IIS service. I recommend using a Windows service to protect yourself from IIS somewhat-randomly restarting your application pool in the middle of a big job.
WebSockets
Another option worth some exploration on your part is to use WebSockets, which allow the server to contact a modern browser when the process is complete. The main advantage of this approach is that the client does not need to busily poll the service for updates. Its primary disadvantage is that WebSockets are new enough that there are still plenty of browsers that could not be clients for such a service.
Good luck!
Let me give a back ground for everybody before I go to my problem. My company hosts website for many clients, my company also contracts some of the work to another company.
So when we first set up a website with all the informations to our clients, we pass that information to the other company we contracted and three of us have the same data. Problem is once the site is up and running, our clients will change some data and when ever they do that we should be able to update our contracted company.
The way we transfer data to the contracted company is by using a web service (httppost, xml data). Now my question is what it the best way to write a program which sends updated data to the contracted company everytime our clients change some data.
1) Write a windows service having a timer inside my code where every 30min or so connects to the database and find all changes and send it to the contracted company
2) Write the same code as #1 (with out the timer in it) but this time make it a simple program and let windows scheduler wake it every 30min
3) Any other suggestion you may have
Techenologies available for me are VS 2008, SQLServer 2005
Scheduled task is the way to go. Jon wrote up a good summary of why services are not well suited for this sort of thing: http://weblogs.asp.net/jgalloway/archive/2005/10/24/428303.aspx
A service is easy to create and install and is more "professional" feeling so why not go that way? Using a non-service EXE would also work of course and would be slightly easier to get running (permissions, etc.) but I think the difference in setup between the two is nearly negligible.
One possible solution would be to add a timestamp column to your data tables.
Once this is done, you can have one entry in each table that has the last collected time by your contracted company. They can pull all records since that last time and update their records accordingly.
A Windows Service is more self contained, and you can easily configure it to start up automatically when the OS is starting up. You might also need to create additional configuration options, as well as some way to trigger the synchronization immediately.
It will also give you more room to grow your functionality for the service in the future.
A standalone app should be easier to develop though, however you are reliant on the windows scheduler to execute the task always. My experience has been that it is easier to mess up things with the windows scheduler and have it not run, for example in cases where you reboot the OS but no user has logged in.
If you want a more professional approach go with the service, even though it might mean a little bit more work.
A windows service makes more sense in this case. Think about what happens after your server is restarted:
With a Windows Application you need to have someone restart the application, or manually copy a shortcut to the startup folder to make sure the application gets launched
OR,
With a Windows Service you set it to start automatically and forget about it. When the machine reboots your service starts up and continues processing.
One more consideration, what happens when there is an error? A Windows application would likely show an error dialog and wait for input before continuing; whereas a service would log the error in the event log and carry on.
I have been working on many applications which run as windows service or scheduled tasks.
Now, i want to make sure that these applications will be fault tolerant and reliable. For example; i have a service that runs every hour. if the service crashes while its operating or running, i d like the application to run again for the same period (there are several things involved with this including transactions of data processing) , to avoid data loss. moreover, i d like the program to report the error with details. My goal is to avoid data loss and not falling behind for running the program.
I have built a class library that a user can import into a project. Library is supposed to keep information of running instance of the program, ie. program reads and writes information of running interval, running status etc. This data is stored in a database.
I was curious, if there are some best practices to make the scheduled tasks/ windows services fault tolerant and reliable.
Edit : I am talking about independent tasks or services which on different servers. and my goal is to make sure that the service will keep running, report any failures and recover from them.
I'm interested in what other people have to say, but I'll give you a few points that I've stumbled across:
Make an event handler for Unhandled Exceptions. This way you can clean up resources, write to a log file, email an administrator, or anything you need to instead of having it crash.
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(AppUnhandledExceptionEventHandler);
Override any servicebase event handlers you need in the main part of your application. OnStart and OnStop are pretty crucial, but there are many others you can use. http://msdn.microsoft.com/en-us/library/system.serviceprocess.servicebase%28v=VS.71%29.aspx
Beware of timers. Windows forms timers won't work right in a service. User System.Threading.Timers or System.Timers.Timer. Best Timer for using in a Windows service
If you are updating on a thread, make sure you use a lock() or monitor in key sections to make sure everything is threadsafe.
Be careful not to use anything user specific, as a service runs without a specific user context. I noticed some of my SQL connection strings were no longer working for windows authorizations, etc. Also have heard people having trouble with mapped drives.
Never make a service with a UI. In fact for Vista and 7 they make it nearly impossible to do anyway. It shouldn't require user interaction, the most you can do is send a message with a WIN32 function. MSDN claims making interactive services is bad practice. http://msdn.microsoft.com/en-us/library/ms683502%28VS.85%29.aspx
For debugging purposes, it is way cool to make a service run as a console application until you get it doing what you want it to. Awesome tutorial: http://mycomponent.blogspot.com/2009/04/create-debug-install-windows-service-in.html
Anyway, hope that helps a little, but that is just a couple thing I poked around to find on my own.
Something obvious - don't run all your tasks at the same time. Try to schedule them so only one task is using some expensive resource at any time (if possible). For example, if you need to send out newsletters and some specific notifications, schedule them at different times. If two tasks need to clean up something in the database, let the one run after another.
Also schedule tasks to run outside of normal business hours - at night obviously.
There doesn't seem to be many Windows Workflow Foundation gurus out there :(
Here are couple of challenges that I face:
How many workflow runtimes should there be running for in an Asp.Net MVC application? One per application, per session or per request?
How frequently should the workflow runtime be started and stopped? Once per application instance, once per session or once per request?
What are the pros and cons of doing one or another in the above options?
Any comments or suggestions are welcome,
Thanks,
Cullen
You would normally only run one workflow runtime per application. It is possible to define more than one and there may be some complex scenarios where that is desirable but its highly unlikely. I can't see any scenario where multiple runtimes for the same configuration would be run in the same process.
For a web hosted workflow you really need the SqlWorkflowPersistenceService. IIS expects to be able to recycle an application pool with minimul impact on the application. Hence you need idled workflows to be persisted so that they survive such recycles.
On a similar note you should use the ManualWorkflowSchedulerService which plays nice with ASP.NET use of threads, its also really handy in being able to perform end-to-end processing of a request to a response through workflow on a single thread. Just be sure to include the useActiveTimers="true" attribute so that delay activities work.
In line with the above you need to be sure that any active workflow does not take longer to complete or go idle than the application pool's shutdown time limit. Otherwise on recycle IIS may force the process to terminate before a workflow has persisted.
As to starting and stopping the workflow, its again difficult to see a scenario where you wouldn't just want it to start on application start and remain running. I guess if you have a workflow which never idles but just runs from beginning to end and you only run such workflows very occasionally then it might be simpler to start the runtime and the end it afterward. However even that can get messy, I wouldn't bother just start it on app start and be done with it.
How many workflow runtimes should there be running for in an Asp.Net MVC application?
one per application, unless you need more for scalability purposes (too many requests)
How frequently should the workflow runtime be started and stopped?
typically, once per application instance
The pro's and con's are trivial, you can scale better with more session requests and instances, but it takes more overhead to manage them all.
Your best bet is to use just enough of what you need and grow later if necessary.
I have a require ment to read data from a table(SQL 2005) and send that data to other application for every 5 seconds. I am looking for the best approach to do the same.
Right now I am planning to write a console application(.NET and C#) which will read the data from sql server 2005(QUEUE table which will be filled through different applications) and send to other application through TCP/IP(Central server). Run that console application under schedule task for every 5 seconds. I am assuming scheduled task will take care to discard new run event if task is already running(avoid to run concurrent executions).
Does any body come accross similar situation? Please share your experience and advice me for best approach.
Thanks in advance for your valuable time spending for my request.
-Por-hills-
We have done simliar work. If you are going to query a sql database every 5 seconds, be sure to use a stored procedure that is optimized to be very fast. It should not update data unless aboslutely necessary. This approach is typically called 'polling' and I've found that it is acceptable if your sqlserver is not otherwise bogged down with too many other calls.
In approaches we've used, a Windows Service that does the polling works well.
To communicate results to another app, it all depends on what your other app is doing and what type of interface you can make into it, and how quickly you need the results. The WCF class libraries from Microsoft provide many workable approaches for real time communication. My preference is to write to the applications database, and then have the application read the data (if it works for that app). If you need something real time, WCF is the way to go, and I'd suggest using a stateless protocol like http if < 5 sec response time is required, (using standard HTTP posts), or TCP/IP if subsecond response time is required.
since I assume your central storage is also SQL 2005, have you considered using what SQL Server 2005 offers out of the box to achieve your requirements? Rather than pool every 5 seconds, marshal and unmarshal TCP/IP, implement authentication and authorization for the TCP/IP pipe, scale TCP transmission with boxcaring, manage message acknowledgments and retries, deal with central site availability, fragment large messages, implement fairness in transmission and so on and so forth, why not simply use Service Broker? It does all you need and more, out of the box, already tested, already tuned for performance and scalability.
Getting reliable messaging right is not trivial and you should focus your efforts in meeting your business specifics, not reiventing the wheel.
I would recommend writing a Windows Service (since you are C#) that has some timer which runs every 5 seconds. That way you wont be starting and stopping an application all the time, it can run even when there is no one logged into the machine, and it will automatically start when the machine is restarted.
For one of my projects, I needed to do something periodically. I opted for a service and set up a timer that takes care of reading the data. You might consider that solution. It has worked well for me.
I suggest to create a windows service and not an application and to perform the timing yourself - create a timer and execute one step on each timer event. For the communication you have many choices - I would consider using standard technologies like a webservice or Winows Communication Foundation.
Besides this custom solution I would evaluate if the task can be solved using Microsoft Integration Services .
Finally other question comes to mind - why do you need this application? Why doesn't/don't the application(s) consuming the data query the database? Is the expensive polling required? Is it possible for the data producers to signal the availibilty of new data directly to the data consumers?
I am not sure about the details of your project, specifically related to security but maybe it would be better to create an SSIS package and schedule it as a job?