I want to create a service that will monitor changes to web pages i.e. the page content has been updated. I am trying to think of the best way to achieve this and at present I am considering a couple of options. Note that there could be hundreds of pages to monitor and the interval for checking could be seconds or hours (configurable).
Create a windows service for each page to monitor
Create a windows service that spawns a thread for each page to monitor
Now, I am concerned which of these is the best approach and whether these is an alternative I haven't considered. I thought 1 would have the benefit of isolating each monitoring task but would come at the expense of overhead in terms of physical resources and effort to create/maintain. The second would be slightly more complex but cleaner. Obviously it would also lose isolation in that if the service fails then all monitoring will fail.
I have done something similar and I solved it by having a persisted queue (a SQL Server table) that would store the remote Uri along with the interval and a DateTime for the last time it ran.
I can then get all entries that I want to run by selecting the ones that has lastRun + interval < now.
If your smallest interval are in the region of seconds, you probably want to use a ThreadPool, so that you can issue several request at the same time. (Remember to adjust the maxConnections setting in your app.config accordingly).
I would use one Windows service (have a look at the TopShelf project for that) and I would then have Quartz.Net trigger the jobs. With Quartz, you can control whether it has to wait for previous jobs to finish etc.
Creating one Windows Service is the way to go... regarding the failure of this windows Service there are several measures you could take to deal with that - for example configure windows to automatically restart the Windows Service on failure...
I would recommend using a thread pool approach and/or a System.Threading.Timer in combination with a ConcurrentDictionary or ConcurrentQueue .
Related
I am developing a asp.net site that needs hit a few social media sites daily for blanket friend/follower data. I have chosen arvixe business class as my hosting. In the future if we grow, I'd love to get onto a dedicated server and run a windows service, however since that is not in the cards at this point I need another reliable way of running scheduled tasks. I am familiar with running a thread timer from the app_code(global.aspx). However the app pool recycling will cause some problems with the timer. I have never used task scheduling like quartz but have read a lot about it on stackoverflow. I was looking for some advise as to how to approach my goal. One big problem I have using either method is that I will need the crawler threads to sleep for up to an hour regularly due to api call limits. My first thoughts were to use the db to save the starting and ending of a job. When the app pool recycles I would clear out any parts not completed and only start parts that do not have a record of running on that day. What do the experts here think? any good links to sample architecture of this type of scheduling?
It doesn't really matter what method you use, whether you roll your own or use Quartz. You are at the mercy of ASP.NET/IIS because that's where you want to host it.
Do you have a spare computer laying around that can just run a scheduled task and upload data to a hosted database? To be honest, it's possibly safer (depending on your use case) to just do it that way then try to run a scheduler in ASP.NET.
Somewhat along the lines of Bryan's post;
Find a spare computer.
Instead of allowing DB access have it call up a web service on your site. This service call should be the initiator of the process you are trying to do. Don't try to put params into it, just something like "StartProcess()" should work fine.
As far as going to sleep and resuming later take a look at Workflow Foundation. There are some nice built in features to persist state.
Don't expose your DB to the outside world, instead expose that page or web service and wraps some security around that. WCF has some nice built in security features for that.
The best part is when you decide to move off, you can keep your web service and have it called from a Windows Service in the same manner.
As long as you use a persistent job store (like a database) and you write and schedule your jobs so that they can handle things like being killed half way through, having IIS recycle your process is not that big a deal.
The bigger issue is that IIS shuts your site down if it doesn't have traffic. If you can keep your site up, then just make sure you set the misfire policy appropriately and that your jobs store any state data needed to pick up where they left off, you should be able to pull it off.
If you are language-agnostic and don't mind writing your "job-activation-script" in your favourite, Linux-supported language...
One solution that has worked very well for me is:
Getting relatively cheap, stable Linux hosting(from reputable
companies),
Creating a WCF service on your .Net hosted platform that will contain the logic you want to run regularly (RESTfully or SOAP or XMLRPC... whichever suits you),
Handling the calls through your Linux hosted cron jobs, written in your language of choice(I use PHP).
Working very well, like I said. No VPS expense,configurable and externally activated. I have one central place where my jobs are activated, with 99 to 100% uptime(never had any failures).
I have been given a windows service written by a previous intern at my current internship that monitors an archive and alerts specific people through emails and pop-ups should one of the recorded values go outside a certain range. It currently uses a timer to check the archive every 30 seconds, and I have been asked if I would be able to update it to allow a choice of time depending on what "tag" is being monitored. It uses an XML file to keep track of which tags are being monitored. Would creating multiple timers in the service be the most efficient way of going about this? I'm not really sure what approach to take.
The service is written in C# using .NET 3.5.
Depending on the granularity, you could use a single timer that is a common factor of the timing intervals they want. Say they want to put in the XML file that each archive is to be checked every so many minutes. You set up a timer that goes off once a minute, and you check how long it's been since you did each one and whether to do it or not.
If you're getting a chance to re-architect, I would move away from a service to a set of scheduled tasks. Write it so one task does one archive. Then write a controller program that sets up the scheduled tasks (and can stop them, change them etc.) The API for scheduled tasks on Windows 7 is nice and understandable, and unlike a service you can impose restrictions like "don't do it if the computer is on battery" or "only do it if the machine is idle" along with your preferences for what to do if a chance to run the task was missed. 7 or 8 scheduled tasks, each on their own schedule, using the same API of yours, passing in the archive path and the email address, is a lot neater than one service trying to juggle everything at once. Plus the machine will start up faster when you don't have yet another autostart service on it.
Efficient? Possibly not - especially if you have lots of tags, as each timer takes a tiny but finite amount of resources.
An alternative approach might be to have one timer that fires every second, and when that happens you check a list of outstanding requests.
This has the benefit of being easier to debug if things go wrong as there's only one active thread.
As in most code maintenance situations, however, it depends on your existing code, your ability, and how you feel more comfortable.
I woould suggest to just use one timer scheduled at the least common divisor.
For example configure your timer to signal every second and you can handle every interval (1 second, 2 seconds, ...) by counting the according number of timer ticks.
I made dnn scheduler and set to run it on every 1 min. It works when I do something on site. But I need to run some actions when I am not on the site. For example insert record to database with currenct time. Is this possible?
In Host Settings, use Scheduler Mode = Timer Method
This will make the scheduler run in a separate thread that is not triggered by page requests.
If the scheduler runs in the timer method, it won't have access to the current HttpContext.
You will also have to make sure that DNN is kept alive, and IIS doesn't shut down the application due to inactivity. Setting the application pool idle timeout appropriately, and pinging the /Keepalive.aspx should take care of this. Nevertheless, using the DNN scheduler for critical tasks is not a good idea.
See Also:
Creating DotNetNuke Scheduled Jobs
DotNetNuke Scheduler
Explained
If you just want database related things, such as inserting a record, you can use database jobs. You didn't mention what dbms you use but almost every database have almost same functionality under different names.
Doing the equivalent of a Cron job is still a pain in the butt on Windows.
The DNN Scheduler will work if you aren't super concerned about when it runs. What you may need to do is have more logic on your end... if it only runs every 10 minutes, or every hour or whatever you may have to look at your database tables, determine last time it ran and then do whatever it needs to do to 'catch up.' In your case add 60 minutes of info versus every minute.
I'm struggling to think of an example of why I would just write to a table every minute or on some interval. If I needed it for a join table or something convenient to report off of you should generate them in larger chunks.
The other option is to write a small .NET windows service which isn't that hard and have it run every minute. That would be more reliable.
I have to create an app that will read in some info from a db, process the data, write changes back to the db, and then send an email with these changes to some users or groups. I will be writing this in c#, and this process must be run once a week at a particular time. This will be running on a Windows 2008 Server.
In the past, I would always go the route of creating a windows service with a timer and setting the time/day for it to be run in the app.config file so that it can be changed and only have to be restarted to catch the update.
Recently, though, I have seen blog posts and such that recommend writing a console application and then using a scheduled task to execute it.
I have read many posts talking to this very issue, but have not seen a definitive answer about which process is better.
What do any of you think?
Thanks for any thoughts.
If it is a one per week application, why waste the resources for it to be running in the background for the rest of the week.
A console application seems much more appropriate.
The typical rule of thumb that I use is something along these lines. First I ask a few questions.
Frequency of Execution
Frequency of changes to #1
Triggering Mechanism
Basically from here if the frequency of execution is daily or less frequent I'll almost always lean towards a scheduled task. Then looking at the frequency for changes, if there is a high demand for schedule changes, I'll also try to lean towards scheduled tasks, to allow no-coding changes for schedule changes. lastly if there is ever a thought of a trigger other than time, then I'll lean towards windows services to help "future proof" an application. Say for example the requirement changes to be run every time a user drops a file in X folder.
The basic rule I follow is: if you need to be running continuously because events of interest can happen at any time, use a service (or daemon in UNIX).
If you just want to periodically do something, use a scheduled task (or cron).
The clincher here is your phrase "must be run once a week at a particular time" - go for a scheduled task.
If you have only one application and you need it to run once a week may be scheduler will be good as there is no need to have separate service and process running on the system which will be idle most of the time.
I have been working on many applications which run as windows service or scheduled tasks.
Now, i want to make sure that these applications will be fault tolerant and reliable. For example; i have a service that runs every hour. if the service crashes while its operating or running, i d like the application to run again for the same period (there are several things involved with this including transactions of data processing) , to avoid data loss. moreover, i d like the program to report the error with details. My goal is to avoid data loss and not falling behind for running the program.
I have built a class library that a user can import into a project. Library is supposed to keep information of running instance of the program, ie. program reads and writes information of running interval, running status etc. This data is stored in a database.
I was curious, if there are some best practices to make the scheduled tasks/ windows services fault tolerant and reliable.
Edit : I am talking about independent tasks or services which on different servers. and my goal is to make sure that the service will keep running, report any failures and recover from them.
I'm interested in what other people have to say, but I'll give you a few points that I've stumbled across:
Make an event handler for Unhandled Exceptions. This way you can clean up resources, write to a log file, email an administrator, or anything you need to instead of having it crash.
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(AppUnhandledExceptionEventHandler);
Override any servicebase event handlers you need in the main part of your application. OnStart and OnStop are pretty crucial, but there are many others you can use. http://msdn.microsoft.com/en-us/library/system.serviceprocess.servicebase%28v=VS.71%29.aspx
Beware of timers. Windows forms timers won't work right in a service. User System.Threading.Timers or System.Timers.Timer. Best Timer for using in a Windows service
If you are updating on a thread, make sure you use a lock() or monitor in key sections to make sure everything is threadsafe.
Be careful not to use anything user specific, as a service runs without a specific user context. I noticed some of my SQL connection strings were no longer working for windows authorizations, etc. Also have heard people having trouble with mapped drives.
Never make a service with a UI. In fact for Vista and 7 they make it nearly impossible to do anyway. It shouldn't require user interaction, the most you can do is send a message with a WIN32 function. MSDN claims making interactive services is bad practice. http://msdn.microsoft.com/en-us/library/ms683502%28VS.85%29.aspx
For debugging purposes, it is way cool to make a service run as a console application until you get it doing what you want it to. Awesome tutorial: http://mycomponent.blogspot.com/2009/04/create-debug-install-windows-service-in.html
Anyway, hope that helps a little, but that is just a couple thing I poked around to find on my own.
Something obvious - don't run all your tasks at the same time. Try to schedule them so only one task is using some expensive resource at any time (if possible). For example, if you need to send out newsletters and some specific notifications, schedule them at different times. If two tasks need to clean up something in the database, let the one run after another.
Also schedule tasks to run outside of normal business hours - at night obviously.