I am looking to program a simple "data push" service, that extracts data out of a SQL Server database, and deposits a CSV file to a remote FTP site every ten minutes. This service will be running on a remote server, managed over TeamViewer.
There are a few ways I've thought to do this, but would like a bit of advice as to which is the best and most reliable method. A few pro's and cons would also be very helpful from people who have experience in this type of work.
Possible solutions:
Windows service with use of Thread.Sleep(..) to run task every ten minutes
Simple EXE console project that runs as a Windows Scheduler task
Windows service with use of a Timer class
Any other methods?
The program will be written in C#, but I am very flexible in terms of project type, design etc.
The main requirement of this service is to be reliable, and I'd also look to build in an alerts system to notify on failure.
Many thanks
I would favour a scheduled task for this kind of application, as it's far easier to make changes to the schedule at a later date.
There's a previous question along a similar line here: Windows Service or Task Scheduler for maintenance tasks?
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 9 years ago.
Improve this question
I know variations of this topic have already been asked, but here's my situation:
I have about 30 FTP interface applications. Each interface has their own requirements and configuration, but basically it downloads files from a source server – sometimes daily, sometimes every minute (some, possibly even scheduled in the seconds)
For my initial development, I wrote a C# class library that does all FTP work. This application (be it a console app or windows service) will most likely be run under Windows Server 2012.
Now comes the next piece and I’m trying to decide between:
1) Writing a console app (or a powershell script?) that takes command
line inputs plus a configuration file for each interface. I would
schedule this using Windows Task Scheduler. For deployment of these
interfaces, I could create a batch file that uses the “schtasks.exe”
to create and configure the task. One task for each interface. Sounds
easy peasy…
OR
2) Write a windows services application… but here, I am confused. Do I
create and install a service for each one of my interfaces? (i.e. only
thing different might be the config file). Or, do I create a a main
service that spawns off threads for each interface defined in a single
config file?
If I did this as a single service, how do I manage the
maintenance/deployment of this? If I stop the service, would it not
affect all the interfaces? And how do I perform the actual
scheduling? I’ve read the suggestion is to use Quartz.Net scheduler or
just .Net Timers.
---
Some additional thoughts:
Here are some readings on StackOverflow which brings up these topics/concerns:
windows service vs scheduled task
Scheduled console app vs Windows service? When is it appropriate to use each
Task Scheduler Concerns
List item
May need to be logged in? (I’ve read this is not true)
Issues on machine admin password change (I’ve read this is not true)
Issues with running in high-authority accounts (NetworkService, LocalSystem, or a User)
Issue with multiple processes / long running transactions
This would be really bad for me, for example, if two processes tried to download (and delete) the same source FTP file.
Experience of many is that this is not as stable/reliable as Windows Services, especially on earlier operating systems pre Windows7
Less infrastructure support (e.g. failure policies for retry, monitoring, etc.)
Concerns when scheduling in the seconds…
Windows Service Concerns
List item
Potential issues with Timers
More complicated
Console App + Scheduled Task Concerns
List item
Can’t run in background – so hosting server will have command prompts launching
This is a serious problem. If I have 30 FTP Interfaces and each is scheduled to run every minute/hour, that’s a lot of windows!!!
How do I get around this? Use PowerShell scripts instead?
Looking forward to some feedback, and sample code/scripts, if relevant also highly appreciated.
Thanks
You can access your c# class in powershell using
Either add-type or [System.Reflection.Assembly] - check out this SO post for examples - Can I access My Custom .NET Class from PowerShell?
You could do it all in powershell, using start-job and the like, but then your controlling script will need a lot of effort.
I'd be tempted to use Powershell' inbuilt (in version 3) cmdlets for creating/modifying scheduled jobs in task scheduler and custom triggers - http://blogs.technet.com/b/heyscriptingguy/archive/2012/09/18/create-a-powershell-scheduled-job.aspx
Have it schedule jobs that are running .ps1 powershell scripts, one for each transfer or group of related transfers e.g. Monthly_Payroll_xfer.ps1, Weekly_Expenses_xfer.ps1 with logic relevant to that transfer, email alerts on success/failure etc.
The only thing that doesn't solve is same file/destination being processed at the same time but you could potentially have the individual scripts check at start if they are running already - see the accepted answer to this post - Assure only 1 instance of PowerShell Script is Running at any given Time
As for windows prompts, you can run a scheduled task as a user that won't be logged on, and you won't see the powershell command windows as they will be on a different console - be sure not to set -noexit switch of powershell.exe so they do go away when finished.
Had to do this at work for a tool that only runs as GUI, not as a service, and can't be frigged with srvany etc - we setup a scheduled task that runs (at startup) as a service account and runs a powershell script that runs the GUI.exe
We then use powershell scripts to stop-process to kill it, disable/enable the task (like disabling/enabling a service) and one to run the task.
It's a cludge but it works.
Apologies I haven't given much in the way of code examples, doing this on my phone but may give you ideas.
You can also do it all in c# - we have an in-house developed application with SQL back-end that does exactly what you're trying to achieve, very successfully, multi-threaded with very high volumes being chucked at it..
However, my employers' Intellectual Property and other policies prevent me sharing any more about it than that :-(
So this is my first stint with programmatically creating windows service or scheduled tasks and I am confused which one to choose. I had a look at various articles like http://weblogs.asp.net/jgalloway/archive/2005/10/24/428303.aspx , scheduled task or windows service and some more but can' t really decide btween the two
Here is my scenario :
My application will pick up the code paths of a few dlls from the db , execute the DLLs using MSTest.exe and log back the results to the Db. this will probably be repeated every 2-3 hours . Now I am leaning a bit towards scheduled tasks since i won't have to worry about memory related issues but need some expert advice on this.
P.S. : The DLLs contain test methods that make calls to web services of applications deployed on various servers
Thanks in advance for the help
A Scheduled Task would be more appropiate for your scenario. I don't think it make a lot of sense building a scheduling mechanism on a windows service when OS already provides scheduling infraestructure.
A Windows service is more appropiate for processes that have to respond to events at any moment and not at specific and fix periods. That's why they are running all the time. An example of this is the SQL Server Service.
An exception of this could be a task that needs to run every second or so. In that corner case, a Window Service could be the best solution. For your specific schedule, I have no doubts that a scheduled task would fit much more better.
Although this post is several months old, here's a possible resolution in case it's helpful to others for the "Run whether the user is logged on or not" issue : start with a console project then change type to Windows App as mentioned at Run code once and exit formless app: Windows Forms or WPF... does it matter?:
“If you never show a UI, you should start with a WinForms project (WPF projects set extra project metadata that you don't want), then delete the reference to System.Windows.Forms.dll. Alternatively, start with a console project, then change the Output type to Windows Application.”
I am developing a asp.net site that needs hit a few social media sites daily for blanket friend/follower data. I have chosen arvixe business class as my hosting. In the future if we grow, I'd love to get onto a dedicated server and run a windows service, however since that is not in the cards at this point I need another reliable way of running scheduled tasks. I am familiar with running a thread timer from the app_code(global.aspx). However the app pool recycling will cause some problems with the timer. I have never used task scheduling like quartz but have read a lot about it on stackoverflow. I was looking for some advise as to how to approach my goal. One big problem I have using either method is that I will need the crawler threads to sleep for up to an hour regularly due to api call limits. My first thoughts were to use the db to save the starting and ending of a job. When the app pool recycles I would clear out any parts not completed and only start parts that do not have a record of running on that day. What do the experts here think? any good links to sample architecture of this type of scheduling?
It doesn't really matter what method you use, whether you roll your own or use Quartz. You are at the mercy of ASP.NET/IIS because that's where you want to host it.
Do you have a spare computer laying around that can just run a scheduled task and upload data to a hosted database? To be honest, it's possibly safer (depending on your use case) to just do it that way then try to run a scheduler in ASP.NET.
Somewhat along the lines of Bryan's post;
Find a spare computer.
Instead of allowing DB access have it call up a web service on your site. This service call should be the initiator of the process you are trying to do. Don't try to put params into it, just something like "StartProcess()" should work fine.
As far as going to sleep and resuming later take a look at Workflow Foundation. There are some nice built in features to persist state.
Don't expose your DB to the outside world, instead expose that page or web service and wraps some security around that. WCF has some nice built in security features for that.
The best part is when you decide to move off, you can keep your web service and have it called from a Windows Service in the same manner.
As long as you use a persistent job store (like a database) and you write and schedule your jobs so that they can handle things like being killed half way through, having IIS recycle your process is not that big a deal.
The bigger issue is that IIS shuts your site down if it doesn't have traffic. If you can keep your site up, then just make sure you set the misfire policy appropriately and that your jobs store any state data needed to pick up where they left off, you should be able to pull it off.
If you are language-agnostic and don't mind writing your "job-activation-script" in your favourite, Linux-supported language...
One solution that has worked very well for me is:
Getting relatively cheap, stable Linux hosting(from reputable
companies),
Creating a WCF service on your .Net hosted platform that will contain the logic you want to run regularly (RESTfully or SOAP or XMLRPC... whichever suits you),
Handling the calls through your Linux hosted cron jobs, written in your language of choice(I use PHP).
Working very well, like I said. No VPS expense,configurable and externally activated. I have one central place where my jobs are activated, with 99 to 100% uptime(never had any failures).
Let me give a back ground for everybody before I go to my problem. My company hosts website for many clients, my company also contracts some of the work to another company.
So when we first set up a website with all the informations to our clients, we pass that information to the other company we contracted and three of us have the same data. Problem is once the site is up and running, our clients will change some data and when ever they do that we should be able to update our contracted company.
The way we transfer data to the contracted company is by using a web service (httppost, xml data). Now my question is what it the best way to write a program which sends updated data to the contracted company everytime our clients change some data.
1) Write a windows service having a timer inside my code where every 30min or so connects to the database and find all changes and send it to the contracted company
2) Write the same code as #1 (with out the timer in it) but this time make it a simple program and let windows scheduler wake it every 30min
3) Any other suggestion you may have
Techenologies available for me are VS 2008, SQLServer 2005
Scheduled task is the way to go. Jon wrote up a good summary of why services are not well suited for this sort of thing: http://weblogs.asp.net/jgalloway/archive/2005/10/24/428303.aspx
A service is easy to create and install and is more "professional" feeling so why not go that way? Using a non-service EXE would also work of course and would be slightly easier to get running (permissions, etc.) but I think the difference in setup between the two is nearly negligible.
One possible solution would be to add a timestamp column to your data tables.
Once this is done, you can have one entry in each table that has the last collected time by your contracted company. They can pull all records since that last time and update their records accordingly.
A Windows Service is more self contained, and you can easily configure it to start up automatically when the OS is starting up. You might also need to create additional configuration options, as well as some way to trigger the synchronization immediately.
It will also give you more room to grow your functionality for the service in the future.
A standalone app should be easier to develop though, however you are reliant on the windows scheduler to execute the task always. My experience has been that it is easier to mess up things with the windows scheduler and have it not run, for example in cases where you reboot the OS but no user has logged in.
If you want a more professional approach go with the service, even though it might mean a little bit more work.
A windows service makes more sense in this case. Think about what happens after your server is restarted:
With a Windows Application you need to have someone restart the application, or manually copy a shortcut to the startup folder to make sure the application gets launched
OR,
With a Windows Service you set it to start automatically and forget about it. When the machine reboots your service starts up and continues processing.
One more consideration, what happens when there is an error? A Windows application would likely show an error dialog and wait for input before continuing; whereas a service would log the error in the event log and carry on.
I have been working on many applications which run as windows service or scheduled tasks.
Now, i want to make sure that these applications will be fault tolerant and reliable. For example; i have a service that runs every hour. if the service crashes while its operating or running, i d like the application to run again for the same period (there are several things involved with this including transactions of data processing) , to avoid data loss. moreover, i d like the program to report the error with details. My goal is to avoid data loss and not falling behind for running the program.
I have built a class library that a user can import into a project. Library is supposed to keep information of running instance of the program, ie. program reads and writes information of running interval, running status etc. This data is stored in a database.
I was curious, if there are some best practices to make the scheduled tasks/ windows services fault tolerant and reliable.
Edit : I am talking about independent tasks or services which on different servers. and my goal is to make sure that the service will keep running, report any failures and recover from them.
I'm interested in what other people have to say, but I'll give you a few points that I've stumbled across:
Make an event handler for Unhandled Exceptions. This way you can clean up resources, write to a log file, email an administrator, or anything you need to instead of having it crash.
AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(AppUnhandledExceptionEventHandler);
Override any servicebase event handlers you need in the main part of your application. OnStart and OnStop are pretty crucial, but there are many others you can use. http://msdn.microsoft.com/en-us/library/system.serviceprocess.servicebase%28v=VS.71%29.aspx
Beware of timers. Windows forms timers won't work right in a service. User System.Threading.Timers or System.Timers.Timer. Best Timer for using in a Windows service
If you are updating on a thread, make sure you use a lock() or monitor in key sections to make sure everything is threadsafe.
Be careful not to use anything user specific, as a service runs without a specific user context. I noticed some of my SQL connection strings were no longer working for windows authorizations, etc. Also have heard people having trouble with mapped drives.
Never make a service with a UI. In fact for Vista and 7 they make it nearly impossible to do anyway. It shouldn't require user interaction, the most you can do is send a message with a WIN32 function. MSDN claims making interactive services is bad practice. http://msdn.microsoft.com/en-us/library/ms683502%28VS.85%29.aspx
For debugging purposes, it is way cool to make a service run as a console application until you get it doing what you want it to. Awesome tutorial: http://mycomponent.blogspot.com/2009/04/create-debug-install-windows-service-in.html
Anyway, hope that helps a little, but that is just a couple thing I poked around to find on my own.
Something obvious - don't run all your tasks at the same time. Try to schedule them so only one task is using some expensive resource at any time (if possible). For example, if you need to send out newsletters and some specific notifications, schedule them at different times. If two tasks need to clean up something in the database, let the one run after another.
Also schedule tasks to run outside of normal business hours - at night obviously.