I currently have an application which is basically a wrapper for ~10 "LongRunning" Tasks. Each thread should keep running indefinitely, but sometimes they lock up or crash, and sometimes the wrapper app spontaneously exits (I haven't been able to track that down yet). Additionally, the wrapper application can currently only be running for one user, and that user has to be the one to restart the threads or relaunch the whole app.
I currently have a monitor utility to let me know when the threads stop doing work so that they can be manually restarted, but I'd like to automatically restart them instead. I'd also like the wrapper to be available to everyone to check the status of the threads, and for the threads to be running even when the wrapper isn't.
Based on these goals, I think I want to separate the threads into a Windows Service, and convert the wrapper into something which can just connect to the service to check its status and manipulate it.
How would I go about doing this? Is this a reasonable architecture? Should I turn each thread into a separate service, or should I have a single multi-threaded service?
Edit: All the tasks log to the same set of output files (via a TextWriter.Synchronized(StreamWriter)), and I would want to maintain that behavior.
They also all currently share the same database connection, which means I need to get them all to agree to close the connection at the same time when it's necessary. However, if they were split up they could each use their own database connection, and I wouldn't need to worry about synchronizing that. I actually suspect that this step is one of the current failure points, so splitting it up would be a Good Thing.
I would suggest you to stay inside one multithreading service if possible. Just make sure that threads are handled correctly when Service Stop is triggered. Put brake flags inside blocks of code that will take a lot of time to execute. This way you will make your service responsive on Stop event. Log any exceptions and make sure to wait for all threads to exit until service is finally stopped. This will prevent you to run same "task" in multiple threads.
Maintaining one service is in the end easier then multiple services.
Splitting to multiple services would be reasonable if you require some separate functionalities that can run or not beside each other.
I don't think moving the threads to a Windows Service removes any of the problems. The service will still crash randomly and the threads will still exit randomly.
I assume that your long-running tasks implement a kind of worker loop. Wrap the body of that loop in a try-catch and log all exceptions. Don't rethrow them so that the task does not ever exit. Examine the logs to find the bugs.
Related
I have a Windows service that is calling a stored proc over and over (in an infinite loop).
The code looks like this:
while(1)
{
callStoredProc();
doSomethingWithResults();
}
However, how there might be cases where the loop gets stuck with no response, but the service is still technically running.
I imagine there are tools to monitor the health of a service, to let operations teams know to restart it.
But for my scenario this won't help since the service will still be technically running, but it's stuck and can't continue.
What's the best way to ensure this process restarts if this scenario happens?
Would the solution be to use a task scheduler that checks for the heartbeat of this process, and restarts the service if it there's no heartbeat for a period of time? To have another separate thread that monitors the progress of the first process?
Windows services have various recovery options which takes care of question 1. For question 2, the best bet would be to use a timeout approach whereby if the service takes more than X amount of time to complete it restarts or stops what it's doing (I don't know the nature of your service so can't provide implementation detail).
The heartbeat idea would work as well, however, that just becomes another thing to manage/maintain & install.
Hi,
I have a ASP.NET application where I have added a Webservice that contains a "fire and forget" method. When this method is executed it will start a loop (0-99999) and for every loop it will read a xml file and save it to the database.
The problem is that this action will take a couple of hours and it usually ends with a Thread Aborted exception?
I have seen that you can increase the executionTimeout and this is how :
<httpRuntime executionTimeout="604800"/>
<compilation debug="true">
But this does not help?
I have also tried to add a thread.sleep within the loop. If I set it to 500 it will go half way and if I set <100 it will just go a couple of 1000 loops before the thread aborted exception?
How can I solve this?
Don't run the loop inside the web service. Instead, have it in a console app, a winforms app, or possibly even a windows service. Use the web service to start up the other program.
A web service is basically a web page, and asp.net web pages are not meant to host long running processes.
This article does not directly answer your question, but contains a snippet of info relevant to my answer:
http://msdn.microsoft.com/en-us/magazine/dd296718.aspx
However, when the duration of the
operation grows longer than the
typical ASP.NET session duration (20
minutes) or requires multiple actors
(as in my hiring example), ASP.NET
does not offer sufficient support. You
may recall that the ASP.NET worker
processes automatically shut down on
idle and periodically recycle
themselves. This will cause big
problems for long-running operations,
as state held within those processes
will be lost.
and the article is a good read, at any rate. It may offer ideas for you.
Not sure if this is 'the answer', but when you receive the web service call you could consider dispatching the action onto another thread. That could then run until completion. You would want to consider how you ensure that someone doesn't kick off two of these processes simultaneously though.
I have a ASP.NET application where I
have added a Webservice that contains
a "fire and forget" method. When this
method is executed it will start a
loop (0-99999) and for every loop it
will read a xml file and save it to
the database.
Lets not go into that I fhind this approach quite... hm... bad for many reasons (like a mid of the thing reset). I would queue the request, then return, and have a queue listener do the processing with transactional integrity.
Anyhow, what you CAN do is:
Queue a WorkItem for a wpool thread to pick things up.
Return immediately.
Besides that, web services and stuff like this are not a good place for hourly long running processes. Tick off a workflow, handle it separately.
I’m looking for the best way of using threads considering scalability and performance.
In my site I have two scenarios that need threading:
UI trigger: for example the user clicks a button, the server should read data from the DB and send some emails. Those actions take time and I don’t want the user request getting delayed. This scenario happens very frequently.
Background service: when the app starts it trigger a thread that run every 10 min, read from the DB and send emails.
The solutions I found:
A. Use thread pool - BeginInvoke:
This is what I use today for both scenarios.
It works fine, but it uses the same threads that serve the pages, so I think I may run into scalability issues, can this become a problem?
B. No use of the pool – ThreadStart:
I know starting a new thread takes more resources then using a thread pool.
Can this approach work better for my scenarios?
What is the best way to reuse the opened threads?
C. Custom thread pool:
Because my scenarios occurs frequently maybe the best way is to start a new thread pool?
Thanks.
I would personally put this into a different service. Make your UI action write to the database, and have a separate service which either polls the database or reacts to a trigger, and sends the emails at that point.
By separating it into a different service, you don't need to worry about AppDomain recycling etc - and you can put it on an entire different server if and when you want to. I think it'll give you a more flexible solution.
I do this kind of thing by calling a webservice, which then calls a method using a delegate asynchronously. The original webservice call returns a Guid to allow tracking of the processing.
For the first scenario use ASP.NET Asynchronous Pages. Async Pages are very good choice when it comes to scalability, because during async execution HTTP request thread is released and can be re-used.
I agree with Jon Skeet, that for second scenario you should use separate service - windows service is a good choice here.
Out of your three solutions, don't use BeginInvoke. As you said, it will have a negative impact on scalability.
Between the other two, if the tasks are truly background and the user isn't waiting for a response, then a single, permanent thread should do the job. A thread pool makes more sense when you have multiple tasks that should be executing in parallel.
However, keep in mind that web servers sometimes crash, AppPools recycle, etc. So if any of the queued work needs to be reliably executed, then moving it out of process is a probably a better idea (such as into a Windows Service). One way of doing that, which preserves the order of requests and maintains persistence, is to use Service Broker. You write the request to a Service Broker queue from your web tier (with an async request), and then read those messages from a service running on the same machine or a different one. You can also scale nicely that way by simply adding more instances of the service (or more threads in it).
In case it helps, I walk through using both a background thread and Service Broker in detail in my book, including code examples: Ultra-Fast ASP.NET.
I have a Windows Service that performs a long-running process. It is triggered by a timer and the entire process can take a few minutes to complete. When the timer elapses the service instantiates a management object that performs the various tasks, logs the results and then exits.
I have not implemented anything to handle those occasions when the server is shutdown during the middle of the process. It could cause some problems. What is the best practice to handle this?
Can only give vague suggestions since I don't know what task you are actually doing.
If it is something to do w/ database, there is transaction that can be rolled back if it is not committed.
If it involves some file manipulation, perhaps take a look at this article on Transactional NTFS. You can use it in combination w/ TransactionScope object to ensure atomic transaction.
If you are dealing with web services, well the service boundary will dictate when one transaction starts / ends and when the other one begins, use compensation model (if you break something on your part, you need to provide a way later on, after recovery, a way to notify / execute compensation scripts on the other end. (Think about ordering book online and how to handle backorder, cancellation, etc.)
For tracking mechanism, log every steps and the timelines for troubleshooting if something like shutdown occurs.
If your describing essentially a batch process its ok to have a timer that does work at an interval - much of the world works that way.
If its long running, try to keep your units of work, or batches, small enough that your process can at least check to see if its been signaled to stop or not. This will allow the service to exit gracefully instead of essentially ignoring the service stop message.
Somewhere in your timer function you have a property, IsShutdownRequired or some such, that your checking (assuming some loop processing). This property is set to true in the service stop control message, which allows your process to gracefully exit by either not trying to do more work, or as Jimmy suggested, rolling back that work if in a transaction.
Ideally, smaller batches would be better than one big one.
Folks,
I want to develop a long running windows service (it should be working without problems for months), and I wonder what is the better option here:
Use a while(true) loop in the OnStop method
Use a timer to tick each n seconds and trigger my code
Any other options ?
Thanks
Essam
I wouldn't do #1.
I'd either do #2, or I'd spin off a separate thread during OnStart that does the actual work.
Anything but #1
The services manager (or the user, if he's the one activating the controls) expects OnStart() and OnStop() to return in a timely fashion.
The way it's usually done is to start your own thread that keeps things running and ofcourse, listens to an event that might tell it to stop.
Might be worth considering a scheduled task with a short interval. Saves writing a lot of plumbing code and dealing with the peculiarities of Windows Services timers.
Don't mess with the service controller code. If the service wants to stop, you will only make matters worse by using #1. And BTW the service can always crash, in which case your while(true) won't help you a thing.
If you really want to have a "running windows service (it should be working without problems for months)", you'd better make sure your own code is properly and thoroughly tested using unit and integration tests before your run it as a service.
I would NOT recommend #1.
What I’ve done in the past for the exact same scenario/situation is create a scheduled task that runs ever N seconds, kicks off a small script that simply does these 2 things: #1 checks for “IsAlreadyRunning” flag (which is read from the database) #2 If the flag is true, then the script immediately stops end exits. If the flag is false, the script kicks off a separate process (exe) in a new thread (which utilizes a service to perform a task that can be either very short or sometimes really long, depending on the amount of records to process). This process of course sets and resets the IsAlreadyRunning flag to ensure threads do not kick off actions that overlap. I have a service that's been running for years now with this approach and I never had any problems with it. My main process utilizes a web service and bunch of other things to perform some heavy backup operations.
The System.Threading.Timer class would seem appropiate for this sort of usage.
Is it doing a
1 clean up task, or
2 waking up and looking to see if needs to run a task
If it is something like #2, then using MSMQ would be more appropriate. With MSMQ task would get done almost immediately.