I have a Windows service that is calling a stored proc over and over (in an infinite loop).
The code looks like this:
while(1)
{
callStoredProc();
doSomethingWithResults();
}
However, how there might be cases where the loop gets stuck with no response, but the service is still technically running.
I imagine there are tools to monitor the health of a service, to let operations teams know to restart it.
But for my scenario this won't help since the service will still be technically running, but it's stuck and can't continue.
What's the best way to ensure this process restarts if this scenario happens?
Would the solution be to use a task scheduler that checks for the heartbeat of this process, and restarts the service if it there's no heartbeat for a period of time? To have another separate thread that monitors the progress of the first process?
Windows services have various recovery options which takes care of question 1. For question 2, the best bet would be to use a timeout approach whereby if the service takes more than X amount of time to complete it restarts or stops what it's doing (I don't know the nature of your service so can't provide implementation detail).
The heartbeat idea would work as well, however, that just becomes another thing to manage/maintain & install.
Related
Is it possible to send a heartbeat to hangfire (Redis Storage) to tell the system that the process is still alive? At the moment I set the InvisibilityTimeout to TimeSpan.MaxValue to prevent hangfire from restarting the job. But, if the process fails or the server restarts, the job will never be removed from the list of running jobs. So my idea was, to remove the large time out and send a kind of heartbeat instead. Is this possible?
I found https://discuss.hangfire.io/t/hangfire-long-job-stop-and-restart-several-time/4282/2 which deals with how to keep a long-running job alive in Hangfire.
The User zLanger says that jobs are considered dead and restarted once you ...
[...] are hitting hangfire’s invisibilityTimeout. You have two options.
increase the timeout to more than the job will ever take to run
have the job send a heartbeat to let hangfire’s know it’s still alive.
That's not new to you. But interestingly, the follow-up question there is:
How do you implement heartbeat on job?
This remains unanswered there, a hint that that your problem is really not trivial.
I have never handled long-running jobs in Hangfire, but I know the problem from other queuing systems like the former SunGrid Engine which is how I got interested in your question.
Back in the days, I had exactly your problem with SunGrid and the department's computer guru told me that one should at any cost avoid long-running jobs according to some mathematical queuing theory (I will try to contact him and find the reference to the book he quoted). His idea is maybe worth sharing with you:
If you have some job which takes longer than the tolerated maximal running time of the queuing system, do not submit the job itself, but rather multiple calls of a wrapper script which is able to (1) start, (2) freeze-stop, (3) unfreeze-continue the actual task.
This stop-continue can indeed be a suspend (CTRL+Z respectively fg in Linux) on operating-system level, see e.g. unix.stackexchange.com on that issue.
In practice, I had the binary myMonteCarloExperiment.x and the wrapper-script myMCjobStarter.sh. The maximum compute time I had was a day. I would fill the queue with hundreds of calls of the wrapper-script with the boundary condition that only one at a time of them should be running. The script would check whether there is already a process myMonteCarloExperiment.x started anywhere on the compute cluster, if not, it would start an instance. In case there was a suspended process, the wrapper script would forward it and let it run for 23 hours and 55 minutes, and suspend the process then. In any other case, the wrapper script would report an error.
This approach does not implement a job heartbeat, but it does indeed run a lengthy job. It also keeps the queue administrator happy by avoiding that job logs of Hangfire have to be cleaned up.
Further references
How to prevent a Hangfire recurring job from restarting after 30 minutes of continuous execution seems to be a good read
I have multiple windows services which run 24/7 on a server. For logging events etc. I already use log4net but I want to be able to see if all my services are still running. So I've stumbled upon this question and learned about the ServiceController class. Now I've had the idea to make another service in which I create a ServiceController object per service, and use the WaitForStatus method to be notified when any of the services are stopped. I'd be able to check for any statuses externally through a hosted WCF in the servicecontroller service.
But I've also seen the answer to this question which states a ServiceController should be closed and disposed. Would it be bad to let my ServiceController wait 24/7 until any of my services stopped? Or should I use Quartz or a simple Timer to run a check every x amount of time?
Thanks in advance
You shouldn't. There is no mechanism in Windows to let a service status change generate an event. So ServiceController.WaitForStatus() must poll. It is hard-coded to query the service status 4 times per second, a Thread.Sleep(250) hard-codes the poll interval. Use a decompiler to see this for yourself.
So you basically have many threads in your program, doing nothing but sleep for hours. That's pretty ugly, a thread is an expensive OS object. These threads don't burn any core but the OS thread scheduler is still involved, constantly re-activating the threads when their sleep period expires.
If you need this kind of responsiveness to status changes then it is okayish, but keep in mind that it cannot be more responsive than 250 msec. And keep in mind that increasing the interval by using a Timer sounds attractive but do consider the problem with polling. If you do it, say, once a minute and an admin stops and restarts the service in, say, 30 seconds between two polls then you'll never see the status change. Oops.
Consider to use only one thread that queries many ServiceControllers through their Status property. Your own polling code, minus the cost of the threads.
I need to execute an infinite while loop and want to initiate the execution in global.asax.
My question is how exactly should I do it? Should I start a new Thread or should I use Async and Task or anything else? Inside the while loop I need to do await TaskEx.Delay(5000);
How do I do this so it will not block any other processes and will not create memory leaks?
I use VS10,AsyncCTP3,MVC4
EDIT:
public void SignalRConnectionRecovery()
{
while (true)
{
Clients.SetConnectionTimeStamp(DateTime.UtcNow.ToString());
await TaskEx.Delay(5000);
}
}
All I need to do is to run this as a singleton instance globally as long as application is available.
EDIT:SOLVED
This is the final solution in Global.asax
protected void Application_Start()
{
Thread signalRConnectionRecovery = new Thread(SignalRConnectionRecovery);
signalRConnectionRecovery.IsBackground = true;
signalRConnectionRecovery.Start();
Application["SignalRConnectionRecovery"] = signalRConnectionRecovery;
}
protected void Application_End()
{
try
{
Thread signalRConnectionRecovery = (Thread)Application["SignalRConnectionRecovery"];
if (signalRConnectionRecovery != null && signalRConnectionRecovery.IsAlive)
{
signalRConnectionRecovery.Abort();
}
}
catch
{
///
}
}
I found this nice article about how to use async worker: http://www.dotnetfunda.com/articles/article613-background-processes-in-asp-net-web-applications.aspx
And this:
http://code.msdn.microsoft.com/CSASPNETBackgroundWorker-dda8d7b6
But I think for my needs this one will be perfect:
http://forums.asp.net/t/1433665.aspx/1
ASP.NET is not designed to handle this kind of requirement. If you need something to run constantly, you would be better off creating a windows service.
Update
ASP.NET is not designed for long running tasks. It's designed to respond quickly to HTTP requests. See Cyborgx37's answer or Can I use threads to carry out long-running jobs on IIS? for a few reasons why.
Update
Now that you finally mentioned you are working with SignalR, I see that you are trying to host SignalR within ASP.NET, correct? I think you're going about this the wrong way, see the example NuGet package referenced on the project wiki. This example uses an IAsyncHttpHandler to manage tasks.
You can start a thread in your global.asax, however it will only run till your asp.net process get recycled. This will happen at least once a day, or when no one uses of your site. If the process get recycled, the only way the thread is restarted agian, is when you have a hit on your site. So the thread is not running continueuosly.
To get a continues process it is better to start a windows service.
If you do the 'In process' solution, it realy depends on what your are doing. The Thread itself will not cause you any problems in memory or deadlocks. You should add a meganism to stop your thread when the application stops. Otherwise restarting will take a long time, because it will wait for your thread to stop.
This is an old post, but as I was seraching for this, I would like to report that in .NET 4.5.2 there is a native way to do it with QueueBackgroundWorkItem.
Take a look at this post: https://blogs.msdn.microsoft.com/webdev/2014/06/04/queuebackgroundworkitem-to-reliably-schedule-and-run-background-processes-in-asp-net/
MarianoC
It depends what you are trying to accomplish in your while loop, but in general this is the kind of situation where a Windows Service is the best answer. Installing a Windows Service is going to require that you have admin privileges on the web server.
With an infinite loop you end up with a lot of issues regard the Windows message pump. This is the thing that keeps a Windows application alive even when the application isn't "doing" anything. Without it, a program simply ends.
The problem with an infinite loop is that the application is stuck "doing" something, which prevents other applications (or threads) from "doing" their thing. There have been a few workarounds, such as the DoEvents in Windows Forms, but they all have some serious drawbacks when it comes to responsiveness and resource management. (Acceptable on a small LOB application, maybe not on a web server.) Even if the while-loop is on a separate thread, it will use up all available processing power.
Asynchronus programming is really designed more for long-running processes, such as waiting for a database to return a result or waiting for a printer to come online. In these cases, it's the external process that is taking a long time, not a while-loop.
If a Window Service is not possible, then I think your best bet is going to be setting up a separate thread with its own message pump, but it's a bit complicated. I've never done it on a web server, but you might be able to start an Application. This will provide a message pump for you and allow you to respond to Windows events, etc. The only problem is that this is going to start a Windows application (either WPF or WinForms), which may not be desirable on a web server.
What are you trying to accomplish? Is there another way you might go about it?
I found this nice article about how to use async worker, will give it a try. http://www.dotnetfunda.com/articles/article613-background-processes-in-asp-net-web-applications.aspx
And this:
http://code.msdn.microsoft.com/CSASPNETBackgroundWorker-dda8d7b6
But I think for my needs this one will be perfect:
http://forums.asp.net/t/1433665.aspx/1
I currently have an application which is basically a wrapper for ~10 "LongRunning" Tasks. Each thread should keep running indefinitely, but sometimes they lock up or crash, and sometimes the wrapper app spontaneously exits (I haven't been able to track that down yet). Additionally, the wrapper application can currently only be running for one user, and that user has to be the one to restart the threads or relaunch the whole app.
I currently have a monitor utility to let me know when the threads stop doing work so that they can be manually restarted, but I'd like to automatically restart them instead. I'd also like the wrapper to be available to everyone to check the status of the threads, and for the threads to be running even when the wrapper isn't.
Based on these goals, I think I want to separate the threads into a Windows Service, and convert the wrapper into something which can just connect to the service to check its status and manipulate it.
How would I go about doing this? Is this a reasonable architecture? Should I turn each thread into a separate service, or should I have a single multi-threaded service?
Edit: All the tasks log to the same set of output files (via a TextWriter.Synchronized(StreamWriter)), and I would want to maintain that behavior.
They also all currently share the same database connection, which means I need to get them all to agree to close the connection at the same time when it's necessary. However, if they were split up they could each use their own database connection, and I wouldn't need to worry about synchronizing that. I actually suspect that this step is one of the current failure points, so splitting it up would be a Good Thing.
I would suggest you to stay inside one multithreading service if possible. Just make sure that threads are handled correctly when Service Stop is triggered. Put brake flags inside blocks of code that will take a lot of time to execute. This way you will make your service responsive on Stop event. Log any exceptions and make sure to wait for all threads to exit until service is finally stopped. This will prevent you to run same "task" in multiple threads.
Maintaining one service is in the end easier then multiple services.
Splitting to multiple services would be reasonable if you require some separate functionalities that can run or not beside each other.
I don't think moving the threads to a Windows Service removes any of the problems. The service will still crash randomly and the threads will still exit randomly.
I assume that your long-running tasks implement a kind of worker loop. Wrap the body of that loop in a try-catch and log all exceptions. Don't rethrow them so that the task does not ever exit. Examine the logs to find the bugs.
Folks,
I want to develop a long running windows service (it should be working without problems for months), and I wonder what is the better option here:
Use a while(true) loop in the OnStop method
Use a timer to tick each n seconds and trigger my code
Any other options ?
Thanks
Essam
I wouldn't do #1.
I'd either do #2, or I'd spin off a separate thread during OnStart that does the actual work.
Anything but #1
The services manager (or the user, if he's the one activating the controls) expects OnStart() and OnStop() to return in a timely fashion.
The way it's usually done is to start your own thread that keeps things running and ofcourse, listens to an event that might tell it to stop.
Might be worth considering a scheduled task with a short interval. Saves writing a lot of plumbing code and dealing with the peculiarities of Windows Services timers.
Don't mess with the service controller code. If the service wants to stop, you will only make matters worse by using #1. And BTW the service can always crash, in which case your while(true) won't help you a thing.
If you really want to have a "running windows service (it should be working without problems for months)", you'd better make sure your own code is properly and thoroughly tested using unit and integration tests before your run it as a service.
I would NOT recommend #1.
What I’ve done in the past for the exact same scenario/situation is create a scheduled task that runs ever N seconds, kicks off a small script that simply does these 2 things: #1 checks for “IsAlreadyRunning” flag (which is read from the database) #2 If the flag is true, then the script immediately stops end exits. If the flag is false, the script kicks off a separate process (exe) in a new thread (which utilizes a service to perform a task that can be either very short or sometimes really long, depending on the amount of records to process). This process of course sets and resets the IsAlreadyRunning flag to ensure threads do not kick off actions that overlap. I have a service that's been running for years now with this approach and I never had any problems with it. My main process utilizes a web service and bunch of other things to perform some heavy backup operations.
The System.Threading.Timer class would seem appropiate for this sort of usage.
Is it doing a
1 clean up task, or
2 waking up and looking to see if needs to run a task
If it is something like #2, then using MSMQ would be more appropriate. With MSMQ task would get done almost immediately.