I'm developing a Windows Service that will run every 15 minutes, and sends out push notifications to iOS, Android, and BlackBerry users. Each of these device specific operations will run in separate threads in the Windows service. That's all well and good, but there is a chance that we will need to send out up to 50,000 push notifications at a time. If this happens, it could possibly take more than 15 minutes, so before it's time for the next service to run, I want to know if the previous process has finished, and if it hasn't, wait and queue the Windows service to execute once the prior execution is complete. I'm fine with the threading aspect, but I don't know the correct way to implement the scenario that I've described above. Is there some sort of "Wait" or "Queue" mechanism in C#?
Since your service is using a Timer to schedule the work, it can always disable the timer when work begins, and reschedule, as needed, at the end of your work.
This allows a lot of flexibility. If a queue of work took more than 15 minutes, you could decide whether to delay the next one, just start it immediately (potentially running "forever"), skip it entirely, or whatever you needed, as the timer won't run again until it was reenabled.
Related
Is it possible to send a heartbeat to hangfire (Redis Storage) to tell the system that the process is still alive? At the moment I set the InvisibilityTimeout to TimeSpan.MaxValue to prevent hangfire from restarting the job. But, if the process fails or the server restarts, the job will never be removed from the list of running jobs. So my idea was, to remove the large time out and send a kind of heartbeat instead. Is this possible?
I found https://discuss.hangfire.io/t/hangfire-long-job-stop-and-restart-several-time/4282/2 which deals with how to keep a long-running job alive in Hangfire.
The User zLanger says that jobs are considered dead and restarted once you ...
[...] are hitting hangfire’s invisibilityTimeout. You have two options.
increase the timeout to more than the job will ever take to run
have the job send a heartbeat to let hangfire’s know it’s still alive.
That's not new to you. But interestingly, the follow-up question there is:
How do you implement heartbeat on job?
This remains unanswered there, a hint that that your problem is really not trivial.
I have never handled long-running jobs in Hangfire, but I know the problem from other queuing systems like the former SunGrid Engine which is how I got interested in your question.
Back in the days, I had exactly your problem with SunGrid and the department's computer guru told me that one should at any cost avoid long-running jobs according to some mathematical queuing theory (I will try to contact him and find the reference to the book he quoted). His idea is maybe worth sharing with you:
If you have some job which takes longer than the tolerated maximal running time of the queuing system, do not submit the job itself, but rather multiple calls of a wrapper script which is able to (1) start, (2) freeze-stop, (3) unfreeze-continue the actual task.
This stop-continue can indeed be a suspend (CTRL+Z respectively fg in Linux) on operating-system level, see e.g. unix.stackexchange.com on that issue.
In practice, I had the binary myMonteCarloExperiment.x and the wrapper-script myMCjobStarter.sh. The maximum compute time I had was a day. I would fill the queue with hundreds of calls of the wrapper-script with the boundary condition that only one at a time of them should be running. The script would check whether there is already a process myMonteCarloExperiment.x started anywhere on the compute cluster, if not, it would start an instance. In case there was a suspended process, the wrapper script would forward it and let it run for 23 hours and 55 minutes, and suspend the process then. In any other case, the wrapper script would report an error.
This approach does not implement a job heartbeat, but it does indeed run a lengthy job. It also keeps the queue administrator happy by avoiding that job logs of Hangfire have to be cleaned up.
Further references
How to prevent a Hangfire recurring job from restarting after 30 minutes of continuous execution seems to be a good read
I have a Windows service that is calling a stored proc over and over (in an infinite loop).
The code looks like this:
while(1)
{
callStoredProc();
doSomethingWithResults();
}
However, how there might be cases where the loop gets stuck with no response, but the service is still technically running.
I imagine there are tools to monitor the health of a service, to let operations teams know to restart it.
But for my scenario this won't help since the service will still be technically running, but it's stuck and can't continue.
What's the best way to ensure this process restarts if this scenario happens?
Would the solution be to use a task scheduler that checks for the heartbeat of this process, and restarts the service if it there's no heartbeat for a period of time? To have another separate thread that monitors the progress of the first process?
Windows services have various recovery options which takes care of question 1. For question 2, the best bet would be to use a timeout approach whereby if the service takes more than X amount of time to complete it restarts or stops what it's doing (I don't know the nature of your service so can't provide implementation detail).
The heartbeat idea would work as well, however, that just becomes another thing to manage/maintain & install.
I have determined that I have some intense operations that shouldn't occur in the context of a web request in my EWL web application. I see that EWL supports running a Windows Service which will be perfect for running my intense operations in the background without tying up web request threads and forcing users to wait.
I am to the point where I need to implement void WindowsServiceBase.Tick() but I don't see how often Tick() is called. How often is Tick() called by default and is this configurable?
I also see from the source that the Windows Service sends a "health check" email before calling Tick(). What's the thinking behind this? What if I don't want my email spammed with these emails?
The code in ServiceBaseAdapter is cryptic, but looking very carefully you can see that Tick will be called ten seconds after Init completes and, from then on, ten seconds after the last Tick call completes.
The health check email will go out shortly after midnight every day. It's designed to let you know that the service is still alive.
I'll try and briefy explain what i'm looking to achieve. My intial idea of doing this, is not going to work well in my opinion, so i'm trying to decide how best to plan this.
The first thought was:
I have a list of messages that need to be sent out at scheduled times, each one is stored in a central SQL database.
The intention is to use a Windows service that will have a timer that ticks every 30 mins. So..
30 Mins pass > Call ScheduleMessages()
ScheduleMessages will check the database for any unsent messages that need to go out in the next 30 minutes, it will then mark them in the database as:
ScheduleActivated = 1
For each one it marks as ScheduleActivated = 1 it will fire off a customer time object which inherits from the normal timer, which also includes the properties for the message it needs to send.
It will be set to tick at the time the message is due to go out, will send the message and mark as successful in the database.
The main problem with this is I am going to have timers all over the place, and if there were a few hundred messages sheduled at once, it's probably going to either not perform well, or fall over completely.
After re-evalutating I thought of solution 2
My other idea was to have 1 timer running in the service, which ticks once every 10 minutes. Each time it ticks it would fire off a method that gathers every single message due to be sent at any point up until that time into a list, and then processes them one at a time.
This seems much less resource intensive, but i'm worried that if the timer ticks after 10 minutes, any messages that haven't finished sending, wil be caught in the next tick, and be sent again.
Would it be work to stop the timer once it has been going for 10 minutes, then reset to zero and start again once the messages have been sent.
Is there a 3rd solution to the problem which is better than the above?
We implemented this on one project, what worked for us was:
All messages written to a table with a send time
Service that checks every x mins if there is something to send
When the service sends a message it also marks the message a sent (update sent time from null to actual sent time)
Marking the message avoids resends, and if you want to resend you just set the date to null.
The only problem that we had is that the service ran as a single thread and the number of messages sent was therefore limited. But you would have very many messages and a very small window before this is a problem.
Ditch the fixed interval. Windows has plenty of ways to sleep for a specific amount of time, including the sleep function, waitable timers, etc.
Some of these are available in .NET, for example WaitHandle.WaitAll accepts a sleep time and an event, that way your thread can wait until the next scheduled item but also be woken by a request to modify the schedule.
In my opinion, the scheduling service should only be responsible for checking schedules and any work should be passed off to a separate service. The scheduling service shouldnt care about the work to be scheduled. Try implementing a work item interface that contains an execute method. That way the executing object can handle the internals itself and neednt be aware of the scheduling service. For scheduling have you checked out quartz.net?
I need to implement a windows service that performs database import, and that once a month.
The program receives data via e-mail, and will import them at the end of each month.
Is there a better way than set the program to sleep for max_integer seconds/miliseconds repeatedly ?
I would not do it as windows service. I would run it as a scheduled task.
Your service will just be sleeping for a month and is just a a waste of resources. Let the OS keep track of the time and start your application to do the once a month processing.
If you can avoid writing a window service, you'll make your life easier - scheduled tasks may be a better option here. Windows services are best used for things where there is some constant background activity that needs to happen - not for running tasks over long timescales.
However, if you must make this a windows service, then you don't want to set a long-timer sleep timeout. That's most definitely a problematic approach. What happens if your application is restarted? How will it know how long it's been sleeping for? Or what if the thread is restarted?
A better approach is to write a record somewhere in the database that identifies when the next import should happen as a date/time. You windows service can wake up periodically (every few minutes or hours) and see if the current date/time is greater than that value. If it is, run the import and update the next run date in the database to the next time to run.
If your app is restarted, you simply read the value back from the database and continue as before.