Ok I am sure I am not using the state machine correctly but here is subset of sample code. This is the Appccelerate.StateMachine which used to be bbvcommon.StateMachine.
fsm.In(State.Idle)
.ExecuteOnEntry(() => {
// wake up and check if there are people still standing and if so restart
if(currentlyTalkingTo.Count() > 0)
{
fsm.Fire(Event.PersonFound);
}
})
.On(Event.PersonFound).Goto(State.WaitToRecognizePeople);
fsm.In(State.WaitToRecognizePeople)
.ExecuteOnEntry(() => {
Thread.Sleep(1000);
fsm.Fire(Event.TimeOut);
})
.On(Event.TimeOut).Goto(State.Greet);
The issue is what the best way to handle a sleep? With this code calling fsm.Stop() when shutting the application down sometimes hangs the application. Commenting all the Thread.Sleeps() in the states fixes the issue so the application successfully shuts down.
What is the recommended way of handling states that need to time out and move to another state? Sample code would be appreciated.
The Thread.Sleep would block the state machine, both the PassiveStateMachine and ActiveStateMachine, so that you cannot react to other events anymore during the sleep time. That might be an explanation for why a call to Stop sometimes hangs the explanation.
I assume you don't just want to wait for a given period of time but want to wait for some event under a timeout condition. I would then recommend to start a Timer in the ExecuteOnEntry of WaitToRecognizePeople that, when elapsed, fires the event Event.Timeout.
If you have many states that have associated timeouts, to avoid a lot of duplicated code, you could even implement this in a separate class as a state machine extension, like I did in one of my projects using this library. (see http://www.appccelerate.com/statemachineextensions.html).
Related
we use a custom built C# application ("communicator") to interact with several other locally installed applications (Excel, ...). The communicator gets its inputs from a queue, calls the applications in defined order (via several DLLs, again custom build) to process the inputs and submits the results to the queue. The times required to complete the steps can be predicted (step 1 (app1) completes within 10 seconds, step 2 (app2) completes within 45 seconds, etc) and the total time to process an individual order should not exceed 90 seconds.
Most of the time things run well unattended (as designed), however every now and then one of the called applications gets stuck and cannot proceed without a human interaction. This also stops the queue processing. We look into the causes and try to eliminated them - once the interruption has been detected.
We'd foremost like to reduce the time to detect the interruption - perhaps by using the communicatior or another method to detect that the expected time to complete a step has been exceeded and notify the admin of such occurence (sending an email).
Would you use the communicatior to set a timer at each step and then raise error if the expected time to complete has been exceed (and send email from within its code) or call an external application at the start of a step and its finish and let it do the timing and notifications part to avoid clogging the communicators code?
Are there some code samples you can share for timing and sending emails?
Or perhaps suggest an external application that would accepts start signals from communicator and send an email if it wouln't receive the completion message within a certain timeframe?
Best regards
I originally answered as a comment but I figure I might as well do a proper answer.
How you do this is going to depend on a lot of things. I'm simplifying because I don't know what your code actually looks like.
Here's what a basic outline of the code could look like:
private async Task ExecuteStepInCommunicator(TimeSpan expectedTimeToComplete)
{
// Maybe you want to allow for some buffer as #Oakley suggested.
// This will give it twice as long as the expected time to complete.
expectedTimeToComplete = expectedTimeToComplete.Add(expectedTimeToComplete);
using var alertTimer = new Timer()
{
Interval = expectedTimeToComplete.TotalMilliseconds,
AutoReset = false,
};
alertTimer.Elapsed += this.AlertTimer_Elapsed;
// Replace the Task.Delay with your normal 'Communicator' logic
await Task.Delay(1000);
// Make sure the timer is stopped once your logic has finished.
// If it hasn't run yet, no alert will be sent. If it has run this will do nothing
alertTimer.Stop();
}
private void AlertTimer_Elapsed(object sender, ElapsedEventArgs e)
{
// Send email from here. This will be called only once, when the timer elapses.
// https://learn.microsoft.com/en-us/dotnet/api/system.net.mail.smtpclient.send?view=net-5.0
}
System.Timer will not block your normal code from running. Once the timer finishes, the Elapsed callback will be run on a thread from the threadpool, so it can run even when your communicator code is also running.
Because this is rather simple, I'd suggest keeping it in the same place. You suggested sending off to some other application to do the timing and notifications, but that can add a lot of complexity (and chances for things to go wrong), for little gain.
I'm developing an app that starts a System.Threading.Timer which does some fairly rapid reading/writing to the serial port (every 100ms). The timer's callback method looks something like this:-
if (_timerTaskRunning)
{
Debug.WriteLine("still running");
return;
}
_timerTaskRunning = true;
... do the serial write/read here ...
_timerTaskRunning = false;
The _timerTaskRunning flag is a safeguard to ensure that the delegate doesn't run if the previous timer "cycle" hasn't finished, i.e. it's taking longer than 100ms.
When I first start the app I see around a dozen debug messages from the if statement. It'll then settle down but I see another group of messages 7 or 8 seconds later. It settles down again, and every once in a while I'll see a group of messages appear in varying numbers.
I'm assuming the first group of messages are caused by the timer delegate running slowly due to the app still starting up, objects/UI initialising, etc, etc, while subsequent messages are perhaps caused by garbage collection kicking in every so often and slowing things down? It's not the serial port because I see the same behaviour with a "mock" serial port.
I've tried delaying the timer's first run by a couple of seconds but it makes no difference - I still get a batch of debug messages for the first second or so after the timer starts. It's not the end of the world skipping some of the timer tasks but it would be interesting to know what might be causing them. Is there anything I can do to investigate the cause further, e.g. would perfmon shed any light on things? I haven't used it before so which counters would you suggest?
It sounds like you have a reentrancy problem. Basically your _timerTaskRunning isn't working as a safeguard likely due to race conditions.
Use System.Timers.Timer instead of System.Threading.Timer
Set Timer.AutoReset to false. This will fix your entrancing problem because it won't call you call back until you explicitly want it to.
Call Start() on your timer when you need it to go again (you may need to adjust the interval to account for execution time)
If you have multiple threads that can call start you will need to synchronize the calls to it.
Your _timerTaskRunning gets updated by several threads. You need to use locking to make it thread-safe.
However, I would not use a timer at all. I have implemented a non-renetrant timer here. It uses AutoResetEvent WaitOne with timeout which ensures non-reentry.
Background is the following: A Windows Service which is supposed to perform an action once per day at a given time.
I have currently implemented this by creating a timer and added the ElapsedEventHandler. The event fires every t minutes and it is then checked that we are passed the configured time. If so the action is performed and if not nothing happens.
A colleague asked me if it was not easier just to have a while(true) loop containing a sleep() and then of course the same logic for checking if we are past the time for action.
Question:
Can one say anything about the "robustness" of an event vs. a while(loop)? I am thinking of the situation where the thread "dies" so the while(true) loop exits. Is this more "likely" to happen in the one scenario vs. the other?
I'd vote for neither.
If your service just sits idle for an entire day periodically waking up (and paging code in) to see if "it's time to run", then this is a task better suited for the Windows Task Scheduler. You can programatically install a task to run every day through the task scheduler. Then your code doesn't need to be running at all unless it's time to run. (Or if your service does need to run in the background anyway, the task in the scheduler can signal your service to wake up instead of timer logic).
Both will be equally robust if you use proper error handling.
If you don't use proper error handling they will be equally brittle.
while(true)
{
...
Thread.Sleep(1000);
}
will make your service slow when responding to the standard service events like OnStop.
Besides, where do you put your while loop? In a separate thread? You will get more manual management if you use a loop too.
To summarize: use a timer.
I have an ASP.Net application which fires off some background workers using ThreadPool.QueueUserWorkItem(). Most of the time there will be zero background threads running, however it's possible that there could occationally be up to 5-6 executing at a time. Additionally, most will complete within a couple seconds, but it is possible that they could run for as long as 10 minutes.
Given all the different scenarios that could cause a shutdown of an ASP.Net application, I'd like to have these background processes exit gracefully if possible when the application needs to shutdown and they are in the middle of processing.
I don't anticipate having too much of an issue creating a way to signal the processes to stop their work early and clean up within a few seconds of getting the call that the application is being shut down. Although if anyone has specific advice on this I'd certainly appreciate it.
My two main questions are:
1) When would be the appropriate time to tell the background workers to wrap things up. During Application_End? Dispose()? Or perhaps a third option I'm not aware of.
2) If I wait during the above event for the processes to finish before returning, is there a point where not having Application_End return immediately could cause more serious issues for the application than not shutting down the background jobs nicely.
void Application_End(object sender, EventArgs e)
{
//if this for whatever reason takes a non-trivial time to execute, what problems could I encounter?
SignalBackgroundJobsAndWaitForThemToShutDown();
}
Thanks!
The appropriate time is on the Application_End
Now after you signal your background jobs you must wait for them at that point to end, and then let continue, as its seems that you do.
Have in mine, that is good to place a time out on the wait, not wait for ever or else your pool may have problems, or shutdown, then you need to check your pool setting on the max wait to shutdown the pool, there set a value larger than your wait, or just disabled it.
Also have in mine that if you have pool garden, (more than 1 working pools) then the Application_End is called one time for each pool.
I use the same technique on my sites, the same way I signal my threads to stop the same way, I just also log the final end, and check when they are final end, and also I force my routines to stop and not let them run for many time. The Application_End is called on recycle the pool, or when you open the app_offline.htm file, or when you go to close the web service. Personally I have disable the recycles, I do not need them and I only open the app_offline.htm when I make updates. At that moment I wait my pool threads to stop their work.
For some time now I've been having this recurrent nightmare (read - bug in my application). For some reason, a certain Timer continues to send "Elapsed" events after I stopped it, even though in the event itself the timer "admits" to having been disabled! Check this out:
//Timer is created in Class' Constructor. Class is not static.
public PDAAccess ()
{
ConnectionTimeoutChecker = new System.Timers.Timer(1000);
ConnectionTimeoutChecker.Elapsed += new System.Timers.ElapsedEventHandler(ConnectionTimeoutChecker_Elapsed);
}
void ConnectionTimeoutChecker_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{ //_DeviceConTimeout eventually reaches A LOT MORE than 10.
if (_DeviceConTimeout > 10)
{
ConnectionTimeoutChecker.Stop(); //This is invoked but the timer DOES NOT STOP.
if (OnDeviceSyncTimeout != null) OnDeviceSyncTimeout(this, null); //This gets fired all the time.
}
_DeviceConTimeout++; //This keeps increasing and increasing.
//Worth mentioning: sender = Timer, sender.Enabled = false (!) so then why is this executing?
}
As for where I start it: I start it in a single place, I put a breakpoint there and it doesn't execute more than once. And before you ask: no multiple threads are involved. I do work with threads in this application, but: the timer is not created in a thread, neither is the Class.
Yet the .Stop(); is executed 100 times and the timer still WON'T stop.
I'm completely at a loss here. This behavior is so strange to me, it gives me that embarrassed feeling that I might have been missing something super-obvious. Sometimes writing such a post helps me identify the problem before I hit the submit button (we all know the "Explaining to 3rd party" effect). But it didn't hit me yet so I'm gonna hit the button and... see what you see ::- D.
Shot in the dark: Perhaps the call to OnDeviceSyncTimeout() somehow indirectly causes the timer to be reactivated? I know you said it is started only in a single place, but I’m not really sure how you can be so certain about that.
Well, that doesn't make much sense with the code snippets you posted. But extraordinary problems like this require extraordinary explanations.
The System.Timers.Timer class is a horrid timer. The condition you see, getting the Elapsed event called while the timer is stopped, is inevitable in most any use for it. The problem is that it raises the Elapsed event by using ThreadPool.QueueUserWorkItem(). That thread is subject to the vagaries of the thread pool scheduler and the Windows thread scheduler. QUWI does not guarantee that the thread will run right away. Instead, the thread goes in a queue in the "ready to run" state. The TP scheduler removes it from that queue only when it deems the time right to run a thread. Which for one means that it will be stuck for a while if it already has as many threads running as your machine has CPU cores. Only when threads don't complete will it allow more threads to run. That takes a while, these scheduling decisions only run twice a second.
The Windows thread scheduler plays a role as well. When the machine is loaded, having many threads ready to run then it can take a while before the Elapsed thread gets a turn. What is especially nasty about this problem is that such a race is heavily dependent on what else is going on in your program or on the machine. Runs fine on your dev machine, malfunctions unpredictably in production when the Elapsed event finally manages to run, even though the timer got stopped a long time ago.
Also note that this can build up. You are particularly vulnerable to that because you stop the timer in the Elapsed event. When the TP threads just don't manage to get scheduled, the timer just keeps calling QUWI, addding more TP threads, without your code able to stop the timer.
Well, guesses, but it does explain the problem. Either the synchronous timer in your UI library or System.Thread.Timer should fix it. Do try to use the single-shot version of the latter. A time-out is a fixed time, it shouldn't have to be counted.