My application must ping a (lan) server every 30 seconds in order to let it know I'm alive.
This works fine, except for when a separate process I do not control uses a very large amount of system resources and hangs the entire machine while it does its thing. As a result, my timer tick is delayed until the other process finishes its heavy lifting, and by this time the server already believes I have died.
The other process is set at low priority, and my process is set at normal priority.
My first attempt used a System.Timers.Timer to initiate the ping.
In a second attempt, I tried to roll my own PriorityTimer, to run on a dedicated high priority thread, but that also seems to have little effect.
public sealed class PriorityTimer : IDisposable
{
private int interval;
private bool active;
private Thread timerThread;
private AutoResetEvent resetEvent;
public event EventHandler Elapsed;
public PriorityTimer(int interval)
{
this.interval = interval;
resetEvent = new AutoResetEvent(false);
}
public bool Enabled
{
get { return active; }
set
{
if (active != value)
{
active = value;
if (value)
{
timerThread = new Thread(Wait);
timerThread.IsBackground = true;
timerThread.Priority = ThreadPriority.Highest;
timerThread.Start();
}
else
{
resetEvent.Set();
}
}
}
}
private void Wait()
{
resetEvent.Reset();
while (!resetEvent.WaitOne(interval))
{
if (Elapsed != null)
{
Elapsed(this, EventArgs.Empty);
}
}
}
public void Dispose()
{
Enabled = false;
}
}
Assuming I can not force the offending process to play nice, what can I do to assure my timer elapses at 30 seconds?
this does seem to happen when huge amounts of memory is paged to disk.
You can either try to keep your own working set in memory, or force the other (rogue) process(es) to consume less. The latter strategy is more reliable.
Create a Windows Job Object and configure its maximum memory usage. Assign the problematic processes into the Job. Jobs are capable of limiting CPU usage and memory. They are meant to "jail" processes.
Any operating system that doesn't enforce real-time constraints will generally miss timer requests by some normal delta under contention. Have you been keeping track of your deltas?
Also, I would ask if you're using a garbage collected language. If you are, pauses will definitely skew timer results.
Finally, what's the timer interval you're using to send the heartbeats? As a general rule of thumb, you should send heartbeats twice every interval to account for timing variances.
Related
I've created a watchdog timer (using a System.Windows.Forms.Timer), which triggers if a long period of time expires without receiving a small packet of data:
using System.Windows.Forms;
public class Watchdog
{
private Timer Timer;
public void Go()
{
Timer.Start();
}
public void Reset()
{
Timer.Stop();
Timer.Start();
}
private void OnTimerExpired(object State)
{
Timer.Stop();
DoSomething();
}
public Watchdog()
{
Timer = new Timer();
Timer.Tick += new EventHandler(OnTimerExpired);
Timer.Interval = (1000 * Timeout);
}
}
The main code calls Go(), and then calls Reset() each time a packet is received. If the timer expires, OnTimerExpired() is called.
Since that there may be hundreds of packet receptions per second, and since the main job of the application is to respond to such packets, I'm beginning to wonder if resetting the timer isn't too CPU/OS intensive.
Any idea how calling Timer.Stop()/Timer.Start() this way may impact performance (in terms of latency)?
Use a simple timespan or integer variable as a flag. When the timer ticks, it checks against a Stopwatch object to see how much time has elapsed since the flag was last udpated. If it's longer than your timeout value you trigger your watchdog code.
Now, instead of resetting your timer, other code can just use the stopwatch to update your timespan flag value when a new packet comes in.
You should also either set your timer's tick interval to about 1/2 of what you want the actual timeout duration to be, or have code in the event to set your interval so your next tick event is just a few milliseconds after you would timeout if the connection was severed now. Otherwise you could end up waiting almost twice as long as the timeout duration in the situation where your last packet arrived very soon after a tick event.
Another option, by the way, is to just have a Boolean flag that's set whenever a message comes in. The timer event handler checks that flag and alerts if it's not set. So you have:
private bool GotAMessage = false;
void MessageReceived()
{
// happens whenever a message is received
GotAMessage = true;
}
void OnTimerExpired(object state)
{
if (!GotAMessage)
{
// didn't receive a message in time.
}
GotAMessage = false;
}
A simpler option is to call a method on the WatchDog class that updates a common lastPacketReceived value time a packet is received. Then you only need to start a single timer one time in the WatchDog class that ticks once per timeout interval and compares the current time to the lastPacketReceived value:
public static class WatchDog
{
static object locker = new object();
static long lastPacketReceived;
static Stopwatch stopWatch = new Stopwatch();
static long threshold = 5000;
static WatchDog()
{
Timer watchDogTimer = new Timer(1000);
watchDogTimer.Elapsed += new ElapsedEventHandler(watchDogTimer_Elapsed);
watchDogTimer.Start();
stopWatch.Start();
}
static void watchDogTimer_Elapsed(object sender, ElapsedEventArgs e)
{
lock (locker)
{
if ((stopWatch.ElapsedMilliseconds - lastPacketReceived) > threshold)
{
// threshold exceeded
}
}
}
public static void PacketReceived()
{
lock (locker)
{
lastPacketReceived = stopWatch.ElapsedMilliseconds;
}
}
}
Any idea how calling Timer.Stop()/Timer.Start()
This way may impact performance (in terms of latency)?
None
The amount of resources required to do this is unlikely to be measured. Unless you have a performance problem, don't try to solve a performance problem, at the very least use software to profile the software to see if its an actual problem.
I have a requirement for a timer that has the following behaviour:
Millisecond accuracy
I want the tick event handler to only be called once the current tick handler has completed (much like the winforms timer)
I want exceptions on the main UI thread not to be swallowed up by the thread timer so this requires Invoke/Send instead of BeginInvoke/Post
I've played around with CreateTimerQueueTimer and had some success but at the same time had problems with code reentrance and/or locks when deleting the timer.
I decided to create my own timer so that I could get a better idea of what is going on under the hood so that I can fix the locking and reentrance problems. My code seems to work fine leading me to believe that I may as well use it. Does it look sound?
I've put in a check if the timer is deleted to make sure that the deletion is complete before the timer can be created again. Does that look ok?
Note: I should say that I call timeBeginPeriod(1) and timeEndPeriod(1) inorder to achieve the millisecond accuracy.
(The following code is converted from vb.net to c#, so apologies for any missed mess-ups}
ETA: I've found a problem with it. If the timer is running at an interval of 1 millisecond, and I call, say, Change(300), it locks up # while (this.DeleteRequest). This
must be because the TimerLoop is in the this.CallbackDelegate.Invoke(null) call.
public class MyTimer : IDisposable
{
private System.Threading.TimerCallback CallbackDelegate;
private bool DeleteRequest;
private System.Threading.Thread MainThread;
public MyTimer(System.Threading.TimerCallback callBack)
{
this.CallbackDelegate = callBack;
}
public void Create(int interval)
{
while (this.DeleteRequest) {
System.Threading.Thread.Sleep(0);
}
if (this.MainThread != null) {
throw new Exception("");
}
this.MainThread = new System.Threading.Thread(TimerLoop);
// Make sure the thread is automatically killed when the app is closed.
this.MainThread.IsBackground = true;
this.MainThread.Start(interval);
}
public void Change(int interval)
{
// A lock required here?
if (!this.IsRunning()) {
throw new Exception("");
}
this.Delete();
this.Create(interval);
}
public void Delete()
{
this.DeleteRequest = true;
}
public bool IsRunning()
{
return (this.MainThread != null) && this.MainThread.IsAlive;
}
private void TimerLoop(object args)
{
int interval = (int)args;
Stopwatch sw = new Stopwatch();
sw.Start();
do {
if (this.DeleteRequest) {
this.MainThread = null;
this.DeleteRequest = false;
return;
}
long t1 = sw.ElapsedMilliseconds;
// I want to wait until the operation completes, so I use Invoke.
this.CallbackDelegate.Invoke(null);
if (this.DeleteRequest) {
this.MainThread = null;
this.DeleteRequest = false;
return;
}
long t2 = sw.ElapsedMilliseconds;
int temp = Convert.ToInt32(Math.Max(interval - (t2 - t1), 0));
sw.Reset();
if (temp > 0) {
System.Threading.Thread.Sleep(temp);
}
sw.Start();
} while (true);
}
// The dispose method calls this.Delete()
}
I would recommend using p/Invoke and using the timers from Win32's Timer Queues:
http://msdn.microsoft.com/en-us/library/ms686796(v=vs.85).aspx
One should be mindful that the managed CLR environment has a lot of non-determinism built into it, garbage collection, for instance. Just because your timer is has a period of 1 millisecond doesn't mean that that is necessarily what happens.
Also, the documentation doesn't mention it, but the callback invoked by the timer must be pinned in memory and not garbage collectable, via a GCHandle or other construct. When a timer (or timers, if you kill off a timer queue), the callback will be executed one last time. Not sure whether that happens by the internal wait expiring, or by signalling the internal event handle.
Execution of DeleteTimerQueueTimer() and DeleteTimerQueueEx() can be made synchronous, so they won't return until all timers have signalled and invoked their last callback, but doing that would be suboptimal.
If you don't pin the callbacks and prevent them from being garbage-collected, things will go swimmingly...most of the time. You'll encounter random exceptions.
Also, the callback should be smart enough to bail out if the timer is being deleted, lest it make reference to something that's already GC'd.
μTimer would be a better example!
You can find it here # https://stackoverflow.com/questions/15725711/obtaining-microsecond-precision-using-net-without-platform-invoke?noredirect=1#comment22341931_15725711
It provides accurate wait times down to 1µs and possibly lower depending on your NIC!
Let me know if you need anything else!
Summary:
Within a Windows service & Console Application I am calling a common library that contains a Timer that periodically triggers an action that takes around 30 seconds to complete. This works fine, however...
When a service stop or application exit is called and the timer is in the ElapsedEventHandler I need the service stop/application exit to wait until the event handler has completed.
I have implemented this functionality by having a Boolean InEvent property that is checked when the timer stop method is called.
While this is functional, the question is: Is this the best way to go about doing this? Is there an alternative approach that may serve this purpose better?
The other issue is that I need to avoid the service stop request failing with a "Service failed to respond to stop request"
This is my implementation
public sealed class TimedProcess : IDisposable
{
static TimedProcess singletonInstance;
bool InEvent;
Timer processTimer;
private TimedProcess()
{
}
public static TimedProcess Instance
{
get
{
if (singletonInstance == null)
{
singletonInstance = new TimedProcess();
}
return singletonInstance;
}
}
public void Start(double interval)
{
this.processTimer = new Timer();
this.processTimer.AutoReset = false;
this.processTimer.Interval = interval;
this.processTimer.Elapsed += new ElapsedEventHandler(this.processTimer_Elapsed);
this.processTimer.Enabled = true;
}
public void Stop()
{
if (processTimer != null)
{
while (InEvent)
{
}
processTimer.Stop();
}
}
void processTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
InEvent = true;
// Do something here that takes ~30 seconds
}
catch
{
}
finally
{
InEvent = false;
processTimer.Enabled = true;
}
}
public void Dispose()
{
if (processTimer != null)
{
Stop();
processTimer.Dispose();
}
}
}
And this is how it is called in the service OnStart / console application main:
TimedProcess.Instance.Start(1000);
This is how it is called in service OnStop and application main (pending keypress):
TimedProcess.Instance.Stop();
Probably the easiest and most reliable way is to use a Monitor. Create an object that the main program and the timer callback can access:
private object _timerLock = new object();
Your main program tries to lock that before shutting down:
// wait for timer process to stop
Monitor.Enter(_timerLock);
// do shutdown tasks here
And your timer callback locks it, too:
void processTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if (!Monitor.TryEnter(_timerLock))
{
// something has the lock. Probably shutting down.
return;
}
try
{
// Do something here that takes ~30 seconds
}
finally
{
Monitor.Exit(_timerLock);
}
}
The main program should never release the lock once it's obtained it.
If you want the main program to go ahead and shut down after some period of time, regardless of whether it's obtained the lock, use Monitor.TryEnter. For example, this will wait 15 seconds.
bool gotLock = Monitor.TryEnter(_timerLock, TimeSpan.FromSeconds(15));
The return value is true if it was able to obtain the lock.
By the way, I strongly suggest that you use System.Threading.Timer rather than System.Timers.Timer. The latter squashes exceptions, which can end up hiding bugs. If an exception occurs in your Elapsed event, it will never escape, meaning that you never know about it. See my blog post for more information.
EDIT
Each callback to the System.Timers.Timer is queued on the ThreadPool. Be aware that the System.Timers.Timer can have a race condition (you can read more about it here.) System.Threading.Timer is a slightly nicer wrapper which I prefer to use due to it's simplicity.
You haven't described enough details to know if your particular application could handle that race condition, so it's hard to tell. But given your code, it is possible that there might be a callback queued up for processTimer_Elapsed after Stop() is called.
For the service timeout issue --
One way to do this is to make a call to the ServiceController method WaitForStatus with a timeout. I've done this in the past and it works reasonably well, although I recall there being some edge cases around waiting for a very long time.
See the MSDN reference. A sample use is described here.
One possible alternative seems to be to not do the actual work in the timer callback itself but to just queue a work item from there on the tread pool to do the work. Then you can go ahead and dispose of the timer - anything currently running on the thread pool will remain operational, and your service can respond to the stop request immediately but the thread pool item (if queued) will still get processed.
I want to know a proper way to start and stop a threaded job forced and unforced. Is this the proper way to stop a Thread?
public class ProcessDataJob : IJob
{
private ConcurrentQueue<byte[]> _dataQueue = new ConcurrentQueue<byte[]>();
private volatile bool _stop = false;
private volatile bool _forceStop = false;
private Thread _thread;
private int _timeOut = 1000;
public void Start()
{
_stop = false;
_forceStop = false;
_thread = new Thread(ProcessData);
_thread.Start();
}
private void ProcessData()
{
while (!_stop || _dataQueue.Count > 0)
{
if(_forceStop) return;
byte[] data;
if(_dataQueue.TryDequeue(data))
{
//Process data
//.....//
}
}
}
public void Stop(bool force)
{
_stop = true;
_forceStop = force;
_thread.Join(_timeOut);
}
public void Enqueue(byte[] data)
{
_dataQueue.Enqueue(data);
}
}
There is no proper way to forcibly kill a thread.
There are several ways to do it, but none of them are proper.
Forcibly killing a thread is what you should do only if you need to terminate the program, or unload the appdomain containing the thread, and don´t care about any data structures left dangling in a corrupted/bad/locked state, because they will be gone in a short while as well.
There´s plenty of advice on the internet about how bad/evil Thread.Abort is, so don´t do it.
Instead, write proper cooperative threading. The thread(s) should themselves check a flag (event, volatile bool field, etc.) and then voluntairly exit when nicely asked to do so.
That is the proper way.
This is the way that I've done it in the past, with one difference. I've had threads unexpectedly hang in the past, which means that that loop of yours doesn't come back with an answer. To address this I have all thread using classes, like yours, register with a 'manager' class who is responsible for participating in key things like Forced Stop. The threaded class has a reference to the manager, and when a forced stop is done it calls a method on the manager which effectively is a timer. If by the time the timer has gone off the threaded class hasn't set a state flag to STOPPED then the manager calls abort on it.
The key thing for me was not just the calling 'stop' but the confirmation that 'stop' had occurred, and understanding that it will be a non-deterministic amount of time but that 'after a reasonable amount of time' that I should give up and move on.
You could use the .NET ThreadPool class, this way you don't have to handle the stack yourself.
Let's say I have an existing System.Threading.Timer instance and I'd like to call Change on it to push it's firing time back:
var timer = new Timer(DelayCallback, null, 10000, Timeout.Infinite);
// ... (sometime later but before DelayCallback has executed)
timer.Change(20000, Timeout.Infinite);
I'm using this timer to perform an "idle callback" after a period of no activity. ("Idle" and "no activity" are application-defined conditions in this case...the specifics aren't terribly important.) Every time I perform an "action", I want to reset the timer so that it is always set to fire 10 seconds after that.
However, there is an inherent race condition because when I call Change, I can't tell if the Timer has already fired based on its old settings. (I can, of course, tell if my callback has happened but I can't tell if the CLR's internal timer thread has queued my callback to the threadpool and its execution is imminent.)
Now I know I can call Dispose on the timer instance and re-create it each time I need to "push it back". but this seems less efficient than just changing the existing timer. Of course it may not be...I'll run some micro-benchmarks in a bit and let you all know.
Alternatively, I can always keep track of the expected firing time (via DateTime.Now.AddSeconds(10)) and, if the original Timer fires, ignore it by checking DateTime.Now in the callback. (I have a nagging concern that this may not be 100% reliable on account of the Timer using TimeSpan and my check using DateTime...this may not be an issue but I'm not completely comfortable with it for some reason...)
My questions are:
Is there a good way for me to call Timer.Change and be able to know whether I managed to change it before the callback was queued to the threadpool? (I don't think so, but it doesn't hurt to ask...)
Has anyone else implemented (what I term) a "pushback timer" like this? If so, I'd love to hear how you tackled the problem.
This question is somewhat hypothetical in nature since I already have a couple of working solutions (based on Dispose and based on DateTime.Now)...I'm mainly interested in hearing performance-related suggestions (as I'll be "pushing back" the Timer VERY frequently).
Thanks!
it sounds like what you really want is the application-idle event
System.Windows.Forms.Application.Idle
Im interpreting your questions as a request for an implementatation of the IdleNotifier interface specified below. Also you state that ActionOccured() needs to be fast.
public delegate void IdleCallback();
public interface IdleNotifier
{
// Called by threadpool when more than IdleTimeSpanBeforeCallback
// has passed since last call on ActionOccured.
IdleCallback Callback { set; }
TimeSpan IdleTimeSpanBeforeCallback { set; }
void ActionOccured();
}
I provide an implementation with System.Threading.Timer below.
Important points about the implementation:
We accept that the timer can wake up at any time and make sure this is ok.
Since we assume the timer wakes relatively seldom we can do expensive work at these times.
Since we can do all logic in the timer callback all we need to do to "push the timer" is to remeber when last we pushed it.
Implementation:
public class IdleNotifierTimerImplementation : IdleNotifier
{
private readonly object SyncRoot = new object();
private readonly Timer m_Timer;
private IdleCallback m_IdleCallback = null;
private TimeSpan m_IdleTimeSpanBeforeEvent = TimeSpan.Zero;
// Null means there has been no action since last idle notification.
private DateTime? m_LastActionTime = null;
public IdleNotifierTimerImplementation()
{
m_Timer = new Timer(OnTimer);
}
private void OnTimer(object unusedState)
{
lock (SyncRoot)
{
if (m_LastActionTime == null)
{
m_Timer.Change(m_IdleTimeSpanBeforeEvent, TimeSpan.Zero);
return;
}
TimeSpan timeSinceLastUpdate = DateTime.UtcNow - m_LastActionTime.Value;
if (timeSinceLastUpdate > TimeSpan.Zero)
{
// We are no idle yet.
m_Timer.Change(timeSinceLastUpdate, TimeSpan.Zero);
return;
}
m_LastActionTime = null;
m_Timer.Change(m_IdleTimeSpanBeforeEvent, TimeSpan.Zero);
}
if (m_IdleCallback != null)
{
m_IdleCallback();
}
}
// IdleNotifier implementation below
public void ActionOccured()
{
lock (SyncRoot)
{
m_LastActionTime = DateTime.UtcNow;
}
}
public IdleCallback Callback
{
set
{
lock (SyncRoot)
{
m_IdleCallback = value;
}
}
}
public TimeSpan IdleTimeSpanBeforeCallback
{
set
{
lock (SyncRoot)
{
m_IdleTimeSpanBeforeEvent = value;
// Run OnTimer immediately
m_Timer.Change(TimeSpan.Zero, TimeSpan.Zero);
}
}
}
}
There are many straight-forward performance improvements on this code.
If anyone would be intrested in my first thoughts on this just ask me.
I've actually had to build my own "Timing" class for an MMORPG I've made. It could keep track of over 100,000 "entities" that had timers for processing AI, and other tasks. Based on different actions that could be taken, I would have to momentarily delay an event.
Now, my timing class was completely hand written, so it won't be exactly what you're looking for. But something that you could do that would be similar to the solution I came up with is to do a sort of:
while (sleepyTime > 0)
{
int temp = sleepyTime;
sleepyTime = 0;
Thread.Sleep(temp);
}
// here's where your actual code is.
Then, you can make a "Delay" method that basically just ads to sleepyTime.