We Keep Coding

Enforcing process/thread priority in the face of resource starvation

My application must ping a (lan) server every 30 seconds in order to let it know I'm alive.
This works fine, except for when a separate process I do not control uses a very large amount of system resources and hangs the entire machine while it does its thing. As a result, my timer tick is delayed until the other process finishes its heavy lifting, and by this time the server already believes I have died.
The other process is set at low priority, and my process is set at normal priority.
My first attempt used a System.Timers.Timer to initiate the ping.
In a second attempt, I tried to roll my own PriorityTimer, to run on a dedicated high priority thread, but that also seems to have little effect.
public sealed class PriorityTimer : IDisposable
{
private int interval;
private bool active;
private Thread timerThread;
private AutoResetEvent resetEvent;
public event EventHandler Elapsed;
public PriorityTimer(int interval)
{
this.interval = interval;
resetEvent = new AutoResetEvent(false);
}
public bool Enabled
{
get { return active; }
set
{
if (active != value)
{
active = value;
if (value)
{
timerThread = new Thread(Wait);
timerThread.IsBackground = true;
timerThread.Priority = ThreadPriority.Highest;
timerThread.Start();
}
else
{
resetEvent.Set();
}
}
}
}
private void Wait()
{
resetEvent.Reset();
while (!resetEvent.WaitOne(interval))
{
if (Elapsed != null)
{
Elapsed(this, EventArgs.Empty);
}
}
}
public void Dispose()
{
Enabled = false;
}
}
Assuming I can not force the offending process to play nice, what can I do to assure my timer elapses at 30 seconds?

this does seem to happen when huge amounts of memory is paged to disk.
You can either try to keep your own working set in memory, or force the other (rogue) process(es) to consume less. The latter strategy is more reliable.
Create a Windows Job Object and configure its maximum memory usage. Assign the problematic processes into the Job. Jobs are capable of limiting CPU usage and memory. They are meant to "jail" processes.

Any operating system that doesn't enforce real-time constraints will generally miss timer requests by some normal delta under contention. Have you been keeping track of your deltas?
Also, I would ask if you're using a garbage collected language. If you are, pauses will definitely skew timer results.
Finally, what's the timer interval you're using to send the heartbeats? As a general rule of thumb, you should send heartbeats twice every interval to account for timing variances.

Monitor, lock or volatile?

I have a windows service (.NET 4) that periodically processes a queue, for example every 15 minutes. I use a System.Threading.Timer which is set when the service starts to fire a callback every X milliseconds. Typically each run takes seconds and never collides, but what if I could not assume that - then I want the next run to exit at once if processing is in progress.
This is easily solved with lock, volatile bool or a monitor, but what is actually the appropriate to use in this scenario, or simply the preferred option in general?
I've found other posts that answers almost this scenario (like Volatile vs. Interlocked vs. lock) but need some advice on extending this to a Timer example with immediate exit.

You don't need any locks for this, you should just reschedule next timer execution from within the timer delegate. That should ensure 100% no overlaps.
At the end of timer's event handler call timer.Change(nextRunInMilliseconds, Timeout.Infinite), that way the timer will fire only once, after nextRunInMilliseconds.
Example:
//Object that holds timer state, and possible additional data
private class TimerState
{
public Timer Timer { get; set; }
public bool Stop { get; set; }
}
public void Run()
{
var timerState = new TimerState();
//Create the timer but don't start it
timerState.Timer = new Timer(OnTimer, timerState, Timeout.Infinite, Timeout.Infinite);
//Start the timer
timerState.Timer.Change(1000, Timeout.Infinite);
}
public void OnTimer(object state)
{
var timerState = (TimerState) state;
try
{
//Do work
}
finally
{
//Reschedule timer
if (!timerState.Stop)
timerState.Timer.Change(1000, Timeout.Infinite);
}
}

Well, any of them will do the job. Monitor is usually pretty simple to use via lock, but you can't use lock in this case because you need to specify a zero timeout; as such, the simplest approach is probably a CompareExchange:
private int isRunning;
...
if(Interlocked.CompareExchange(ref isRunning, 1, 0) == 0) {
try {
// your work
} finally {
Interlocked.Exchange(ref isRunning, 0);
}
}
to do the same with Monitor is:
private readonly object syncLock = new object();
...
bool lockTaken = false;
try {
Monitor.TryEnter(syncLock, 0, ref lockTaken);
if (lockTaken) {
// your work
}
} finally {
if(lockTaken) Monitor.Exit(syncLock);
}

I think, that if you find that you need to synchronize timer delegate - you are doing it wrong, and Timer is probably not the class you want to use. Imho its better to :
1) either keep the Timer, but increase the interval value to the point, where its safe to assume, that there will be no issues with threading,
2) or remove Timer and use simple Thread instead. You know, something like:
var t = new Thread();
t.Start(() =>
{
while (!_stopEvent.WaitOne(100))
{
..........
}
});

Waiting for a timer elapsed event to complete before application/service closes/stops

Summary:
Within a Windows service & Console Application I am calling a common library that contains a Timer that periodically triggers an action that takes around 30 seconds to complete. This works fine, however...
When a service stop or application exit is called and the timer is in the ElapsedEventHandler I need the service stop/application exit to wait until the event handler has completed.
I have implemented this functionality by having a Boolean InEvent property that is checked when the timer stop method is called.
While this is functional, the question is: Is this the best way to go about doing this? Is there an alternative approach that may serve this purpose better?
The other issue is that I need to avoid the service stop request failing with a "Service failed to respond to stop request"
This is my implementation
public sealed class TimedProcess : IDisposable
{
static TimedProcess singletonInstance;
bool InEvent;
Timer processTimer;
private TimedProcess()
{
}
public static TimedProcess Instance
{
get
{
if (singletonInstance == null)
{
singletonInstance = new TimedProcess();
}
return singletonInstance;
}
}
public void Start(double interval)
{
this.processTimer = new Timer();
this.processTimer.AutoReset = false;
this.processTimer.Interval = interval;
this.processTimer.Elapsed += new ElapsedEventHandler(this.processTimer_Elapsed);
this.processTimer.Enabled = true;
}
public void Stop()
{
if (processTimer != null)
{
while (InEvent)
{
}
processTimer.Stop();
}
}
void processTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
InEvent = true;
// Do something here that takes ~30 seconds
}
catch
{
}
finally
{
InEvent = false;
processTimer.Enabled = true;
}
}
public void Dispose()
{
if (processTimer != null)
{
Stop();
processTimer.Dispose();
}
}
}
And this is how it is called in the service OnStart / console application main:
TimedProcess.Instance.Start(1000);
This is how it is called in service OnStop and application main (pending keypress):
TimedProcess.Instance.Stop();

Probably the easiest and most reliable way is to use a Monitor. Create an object that the main program and the timer callback can access:
private object _timerLock = new object();
Your main program tries to lock that before shutting down:
// wait for timer process to stop
Monitor.Enter(_timerLock);
// do shutdown tasks here
And your timer callback locks it, too:
void processTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if (!Monitor.TryEnter(_timerLock))
{
// something has the lock. Probably shutting down.
return;
}
try
{
// Do something here that takes ~30 seconds
}
finally
{
Monitor.Exit(_timerLock);
}
}
The main program should never release the lock once it's obtained it.
If you want the main program to go ahead and shut down after some period of time, regardless of whether it's obtained the lock, use Monitor.TryEnter. For example, this will wait 15 seconds.
bool gotLock = Monitor.TryEnter(_timerLock, TimeSpan.FromSeconds(15));
The return value is true if it was able to obtain the lock.
By the way, I strongly suggest that you use System.Threading.Timer rather than System.Timers.Timer. The latter squashes exceptions, which can end up hiding bugs. If an exception occurs in your Elapsed event, it will never escape, meaning that you never know about it. See my blog post for more information.

EDIT
Each callback to the System.Timers.Timer is queued on the ThreadPool. Be aware that the System.Timers.Timer can have a race condition (you can read more about it here.) System.Threading.Timer is a slightly nicer wrapper which I prefer to use due to it's simplicity.
You haven't described enough details to know if your particular application could handle that race condition, so it's hard to tell. But given your code, it is possible that there might be a callback queued up for processTimer_Elapsed after Stop() is called.
For the service timeout issue --
One way to do this is to make a call to the ServiceController method WaitForStatus with a timeout. I've done this in the past and it works reasonably well, although I recall there being some edge cases around waiting for a very long time.
See the MSDN reference. A sample use is described here.

One possible alternative seems to be to not do the actual work in the timer callback itself but to just queue a work item from there on the tread pool to do the work. Then you can go ahead and dispose of the timer - anything currently running on the thread pool will remain operational, and your service can respond to the stop request immediately but the thread pool item (if queued) will still get processed.

System.Threading.Timer keep reference to it

According to [http://msdn.microsoft.com/en-us/library/system.threading.timer.aspx][1] you need to keep a reference to a System.Threading.Timer to prevent it from being disposed.
I've got a method like this:
private void Delay(Action action, Int32 ms)
{
if (ms <= 0)
{
action();
}
System.Threading.Timer timer = new System.Threading.Timer(
(o) => action(),
null,
ms,
System.Threading.Timeout.Infinite);
}
Which I don't think keeps a reference to the timer, I've not seen any problems so far, but that's probably because the delay periods used have been pretty small.
Is the code above wrong? And if it is, how to I keep a reference to the Timer? I'm thinking something like this might work:
class timerstate
{
internal volatile System.Threading.Timer Timer;
};
private void Delay2(Action action, Int32 ms)
{
if (ms <= 0)
{
action();
}
timerstate state = new timerstate();
lock (state)
{
state.Timer = new System.Threading.Timer(
(o) =>
{
lock (o)
{
action();
((timerstate)o).Timer.Dispose();
}
},
state,
ms,
System.Threading.Timeout.Infinite);
}
The locking business is so I can get the timer into the timerstate class before the delegate gets invoked. It all looks a little clunky to me. Perhaps I should regard the chance of the timer firing before it's finished constructing and assigned to the property in the timerstace instance as negligible and leave the locking out.

Your second approach wouldn't keep the reference either. After the end of the Delay2-block, the reference to state is gone so the Garbage Collector will collect it ... then your reference to Timer is gone, too and it will be collected and disposed.
class MyClass
{
private System.Threading.Timer timer;
private void Delay(Action action, Int32 ms)
{
if (ms <= 0)
{
action();
}
timer = new System.Threading.Timer(
(o) => action(),
null,
ms,
System.Threading.Timeout.Infinite);
}
}

Update
Thinking about your problem a bit more generally, I think what you're actually trying to accomplish here is achievable in a much simpler way, without using a System.Threading.Timer at all.
Is this basically what you want your method to do? Perform action after a specified number of milliseconds? If so, I would suggest something like the following alternative implementation instead:
private void Delay(Action action, int ms)
{
if (ms <= 0)
{
action();
return;
}
System.Threading.WaitCallback delayed = state =>
{
System.Threading.Thread.Sleep(ms);
action();
};
System.Threading.ThreadPool.QueueUserWorkItem(delayed);
}
...by the way, are you aware that in the code you posted, specifying a non-zero value for ms will cause action to be executed twice?
Original Answer
The timerstate class really isn't necessary. Just add a System.Threading.Timer member to whatever class contains your Delay method; then your code should look like this:
public class Delayer
{
private System.Threading.Timer _timer;
private void Delay(Action action, Int32 ms)
{
if (ms <= 0)
{
action();
}
_timer = new System.Threading.Timer(
(o) => action(),
null,
ms,
System.Threading.Timeout.Infinite);
}
}
Now, I see that you are specifying the period argument of the timer's constructor as System.Threading.Timeout.Infinite (-1). What this means is that you intend for your timer to call action once, after ms has elapsed; am I right? If this is the case, then there's actually not much need to worry about the timer being disposed anyway (i.e., it will be, and that's fine), assuming a relatively low value for ms.
Anyway, if you're going to hold onto an instance of an IDisposable object (like System.Threading.Timer), you should generally dispose of that member when your object (i.e., this instance) is disposed of. I believe System.Threading.Timer has a finalizer that will cause it to be disposed of eventually anyway, but it's best to dispose of things as soon as you don't need them anymore. So:
public class Delayer : IDisposable
{
// same code as above, plus...
public void Dispose()
{
_timer.Dispose();
}
}

I read from your comments to the existing answers that you can have 0..n Actions and so you would have 0..n Timers, too. Is that right? In this case you should do one of the following:
Keep a List/Dictionary of timers, but in this case you have to remove the timer after firing.
Build a scheduler: have 1 Timer, that fired regulary, for each Delay-call add the action an the calculated time when it should run into a List/Dictionary, each time the timer fired, check you list and run&remove the Action. You can even build this scheduler that it sorts the Actions by execution time an sets the Timer to an adequate interval.

The code "working" is indeed a side-effect of a non-deterministic Garbage Collection / finalizers.
This code, running in LINQ Pad as C# Statements, shows the issue - no messages will be logged because the Timer is GC'ed (and the finalizer is called and it cleans up the internal timer resources..)
new System.Threading.Timer((o) => { "Hi".Dump(); }, this, 100, 100);
GC.Collect();
Thread.Sleep(2000);
However, comment out the "GC.Collect" statement and messages will be logged for 2 seconds as Garbage Collection is not [immediately] performed the Timer's finalizer is not called prior to the program ending.
Since the behavior is non-deterministic, it should also be considered a bug to rely on :}
The same issue exists in the follow on code because a strong reference is required to ensure an object is not GC'ed - in that example, there is still no reference kept to the timer wrapper object so the same issue exists, albeit with one more level of indirection..

"Atomically" changing a System.Threading.Timer

Let's say I have an existing System.Threading.Timer instance and I'd like to call Change on it to push it's firing time back:
var timer = new Timer(DelayCallback, null, 10000, Timeout.Infinite);
// ... (sometime later but before DelayCallback has executed)
timer.Change(20000, Timeout.Infinite);
I'm using this timer to perform an "idle callback" after a period of no activity. ("Idle" and "no activity" are application-defined conditions in this case...the specifics aren't terribly important.) Every time I perform an "action", I want to reset the timer so that it is always set to fire 10 seconds after that.
However, there is an inherent race condition because when I call Change, I can't tell if the Timer has already fired based on its old settings. (I can, of course, tell if my callback has happened but I can't tell if the CLR's internal timer thread has queued my callback to the threadpool and its execution is imminent.)
Now I know I can call Dispose on the timer instance and re-create it each time I need to "push it back". but this seems less efficient than just changing the existing timer. Of course it may not be...I'll run some micro-benchmarks in a bit and let you all know.
Alternatively, I can always keep track of the expected firing time (via DateTime.Now.AddSeconds(10)) and, if the original Timer fires, ignore it by checking DateTime.Now in the callback. (I have a nagging concern that this may not be 100% reliable on account of the Timer using TimeSpan and my check using DateTime...this may not be an issue but I'm not completely comfortable with it for some reason...)
My questions are:
Is there a good way for me to call Timer.Change and be able to know whether I managed to change it before the callback was queued to the threadpool? (I don't think so, but it doesn't hurt to ask...)
Has anyone else implemented (what I term) a "pushback timer" like this? If so, I'd love to hear how you tackled the problem.
This question is somewhat hypothetical in nature since I already have a couple of working solutions (based on Dispose and based on DateTime.Now)...I'm mainly interested in hearing performance-related suggestions (as I'll be "pushing back" the Timer VERY frequently).
Thanks!

it sounds like what you really want is the application-idle event
System.Windows.Forms.Application.Idle

Im interpreting your questions as a request for an implementatation of the IdleNotifier interface specified below. Also you state that ActionOccured() needs to be fast.
public delegate void IdleCallback();
public interface IdleNotifier
{
// Called by threadpool when more than IdleTimeSpanBeforeCallback
// has passed since last call on ActionOccured.
IdleCallback Callback { set; }
TimeSpan IdleTimeSpanBeforeCallback { set; }
void ActionOccured();
}
I provide an implementation with System.Threading.Timer below.
Important points about the implementation:
We accept that the timer can wake up at any time and make sure this is ok.
Since we assume the timer wakes relatively seldom we can do expensive work at these times.
Since we can do all logic in the timer callback all we need to do to "push the timer" is to remeber when last we pushed it.
Implementation:
public class IdleNotifierTimerImplementation : IdleNotifier
{
private readonly object SyncRoot = new object();
private readonly Timer m_Timer;
private IdleCallback m_IdleCallback = null;
private TimeSpan m_IdleTimeSpanBeforeEvent = TimeSpan.Zero;
// Null means there has been no action since last idle notification.
private DateTime? m_LastActionTime = null;
public IdleNotifierTimerImplementation()
{
m_Timer = new Timer(OnTimer);
}
private void OnTimer(object unusedState)
{
lock (SyncRoot)
{
if (m_LastActionTime == null)
{
m_Timer.Change(m_IdleTimeSpanBeforeEvent, TimeSpan.Zero);
return;
}
TimeSpan timeSinceLastUpdate = DateTime.UtcNow - m_LastActionTime.Value;
if (timeSinceLastUpdate > TimeSpan.Zero)
{
// We are no idle yet.
m_Timer.Change(timeSinceLastUpdate, TimeSpan.Zero);
return;
}
m_LastActionTime = null;
m_Timer.Change(m_IdleTimeSpanBeforeEvent, TimeSpan.Zero);
}
if (m_IdleCallback != null)
{
m_IdleCallback();
}
}
// IdleNotifier implementation below
public void ActionOccured()
{
lock (SyncRoot)
{
m_LastActionTime = DateTime.UtcNow;
}
}
public IdleCallback Callback
{
set
{
lock (SyncRoot)
{
m_IdleCallback = value;
}
}
}
public TimeSpan IdleTimeSpanBeforeCallback
{
set
{
lock (SyncRoot)
{
m_IdleTimeSpanBeforeEvent = value;
// Run OnTimer immediately
m_Timer.Change(TimeSpan.Zero, TimeSpan.Zero);
}
}
}
}
There are many straight-forward performance improvements on this code.
If anyone would be intrested in my first thoughts on this just ask me.

I've actually had to build my own "Timing" class for an MMORPG I've made. It could keep track of over 100,000 "entities" that had timers for processing AI, and other tasks. Based on different actions that could be taken, I would have to momentarily delay an event.
Now, my timing class was completely hand written, so it won't be exactly what you're looking for. But something that you could do that would be similar to the solution I came up with is to do a sort of:
while (sleepyTime > 0)
{
int temp = sleepyTime;
sleepyTime = 0;
Thread.Sleep(temp);
}
// here's where your actual code is.
Then, you can make a "Delay" method that basically just ads to sleepyTime.