Using lock statement within a loop in C#

Using lock statement within a loop in C# - c#

Lets take the sample class SomeThread where we are attempting to prevent the DoSomething methods from being called after the Running property is set to false and Dispose is called by the OtherThread class because if they are called after the Dispose method is the world would end as we know it.
It feels like there is a chance for something evil to happen because of the loop. That at the point where it starts the next loop and before the lock is taken before calling the DoSomething methods, Running could be changed to false, and Disposed called before it hits the lock. In this scenario life would not be good.
I was looking at ways to handle this when using a loop in a simple easy to maintain method. For the record I did considered the Double Lock Check patterned, however it is does not seem to be recommend for C#.
Warning: This is a simplified example to try to make it easy to focus on the issue with the loop and locking within one. If I didn't elaborate enough some place please let me know and I will do my best to fill in any details.
public class SomeThread : IDisposable
{
private object locker = new object();
private bool running = false;
public bool Running
{
get
{
lock(locker)
{
return running;
}
}
set
{
lock(locker)
{
running = value;
}
}
}
public void Run()
{
while (Running)
{
lock(locker)
{
DoSomething1();
DoSomething2();
}
}
}
private void DoSomething1()
{
// something awesome happens here
}
private void DoSomething2()
{
// something more awesome happens here
}
public void Dispose()
{
lock (locker)
{
Dispose1();
Dispose2();
}
}
private void Dispose1()
{
// something awesome happens here
}
private void Dispose2()
{
// something more awesome happens here
}
}
public class OtherThread
{
SomeThread st = new SomeThread();
public void OnQuit()
{
st.Running = false;
st.Dispose();
Exit();
}
}

Take a step back.
Start by specifying all the desirable and undesirable characteristics before you start to write a solution. A few that come immediately to mind:
The "work" is done on thread W. The "UI" is done on thread U.
The work is done in "units of work". Each unit of work is "short" in duration, for some definition of "short". Let's call the method that does the work M().
The work is done continuously by W, in a loop, until U tells it to stop.
U calls a cleanup method, D(), when all the work is done.
D() must not ever run before or while M() is running.
Exit() must be called after D(), on thread U.
U must never block for a "long" time; it is acceptable for it to block for a "short" time.
No deadlocks, and so on.
Does this sum up the problem space?
First off, I note that it seems at first glance that the problem is that U must be the caller of D(). If W were the caller of D(), then you wouldn't have to worry; you'd just signal W to break out of the loop, and then W would call D() after the loop. But that just trades one problem for another; presumably in this scenario, U must wait for W to call D() before U calls Exit(). So moving the call to D() from U to W doesn't actually make the problem easier.
You've said that you don't want to use double-checked locking. You should be aware that as of CLR v2, the double-checked locking pattern is known to be safe. The memory model guarantees were strengthened in v2. So it is probably safe for you to use double-checked locking.
UPDATE: You asked for information on (1) why is double-checked locking safe in v2 but not in v1? and (2) why did I use the weasel-word "probably"?
To understand why double-checked locking is unsafe in the CLR v1 memory model but safe in the CLR v2 memory model, read this:
http://web.archive.org/web/20150326171404/https://msdn.microsoft.com/en-us/magazine/cc163715.aspx
I said "probably" because as Joe Duffy wisely says:
once you venture even slightly outside
of the bounds of the few "blessed"
lock-free practices [...] you are
opening yourself up to the worst kind
of race conditions.
I do not know if you are planning on using double-checked locking correctly, or if you're planning on writing your own clever, broken variation on double-checked locking that in fact dies horribly on IA64 machines. Hence, it will probably work for you, if your problem is actually amenable to double checked locking and you write the code correctly.
If you care about this you should read Joe Duffy's articles:
http://www.bluebytesoftware.com/blog/2006/01/26/BrokenVariantsOnDoublecheckedLocking.aspx
and
http://www.bluebytesoftware.com/blog/2007/02/19/RevisitedBrokenVariantsOnDoubleCheckedLocking.aspx
And this SO question has some good discussion:
The need for volatile modifier in double checked locking in .NET
Probably it is best to find some other mechanism other than double-checked locking.
There is a mechanism for waiting for one thread which is shutting down to complete -- thread.Join. You could join from the UI thread to the worker thread; when the worker thread is shut down, the UI thread wakes up again and does the dispose.
UPDATE: Added some information on Join.
"Join" basically means "thread U tells thread W to shut down, and U goes to sleep until that happens". Brief sketch of the quit method:
// do this in a thread-safe manner of your choosing
running = false;
// wait for worker thread to come to a halt
workerThread.Join();
// Now we know that worker thread is done, so we can
// clean up and exit
Dispose();
Exit();
Suppose you didn't want to use "Join" for some reason. (Perhaps the worker thread needs to keep running in order to do something else, but you still need to know when it is done using the objects.) We can build our own mechanism that works like Join by using wait handles. What you need now are two locking mechanisms: one that lets U send a signal to W that says "stop running now" and then another that waits while W finishes off the last call to M().
What I would do in this circumstance is:
make a thread-safe flag "running". Use whatever mechanism you are comfortable with to make it thread safe. I would personally start with a lock dedicated to it; if you decide later that you can go with lock-free interlocked operations on it then you can always do that later.
make an AutoResetEvent to act as a gate on the dispose.
So, brief sketch:
UI thread, startup logic:
running = true
waithandle = new AutoResetEvent(false)
start up worker thread
UI thread, quit logic:
running = false; // do this in a thread-safe manner of your choosing
waithandle.WaitOne();
// WaitOne is robust in the face of race conditions; if the worker thread
// calls Set *before* WaitOne is called, WaitOne will be a no-op. (However,
// if there are *multiple* threads all trying to "wake up" a gate that is
// waiting on WaitOne, the multiple wakeups will be lost. WaitOne is named
// WaitOne because it WAITS for ONE wakeup. If you need to wait for multiple
// wakeups, don't use WaitOne.
Dispose();
waithandle.Close();
Exit();
worker thread:
while(running) // make thread-safe access to "running"
M();
waithandle.Set(); // Tell waiting UI thread it is safe to dispose
Notice that this relies on the fact that M() is short. If M() takes a long time then you can wait a long time to quit the application, which seems bad.
Does that make sense?
Really though, you shouldn't be doing this. If you want to wait for the worker thread to shut down before you dispose an object it is using, just join it.
UPDATE: Some additional questions raised:
is it a good idea to wait without a timeout?
Indeed, note that in my example with Join and my example with WaitOne, I do not use the variants on them that wait for a specific amount of time before giving up. Rather, I call out that my assumption is that the worker thread shuts down cleanly and quickly. Is this the correct thing to do?
It depends! It depends on just how badly the worker thread behaves and what it is doing when it is misbehaving.
If you can guarantee that the work is short in duration, for whatever 'short' means to you, then you don't need a timeout. If you cannot guarantee that, then I would suggest first rewriting the code so that you can guarantee that; life becomes much easier if you know that the code will terminate quickly when you ask it to.
If you cannot, then what's the right thing to do? The assumption of this scenario is that the worker is ill-behaved and does not terminate in a timely manner when asked to. So now we've got to ask ourselves "is the worker slow by design, buggy, or hostile?"
In the first scenario, the worker is simply doing something that takes a long time and for whatever reason, cannot be interrupted. What's the right thing to do here? I have no idea. This is a terrible situation to be in. Presumably the worker is not shutting down quickly because doing so is dangerous or impossible. In that case, what are you going to do when the timeout times out??? You've got something that is dangerous or impossible to shut down, and its not shutting down in a timely manner. Your choices seem to be (1) do nothing, (2) do something dangerous, or (3) do something impossible. Choice three is probably out. Choice one is equivalent to waiting forever, whcih we've already rejected. That leaves "do something dangerous".
Knowing what the right thing to do in order to minimize harm to user data depends upon the exact circumstances that are causing the danger; analyse it carefully, understand all the scenarios, and figure out the right thing to do.
Now suppose the worker is supposed to be able to shut down quickly, but does not because it has a bug. Obviously, if you can, fix the bug. If you cannot fix the bug -- perhaps it is in code you do not own -- then again, you are in a terrible fix. You have to understand what the consequences are of not waiting for already-buggy-and-therefore-unpredictable code to finish before disposing of the resources that you know it is using right now on another thread. And you have to know what the consequences are of terminating an application while a buggy worker thread is still busy doing heaven only knows what to operating system state.
If the code is hostile and is actively resisting being shut down then you have already lost. You cannot halt the thread by normal means, and you cannot even thread abort it. There is no guarantee whatsoever that aborting a hostile thread actually terminates it; the owner of the hostile code that you have foolishly started running in your process could be doing all of its work in a finally block or other constrained region which prevents thread abort exceptions.
The best thing to do is to never get into this situation in the first place; if you have code that you think is hostile, either do not run it at all, or run it in its own process, and terminate the process, not the thread when things go badly.
In short, there's no good answer to the question "what do I do if it takes too long?" You are in a terrible situation if that happens and there is no easy answer. Best to work hard to ensure you don't get into it in the first place; only run cooperative, benign, safe code that always shuts itself down cleanly and rapidly when asked.
What if the worker throws an exception?
OK, so what if it does? Again, better to not be in this situation in the first place; write the worker code so that it does not throw. If you cannot do that, then you have two choices: handle the exception, or don't handle the exception.
Suppose you don't handle the exception. As of I think CLR v2, an unhandled exception in a worker thread shuts down the whole application. The reason being, in the past what would happen is you'd start up a bunch of worker threads, they'd all throw exceptions, and you'd end up with a running application with no worker threads left, doing no work, and not telling the user about it. It is better to force the author of the code to handle the situation where a worker thread goes down due to an exception; doing it the old way effectively hides bugs and makes it easy to write fragile applications.
Suppose you do handle the exception. Now what? Something threw an exception, which is by definition an unexpected error condition. You now have no clue whatsoever that any of your data is consistent or any of your program invariants are maintained in any of your subsystems. So what are you going to do? There's hardly anything safe you can do at this point.
The question is "what is best for the user in this unfortunate situation?" It depends on what the application is doing. It is entirely possible that the best thing to do at this point is to simply aggressively shut down and tell the user that something unexpected failed. That might be better than trying to muddle on and possibly making the situation worse, by, say, accidentally destroying user data while trying to clean up.
Or, it is entirely possible that the best thing to do is to make a good faith effort to preserve the user's data, tidy up as much state as possible, and terminate as normally as possible.
Basically, both your questions are "what do I do when my subsystems do not behave themselves?" If your subsystems are unreliable, either make them reliable, or have a policy for how you deal with an unreliable subsystem, and implement that policy. That's a vague answer I know, but that's because dealing with an unreliable subsystem is an inherently awful situation to be in. How you deal with it depends on the nature of its unreliability, and the consequences of that unreliability to the user's valuable data.

Check Running again inside the lock:
while (Running)
{
lock(locker)
{
if(Running) {
DoSomething1();
DoSomething2();
}
}
}
You could even rewrite this as a while(true)...break, which would probably be preferable.

Instead of using a bool for Running, why not use an Enum with states of Stopped, Starting, Running, and Stopping?
That way, you break out of the loop when Running gets set to Stopping, and do your Disposing. Once that's done, Running gets set to Stopped. When OnQuit() sees Running set to Stopped, it will go ahead and exit.
Edit: Here's code, quick and dirty, not tested, etc.
public class SomeThread : IDisposable
{
private object locker = new object();
private RunState running = RunState.Stopped;
public enum RunState
{
Stopped,
Starting,
Running,
Stopping,
}
public RunState Running
{
get
{
lock(locker)
{
return running;
}
}
set
{
lock(locker)
{
running = value;
}
}
}
public void Run()
{
while (Running == RunState.Running)
{
lock(locker)
{
DoSomething1();
DoSomething2();
}
}
Dispose();
}
private void DoSomething1()
{
// something awesome happens here
}
private void DoSomething2()
{
// something more awesome happens here
}
public void Dispose()
{
lock (locker)
{
Dispose1();
Dispose2();
}
Running = RunState.Stopped;
}
private void Dispose1()
{
// something awesome happens here
}
private void Dispose2()
{
// something more awesome happens here
}
}
public class OtherThread
{
SomeThread st = new SomeThread();
public void OnQuit()
{
st.Running = SomeThread.RunState.Stopping;
while (st.Running == SomeThread.RunState.Stopping)
{
// Do something while waiting for the other thread.
}
Exit();
}
}

Related

How to pause a thread at once in C#?

I use ManualResetEnvent to pause/continue a thread. The code example is below.
private _rstEvent = new ManualResetEvent(true);
public void DoSomeWork()
{
while(judgementValue)
{
_rstEvent.WaitOne();
...
}
}
public void Pause()
{
_rstEvent.Reset();
}
public void Continue()
{
_rstEvent.Set();
}
The problem is what if the while loop is large, which means every loop in the while statement has many operations to do. The thread will keep going until meet the next _rstEvent.WaitOne();. Is there a way to pause the thread at once except the deprecated suspend?

The problem with Suspend, Abort and the like is that you're trying to do something to another thread which could literally be doing anything at the time.
It has been repeatedly observed that this can lead to difficult to diagnose bugs where e.g. none of your current threads can obtain a lock because the thread that held the lock has been suspended or aborted and thus will never release the lock (substitute in any other resource also).
This is why modern threading mechanisms are built around cooperative primitives where the thread itself checks (at appropriate moments) whether it's being asked to suspend or abort and can ensure that it only does so when not holding any resources.
There's no reason your loop code cannot check the event multiple times per iteration, whenever it is appropriate.

How can I make a interrupt in C#? [duplicate]

I understand Thread.Abort() is evil from the multitude of articles I've read on the topic, so I'm currently in the process of ripping out all of my abort's in order to replace it for a cleaner way; and after comparing user strategies from people here on stackoverflow and then after reading "How to: Create and Terminate Threads (C# Programming Guide)" from MSDN both which state an approach very much the same -- which is to use a volatile bool approach checking strategy, which is nice, but I still have a few questions....
Immediately what stands out to me here, is what if you do not have a simple worker process which is just running a loop of crunching code? For instance for me, my process is a background file uploader process, I do in fact loop through each file, so that's something, and sure I could add my while (!_shouldStop) at the top which covers me every loop iteration, but I have many more business processes which occur before it hits it's next loop iteration, I want this cancel procedure to be snappy; don't tell me I need to sprinkle these while loops every 4-5 lines down throughout my entire worker function?!
I really hope there is a better way, could somebody please advise me on if this is in fact, the correct [and only?] approach to do this, or strategies they have used in the past to achieve what I am after.
Thanks gang.
Further reading: All these SO responses assume the worker thread will loop. That doesn't sit comfortably with me. What if it is a linear, but timely background operation?

Unfortunately there may not be a better option. It really depends on your specific scenario. The idea is to stop the thread gracefully at safe points. That is the crux of the reason why Thread.Abort is not good; because it is not guaranteed to occur at safe points. By sprinkling the code with a stopping mechanism you are effectively manually defining the safe points. This is called cooperative cancellation. There are basically 4 broad mechanisms for doing this. You can choose the one that best fits your situation.
Poll a stopping flag
You have already mentioned this method. This a pretty common one. Make periodic checks of the flag at safe points in your algorithm and bail out when it gets signalled. The standard approach is to mark the variable volatile. If that is not possible or inconvenient then you can use a lock. Remember, you cannot mark a local variable as volatile so if a lambda expression captures it through a closure, for example, then you would have to resort to a different method for creating the memory barrier that is required. There is not a whole lot else that needs to be said for this method.
Use the new cancellation mechanisms in the TPL
This is similar to polling a stopping flag except that it uses the new cancellation data structures in the TPL. It is still based on cooperative cancellation patterns. You need to get a CancellationToken and the periodically check IsCancellationRequested. To request cancellation you would call Cancel on the CancellationTokenSource that originally provided the token. There is a lot you can do with the new cancellation mechanisms. You can read more about here.
Use wait handles
This method can be useful if your worker thread requires waiting on an specific interval or for a signal during its normal operation. You can Set a ManualResetEvent, for example, to let the thread know it is time to stop. You can test the event using the WaitOne function which returns a bool indicating whether the event was signalled. The WaitOne takes a parameter that specifies how much time to wait for the call to return if the event was not signaled in that amount of time. You can use this technique in place of Thread.Sleep and get the stopping indication at the same time. It is also useful if there are other WaitHandle instances that the thread may have to wait on. You can call WaitHandle.WaitAny to wait on any event (including the stop event) all in one call. Using an event can be better than calling Thread.Interrupt since you have more control over of the flow of the program (Thread.Interrupt throws an exception so you would have to strategically place the try-catch blocks to perform any necessary cleanup).
Specialized scenarios
There are several one-off scenarios that have very specialized stopping mechanisms. It is definitely outside the scope of this answer to enumerate them all (never mind that it would be nearly impossible). A good example of what I mean here is the Socket class. If the thread is blocked on a call to Send or Receive then calling Close will interrupt the socket on whatever blocking call it was in effectively unblocking it. I am sure there are several other areas in the BCL where similiar techniques can be used to unblock a thread.
Interrupt the thread via Thread.Interrupt
The advantage here is that it is simple and you do not have to focus on sprinkling your code with anything really. The disadvantage is that you have little control over where the safe points are in your algorithm. The reason is because Thread.Interrupt works by injecting an exception inside one of the canned BCL blocking calls. These include Thread.Sleep, WaitHandle.WaitOne, Thread.Join, etc. So you have to be wise about where you place them. However, most the time the algorithm dictates where they go and that is usually fine anyway especially if your algorithm spends most of its time in one of these blocking calls. If you algorithm does not use one of the blocking calls in the BCL then this method will not work for you. The theory here is that the ThreadInterruptException is only generated from .NET waiting call so it is likely at a safe point. At the very least you know that the thread cannot be in unmanaged code or bail out of a critical section leaving a dangling lock in an acquired state. Despite this being less invasive than Thread.Abort I still discourage its use because it is not obvious which calls respond to it and many developers will be unfamiliar with its nuances.

Well, unfortunately in multithreading you often have to compromise "snappiness" for cleanliness... you can exit a thread immediately if you Interrupt it, but it won't be very clean. So no, you don't have to sprinkle the _shouldStop checks every 4-5 lines, but if you do interrupt your thread then you should handle the exception and exit out of the loop in a clean manner.
Update
Even if it's not a looping thread (i.e. perhaps it's a thread that performs some long-running asynchronous operation or some type of block for input operation), you can Interrupt it, but you should still catch the ThreadInterruptedException and exit the thread cleanly. I think that the examples you've been reading are very appropriate.
Update 2.0
Yes I have an example... I'll just show you an example based on the link you referenced:
public class InterruptExample
{
private Thread t;
private volatile boolean alive;
public InterruptExample()
{
alive = false;
t = new Thread(()=>
{
try
{
while (alive)
{
/* Do work. */
}
}
catch (ThreadInterruptedException exception)
{
/* Clean up. */
}
});
t.IsBackground = true;
}
public void Start()
{
alive = true;
t.Start();
}
public void Kill(int timeout = 0)
{
// somebody tells you to stop the thread
t.Interrupt();
// Optionally you can block the caller
// by making them wait until the thread exits.
// If they leave the default timeout,
// then they will not wait at all
t.Join(timeout);
}
}

If cancellation is a requirement of the thing you're building, then it should be treated with as much respect as the rest of your code--it may be something you have to design for.
Lets assume that your thread is doing one of two things at all times.
Something CPU bound
Waiting for the kernel
If you're CPU bound in the thread in question, you probably have a good spot to insert the bail-out check. If you're calling into someone else's code to do some long-running CPU-bound task, then you might need to fix the external code, move it out of process (aborting threads is evil, but aborting processes is well-defined and safe), etc.
If you're waiting for the kernel, then there's probably a handle (or fd, or mach port, ...) involved in the wait. Usually if you destroy the relevant handle, the kernel will return with some failure code immediately. If you're in .net/java/etc. you'll likely end up with an exception. In C, whatever code you already have in place to handle system call failures will propagate the error up to a meaningful part of your app. Either way, you break out of the low-level place fairly cleanly and in a very timely manner without needing new code sprinkled everywhere.
A tactic I often use with this kind of code is to keep track of a list of handles that need to be closed and then have my abort function set a "cancelled" flag and then close them. When the function fails it can check the flag and report failure due to cancellation rather than due to whatever the specific exception/errno was.
You seem to be implying that an acceptable granularity for cancellation is at the level of a service call. This is probably not good thinking--you are much better off cancelling the background work synchronously and joining the old background thread from the foreground thread. It's way cleaner becasue:
It avoids a class of race conditions when old bgwork threads come back to life after unexpected delays.
It avoids potential hidden thread/memory leaks caused by hanging background processes by making it possible for the effects of a hanging background thread to hide.
There are two reasons to be scared of this approach:
You don't think you can abort your own code in a timely fashion. If cancellation is a requirement of your app, the decision you really need to make is a resource/business decision: do a hack, or fix your problem cleanly.
You don't trust some code you're calling because it's out of your control. If you really don't trust it, consider moving it out-of-process. You get much better isolation from many kinds of risks, including this one, that way.

The best answer largely depends on what you're doing in the thread.
Like you said, most answers revolve around polling a shared boolean every couple lines. Even though you may not like it, this is often the simplest scheme. If you want to make your life easier, you can write a method like ThrowIfCancelled(), which throws some kind of exception if you're done. The purists will say this is (gasp) using exceptions for control flow, but then again cacelling is exceptional imo.
If you're doing IO operations (like network stuff), you may want to consider doing everything using async operations.
If you're doing a sequence of steps, you could use the IEnumerable trick to make a state machine. Example:
<
abstract class StateMachine : IDisposable
{
public abstract IEnumerable<object> Main();
public virtual void Dispose()
{
/// ... override with free-ing code ...
}
bool wasCancelled;
public bool Cancel()
{
// ... set wasCancelled using locking scheme of choice ...
}
public Thread Run()
{
var thread = new Thread(() =>
{
try
{
if(wasCancelled) return;
foreach(var x in Main())
{
if(wasCancelled) return;
}
}
finally { Dispose(); }
});
thread.Start()
}
}
class MyStateMachine : StateMachine
{
public override IEnumerabl<object> Main()
{
DoSomething();
yield return null;
DoSomethingElse();
yield return null;
}
}
// then call new MyStateMachine().Run() to run.
>
Overengineering? It depends how many state machines you use. If you just have 1, yes. If you have 100, then maybe not. Too tricky? Well, it depends. Another bonus of this approach is that it lets you (with minor modifications) move your operation into a Timer.tick callback and void threading altogether if it makes sense.
and do everything that blucz says too.

Perhaps the a piece of the problem is that you have such a long method / while loop. Whether or not you are having threading issues, you should break it down into smaller processing steps. Let's suppose those steps are Alpha(), Bravo(), Charlie() and Delta().
You could then do something like this:
public void MyBigBackgroundTask()
{
Action[] tasks = new Action[] { Alpha, Bravo, Charlie, Delta };
int workStepSize = 0;
while (!_shouldStop)
{
tasks[workStepSize++]();
workStepSize %= tasks.Length;
};
}
So yes it loops endlessly, but checks if it is time to stop between each business step.

You don't have to sprinkle while loops everywhere. The outer while loop just checks if it's been told to stop and if so doesn't make another iteration...
If you have a straight "go do something and close out" thread (no loops in it) then you just check the _shouldStop boolean either before or after each major spot inside the thread. That way you know whether it should continue on or bail out.
for example:
public void DoWork() {
RunSomeBigMethod();
if (_shouldStop){ return; }
RunSomeOtherBigMethod();
if (_shouldStop){ return; }
//....
}

Instead of adding a while loop where a loop doesn't otherwise belong, add something like if (_shouldStop) CleanupAndExit(); wherever it makes sense to do so. There's no need to check after every single operation or sprinkle the code all over with them. Instead, think of each check as a chance to exit the thread at that point and add them strategically with this in mind.

All these SO responses assume the worker thread will loop. That doesn't sit comfortably with me
There are not a lot of ways to make code take a long time. Looping is a pretty essential programming construct. Making code take a long time without looping takes a huge amount of statements. Hundreds of thousands.
Or calling some other code that is doing the looping for you. Yes, hard to make that code stop on demand. That just doesn't work.

Long running process suspended

I have a .NET 2.0 console application running on a Windows Server GoDaddy VPS in the Visual Studio 2010 IDE in debug mode (F5).
The application periodically freezes (as if the garbage collector has temporarily suspended execution) however on the rare occasion it never resumes execution!
I've been diagonosing this for months, and am running out of ideas.
The application runs as fast as it can (it uses 100% CPU usage), but at normal priority. It is also multi-threaded.
When the application freezes, I can unfreeze it using the VS2010 IDE by pausing/unpausing the process (since it's running in the debugger).
The location of last execution, when I pause the frozen process, seems irrelevant.
While frozen, the CPU usage is still 100%.
Upon unfreezing it, it runs perfectly fine until the next freeze.
The server might run 70 days between freezes, or it might only make it 24 hours.
Memory usage remains relatively constant; no evidence of any sort of memory leak.
Anyone have any tips for diagnosing what exactly is happening?

It is also multi-threaded
That's the key part of the problem. You are describing a very typical way in which a multi-threaded program can misbehave. It is suffering from deadlock, one of the typical problems with threading.
It can be narrowed down a bit further from the info, clearly your process isn't completely frozen since it still consumes 100% cpu. You probably have a hot wait-loop in your code, a loop that spins on another thread signaling an event. Which is likely to induce an especially nasty variety of deadlock, a live-lock. Live-locks are very sensitive to timing, minor changes in the order in which code runs can bump it into a live-lock. And back out again.
Live-locks are extraordinarily difficult to debug since attempting to do so makes the condition disappear. Like attaching a debugger or breaking the code, enough to alter the thread timing and bump it out of the condition. Or adding logging statements to your code, a common strategy to debug threading problems. Which alters the timing due to the logging overhead which in turn can make the live-lock entirely disappear.
Nasty stuff and impossible to get help with such a problem from a site like SO since it is extremely dependent on the code. A thorough review of the code is often required to find the reason. And not infrequently a drastic rewrite. Good luck with it.

Does the application have "dead lock recover/prevention" code? That is, locking with timout, then trying again, perhaps after sleep?
Does the application check error codes (return values or exceptions) and repeatedly retry in case of error anywhere?
Note that such looping can also happen through event loop, where your code is only in some event handler. It does not have to be an actual loop in your own code. Though this is probably not the case, if application is frozen, indicating blocked event loop.
If you have anything like above, you could try to mitigate the problem by making timeouts and sleeps to be of random interval, as well as adding short random-duration sleeps to cases where error might produce dead-/livelock. If such a loop is performance-sensitive, add a counter and only start sleeping with random, perhaps increasing interval after some number of failed retries. And make sure any sleep you add does not sleep while something is locked.
If the situation would happen more often, you could also use this to bisect your code and pinpoint which loops (because 100% CPU usage means, some very busy loops are spinning) are responsible. But from the rarity of issue, I gather you're going to be happy if the problem just goes away in practice ;)

Well three things here...
First of all, start using the server GC of .NET: http://msdn.microsoft.com/en-us/library/ms229357.aspx . That will probably keep your application non-blocked.
Second, if you can do that on your VM: check for updates. This always seems evident, but I've seen numerous occasions where a simple windows update fixes strange issues.
Third, I'd like to make a point about the lifetime of an object, which can be one of the issues here. This is quite a long story what happens, so bear with me.
The lifetime of an object is basically construction - garbage collection - finalization. All three processes run in a separate thread. The GC passes data to the finalization thread which has a queue that calls the 'destructors'.
So what if you have a finalizer that does something strange, say something like:
public class FinalizerObject
{
public FinalizerObject(int n)
{
Console.WriteLine("Constructed {0}", n);
this.n = n;
}
private int n;
~FinalizerObject()
{
while (true) { Console.WriteLine("Finalizing {0}...", n); System.Threading.Thread.Sleep(1000); }
}
}
Because the finalizers run in a separate thread that processes the queue, having a single finalizer that does something stupid is a serious problem for your application. You can see this by using the above class 2 times:
static void Main(string[] args)
{
SomeMethod();
GC.Collect(GC.MaxGeneration);
GC.WaitForFullGCComplete();
Console.WriteLine("All done.");
Console.ReadLine();
}
static void SomeMethod()
{
var obj2 = new FinalizerObject(1);
var obj3 = new FinalizerObject(2);
}
Notice how you end up with a small memory leak and if you remove the Thread.Sleep also with a 100% CPU process - even though your main thread is still responding. Because they're different threads, from here on it's quite easy to block the entire process - for example by using a lock:
static void Main(string[] args)
{
SomeMethod();
GC.Collect(GC.MaxGeneration);
GC.WaitForFullGCComplete();
Thread.Sleep(1000);
lock (lockObject)
{
Console.WriteLine("All done.");
}
Console.ReadLine();
}
static object lockObject = new Program();
static void SomeMethod()
{
var obj2 = new FinalizerObject(1, lockObject);
var obj3 = new FinalizerObject(2, lockObject);
}
[...]
~FinalizerObject()
{
lock (lockObject) { while (true) { Console.WriteLine("Finalizing {0}...", n); System.Threading.Thread.Sleep(1000); } }
}
So I can see you thinking 'Are you serious?'; the fact is that you might be doing something like this without even realizing this. This is where 'yield' comes into the picture:
IEnumerable's from 'yield' are actually IDisposable and as such implement the IDisposable pattern. Combine your 'yield' implementation with a lock, forget to call IDisposable by enumerating it with 'MoveNext' etc. and you get some pretty nasty behavior that reflects the above. Especially because finalizers are called from the finalization queue by a separate thread (!). Combine it with an endless loop or thread unsafe code, and you will get some pretty nasty unexpected behavior, which will be triggered in exceptional cases (when memory runs out, or when the GC things it should do something).
In other words: I'd check your disposables and finalizers and be very critical about them. Check if 'yield' has implicit finalizers and make sure you call IDisposable from the same thread. Some examples of things that you have be be wary of:
try
{
for (int i = 0; i < 10; ++i)
{
yield return "foo";
}
}
finally
{
// Called by IDisposable
}
and
lock (myLock) // 'lock' and 'using' also trigger IDisposable
{
yield return "foo";
}

Is using Thread.Abort() and handling ThreadAbortException in .NET safe practice?

I need to develop a multithreaded Azure worker role in C# - create multiple threads, feed requests to them, each request might require some very long time to process (not my code - I'll call a COM object to do actual work).
Upon role shutdown I need to gracefully stop processing. How do I do that? Looks like if I just call Thread.Abort() the ThreadAbortException is thrown in the thread and the thread can even use try-catch-finally (or using) to clean up resources. This looks quite reliable.
What bothers me is that my experience is mostly C++ and it's impossible to gracefully abort a thread in an unmanaged application - it will just stop without any further processing and this might leave data in inconsistent state. So I'm kind of paranoid about whether anything like that happens in case I call Thread.Abort() for a busy thread.
Is it safe practice to use Thread.Abort() together with handling ThreadAbortException? What should I be aware of if I do that?

Is using Thread.Abort() and handling ThreadAbortException in .NET safe practice?
TL;DR version: No, isn't.
Generally you're safe when all type invariants (whether explicitly stated or not) are actually true. However many methods will break these invariants while running, only to reach a new state when they are again true at the end. If the thread is idle in a state with invariants held you'll be OK, but in that case better to use something like an event to signal the thread to exit gracefully (ie. you don't need to abort).
An out-of-band exception1 thrown in a thread while in such a invariants-not-true, ie. invalid, state is where the problems start. These problems include, but are certainly not limited to, mutually inconsistent field and property values (data structures in an invalid state), locks not exited, and events representing "changes happened" not fired.
In many cases it is possible to deal with these in clean up code (eg. a finally block), but then consider what happens when the out-of-band exception occurs in that clean up code? This leads to clean up code for the clean up code. But then that code is it self vulnerable so you need clean up code for the clean up code of the clean up code… it never ends!
There are solutions, but they are not easy to design (and tends to impact your whole design), and even harder to test—how to re-create all the cases (think combinatorial explosion). Two possible routes are:
Work on copies of state, update the copies and then atomically swap current for new state. If there is an out-of-band exception then the original state remains (and finalisers can clean up the temporary state).
This is rather like the function of database transactions (albeit RDBMSs work with locks and transaction log files).
It is also similar to the approaches to achieving the "strong exception guarantee" developed in the C++ community in response to a paper questioning if exceptions could ever be safe (C++ of course has no GC/finaliser queue to clean up discarded objects). See Herb Sutters "Guru of the Week #8: CHALLENGE EDITION: Exception Safety" for the solution.
In practice this is hard to achieve unless your state can be encapsulated in a single reference.
Look at "Constrained Execution Regions", but not the limitations on what you can do in these cases. (MSDN Magazine had an introductory article (introduction to the subject, not introductory level), from .NET 2 beta period2).
In practice if you have to do this, using approach #2 to manage the state change under #1 is probably the best approach, but getting it right, and then validating that it is correct (and the correctness is maintained) is hard.
Summary: It's a bit like optimisation: rule 1: don't do it; rule 2 (experts only): don't do it unless you have no other option.
1 A ThreadAbortException is not the only such exception.
2 So details have possibly changed.

One example where it's problematic to abort a thread.
using(var disposable=new ClassThatShouldBeDisposed())
{
...
}
Now the Thread abortion happes after the constructor of the class has finished but before the assignment to the local variable. So it won't be disposed. Eventually the finalizer will run, but that can be much later.
Deterministic disposing and thread abortion don't work well together. The only way I know to get safe disposing when using thread abortion is putting all the critical code inside a finally clause.
try
{//Empty try block
}
finally
{
//put all your code in the finally clause to fool thread abortion
using(var disposable=new ClassThatShouldBeDisposed())
{
...
}
}
This works because thread abortion allows finally code to execute. Of course this implies that the thread abortion will simply not work until the code leaves the finally block.
One way to get your code to work correctly with thread abortion is using the following instead of the using statement. Unfortunately it's very ugly.
ClassThatShouldBeDisposed disposable=null;
try
{
try{}finally{disposable=new ClassThatShouldBeDisposed();}
//Do your work here
}
finally
{
if(disposable!=null)
disposable.Dispose();
}
Personally I just assume threads never get aborted(except when unloading the AppDomain) and thus write normal using based code.

It's very difficult to handle the TheadAbortException correctly, because it can be thrown in the middle of whatever code the thread is executing.
Most code is written with the assumption that some actions, for example int i = 0; never causes an exception, so the critical exception handling is only applied to code that actually can cause an exception by itself. When you abort a thread, the exception can come in code that is not prepared to handle it.
The best practice is to tell the thread to end by itself. Create a class for the method that is running the thread, and put a boolean variable in it. Both the code that started the thread and the method running the thread can access the variable, so you can just switch it to tell the thread to end. The code in the thread of course have to check the value periodically.

Thread.Abort is an unsafe way of killing the thread.
It rises an asynchronous ThreadAbortException which is a special exception that can be caught, but it will automatically be raised again at the end of the catch block
It can leave the state corrupted, and your application becomes unstable
TAE is raised in the other thread
The best practise is to use wrappers that support work cancellation, such as the Task class or use volatile bool. Instead of Thread.Abort consider using Thread.Join which will block the calling thread until the working thread is disposed of.
Some links:
- How To Stop a Thread in .NET (and Why Thread.Abort is Evil)
- Managed code and asynchronous exception hardening
- The dangers of Thread.Abort

As others have mentioned, aborting a thread is probably not a good idea. However, signalling a thread to stop with a bool may not work either, because we have no guarantee that the value of a bool will be synchronized across threads.
It may be better to use an event:
class ThreadManager
{
private Thread thread = new Thread(new ThreadStart(CallCOMMethod));
private AutoResetEvent endThread = new AutoResetEvent(false);
public ThreadManager()
{
thread.Start();
}
public StopThread()
{
endThread.Set();
}
private void CallCOMMethod()
{
while (!endThread.WaitOne())
{
// Call COM method
}
}
}
Since the COM method is long running you may just need to "bite the bullet" and wait for it to complete its current iteration. Is the information computed during the current iteration of value to the user?
If not, one option my be:
Have the ThreadManager itself run on a separate thread from the UI which checks for the stop notification from the user relatively often.
When the user requests that the long running operation be stopped, the UI thread can immediately return to the user.
The ThreadManager waits for the long running COM operation to complete its current iteration, then it throws away the results.

It's considered best practice to just let the thread's method return:
void Run() // thread entry function
{
while(true)
{
if(somecondition) // check for a terminating condition - usually "have I been disposed?"
break;
if(workExists)
doWork();
Thread.Sleep(interval);
}
}

Please get simple idea from here as for your requirement, check thead isalive property, then abort your thread.............................................................
ThreadStart th = new ThreadStart(CheckValue);
System.Threading.Thread th1 = new Thread(th);
if(taskStatusComleted)
{
if (th1.IsAlive)
{
th1.Abort();
}
}
private void CheckValue()
{
//my method....
}

Question about terminating a thread cleanly in .NET

I understand Thread.Abort() is evil from the multitude of articles I've read on the topic, so I'm currently in the process of ripping out all of my abort's in order to replace it for a cleaner way; and after comparing user strategies from people here on stackoverflow and then after reading "How to: Create and Terminate Threads (C# Programming Guide)" from MSDN both which state an approach very much the same -- which is to use a volatile bool approach checking strategy, which is nice, but I still have a few questions....
Immediately what stands out to me here, is what if you do not have a simple worker process which is just running a loop of crunching code? For instance for me, my process is a background file uploader process, I do in fact loop through each file, so that's something, and sure I could add my while (!_shouldStop) at the top which covers me every loop iteration, but I have many more business processes which occur before it hits it's next loop iteration, I want this cancel procedure to be snappy; don't tell me I need to sprinkle these while loops every 4-5 lines down throughout my entire worker function?!
I really hope there is a better way, could somebody please advise me on if this is in fact, the correct [and only?] approach to do this, or strategies they have used in the past to achieve what I am after.
Thanks gang.
Further reading: All these SO responses assume the worker thread will loop. That doesn't sit comfortably with me. What if it is a linear, but timely background operation?

Well, unfortunately in multithreading you often have to compromise "snappiness" for cleanliness... you can exit a thread immediately if you Interrupt it, but it won't be very clean. So no, you don't have to sprinkle the _shouldStop checks every 4-5 lines, but if you do interrupt your thread then you should handle the exception and exit out of the loop in a clean manner.
Update
Even if it's not a looping thread (i.e. perhaps it's a thread that performs some long-running asynchronous operation or some type of block for input operation), you can Interrupt it, but you should still catch the ThreadInterruptedException and exit the thread cleanly. I think that the examples you've been reading are very appropriate.
Update 2.0
Yes I have an example... I'll just show you an example based on the link you referenced:
public class InterruptExample
{
private Thread t;
private volatile boolean alive;
public InterruptExample()
{
alive = false;
t = new Thread(()=>
{
try
{
while (alive)
{
/* Do work. */
}
}
catch (ThreadInterruptedException exception)
{
/* Clean up. */
}
});
t.IsBackground = true;
}
public void Start()
{
alive = true;
t.Start();
}
public void Kill(int timeout = 0)
{
// somebody tells you to stop the thread
t.Interrupt();
// Optionally you can block the caller
// by making them wait until the thread exits.
// If they leave the default timeout,
// then they will not wait at all
t.Join(timeout);
}
}

If cancellation is a requirement of the thing you're building, then it should be treated with as much respect as the rest of your code--it may be something you have to design for.
Lets assume that your thread is doing one of two things at all times.
Something CPU bound
Waiting for the kernel
If you're CPU bound in the thread in question, you probably have a good spot to insert the bail-out check. If you're calling into someone else's code to do some long-running CPU-bound task, then you might need to fix the external code, move it out of process (aborting threads is evil, but aborting processes is well-defined and safe), etc.
If you're waiting for the kernel, then there's probably a handle (or fd, or mach port, ...) involved in the wait. Usually if you destroy the relevant handle, the kernel will return with some failure code immediately. If you're in .net/java/etc. you'll likely end up with an exception. In C, whatever code you already have in place to handle system call failures will propagate the error up to a meaningful part of your app. Either way, you break out of the low-level place fairly cleanly and in a very timely manner without needing new code sprinkled everywhere.
A tactic I often use with this kind of code is to keep track of a list of handles that need to be closed and then have my abort function set a "cancelled" flag and then close them. When the function fails it can check the flag and report failure due to cancellation rather than due to whatever the specific exception/errno was.
You seem to be implying that an acceptable granularity for cancellation is at the level of a service call. This is probably not good thinking--you are much better off cancelling the background work synchronously and joining the old background thread from the foreground thread. It's way cleaner becasue:
It avoids a class of race conditions when old bgwork threads come back to life after unexpected delays.
It avoids potential hidden thread/memory leaks caused by hanging background processes by making it possible for the effects of a hanging background thread to hide.
There are two reasons to be scared of this approach:
You don't think you can abort your own code in a timely fashion. If cancellation is a requirement of your app, the decision you really need to make is a resource/business decision: do a hack, or fix your problem cleanly.
You don't trust some code you're calling because it's out of your control. If you really don't trust it, consider moving it out-of-process. You get much better isolation from many kinds of risks, including this one, that way.

The best answer largely depends on what you're doing in the thread.
Like you said, most answers revolve around polling a shared boolean every couple lines. Even though you may not like it, this is often the simplest scheme. If you want to make your life easier, you can write a method like ThrowIfCancelled(), which throws some kind of exception if you're done. The purists will say this is (gasp) using exceptions for control flow, but then again cacelling is exceptional imo.
If you're doing IO operations (like network stuff), you may want to consider doing everything using async operations.
If you're doing a sequence of steps, you could use the IEnumerable trick to make a state machine. Example:
<
abstract class StateMachine : IDisposable
{
public abstract IEnumerable<object> Main();
public virtual void Dispose()
{
/// ... override with free-ing code ...
}
bool wasCancelled;
public bool Cancel()
{
// ... set wasCancelled using locking scheme of choice ...
}
public Thread Run()
{
var thread = new Thread(() =>
{
try
{
if(wasCancelled) return;
foreach(var x in Main())
{
if(wasCancelled) return;
}
}
finally { Dispose(); }
});
thread.Start()
}
}
class MyStateMachine : StateMachine
{
public override IEnumerabl<object> Main()
{
DoSomething();
yield return null;
DoSomethingElse();
yield return null;
}
}
// then call new MyStateMachine().Run() to run.
>
Overengineering? It depends how many state machines you use. If you just have 1, yes. If you have 100, then maybe not. Too tricky? Well, it depends. Another bonus of this approach is that it lets you (with minor modifications) move your operation into a Timer.tick callback and void threading altogether if it makes sense.
and do everything that blucz says too.

Perhaps the a piece of the problem is that you have such a long method / while loop. Whether or not you are having threading issues, you should break it down into smaller processing steps. Let's suppose those steps are Alpha(), Bravo(), Charlie() and Delta().
You could then do something like this:
public void MyBigBackgroundTask()
{
Action[] tasks = new Action[] { Alpha, Bravo, Charlie, Delta };
int workStepSize = 0;
while (!_shouldStop)
{
tasks[workStepSize++]();
workStepSize %= tasks.Length;
};
}
So yes it loops endlessly, but checks if it is time to stop between each business step.

You don't have to sprinkle while loops everywhere. The outer while loop just checks if it's been told to stop and if so doesn't make another iteration...
If you have a straight "go do something and close out" thread (no loops in it) then you just check the _shouldStop boolean either before or after each major spot inside the thread. That way you know whether it should continue on or bail out.
for example:
public void DoWork() {
RunSomeBigMethod();
if (_shouldStop){ return; }
RunSomeOtherBigMethod();
if (_shouldStop){ return; }
//....
}

Instead of adding a while loop where a loop doesn't otherwise belong, add something like if (_shouldStop) CleanupAndExit(); wherever it makes sense to do so. There's no need to check after every single operation or sprinkle the code all over with them. Instead, think of each check as a chance to exit the thread at that point and add them strategically with this in mind.

All these SO responses assume the worker thread will loop. That doesn't sit comfortably with me
There are not a lot of ways to make code take a long time. Looping is a pretty essential programming construct. Making code take a long time without looping takes a huge amount of statements. Hundreds of thousands.
Or calling some other code that is doing the looping for you. Yes, hard to make that code stop on demand. That just doesn't work.

We Keep Coding

C# (C-Sharp) is a programming language developed by Microsoft that runs on the .NET Framework.