Situation
Well, as you can see below, I have a main App, creating a thread, which creates a bunch of background workers. When the RunWorkerCompleted is fired on some BG worker, I sometimes end up dropping an Unhandled Exception, (for a reason I have clearly pinpointed, but this is not the matter).
So the Logs clearly show that my UnhandledException handler is entered and reports the exception.
I know I am deliberately misusing the OnUnhandledException by stopping the App from exiting, using a Console.Read(). I did it on purpose because I want a human intervention/check before terminating the app.
public static void OnUnhandledException(object sender, UnhandledExceptionEventArgs e)
{
Logger.log("UNHANDLED EXCEPTION : " + e.ExceptionObject.ToString());
Mail.Sendmail("ADMIN ALERT : " + e.ExceptionObject.ToString());
Console.Read(); // Yes, this is an ungraceful trick, I confess.
}
However, what was supposed just to be a "pause" before manual exit turned out to keep the App alive, ie, the Thread is still generating workers, and running as if nothing happened.
Even Weirder : The UnhandledException is still being dropped from time to time when it happens again. And it is logged each time, behaving just as if it was a plain ol' try catch.
(This was the default behaviour before .Net 3.0)
Question
So, why is everything happening as if the UnhandledException was not thrown, and the thread and BG's keeping running as if nothing happened ? My guess is that as long as the OnUnhandledException handler is not done, and the App is still alive, all running threads that are still alive keep on living in a free world and doing their job.
This gives me a bad temptation, which is to keep this design, as a try/catch to unHandled Exception. I know this is not a tidy idea, as you never know if the exception is serious or something that could be skipped. However, I would really prefer my program to keep running all day and watch the log report / mail alerts and decide for myself if it needs to be restarted or not.
As this is a server application that needs to be running 24/7 I wanted to avoid ill-timed and repetitive interruptions due to minor still unhandled exceptions.
The biggest advantage to that being that the server program keeps running while we are tracking down unhandled exceptions one by one and deliver patches to handle them as they occur.
Thank you for your patience, and please feel free to give feedback.
PS : No, I did not smoke pot.
My guess is that as long as the OnUnhandledException handler is not
done, and the App is still alive, all running threads that are still
alive keep on living in a free world and doing their job.
Correct. But your other threads are now doing their job in a potentially unstable process. There are good reasons for terminating the App when an unhandled exception happens.
I would really prefer my program to keep running
Besides the (potential) stability problems, you would accumulate the halted state of whatever is involved in the unhandled handling. My guess is that you would at least leak a Thread each time it happens. And you can't spare too many of them.
So it would be a very bad design, and only delay the crashing.
First of all you need to subscribe the UnhadledThreadException handler. There you can not "Hijack" the exception, but you have access to it and also to the thread it belongs. What you can do now is putting the throwing Thread into suspetion. The Thread will be still alive!!
This however is a bad, dirty hack! Do not do this!
Take a look at this article for some details.
I recommend switching your threading model to TPL (design remains). There you have access to those ugly cross-thread excepions and can handle/supress em gracefully.
If you want to allow a BackgroundWorker thread to fail without killing the whole process, there's a cleaner way to do this.
First, intercept the RunWorkerCompleted event. This event is raised when the BackgroundWorker thread terminates, either normally or via an unhandled exception. Then in your event handler, check for the unhandled exception:
// Runs when the BackgroundWorker thread terminates.
private void ThreadFinished(object sender, RunWorkerCompletedEventArgs e)
{
// Process any unhandled exception
if (e.Error != null)
{
this.LogError(e.Error.Message, e.Error.StackTrace, blah, blah);
In this way, you can let the BackgroundWorker thread die and log any exception without terminating the whole process.
Related
I have a situation on a production machine where a thread in my Windows Service appears simply to die, without throwing an exception. To date my logging hasn't been verbose enough to pinpoint the exact line of code where it is dying; I've deployed a new version with more verbose logging for this purpose. But until I get a smoking gun, my suspicion falls on the line of code where I create a new DB context.
The error is not predictable, except that it tends to happen during periods of high activity, and is often correlated with other threads throwing DB timeout exceptions (hence my suspicion above). Exceptions I can handle. Dead threads I can't.
Any ideas why a thread might die silently, or otherwise simply freeze? Or what to do about it?
EDIT: To be clear, the code is surrounded by a try-catch block, and the catch does some logging (using log4net). So does the "finally". I know it's working because other threads have left logs when they threw exceptions. All that I see in my log is that thread x hits a certain debug point, then is never seen or heard from again, and the work it was supposed to do remains not done.
No, not really (but kind of). Threads don't die, they finish. A thread can finish successfully or by failure--if a thread fails then its exception is stored until it is handled. Depending on how you instantiate the thread it's possible that you "fire and forget", meaning that if an exception occurs you have no code to handle and retrieve it. This is doubly bad, as it results in unreleased resources (they're waiting for you to retrieve and handle the exception).
However, you haven't provided any details of what you've diagnosed. What threading approach/framework are you using (there's quite a few)? Are you watching the processes threads and seeing it suddenly disappear? Are you taking heap snapshots, or attaching to the running process after it appears to have died?
If you have access to the system i would debug it with: Visual Studio 2013 Remote Debugger. This is often a good choice if you can't reproduce the error in your development environment.
Threads do not silently freeze or crash, check whether you have some empty catch blocks in your code or it runs into an infinite loop (or very long timeout at any point).
Maybe you have some code for us.
I am using an external component which periodically shoots events from a worker thread. In my event handler I use a Dispatcher to invoke some method on the main thread. This works nicely...
private void HandleXYZ(object sender, EventArgs e)
{
...
if(OnTrigger != null)
dispatcher.Invoke(OnTrigger, new TimeSpan(0, 0, 1), e);
}
However, when the program shuts down and the external component Dispose()s, the program sometimes hangs (and can only be seen and killed in the task manager).
When I look at what is happening it looks like "the component" is waiting for the event to return on the main thread (it stays in the Dispose() method), while the worker thread waits for the dispatcher to invoke the mentioned call to the main thread (it hangs in the dispatcher.Invoke-line).
For now I solved the shutdown problem by adding a timeout to the Invoke, which seems to work but feels wrong.
Is there a cleaner way to do something like this? Can I force the main thread to take some time for jobs from other threads before shutting down?
I have tried to "disconnect" the event before shutting down, but that does not help, because the dispatcher is(could be) already waiting, when the program start to shut down...
PS: external component means here that I do not have access to the source code...
Yes, this is a common source of deadlock. It hangs because the dispatcher exited the dispatcher loop it won't respond to Invoke requests anymore. A quick cure is to use BeginInvoke instead, it doesn't wait for the invoke target to finish executing. Another quickie is to set the worker thread's IsBackground property to True so the CLR will kill it.
These are quick fixes and they may well work for you. Certainly on your dev machine, but if you have a nagging feeling that it may still go wrong then you're right, not observing a deadlock or threading race does not prove they are not present. There are two "good" ways to do it completely safely:
don't allow the main thread to exit until you are sure that the worker thread terminated and can no longer raise events. This answer shows the pattern.
terminate the program forcefully with Environment.Exit(). This is very crude but very effective, a sledgehammer you'll only reach for when you have a heavily threaded program where the UI thread is only second citizen. Odd as this may sound as a suitable approach, the new C++ language standard has elevated it to a supported way to terminate a program. You can read more about it in this answer. Do note how it allows for cleanup functions to be registered, you'll have to do something similar with, say, the AppDomain.ProcessExit event. Focus on the first bullet before you do this.
As for the event subscriptions, it is indeed a good idea to clean them up when you know that a particluar object is not needed anymore. Otherwise you would risk creating memory leaks. You might also want to have a look at the weak event pattern (MSDN).
Regarding the deadlock itself, without knowing your code, we can only guess.
I do not see the HandleXYZ() as a culprit, I would rather check your IDisposable() implemntaion. Have a look at the MSDN documentation and compare it to your implementation.
I suppose that somewhere in there in your implementation some method calls are made that depend on the timing of the GarbageCollector, which is indeterministic: Sometimes it may work out in your case, sometime it may not.
I'm looking for some guidance on best practices on handling exceptions in events. Currently, when exceptions are thrown in my application, the exception message is displayed in a pop up dialog and then the application restarts after hitting ok. The issue I'm seeing is that a lot of exceptions are occurring in the event handlers of some third party libraries and these exceptions are swallowed and never displayed since they are on a background thread. Here are a few solutions that various people have thought of and I would like to know of any of these are the best approach.
Forward the background thread to the UI thread in every event handler in the application.
Wrap the events in another class which has a try/catch around every method that invokes the event. The catch will then forward the exception to the UI thread if one occurs.
Get access to the third party libraries and put try/catch around where the events are being invoked, which could then be forwarded to the main application by a global event.
Lets discuss your options the way you listed them:
1) Forward the background thread to the UI thread in every event
handler in the application.
You should not and you can't! because:
You can't pretend if that event "which may comes from a third party" starts another background thread
or timer.
The whole point of having background threads is not
to freeze the UI, you will then struggle the user interface by a long execution codes.
2) Wrap the events in another class which has a try/catch around every method that invokes the event. The catch will then forward the exception to the UI thread if one occurs.
The same point as the previous. you can't pretend it that event starts another thread..
3) Get access to the third party libraries and put try/catch around where the events are being invoked, which could then be forwarded to the main application by a global event.
This is a bad choice, really, You should not change that code of third party libraries even if you can. because:
You are not the author for those libraries.
Whenever you want to update the library version or even change the libraries, you have to rewrite there code..
So what you have to do? The answer is some sort of what you are currently doing:
Whenever any exception thrown, log the exception restart your process, and at some point in your process send the logged bugs to your email and then start fixing them.
None of the above. Instead hook up the events on the Application and AppDomain for unhandled exceptions.
AppDomain.CurrentDomain.UnhandledException Event
Application.DispatcherUnhandledException Event
Further Information: The global exception handling event for WPF applications Application.DispatcherUnhandledException fires only for exceptions thrown on the main UI thread. However the AppDomain.CurrentDomain.UnhandledException should fire on any unhandled exception in any thread (unfortunately there is no preventing the application from shutting down after ward it reaches here).
Did some quick research on best practices and found that it is recommend that you manually handle exceptions on the background threads with try\catch block. Read the exception part on this page http://www.albahari.com/threading/ and also take a look at this StackOverflow question Catching unhandled exception on separate threads
If you're hooking the AppDomain.UnhandledException event, then the issue isn't the fact that they might be calling back on a background thread, but rather than the third-party library is explicitly swallowing exceptions thrown by the handlers. This is a badly behaving library, but, since you're writing the event handler code, you can catch the exceptions in the event handler and force the application to shut down.
Since the third-party library can't be stopped by throwing an exception, you can force the thread to terminate by calling:
Thread.CurrentThread.Abort();
Thread.Abort() is generally considered a bad practice, but it's slightly less dangerous here, as you are aborting yourself and therefore won't be injecting the exception into potentially nasty locations (you also know that the thread is not in an unmanaged context, so it will abort right away.) It's still a nasty hack, but the third-party application isn't giving you much choice. ThreadAbortException cannot be 'stopped', so the third-party code will terminate its thread as soon as it reaches the end of its exception handlers (even if they try to swallow it). If the third-party library is really nasty, it might invoke lots of code in its catch or finally blocks, even using dirty tricks to stay alive like launching other threads, but it would have to be pretty malicious to do this.
To get the friendly error message behavior, you can marshal the Exception to your UI thread using a SynchronizationContext or Dispatcher to call your exception-handler display (the easiest way may be to just re-throw the exception; I recommend embedding it as an InnerException so that you don't lose stack trace.)
If you really distrust the library (and don't want to give it the opportunity to stay alive even through its catch and finally blocks), you can explicitly marshal to the UI thread with a blocking invoke (to display your error message in a friendly way) and then call Environment.FailFast() to kill your process. This is a particularly harsh way to terminate (it will not call any cleanup code and literally tears down the process as quickly as possible), but it minimizes the possibility for any potentially damaging side effects if application state has become very corrupted. One possible thing to watch for here is that the UI thread could be blocked by some other UI invocation tied to the third-party code. If this happens, the application will deadlock. Again, this would be a pretty malicious thing for it to do, so you can probably ignore it.
I have a windows service, that is occasionally crashing and stopping when an unhandled ThreadInteruptedException is thrown. The main function of this service is to run a sync every x minutes (user defined timespan). I only once use the Thread.Interrupt() method twice (because I know how messy it can be) -
A client can connect via a TCP connection, but if the data is being sent to the client too fast for it to deal with it, it can send a "pause" signal back to the service, and then will resume when all the data has been handled (these could be wireless devices and could have multiple clients running at different speeds, so I didn't want to leave any of this in the TCP buffers). Pause then puts the thread to sleep, and resume interrupts this and continues.
The other use is in a similar situation when the TCP connection is originally connecting.
However, I've let this service run and this exception is thrown even when I do not connect any clients! No TCP connections are being made, and therefore neither of these interrupt methods are reached! I have put try statements to catch this exception around every Thread.Sleep() in the service, but this exception is still being raised!
I'm sure I'm missing something really obvious, but it's just not jumping out at me and I've been looking for days!
Has anyone ever encountered this
before?
Does anyone have any cheap
alternatives to sleeping and
interrupting a thread (not
while(paused) continue;)
Does anyone know of an easy way of
debugging an unhandled exception
like this? I can't even see the
stack to find out where this is
coming from!
Can any other method throw a ThreadInterruptedException? Or can any other methods cause a sleeping thread to be interrupted?
Thanks in advance
For anyone reading this. My accepted answer would be Hans Passant's comment. This solution worked just as well and got rid of any ThreadInterruptedExceptions
A TIE doesn't fall from the sky. Improve your code by removing the Interrupt() calls and replacing them with Auto/ManualResetEvent.Set().
I can't see that exceptions are being raised from methods that aren't even called.
Also, the ThreadInteruptedException is not raised by the Sleep method, but by the Interupt method - so try / catch on Sleep seems irrelevant here.
From MSDN:
You can interrupt a waiting thread by
calling Thread.Interrupt on the
blocked thread to throw a
ThreadInterruptedException, which
breaks the thread out of the blocking
call. The thread should catch the
ThreadInterruptedException and do
whatever is appropriate to continue
working. If the thread ignores the
exception, the runtime catches the
exception and stops the thread.
I have an unhandled exception handler. It shows a nice GUI and allows users to send an error report. Users can even leave their name and phone number and things, and our support department calls them back. Works well, looks good, makes customers less angry. In theory, anyway.
The problem is that my application uses background threads, and the threads don't seem to care if an exception was thrown on, say, the GUI thread (which makes sense), and just continue their work. That eventually results in a WER dialog poping up if the user lets my custom exception handler window stay open long enough, making it look like the error handler itself crashed.
I don't have access to the thread objects in the scope of the exception handler, so I can't suspend them. Making the thread objects globally accessible is not a solution either. My workaround for now is to use something like Globals.Crashed = true; in my exception handler, and to have my thread methods check that property at every loop iteration. Not perfect, but it minimizes the damage.
Does anyone know a less-hacky method? Is my approach wrong? Do I have to do it like WER does and launch an external program that suspends the main program and shows the error UI?
If you have an unhandled, unknown exception, you can assume that ANYTHING has happend and that your program might fail to do even the most simple thing. Consider e.g. the case that it has consumed all available memory - then you won't be able to send the error report either, because it probably requires memory to be allocated.
A good approach is to write a separate small application that just does the error reporting. That application can pick up the details to report from a file. That way your unknown exception handler would:
Dump the info to a file in the temp directory.
Start the error reporting app with the file name as an argument.
Terminate the failing process, before it does something stupid.
The temp file should be removed by the error reporting app.
You could track all your threads in a global Collection object, so that when your handler executes, it could simply iterate through the collection object and abort the threads there.
Take a look at the code in this question, Suspend Process in C#, you'll need to tweak it so as to not suspend your GUI thread and any that aren't background ones you've started, but it should do the trick.
The better option, however, is to try and launch your error report GUI as a separate process, passing any required information to it, and then kill the original process from your unhandled exception handler, rather than allowing anything to run in a potentially corrupt state.