I have a big and complex process that runs on a production environment that's basically a WPF user interface developed in C#. It also hosts threads and DLL's written in C++ unmanaged and managed code.
Typically, if an exception raises, it's caught and the related stack dump is written in a log file for post-mortem debugging purposes. Unfortunately, from time to time, the application crashes without writing any information in the log so we have no clue about who's causing the crash.
Does anybody know how to detect and eventually trace all the causes that make the application crash and are not detected with a simple try-catch block?
To give an example I saw that StackOverflow Exception is not caught and also errors like 0xc0000374 coming from unmanaged code are not detected. This is not a matter of debugging it. I know I can attach a debugger to the system and try to reproduce the problem. But as I told this is a production system and I have to analyze issues coming from the field after the problem occurred.
Unlike C# exceptions, C++ exceptions do not catch hardware exceptions such as access violations or stack overflows since C++ apps run unmanaged and directly on the cpu.
For post-crash analysis I would suggest using something like breakpad. breakpad will create a dump file which will give you very useful information such as a call-stack, running threads and stack/heap memory depending on your configuration.
This way you would not need to witness the crash happening or even try to reproduce it which I know from experience can be very difficult. All you would need is a way to retrieve these crash dumps from your users devices.
You can log exception by subscribing to AppDomain.UnhandledException event. Its args.ExceptionObject argument is of type object and is designed not to be limited by C# exceptions, so you can call ToString method to log it somewhere.
Also check MSDN docs for its limitations. For instance:
Starting with the .NET Framework 4, this event is not raised for exceptions that corrupt the state of the process, such as stack overflows or access violations, unless the event handler is security-critical and has the HandleProcessCorruptedStateExceptionsAttribute attribute.
Solved ! I followed Mohamad Elghawi suggestion and I integrated breakpad. After I struggled a lot in order to make it compiling, linking and working under Visual Studio 2008, I was able to catch critical system exceptions and generate a crash dump. I'm also able to generate a dump on demand, this is useful when the application stuck for some reason and I'm able to identify this issue with an external thread that monitors all the others.
Pay attention ! The visual studio solution isn't included in the git repo and the gyp tool, in contradiction as wrongly mentioned in some threads, it's also not there. You have to download the gyp tool separately and work a bit on the .gyp files inside the breadpad three in order to generate a proper solution. Furthermore some include files and definitions are missing if you want to compile it in Visual Studio 2008 so you have also to manage this.
Thanks guys !
Related
My application sometimes crash. but the visual studio always just output:
XXAplication return -1073741819 (0xc0000005) 'Access violation'。
So I even don't know which part is crash, or which line is crash.
the application is very big. So Is there any way can know or understand the crash position or any way can Debug it?
UPDATE
Sorry guys, Enable : "Common Language Runtime Exceptions" not any
help. still just output (0xc0000005) 'Access violation'。
Generally it's very hard, a stack trace may not lead you anywhere or it may even mislead you. Even in a native platform like C++ it would be hard, but at least they have tools to help diagnose these problems.
An access violation error means memory corruption. .net is a memory managed platform, so technically it wouldn't be possible.
But it is possible when you or the framework or a 3rd party library uses unsafe code or calls unmanaged APIs. When one of those corrupts memory, an error may not be thrown instantly. The error could surface anywhere else.
What you can do is look through the components you are using and try to upgrade them to the latest versions, or try to temporarily disable/replace them.
In my experience for example the System.DirectoryServices assembly caused a lot of those errors on an old Windows server. We upgraded to a newer Windows and newer .net and haven't seen this error ever since.
Sounds like you're running the app from VS, so turning on first chance exceptions in the IDE is probably the easiest way to figure out where the error is actually being thrown. If you go to the Debug | Exceptions... menu item, this will bring up a dialog where you can check the "Thrown" checkbox for "Common Language Runtime Exceptions". This will cause the debugger to break where the exception is thrown, even if it is handled by code.
In my case, I have an array of an obj1 and I wanted to send it to the server. Inside of my object I have a List of obj2. Inside obj2 I have a reference to obj1. As you probably guess, I have a overflow exception inside my object and System can not convert it to object and Access Violation occurred.
Try to find something like this in your new updated codes.
I have inherited a large and complex C# windows service project that crashes every now and then. The logging system is not logging any messages which I initially thought strange but I now understand that logging might fail if there's a stack overflow or out-of-memory exception.
So one of the tasks that I have is to try and find any recursive functions that might blow the stack. Is there any tooling in VS2010 or other code analysis software that would help detect recursive code?
As a second question: What else could cause logging to fail in a windows service?
(Project uses VS2010 but still targets .net 3.5 with C# 3.0)
Download Debug Diagnostic Tool, point it to your service and add stack overflow in the exception lists and let it run. When the service fails it will dump the memory. Open the dump in Visual Studio and check all stacks on all threads to identify the offensive code. You might need the original debugging symbols for your service to get intelligible inforamtion.
More about memory dumps debugging with VS2010 here. More about debugging this kind of problems with Tess Ferrandez watch this
Update: Tutorial on a stack overflow exception with details. It is based on a web app in IIS but you can easily apply the same technique to a service, it is just the way you take the memory dump that is different.
HTH
Are you attaching to the AppDomain.UnHandledException event? It should raise an event if an unhandled exception occurs. Also, have you checked the Eventlog?
It's very difficult to try and guess what could cause your service to crash. If you are attached to the event I mentioned then I guess it could only really be one of a few events, a StackOverflow exception being one. If you're not attaching to that event it could be anything.
If you're really at a loss you can always try to run the service as a console application from within Visual Studio. Visual Studio should then show you the error if it does occur. This is not always possible depending on your environment.
I'm developing a GUI app in C#.
It's a multithread app, and I would like to wrap all the threads (some of them I don't open, e.g. NetClient.StartDownload which is a none blocking function) with a try/catch statement so that if an exception is thrown and uncaught, I could log it and report to base.
I tried using Application.ThreadException and AppDomain.CurrentDomain.UnhandledException, but they seem to only catch GUI exceptions.
Is there a different way I need to handle this?
The code in each thread would need to have a try-catch. Exceptions are not marshalled across threads... and an unhandled exception just brings the application down.
To catch all Exceptions - just use the base Exception type.
catch(Exception e) { // log e }
Updated:
You could take a look at AppDomain.UnhandledException - handle this event for logging unhandled exceptions on any thread ; however you can't stop the application from going down. For more check out
http://www.albahari.com/threading/#_Introduction - towards the end of that page.
The best answer is: don't. There's already a system for doing this built right into the operating system.
If you let the exceptions "bubble up" and out of the program, modern OS'es include Windows Error Reporting, which will give the user the option of sending an error report to Microsoft. (Note: do not send error reports automatically - think long and hard about the legal ramifications regarding privacy laws...)
The error report includes not just the exception information and full stack trace, but also includes a good portion of your process' memory, as well as exactly which OS modules were loaded (so you can even tell whether the client had a relevant Windows Update patch applied). It wraps up all of this information into a minidump file (and also includes a few XML files with additional info).
This error report is uploaded to Microsoft's servers, so you don't have to worry about setting up and maintaining a "call home" server. All error reports for your programs are categorized using advanced heuristics into "buckets", with each "bucket" containing reports that are likely to be caused by the same bug.
Microsoft exposes all of this information to you (as the software publisher) through a site called Winqual. Using Winqual, you can examine all of the error reports, and use their heuristical bucketizing algorithms to decide which bugs are most impacting your customers. You can also download each individual report for more detailed investigation.
If you have a symbol server and source control server set up in your organization (and you certainly should), then you can just load the minidump from a downloaded error report right into Visual Studio, and it will automagically check the old source out of source control and allow you to inspect the exact state of the program at the moment it crashed.
Finding and fixing bugs has never been this easy.
To summarize, here's what you need to do:
Set up a symbol server and a source server in your organization. Establish a release policy to ensure that all releases get source-indexed pdbs added to the symbol server before they get released to the customer.
Establish an account with Winqual. You'll need an Authenticode code-signing certificate; if you get a non-VeriSign code-signing cert, then you'll also have to spend $100 for an "organizational certificate" from VeriSign.
Modify your release policy to include creating mapping files for your releases. These mapping files are uploaded to Winqual before shipping the release.
Do not catch unexpected exceptions. If you do ever need to catch them, be sure to rethrow using throw and not throw ex, so the stack trace and original program state is preserved.
For more information, I can highly recommend Debugging .NET 2.0 Applications by John Robbins. In spite of the "2.0" in the title, this book is still completely relevant today (except that the only source server example is using Visual SourceSafe, which was a complete joke even back then). Another good book if you need to do a lot of debugging is Advanced .NET Debugging, though it is starting to show its age, especially since the new VS2010 debugging advancements make a lot of its techniques out of date.
We need to catch exceptions at the smallest possible unit. As mentioned by Gishu, Exceptions which occur in the threads do not get propgated back to the main thread in many cases.
I had blogged about a similar experience some time back at WCF service unhandled exception error
I know I'm opening myself to a royal flaming by even asking this, but I thought I would see if StackOverflow has any solutions to a problem that I'm having...
I have a C# application that is failing at a client site in a way that I am unable to reproduce locally. Unfortunately, it is very difficult (impossible) for me to get any information that at all helps in isolating the source of the problem.
I have in place a rather extensive error monitoring framework which is watching for unhandled exceptions in all the usual places:
Backstop exception handler in threads I control
Application.ThreadException for WinForms exceptions
AppDomain.CurrentDomain.UnhandledException
Which logs detailed information in a place where I have access to them.
This has been very useful in the past to identify issues in production code, but has not been giving me any information at about the current series of issues.
My best guess is that the core issue is one of the "rude" exception types (thread abort, out of memory, stack overflow, access violation, etc.) that are escalating to a rude shutdown that are ripping down the process before I have a chance to see what is going on.
Is there anything that I can be doing to snapshot information as my process is crashing that would be useful? Ideally, I would be able to write out my custom log format, but I would be happy if I could have a reliable way of ensuring that a crash dump is written somewhere.
I was hoping that I could implement class deriving from CriticalFinalizerObject and have it spit a last-chance error log out when it is disposing, but that doesn't seem to be triggered in the StackOverflow scenario which I tested.
I am unable to use Windows Error Reporting and friends due to the lack of a code signing certificate.
I'm not trying to "recover" from arbitrary exceptions, I'm just trying to make a note of what went wrong on the way down.
Any ideas?
You could try creating a minidump file. This is a C++ API, but it should be possible to write a small C++ program that starts your application keeps a handle to the process, waits on the process handle, and then uses the process handle to create a minidump when the application dies.
If you have done what you claim:
Try-Catch on the Application.Run
Unhandled Domain Exceptions
Unhandled Thread Exceptions
Try Catch handlers in all threads
Then you would have caught the exception except perhaps if it is being thrown by a third party or COM component.
You certainly haven't given enough information.
What events does the client say leads up to the exception?
What COM or third party components do you use? (Do you properly instance and reference these components? Do you pass valid arguments to COM function calls?)
Do you make use of any un-managed - un-safe code?
Are you positive that you have all throw-capable calls covered with try-catch?
I'm just saying that no-one can offer you any helpful advice unless you post a heck of lot more information and even at that we probably can only speculate as to the source of you problem.
Have a set of fresh eyes look at your code.
Some errors cannot be caught by logging.
See this similar question for more details:
StackOverflowException in .NET
Here's a link explaining asynchronous exceptions (and why you can't recover from them):
http://www.bluebytesoftware.com/blog/PermaLink.aspx?guid=c1898a31-a0aa-40af-871c-7847d98f1641
we have a dotnet 2.0 desktop winforms app and it seems to randomly crash. no stack trace, vent log, or anything. it just dissapears.
There are a few theories:
machine simply runs out of resources. some people have said you will always get a window handle exception or a gdi exception but others say it might simply cause crashes.
we are using wrappers around non managed code for 2 modules. exceptions inside either of these modules could cause this behavior.
again, this is not reproducible so i wanted to see if there were any suggestions on how to debug better or anything i can put on the machine to "catch" the crash before it happens to help us understand whats going on.
Your best bet is to purchase John Robbins' book "Debugging Microsoft .NET 2.0 Applications". Your question can go WAY deeper than we have room to type here.
Sounds for me like you need to log at first - maybe you can attach with PostSharp a logger to your methods (see Log4PostSharp) . This will certainly slow you down a lot and produce tons of messages. But you should be able to narrow the problematic code down ... Attach there more logs - remove others. Maybe you can stress-test this parts, later. If the suspect parts are small enough you might even do a code review there.
I know, your question was about debugging - but this could be an approach, too.
You could use Process Monitor from SysInternals (now a part of Microsoft) filtered down to just what you want to monitor. This would give you a good place to start. Just start with a tight focus or make sure you have plenty of space for the log file.
I agree with Boydski. But I also offer this suggestion. Take a close look at the threads if you're doing multi-threading. I had an error like this once that took a long long time to figure out and actually ended up with John Robbins on the phone helping out with it. It turned out to be improper exception handling with threads.
Run your app, with pdb files, and attach WinDbg, make run.
Whem crash occur WinDbg stop app.
Execute this command to generate dump file :
.dump /ma c:\myapp.dmp
An auxiliary tool of analysis is ADPlus
Or try this :
Capturing user dumps using Performance alert
How to use ADPlus to troubleshoot "hangs" and "crashes"
Debugging on the Windows Platform
Do you have a try/catch block around the line 'Application.Run' call that starts the GUI thread? If not do add one and put some logging in for any exceptions thrown there.
You can also wrie up the Application.ThreadException event and log that in case that gives you some more hints.
you should use a global exception handler:
http://msdn.microsoft.com/en-us/library/system.windows.forms.application.threadexception.aspx
I my past I have this kind of behaviour primarily in relation to COM objects not being released and/or threading issues. Dive into the suggestions that you have gotten here already, but I would also suggest to look into whether you properly release the non-managed objects, so that they do not leak memory.