There is a C#-program which hangs pretty rare. Execution of the program takes place on a remote machines and to start debugger is not an option. Run external profiler is more realistic, but also conjugate with huge difficulties. How can you determine the point of the program hang without profiler or debugger?
Option "detailed logging on FS" is poorly suited. The program consists of about 20 thousand lines of code and hangs not often.
I have tried Process Explorer but it works very strange (or I have not understood it). If you have managed to "catch" the moment when thread entered into an infinite loop, it is possible to see the stack in that moment. But this thread disappears quite quickly (whether in PE or it is really killed by the environment).
The option to create another application, application-monitor, is acceptable. If you can say how to create a dump of the main process or to obtain information about threads of the main process, it would be great. If you have some ready tools, it would be even better.
When an application crashes, it should normally be logged into Window's Application Event Log. It's not extremely detailed, but should give pretty solid clues anyway without any external tools needed.
To get there, you can either search "Event Log" in the Start Menu or find it in the Control Panel. It is located in the Administrative Tools section.
Once you're in the Event Viewer, open the Windows Logs item on the left then select Application. You should be able to find your application in the list using the Source column.
At the bottom you'll find the error detail, timestamp and a couple more infos which can help you debug your application.
Picture taken from Cyberlink.com
By 'hang' do you mean the program stops working until it is restarted or that the program pauses for an unusual amount of time. If the latter it could be in a heavy GC collection. If it's the former and you suspect some sort of infinite loop then in task manager (or process explorer) you should see it pretty much eating up one of the processor cores. For example if you have four cores and a program in hung in a tight loop, you will see roughly 25% cpu usage in the performance panel (assuming an otherwise lightly loaded machine).
MS supports managed debugging, see Debugging Managed Code Using the Windows Debugger You can use the sos extension to break the code execution and look at the state of the program. You might want to have the programs pdb handy if you take this approach.
Related
So this is a weird one.
I created a WPF application using MahApps for the GUI. So far my testing indicates that the app works fine on several different machines. Of course this is not the case on the client's machine.
The client makes use of Terminal Services and Windows Server 2008R2. Several users can be logged into their own version of the server at anytime. The app starts up fine once or twice, but after a day or so, it no longer opens up.
The app doesn't show up in the Application tab of Task Manager, but its process can be seen to be running in Processes Tab of Task Manager.
To be honest, I'm completely stumped. I had a look at the event manager log and couldn't find anything indicative of a problem. (Of course I might have missed something). I saw another SO question suggesting to disable hardware acceleration, but I'm not if that would help.
Any and all ideas would be greatly appreciated.
EDIT:
I thought I might mention the only thing that helps is if we restart the client machine.
EDIT:
I think I have isolated the issue to integration with Twain (should probably have mentioned that as another possible factor). I think the Twain library (unmanaged code) somehow stalls without sending back an error. Disabling it has "fixed" the issue.
This somehow relates to Twain and multi-session setups. I'm almost sure of it.
First you can analyze the wait chain in Windows Resource Monitor to check if there are any resources the process is waiting for. (You can find more information about the wait chain here or here.)
If you don't find any viable suspects there, you can create a memory dump of the hanging process and analyze the call stacks. If you don't know how to create one, you can read about it here. If you want to use Windows Task Manager and your OS is 64-bit then please be aware that you need to use the same bitness of Task Manager as the application.
That is: If your application is 64-bit then you have to use C:\Windows\System32\taskmgr.exe and if it's 32-bit you have to use C:\Windows\SysWOW64\taskmgr.exe. If you forget this important step you'll just get an unusable dump full of gibberish.
After you got the memory dump you can either load it into WinDbg (using the same bitness as the application) or Visual Studio (best to use 2015 or later) and analyze the call stacks of all running threads.
You can download WinDbg here and read about the necessary WinDbg configuration here. For the list of all threads you need to use this SOS command.
If you need help in loading memory dumps into Visual Studio you can find more information here.
After you've looked at the call stacks you most definitely find the answer what is waiting on what resources and is thus preventing the shutdown or startup of the application. It can either be a classic deadlock or an external resource like writing/reading of a file or some other waiting without a timeout like accessing a database or an URL that can't be reached at the moment. And of course it can also be just an infinite loop - if it doesn't consume much CPU then perhaps with some kind of DoEvents in between.
And last but very not least: If you are really interested what can be analyzed if an application hangs you can read about an example analysis done by the absolutely awesome great Mark Russinovich here.
I have a C# .NET 4.6 console application that is supposed to run continuously (over days/months). However, after a non deterministic duration, all running threads will freeze for no apparent reason (CPU usage is at 0%, memory is not particularly high), and trying to attach the application to an instance of Visual Studio 2015 for debugging will fail (pressing "pause" will cause Visual Studio to stop responding!).
I inspected the parallel stack traces (captured via a dump in the process explorer) and could not find any sign of a deadlock (which would otherwise be the obvious culprit).
Here are for example 2 parallel stacks that are frozen (not even in my code but in the DirectoryInfo.cs core library, and ServiceStack OrmLite library), even though there are absolutely no reasons for them to be stuck like this.
I have previously noticed this behavior of freezing on other parts of code so it really seems these libraries are "victims" of the freeze and not responsible for it. Even if there were a deadlock which I could not see, it should not prevent these threads from completing as they are not waiting for anything.
Finally, killing the process and restarting it will always allow the previously frozen operations to run successfully.
Do you have any clue on what could be causing this kind of weird behavior/have any advice on tools to be used to get more insight?
Seems to be both threads are hanging while reading information(execute reader reads data from file, Enumerator trying to read data from file system for directory information). Is the file system is accessible at that point of time? Are you able to access the directory in which the reading is happening ?
I have a client/server app. The server component runs, uses WCF in a 'remoting' fashion (binary formatter, session objects).
If I start the server component and launch the client, the first task the server does completes in <0.5sec.
If I start the server component with VS debugger attached, and then launch the client, the task takes upwards of 20sec to complete.
There are no code changes - no conditional compilation changes. The same occurs whether I have the server component compiled and running in 32-bit, 64-bit, with the VS hosting process, without the VS hosting process, or any combination of those things.
Possibly important: If I use the VS.NET profiler (sampling mode), then the app runs as quick as if there were no debugger attached. So I can't diagnose it that way. Just checked, instrumentation mode also runs quickly. Same for the concurrency profiling mode, works quickly.
Key data:
The app uses fairly heavy multithreading (40 threads in the standard thread pool). Creating the threads happens quickly regardless and is not a slow point. There are many locks, WaitHandles and Monitor patterns
The app raises no exceptions at all.
The app creates no console output.
The app is entirely managed code.
The app does map a few files on disk to a MemoryMappedFile: 1x750MB and 12x8MB and a few smaller ones
Measured performance:
CPU use is minimal in both cases; when debugger is attached, CPU sits at <1%
Memory use is minimal in both cases; maybe 50 or 60MB in both cases
There are plenty of page faults happening (ref MMF), however they happen more slowly when the debugger is attached
If the VS hosting process is not used, or basically the 'remote debugging monitor' comes into play, then that uses a decent amount CPU and creates a good number of page faults. But that's not the only time the problem is occurring
The performance difference is seen regardless of how the client is run. The only variable being changed is the server component being run via 'Start with debugging' vs launched from Explorer.
My ideas:
WCF slow when debugged?
MemoryMappedFiles slow when debugged?
40 threads used - slow to debug? Perhaps Monitors/locks notify debugger? Thread scheduling becomes strange/context switches very infrequent?
Cosmic background radiation granting intelligence and cruel sense of humour to VS
All seem stupidly unlikely.
So, my questions:
Why is this happening?
If #1 unknown, how can I diagnose / find out?
Since this is one of the first results when googling for this issue I would like to add my problem solution here in the hopes of saving someone 2 hours of research like in my case.
My code slowed down from 30 seconds without debugger attached to 4 minutes with debugger. because I forgot to remove a conditional breakpoint. These seem to slow down execution tremendously, so watch out for those
Exceptions can notably impact the performance of an application. There are two types of exceptions: 1st chance exceptions (the one gracefully handled with a try/catch block), and unhandled exceptions (that will eventually crash the application).
By default, the debugger does not show 1st chance exceptions, it just shows unhandled exceptions. And by default, it also shows only exceptions occurring in your code. However, even if it does not show them, it still handles them, so its performance may be impacted (especially in load tests, or big loop runs).
To enable 1st chance exceptions display in Visual Studio, click on "Debug | Exceptions" to invoke the Exceptions dialog, and check "Thrown" on the "Common language runtime" section (you can be more specific and choose wich 1st chance exception you want to see).
To enable 1st chance exceptions display originating from anywhere in the application, not just from your code, click on "Tools | Options | Debugging | General" and disable the "Enable Just My Code" option.
And for these specific "forensics mode" cases, I also strongly recommend to enable .NET Framework Source Stepping (it requires "Enable Just My Code" to be disabled). It's very useful to understand what's going on, sometimes just looking at the call stack is very inspiring - and helpful especially in the case of cosmic radiation mixup :-)
Two related interesting articles:
How to debug crashes and hangs
Configuring Visual Studio to Debug .NET Framework Source Code
Possible causes:
Various special kinds of breakpoints such as:
Conditional breakpoints
Memory changed breakpoints
Function breakpoints
Having the "Enable native code debugging" option checked.
This option makes debug runs slow as molasses.
This option is not under Tools -> Options -> Debug, (that would make too much sense,) it is under Project -> Properties -> Debug
Excessive use of System.Diagnostics.Debug.Write().
My benchmarks show that 1000 invocations of Debug.WriteLine() take only 10 milliseconds when running without debugging, but a whole 500 milliseconds when debugging. Apparently the Visual Studio Debugger, when active, intercepts DotNet debug output, and does extremely time-consuming stuff with it. (Over decades of using Microsoft products, we have come to expect nothing less from Microsoft.)
Replacing Debug.WriteLine() with kernel32.dll -> PInvoke -> OutputDebugStringW() does not help, because when running a DotNet application, Visual Studio completely ignores kernel32 debug output and only displays DotNet debug output, which is a completely different thing. (And I suppose that anything else would, again, make too much sense.)
Excessive amount of exceptions being thrown and caught, as another answer suggests.
Throwing an exception under DotNet is a mind-bogglingly slow operation.
Collecting a stack trace under DotNet is an insanely slow operation.
I have a longstanding C# .NET 3.5 application 'freeze' which I am at a loss with. There are two C# executables. One has a full UI, the other runs as a tray app. They both communicate via WCF to a third service app, also running in the tray.
Randomly the UI thread of main Winforms app will deadlock. Mysteriously if I quit the tray app the UI of this app will unlock.
Whenever I attach the debugger to either app I learn nothing useful. The UI thread is blocked in the frozen app on the Application.Run method. All other threads are either sleeping, or blocked on Invokes onto the UI thread.
Also mysteriously another running application like Photoshop will behave strangely whilst this deadlock is in place. Quitting the tray app sorts this too.
All I can deduce is that something is going wrong with the main Windows-level message pump, but I don't really understand how I can debug further into this. I've installed the framework source code and can see the deadlocked app is stuck in a while loop in :
Application.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop
but don't really understand enough to do anything with this information.
Does anyone have any advice at all on where to look further? I've been chasing this random deadlock bug for months.
Thanks,
Nick
I think this could be a red herring since this one is in Visual Studio SDK so really your debugging freezes.
I have had to debug a few work related/work unrelated freezes and they are very very nasty and require meticulous instrumentation and code review. So be patient!
Here is a few pieces of advice from me:
1) You will see a few red herrings on the way so be careful not to get bugged down on them and confuse manifestations of the problem with the cause itself.
2) What is the timing of this freeze? How long does it take? A TCP connection time out usually takes 23 seconds while a database connection times out in 30, a command in 120 seconds (could be different on different settings) so the time it takes is a big clue. If it does not resolve by itself and you have to close one application to get rid of it, it is almost certainly a thread or database deadlock.
3) Use sysinternal's Process Explorer and Process Monitor to see what they are doing and at what point they freeze. The last sactivity could give you a hint not always.
4) I know it will take sometime but start writing tracing in your code so that you find the axact location of the issue and from then on, it usually takes a few hours to days to find the problem.
5) If you have more info, post another question and let me know.
All:
I am using some custom Performance Counters that I have created. These are multi-instance, with a lifetime of "Process".
the problem: When I'm debugging in VS, if I stop the process and then start it again, I get an exception when my code attempts to create my performance counters. The exception indicates that the peformance counters already exist and that I cannot create them until the owning process releases them.
Once I get this error, there seems to be only 1 way out -- I have to close and restart Visual Studio -- it's as though VS gets ownership of my Process Lifetime performance counters even though it was really created by the owned process. Any idea what I can do about this?
BTW: the problem only seems to surface if my code actually writes to a performance counter before it is shut down.
I think you're doing battle with the Visual Studio hosting process. It is a helper .exe that hosts the CLR to improve the debugging experience, it is always running while you've got a project loaded into VS. Project + Properties, Debug tab, scroll down, uncheck the "Enable the Visual Studio hosting process" checkbox.
This does affect the debugging session somewhat, most notable is that the output written by Console.WriteLine() in your program no longer shows up in the Output window. Some obscure security options, not at all well documented. I doubt you'll have a problem.