What is my next step to track down my app crash?
I have a c# winforms app which seems to randomly close without warning. I have run a dump file though winDbg but it hasn't narrowed down my search. Here's what it has given me.
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
Unable to load image C:\Windows\assembly\NativeImages_v4.0.30319_32\mscorlib\225759bb87c854c0fff27b1d84858c21\mscorlib.ni.dll, Win32 error 0n2
*** WARNING: Unable to verify checksum for mscorlib.ni.dll
*** WARNING: Unable to verify checksum for System.ni.dll
Failed to request MethodData, not in JIT code range
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 39186
Key : Analysis.DebugAnalysisProvider.CPP
Value: Create: 8007007e on IT21023
Key : Analysis.DebugData
Value: CreateObject
Key : Analysis.DebugModel
Value: CreateObject
Key : Analysis.Elapsed.mSec
Value: 61046
Key : Analysis.Memory.CommitPeak.Mb
Value: 162
Key : Analysis.System
Value: CreateObject
Key : CLR.BuiltBy
Value: NETFXREL2
Key : CLR.Engine
Value: CLR
Key : CLR.Version
Value: 4.6.1055.0
Key : Timeline.Process.Start.DeltaSec
Value: 230
Key : WER.OS.Branch
Value: win8_rtm
Key : WER.OS.Timestamp
Value: 2012-07-25T12:47:00Z
Key : WER.OS.Version
Value: 6.2.9200.16384
Key : WER.Process.Version
Value: 1.0.0.0
ADDITIONAL_XML: 1
OS_BUILD_LAYERS: 1
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 00000970
PROCESS_NAME: app.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE_STR: 80000003
MISSING_CLR_SYMBOL: 0
STACK_TEXT:
0074e4d4 772b28a2 772b9f60 0012020e 753bca8a ntdll!NtGetContextThread+0xc
0074e4e0 753bca8a 0074e5c4 0074e5b0 74a7bd90 ntdll!_SEH_epilog4_GS+0xa
0074e5a8 772b214f 0e990248 772b2128 fe9c0cf3 KERNELBASE!CheckSpecialLocales+0xd7
0074e628 772b24cc 00000000 0f27fc30 0e9905c0 ntdll!RtlpFreeHeap+0x667
0074e64c 746026cd 0e990000 00000000 0f27fc30 ntdll!RtlFreeHeap+0x206
0074e660 74600aa1 0f27fc30 0074ed58 0f250048 dbghelp!Win32LiveAllocationProvider::Free+0x19
0074e8a0 0f276660 0074e8c0 00000340 0074ed58 dbghelp!Win32LiveSystemProvider::QueryBuildString+0x1e4
WARNING: Frame IP not in any known module. Following frames may be wrong.
00000000 00000000 00000000 00000000 00000000 0xf276660
STACK_COMMAND: ~0s; .ecxr ; kb
SYMBOL_NAME: dbghelp!Win32LiveAllocationProvider::Free+19
MODULE_NAME: dbghelp
IMAGE_NAME: dbghelp.dll
FAILURE_BUCKET_ID: BREAKPOINT_80000003_dbghelp.dll!Win32LiveAllocationProvider::Free
OS_VERSION: 6.2.9200.16384
BUILDLAB_STR: win8_rtm
OSPLATFORM_TYPE: x86
OSNAME: Windows 8
IMAGE_VERSION: 6.2.9200.16384
FAILURE_ID_HASH: {0bbdb302-68ba-7226-fd2a-f9b8c4d5b299}
Followup: MachineOwner
---------
It doesn't seem to throw any clues which link it back to the app.
The dump file it self came from bugsplat would this be the cause? I tried it in WinDbg as bugsplat also couldn't offer any help, the active thread looks like this.
ntdll!NtGetContextThread
ntdll!_SEH_epilog4_GS
KERNELBASE!CheckSpecialLocales
ntdll!RtlpFreeHeap
ntdll!RtlFreeHeap
dbghelp!Win32LiveAllocationProvider::Free
dbghelp!Win32LiveSystemProvider::QueryBuildString
0xf276660
[InlinedCallFrame: 0074ee98]
08068e3e
08068bf1
08068bf1
080683d3
08067aef
6b8d9eda
System.Windows.Forms.Control.InvokeMarshaledCallbacks()
System.Windows.Forms.Control.WndProc(System.Windows.Forms.Message ByRef)
[InlinedCallFrame: 0074f074]
System.Windows.Forms.DataGridView.WndProc(System.Windows.Forms.Message ByRef)
System.Windows.Forms.Control+ControlNativeWindow.OnMessage(System.Windows.Forms.Message ByRef)
System.Windows.Forms.Control+ControlNativeWindow.WndProc(System.Windows.Forms.Message ByRef)
System.Windows.Forms.NativeWindow.Callback(IntPtr, Int32, IntPtr, IntPtr)
[InlinedCallFrame: 0074f2b4]
DomainBoundILStubClass.IL_STUB_PInvoke(MSG ByRef)
[InlinedCallFrame: 0074f2b4] System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG ByRef)
System.Windows.Forms.Application+ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr, Int32, Int32)
[InlinedCallFrame: 0074f2ec]
System.Windows.Forms.Application+ThreadContext.RunMessageLoopInner(Int32, System.Windows.Forms.ApplicationContext)
System.Windows.Forms.Application+ThreadContext.RunMessageLoop(Int32, System.Windows.Forms.ApplicationContext)
System.Windows.Forms.Application.Run(System.Windows.Forms.Form)
app.Program.Main() [C:\Projects\App\App\Program.cs # 55]
Which manages to link back to Application.Run command but not to anywhere else.
Am I missing something which can help me find what the cause of the crash is, or have I been over looking some information?
# # Update # #
The error turned out to be from the line.
System.Windows.Forms.Control.InvokeMarshaledCallbacks()
Which was being produced from a BeginInvoke call.
If you use the BugSplat .NET SDK, you'll get a managed call stack as well as the native call stack. If you need help with that, feel free to reach out to support#bugsplat.com
Related
I'm relatively new to WinDbg, so I'm hoping this is just something obvious I'm missing. I have a .NET assembly that contains the function calls InitializeProcThreadAttributeList, UpdateProcThreadAttribute and DeleteProcThreadAttributeList, defined using P/Invoke. All three of those functions are imported from the "kernel32.dll" library. My goal is to trace these functions to understand the ntdll sys calls being used.
I tried to set a breakpoint with this command:
bp KERNEL32!InitializeProcThreadAttributeList
, but got this error:
Couldn't resolve error at 'KERNEL32!InitializeProcThreadAttributeList'
Next, I searched for any sign of this function using the following command:
x kernel32!*procthread*
I got this reply:
00007ffd`c7598588 KERNEL32!_imp_InitializeProcThreadAttributeList = <no type information>
I then tried to set a breakpoint on this stub? function like this:
bp kernel32!_imp_InitializeProcThreadAttributeList
Although it says the breakpoint has been defined, when I continue execution, it errors with this:
Unable to insert breakpoint 0 at 00007ffd`c7598588, Win32 error 0n998
"Invalid access to memory location."
I also tried setting the breakpoint with "bu", but that produced the same error.
Does anyone know how to set a breakpoint in this situation? Thank you in advance!
Most of the functions in kernel32 have been implemented in kernelbase.dll
and kernel32 only contains a thunk
the telltale sign is the IMP denoting imported from
_imp_InitializeProcThreadAttributeList
try setting bp kernelbase!InitializeProcThreadAttributeList
0:000> x /v kernel32!InitializeProcThreadAttributeList
pub func 761f4fc1 0 kernel32!InitializeProcThreadAttributeList (<no parameter info>)
0:000> u kernel32!InitializeProcThreadAttributeList l1
kernel32!InitializeProcThreadAttributeList:
761f4fc1 ff25c0181476 jmp dword ptr [kernel32!_imp__InitializeProcThreadAttributeList (761418c0)]
0:000> ? poi(kernel32!_imp__InitializeProcThreadAttributeList)
Evaluate expression: 1978493618 = 75ed6ab2
0:000> ln poi(kernel32!_imp__InitializeProcThreadAttributeList)
Exact matches:
KERNELBASE!InitializeProcThreadAttributeList (<no parameter info>)
0:000> bp poi(kernel32!_imp__InitializeProcThreadAttributeList)
0:000> bl
0 e 75ed6ab2 0001 (0001) 0:**** KERNELBASE!InitializeProcThreadAttributeList
I need to inspect code, that JIT emits for generic method with different struct parameters. I read articles about WinDbg and SOS.dll and it's possible to inspect non-generic methods. But for generic methods, which should be JIT-ed for each struct type there is no JIT-ed code in method table, where can i find it?
namespace ConsoleApp1
{
class Program
{
public void Foo<T>(T arg)
{
//some code here
}
static void Main(string[] args)
{
var program = new Program();
program.Foo("test");
program.Foo(1.0);
program.Foo(new Guid());
program.Foo((byte)1);
program.Foo((char)1);
program.Foo(1);
}
}
}
WinDbg:
0:006> .loadby sos clr
0:006> !Name2EE ConsoleApp1!ConsoleApp1.Program
Module: 00007ffa3abf4118
Assembly: ConsoleApp1.exe
Token: 0000000002000002
MethodTable: 00007ffa3abf5a18
EEClass: 00007ffa3abf24b8
Name: ConsoleApp1.Program
Dump method table
0:006> !DumpMT -md 00007ffa3abf5a18
EEClass: 00007ffa3abf24b8
Module: 00007ffa3abf4118
Name: ConsoleApp1.Program
mdToken: 0000000002000002
File: D:\repos\ConsoleApp1\ConsoleApp1\bin\Debug\net462\ConsoleApp1.exe
BaseSize: 0x18
ComponentSize: 0x0
Slots in VTable: 7
Number of IFaces in IFaceMap: 0
--------------------------------------
MethodDesc Table
Entry MethodDesc JIT Name
00007ffa9759c080 00007ffa970f7fb8 PreJIT System.Object.ToString()
00007ffa97604360 00007ffa970f7fc0 PreJIT System.Object.Equals(System.Object)
00007ffa97674770 00007ffa970f7fe8 PreJIT System.Object.GetHashCode()
00007ffa975cf570 00007ffa970f8000 PreJIT System.Object.Finalize()
00007ffa3ad005c0 00007ffa3abf5a10 JIT ConsoleApp1.Program..ctor()
00007ffa3ad00078 00007ffa3abf59e0 NONE ConsoleApp1.Program.Foo(!!0) // "base" IL code
00007ffa3ad00480 00007ffa3abf5a00 JIT ConsoleApp1.Program.Main(System.String[])
And ofc I got no code for Foo
0:006> !U 00007ffa3abf59e0
Not jitted yet
Is it any ideas?
I took your code and compiled it as x86 debug build. I then attached to the process and waited until it terminated.
I find the same result as you:
Entry MethodDe JIT Name
[...]
00250038 00204d34 NONE JitExample2.Program.Foo(!!0)
Here it says NONE, which has the meaning "not jitted".
0:000> !dumpmd 00204d34
Method Name: JitExample2.Program.Foo(!!0)
[...]
IsJitted: no
So the answer seems to be confirmed: it's not jitted.
I don't know any official method of getting a list of jitted generic methods. But with the help of !bpmd it's possible:
0:000> !bpmd JitExample2 JitExample2.Program.Foo
Found 1 methods in module 00204024...
MethodDesc = 00204d34
Setting breakpoint: bp 0025074A [JitExample2.Program.Foo[[System.Char, mscorlib]](Char)]
Setting breakpoint: bp 002506EA [JitExample2.Program.Foo[[System.Byte, mscorlib]](Byte)]
Setting breakpoint: bp 00250687 [JitExample2.Program.Foo[[System.Guid, mscorlib]](System.Guid)]
Setting breakpoint: bp 00250627 [JitExample2.Program.Foo[[System.Double, mscorlib]](Double)]
Setting breakpoint: bp 002505B0 [JitExample2.Program.Foo[[System.__Canon, mscorlib]](System.__Canon)]
Setting breakpoint: bp 002507AA [JitExample2.Program.Foo[[System.Int32, mscorlib]](Int32)]
You can then use the addresses after bp to show the JIT code:
0:000> !u 0025074A
Normal JIT generated code
JitExample2.Program.Foo[[System.Char, mscorlib]](Char)
Begin 00250728, size 4b
C:\Users\For example John\JitExample2\Program.cs # 8:
00250728 55 push ebp
00250729 8bec mov ebp,esp
0025072b 83ec10 sub esp,10h
0025072e 33c0 xor eax,eax
[...]
I have a .NET windows service application with a memory leak.
Overview:
The service connects to another service via TCP, and writes data to a SQL server database. The service normally runs between 100 and 300 MB depending on how much data it has queued up. It generally takes 3 + days before I noticed the service's memory is getting out of hand, and will have a Out Of Memory exception within 2 weeks of running.
I think the issue might be related to a massive number of objects that are not leaving the finalization queue.
I have attached to the service with Windbg and run the !FinalizationQueue command and get:
0:029> !finalizequeue
SyncBlocks to be cleaned up: 0
Free-Threaded Interfaces to be released: 0
MTA Interfaces to be released: 0
STA Interfaces to be released: 0
generation 0 has 0 finalizable objects (32c69ea4->32c69ea4)
generation 1 has 266 finalizable objects (32c69a7c->32c69ea4)
generation 2 has 23191 finalizable objects (32c53020->32c69a7c)
Ready for finalization 5124616 objects (32c69ea4->33ff66c4)
Statistics for all finalizable objects (including all objects ready for finalization):
MT Count TotalSize Class Name
73e0b3ec 1 16 System.Threading.Gen2GcCallback
73e16648 1 20 Microsoft.Win32.SafeHandles.SafeFileMappingHandle
73e165f8 1 20 Microsoft.Win32.SafeHandles.SafeViewOfFileHandle
73df36f4 1 20 System.Security.Cryptography.SafeKeyHandle
...
73e2b208 36 720 Microsoft.Win32.SafeHandles.SafeWaitHandle
731a7754 33 924 System.Net.SafeFreeCredential_SECURITY
731a2d58 10 1240 System.Diagnostics.PerformanceCounter
59d01074 28 1680 System.Management.ManagementObject
73e2a8c4 175 2100 System.WeakReference
70b0385c 48 6144 System.Data.SqlClient.SqlConnection
731b2e94 201 6432 System.ComponentModel.Container
70b03ef4 86 14104 System.Data.SqlClient.SqlCommand
70b06014 712 14240 System.Data.SqlClient.SNIPacket
73e15b34 1248 24960 System.Threading.ThreadPoolWorkQueueThreadLocals
73e26c74 2238 44760 Microsoft.Win32.SafeHandles.SafeRegistryHandle
**73e2e424 1903 98956 System.Threading.Thread
70b08530 986 295800 System.Data.DataTable
70b08b08 20802 2662656 System.Data.DataColumn
6fab76dc 379139 7582780 System.Transactions.SafeIUnknown
73e30008 4740054 208562376 System.Threading.ReaderWriterLock**
I ran: !DumpHeap on the ReaderWriterLock and got:
0:029> !DumpHeap /d -mt 73e30008
... Omitted
Address MT Size
00c51f58 73e30008 44
00c5644c 73e30008 44
00c5ebc0 73e30008 44
00c81420 73e30008 44
... Omitted
I was trying to find out why there are millions of ReaderWriterLocks in the finalization queue, so I ran GCRoot on several of the addresses from the dumpheap and get:
0:029> !GCRoot 00c5ebc0
HandleTable:
005413cc (pinned handle)
-> 01d133c8 System.Object[]
-> 00cd2820 System.Diagnostics.TraceSource
-> 00cd29ac System.Diagnostics.SourceSwitch
-> 00c71704 System.Diagnostics.SwitchElementsCollection
-> 00c7177c System.Configuration.ConfigurationValues
-> 00c5fbec System.Configuration.RuntimeConfigurationRecord
-> 00c5ebec System.Configuration.RuntimeConfigurationRecord
-> 00c5eb74 System.Configuration.Internal.InternalConfigRoot
-> **00c5ebc0** System.Threading.ReaderWriterLock
Found 1 unique roots (run '!GCRoot -all' to see all roots).
I ran !do on the lock address:
0:029> !do 00c5ebc0
Name: System.Threading.ReaderWriterLock
MethodTable: 73e30008
EEClass: 73a6ff54
Size: 44(0x2c) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
73e2c00c 4001890 4 System.IntPtr 1 instance 0 _hWriterEvent
73e2c00c 4001891 8 System.IntPtr 1 instance 0 _hReaderEvent
73e2c00c 4001892 c System.IntPtr 1 instance 0 _hObjectHandle
73e2f6bc 4001893 10 System.Int32 1 instance 0 _dwState
73e2f6bc 4001894 14 System.Int32 1 instance 0 _dwULockID
73e2f6bc 4001895 18 System.Int32 1 instance 3 _dwLLockID
73e2f6bc 4001896 1c System.Int32 1 instance 0 _dwWriterID
73e2f6bc 4001897 20 System.Int32 1 instance 3 _dwWriterSeqNum
73e2c494 4001898 24 System.Int16 1 instance 0 _wWriterLevel
I wanted to see all the modules that had the ReaderWriterLock loaded, so I ran the Name2EE command on the System.Threading.ReaderWriterLock and got:
0:029> !Name2EE * System.Threading.ReaderWriterLock
Module: 739f1000
Assembly: mscorlib.dll
Token: 020004c8
MethodTable: 73e30008
EEClass: 73a6ff54
Name: System.Threading.ReaderWriterLock
Module: 00573fbc
Assembly: J_ToDatabase.exe
Module: 00574e0c
Assembly: JLibrary.dll
Module: 72f31000
Assembly: System.ServiceProcess.dll
Module: 72f71000
Assembly: System.dll
Module: 00578034
Assembly: JConfigHelper.dll
Module: 72141000
Assembly: System.Windows.Forms.dll
Module: 72da1000
Assembly: System.Drawing.dll
Module: 0057ac64
Assembly: JLib.dll
Module: 0057ba1c
Assembly: J_DB_Deploy.dll
Module: 0057c8a0
Assembly: JModel.dll
Module: 03571d7c
Assembly: JLicense.dll
Module: 71211000
Assembly: System.Xml.dll
Module: 03574074
Assembly: JSecureBase.dll
Module: 03574804
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 71921000
Assembly: System.Configuration.dll
Module: 71a11000
Assembly: System.Core.dll
Module: 0357539c
Assembly:
Module: 03575c7c
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 59ce1000
Assembly: System.Management.dll
Module: 03577b2c
Assembly: Localization.dll
Module: 03578b6c
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 03579070
Assembly:
Module: 03579840
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 709f1000
Assembly: System.Data.dll
Module: 702f1000
Assembly: System.Security.dll
Module: 70661000
Assembly: System.Numerics.dll
Module: 6faa1000
Assembly: System.Transactions.dll
Module: 6f4e1000
Assembly: System.EnterpriseServices.dll
Module: 6f471000
Assembly: System.EnterpriseServices.Wrapper.dll
Module: 043a2cbc
Assembly: JTPlib.dll
Module: 043a3af8
Assembly: JStore.Core.dll
Module: 043a4920
Assembly: SystemEventArgsUtility.dll
Module: 043a7378
Assembly: JMAFLibrary.dll
Module: 043ace28
Assembly: J_A_Library.dll
Module: 03c0afb8
Assembly: DatabaseSharedLibrary.dll
Module: 03ab1c24
Assembly:
Module: 03ab23f8
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 03ab28f8
Assembly:
Module: 03ab30c8
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Module: 03ab35c8
Assembly:
...
**NOTE JSecure showed up like this over 800 times!**
...
Module: 03ab3d98
Assembly: JSecure, Version=17.13.21.82, Culture=neutral
Assembly: JSecure, Version=17.13.21.82, Culture=neutral This seems to be part of the problem because it showed up over 800 times, I omitted all instances from the results because it was too much text.
Notes about JSecure, it is a dll that manages the licensing of my service. The dll is loaded into my service via
System.Reflection.Assembly.Load
method because of security reasons.
The two major questions I have are
What is causing JSecure at least seem to be loaded over 800 times?
What is causing millions of System.Threading.ReaderWriterLock to be left in the finalization queue?
Any clues on how to track down this memory leak further?
Update 1
Per suggestion in the comment, I have dug into the finalizer thread more.
When I run !Threads, I get 1905 results, though I think most of the threads are also in the finalizerqueue, there are only about 20 active threads. When I change to the Finalizer thread, and run kb to get the stack trace, I get:
0:002> kb
# ChildEBP RetAddr Args to Child
00 02e2f0e0 75892cc7 000003d4 00000000 00000000 ntdll!NtWaitForSingleObject+0xc
01 02e2f154 75892c02 000003d4 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x99
02 02e2f168 75ff9839 000003d4 ffffffff 0075d59c KERNELBASE!WaitForSingleObject+0x12
03 (Inline) -------- -------- -------- -------- combase!MTAThreadWaitForCall+0x43 [d:\blue\com\combase\dcomrem\channelb.cxx # 5657]
04 02e2f198 7609d524 006e1568 006e5808 02e2f404 combase!MTAThreadDispatchCrossApartmentCall+0x1ed [d:\blue\com\combase\dcomrem\chancont.cxx # 193]
05 (Inline) -------- -------- -------- -------- combase!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x33a2 [d:\blue\com\combase\dcomrem\channelb.cxx # 5052]
06 02e2f2ec 75f7caea 0075d59c 02e2f404 02e2f3cc combase!CRpcChannelBuffer::SendReceive2+0x62d [d:\blue\com\combase\dcomrem\channelb.cxx # 4796]
07 (Inline) -------- -------- -------- -------- combase!ClientCallRetryContext::SendReceiveWithRetry+0x2e [d:\blue\com\combase\dcomrem\callctrl.cxx # 1090]
08 (Inline) -------- -------- -------- -------- combase!CAptRpcChnl::SendReceiveInRetryContext+0x81 [d:\blue\com\combase\dcomrem\callctrl.cxx # 715]
09 02e2f34c 75fc1789 0075d59c 02e2f404 02e2f3cc combase!DefaultSendReceive+0x9e [d:\blue\com\combase\dcomrem\callctrl.cxx # 671]
0a (Inline) -------- -------- -------- -------- combase!CAptRpcChnl::SendReceive+0x38 [d:\blue\com\combase\dcomrem\callctrl.cxx # 752]
0b 02e2f3b0 7609a010 0075d59c 02e2f404 02e2f3cc combase!CCtxComChnl::SendReceive+0x248 [d:\blue\com\combase\dcomrem\ctxchnl.cxx # 735]
0c 02e2f3d4 77865769 0073be14 02e2f430 77865740 combase!NdrExtpProxySendReceive+0x5c [d:\blue\com\combase\ndr\ndrole\proxy.cxx # 2017]
0d 02e2f3ec 778e6c1b ee46341f 0073be14 02e2f850 rpcrt4!NdrpProxySendReceive+0x29
0e 02e2f818 76099e1e 75f6d8f8 75f72ff0 02e2f850 rpcrt4!NdrClientCall2+0x22b
0f 02e2f838 75f7c46f 02e2f850 00000008 02e2f8d8 combase!ObjectStublessClient+0x6c [d:\blue\com\combase\ndr\ndrole\i386\stblsclt.cxx # 215]
10 02e2f848 75fc1c07 0073be14 02e2f888 006ce0a8 combase!ObjectStubless+0xf [d:\blue\com\combase\ndr\ndrole\i386\stubless.asm # 171]
11 02e2f8d8 75fc3024 006ce098 74d08aa9 02e2f99c combase!CObjectContext::InternalContextCallback+0x1e4 [d:\blue\com\combase\dcomrem\context.cxx # 4428]
12 02e2f92c 74d086b1 006ce0a8 74d08aa9 02e2f99c combase!CObjectContext::ContextCallback+0xbc [d:\blue\com\combase\dcomrem\context.cxx # 4332]
13 02e2fa24 74d08782 74d081d7 02e2fab4 ec1c2444 clr!CtxEntry::EnterContext+0x243
14 02e2fa5c 74d087c3 74d081d7 02e2fab4 00000000 clr!RCW::EnterContext+0x3a
15 02e2fa78 74d085a0 006ce150 ec1c2490 752963e8 clr!RCWCleanupList::ReleaseRCWListInCorrectCtx+0xc0
16 02e2fad4 74cd19c5 ec1c253c 7529562c 00000000 clr!RCWCleanupList::CleanupAllWrappers+0x14d
17 02e2fb24 74cd1bc8 006ae690 00000001 ec1c24e0 clr!SyncBlockCache::CleanupSyncBlocks+0xd0
18 02e2fb3c 74cd3ad8 ec1c2550 02e2fc58 02e2fcd0 clr!Thread::DoExtraWorkForFinalizer+0x7f
19 02e2fb84 74d09b0e 02e2fcd0 006bdcc8 02e2fcd0 clr!WKS::GCHeap::FinalizerThreadWorker+0x232
1a 02e2fb98 74d09b78 ec1c2224 02e2fcd0 00000000 clr!ManagedThreadBase_DispatchInner+0x67
1b 02e2fc3c 74d09c45 ec1c2280 00000000 00000040 clr!ManagedThreadBase_DispatchMiddle+0x7e
1c 02e2fc98 74c852e2 ec1c230c 74c8517c 00000000 clr!ManagedThreadBase_DispatchOuter+0x5b
1d 02e2fd14 74cd5f91 00000000 00000000 00000000 clr!WKS::GCHeap::FinalizerThreadStart+0x1a0
1e 02e2fdb4 77917c04 006bd910 77917be0 ee68e49d clr!Thread::intermediateThreadProc+0x4d
1f 02e2fdc8 77cdab8f 006bd910 ee10b078 00000000 kernel32!BaseThreadInitThunk+0x24
20 02e2fe10 77cdab5a ffffffff 77cbffdf 00000000 ntdll!__RtlUserThreadStart+0x2f
21 02e2fe20 00000000 74cd5f48 006bd910 00000000 ntdll!_RtlUserThreadStart+0x1b
I'm trying to track down a NullReferenceException from a dump. The NullReferenceException is not the crashing exception, rather the crashing exception is a TargetInvocationException with an InnerException which is the NullReferenceException.
I'm using Windbg with SOS, I use the the command analyze -v and this gives me the call stack of the NullReferenceException:
EXCEPTION_OBJECT: !pe f6cb150
Exception object: 000000000f6cb150
Exception type: System.NullReferenceException
Message: Object reference not set to an instance of an object.
InnerException: <none>
StackTrace (generated):
SP IP Function
000000002CD9D8C0 000007FF01E7C639 MyDll!DoSomething2()+0xe99
000000002CD9DBE0 000007FF01E7B11D MyDll!DoSomething1()+0x43d
000000002CD9DD20 000007FF01E7AB11 MyDll!WorkerDoWork(System.Object, System.ComponentModel.DoWorkEventArgs)+0x51
000000002CD9DD80 000007FEEA68A0F2 System_ni!System.ComponentModel.BackgroundWorker.WorkerThreadStart(System.Object)+0x62
Notice, that I get the method names with byte offsets, but no line numbers. DoSomething2 is a large function, so it's not obvious where the NullReferenceException occurred.
I attempted to follow the instructions in Tess Ferrandez's blog:
.Net exceptions - Tracking down where in the code the exceptions occurred
But I got stuck early on where I attempt to determine the method descriptor for the method DoSomething2 using !ip2md with the IP of DoSomething2: 7FF01E7C639:
> !ip2md 7FF01E7C639
Failed to request MethodData, not in JIT code range
Note that the !ip2md command succeeds on the IP of the method where the TargetInvocationException occurred.
Question:
Where can I go from here to narrow down what line in DoSomething2 is crashing?
Note that I cannot reproduce the crash, so all I have this (and several duplicate) dumps.
Additional Notes:
.NET 4.0
Windbg Version: 6.12.0002.633 AMD64
I'm new to Windbg: so the more info, the better
Edit 1
When I don't have symbols set up properly, I get the following:
STACK_TEXT:
00000000`2cd9d8c0 00000000`ffffffff MyDll!Unknown_0xe99+0xe99
00000000`2cd9dbe0 00000000`ffffffff MyDll!Unknown_0x43d+0x43d
00000000`2cd9dd20 00000000`ffffffff MyDll!Unknown_0x51+0x51
00000000`2cd9dd80 00000000`ffffffff system_ni! System.ComponentModel.BackgroundWorker.WorkerThreadStart+0x62
When I set it up to point to my symbol server and turn on !sym noisy, it appears to load symbols properly:
0:000> ld MyDll
DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\MyDll.dll - file not found
SYMSRV: c:\symbols\MyDll.dll\4F3D6F4B154000\MyDll.dll not found
SYMSRV: http://msdl.microsoft.com/download/symbols/MyDll.dll/4F3D6F4B154000/MyDll.dll not found
SYMSRV: \\mysymbolserver\store\Mydll.dll\4F3D6F4B154000\file.ptr
SYMSRV: MyDll.dl_ from \\mysymbolserver\store: uncompressed
DBGHELP: c:\symbols\MyDll.dll\4F3D6F4B154000\MyDll.dll - OK
DBGENG: c:\symbols\MyDll.dll\4F3D6F4B154000\MyDll.dll - Mapped image memory
SYMSRV: c:\symbols\MyDll.pdb\8AFC2BE7529A41289FA9FBCEDB6836161\Mydll.pdb not found
SYMSRV: http://msdl.microsoft.com/download/symbols/Mydll.pdb/8AFC2BE7529A41289FA9FBCEDB6836161/MyDll.pdb not found
SYMSRV: \\mysymbolserver\store\MyDll.pdb\8AFC2BE7529A41289FA9FBCEDB6836161\file.ptr
SYMSRV: MyDll.pd_ from \\mysymbolserver\store: uncompressed
DBGHELP: MyDll - private symbols & lines
c:\symbols\MyDll.pdb\8AFC2BE7529A41289FA9FBCEDB6836161\MyDll.pdb
Symbols loaded for MyDll
Edit 2
I tried using !name2ee as follows:
0:000> !name2ee MyDll!MyType.DoSomething2
Module: 000007ff004995b8
Assembly: Autodesk.DataManagement.Client.Framework.Vault.dll
<invalid module token>
So, no luck there. But then I almost seemed to get somewhere with this:
0:000> !name2ee MyDll.dll!MyNamespace.MyType
Module: 000007ff004995b8
Assembly: MyDll.dll
Token: 000000000200008c
MethodTable: 000007ff01b2e258
EEClass: 000007ff01b415e0
Name: MyNamespace.MyType
0:000> !dumpmt -md 7ff01b2e258
EEClass: 000007ff01b415e0
Module: 000007ff004995b8
Name: MyNamspace.MyType
mdToken: 000000000200008c
File: C:\Program Files\MyCompany\MyProduct\Bin\MyDll.dll
BaseSize: 0x30
ComponentSize: 0x0
Slots in VTable: 31
Number of IFaces in IFaceMap: 2
--------------------------------------
MethodDesc Table
Entry MethodDesc JIT Name
000007feeb31a2c0 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007feeb3689f0 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007feeb3688c0 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007feeb353440 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01b01300 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01e89140 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01b9c080 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01f45f40 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01a9b358 000007ff01b2e128 NONE MyType.DoSomething3()
000007ff01a9b360 000007ff01b2e130 NONE MyType.DoSomething4()
000007ff01a9b368 000007ff01b2e138 NONE MyType.DoSomething5()
000007ff01e79800 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff020fea80 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01a9b3b0 000007ff01b2e1b0 NONE MyType.DoSomething6()
000007ff01a9b3b8 000007ff01b2e1b8 NONE MyType.DoSomething7()
000007ff01a9b328 000007ff01b2e0f0 NONE MyType..ctor()
000007ff01b01280 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01e7a810 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01e7aac0 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01e83240 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01f19520 000007ff01b2e178 JIT MyType.RunWorkerCompleted(System.Object, System.ComponentModel.RunWorkerCompletedEventArgs)
000007ff01e7ace0 0000000000000000 JIT 0000000000000000 is not a MethodDesc
000007ff01e7b7a0 0000000000000000 JIT 0000000000000000 is not a MethodDesc
000007ff01e7b710 0000000000000000 JIT 0000000000000000 is not a MethodDesc
000007ff01e7d2b0 0000000000000000 JIT 0000000000000000 is not a MethodDesc
000007ff01b015f0 0000000000000000 JIT 0000000000000000 is not a MethodDesc
000007ff01b88ce0 0000000000000000 JIT 0000000000000000 is not a MethodDesc
000007ff01a9b3e0 000007ff01b2e200 NONE MyType.DoSomething8()
000007ff01b921e0 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01b933b0 0000000000000000 NONE 0000000000000000 is not a MethodDesc
000007ff01b93870 0000000000000000 NONE 0000000000000000 is not a MethodDesc
I'm guessing all the missing entries (those listed with "is not a MethodDesc") are due to this not being a full mini-dump. Is that right?
It looks like WinDbg doesn't pick up symbols for your DLL. You can look into that by setting the symbol path and using !sym noisy to troubleshoot as necessary.
I can't say why !ip2md doesn't work in this case, but there are other ways to get the code for DoSomething2. Try !name2ee on the method name, e.g. !name2ee *!TypeName.DoSomething2 or you can get it via the type itself like this !name2ee *!Namespace.TypeName and then !dumpmt -md <method table> on the method table you get from !name2ee.
Once you have the code, the !u command can show you an .NET annotated dump of the assembly code. By using the offset from the exception, you may be able to determine the nature of the NullReferenceException.
You are probably using a stripped PDB file, the default generated for a project in the Release build. All file and line number info is removed from such a file.
Switch to the Release configuration, Project + Properties, Build tab, Advanced, Debug Info = "full".
This is intentional btw, line number info is not very accurate for the Release build. The jitter optimizer moves code around so you'll need to keep in mind that a displayed line number is an approximation.
In a comment above, you mention that the || command yields "User mini dump". In order to properly debug .NET code, you need a full dump, which would indicate "Full memory user mini dump" from the || command. I think this is your problem. Without access to the full loader heaps, it's not possible to map a code address back to a .NET method, so you can't get a stack trace. If you can reproduce this problem, capture a full dump. You can use ADPlus, ProcDump or DebugDiag to capture a dump on crash.
My program throws an error which it cannot handle by a catch(Exception e) block and then it crashes:
Access Violation Corrupted State Exception.
This is the weird thing, because, as I know, corrupted state exceptions are thrown from unmanaged code, while here I get this exception while calling a StringBuilder method.
The code runs in a background thread and crashes from time to time which cannot be easily reproduced. So I attached WinDbg to the process and have the following stack of the exception:
000000001dabd8c8 000007feea129a1d [HelperMethodFrame: 000000001dabd8c8]
000000001dabda00 000007fee90cfce8 System.Text.StringBuilder.ExpandByABlock(Int32)
000000001dabda40 000007fee90cfba4 System.Text.StringBuilder.Append(Char*, Int32)
000000001dabdaa0 000007fee9102955 System.Text.StringBuilder.Append(System.String, Int32, Int32)
000000001dabdaf0 000007ff00bf5ce3 MineUtils.Common.Strings.Strings.Replace(System.String, System.String, System.String, Boolean, Boolean)
000000001dabdb90 000007ff00bf5a59 MineUtils.Common.Strings.Strings.RemoveSubstrings(System.String, System.String, System.String, Boolean) [D:\Programs\Visual Studio 2005 Projects\MineUtils.Common\Strings\Strings.Common-Main.cs # 1481
WinDbg shows this exception occurred:
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000007feea129a1d (clr!WKS::gc_heap::find_first_object+0x0000000000000092)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000003d80
Attempt to read from address 0000000000003d80
I read such exceptions can be handled with a method attribute [HandleProcessCorruptedStateExceptions], but why does this exception ever occur if I only use StringBuilder?
This is the previous WinDbg analysis (StringBuilder.ToString() causes the exception):
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
clr!WKS::gc_heap::find_first_object+92
000007fe`ea129a1d f70100000080 test dword ptr [rcx],80000000h
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000007feea129a1d (clr!WKS::gc_heap::find_first_object+0x0000000000000092)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000001
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000001c98
Attempt to read from address 0000000000001c98
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: 0000000000001c98
READ_ADDRESS: 0000000000001c98
FOLLOWUP_IP:
clr!WKS::gc_heap::find_first_object+92
000007fe`ea129a1d f70100000080 test dword ptr [rcx],80000000h
MOD_LIST: <ANALYSIS/>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK:
(TransitionMU)
000000001AB7DFC0 000007FEE90CFE07 mscorlib_ni!System.Text.StringBuilder.ToString()+0x27
000000001AB7E010 000007FF00C750A9 SgmlReaderDll!Sgml.Entity.ScanToken(System.Text.StringBuilder, System.String, Boolean)+0x169
000000001AB7E080 000007FF00C760E6 SgmlReaderDll!Sgml.SgmlDtd.ParseParameterEntity(System.String)+0xc6
000000001AB7E0F0 000007FF00C76FD8 SgmlReaderDll!Sgml.SgmlDtd.ParseModel(Char, Sgml.ContentModel)+0x298
000000001AB7E160 000007FF00C7701C SgmlReaderDll!Sgml.SgmlDtd.ParseModel(Char, Sgml.ContentModel)+0x2dc
000000001AB7E1D0 000007FF00C7701C SgmlReaderDll!Sgml.SgmlDtd.ParseModel(Char, Sgml.ContentModel)+0x2dc
000000001AB7E240 000007FF00C76BA5 SgmlReaderDll!Sgml.SgmlDtd.ParseContentModel(Char)+0x65
000000001AB7E290 000007FF00C763D7 SgmlReaderDll!Sgml.SgmlDtd.ParseElementDecl()+0xe7
000000001AB7E320 000007FF00C747A1 SgmlReaderDll!Sgml.SgmlDtd.Parse()+0xc1
000000001AB7E370 000007FF00C73EF5 SgmlReaderDll!Sgml.SgmlDtd.Parse(System.Uri, System.String, System.IO.TextReader, System.String, System.String, System.Xml.XmlNameTable)+0x175
000000001AB7E410 000007FF00C73B33 SgmlReaderDll!Sgml.SgmlReader.LazyLoadDtd(System.Uri)+0x163
000000001AB7E480 000007FF00C737B9 SgmlReaderDll!Sgml.SgmlReader.OpenInput()+0x19
000000001AB7E4E0 000007FF00C7334C SgmlReaderDll!Sgml.SgmlReader.Read()+0x1c
000000001AB7E530 000007FEE5983C4C System_Xml_ni!System.Xml.XmlLoader.Load(System.Xml.XmlDocument, System.Xml.XmlReader, Boolean)+0xac
000000001AB7E590 000007FEE5983730 System_Xml_ni!System.Xml.XmlDocument.Load(System.Xml.XmlReader)+0x90
...
000000001AB7F0A0 000007FEE97ED792 mscorlib_ni!System.Threading.Tasks.Task.Execute()+0x82
000000001AB7F100 000007FEE90A181C mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0xdc
000000001AB7F160 000007FEE97E7F95 mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)+0x1b5
000000001AB7F1E0 000007FEE97E7D90 mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry(Boolean)+0xb0
000000001AB7F220 000007FEE90EBA83 mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+0x193
000000001AB7F2C0 000007FEE90EB8D5 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()+0x35
(TransitionUM)
EXCEPTION_OBJECT: !pe 2a61228
Exception object: 0000000002a61228
Exception type: System.ExecutionEngineException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131506
MANAGED_OBJECT_NAME: System.ExecutionEngineException
MANAGED_STACK_COMMAND: _EFN_StackTrace
LAST_CONTROL_TRANSFER: from 000007feea12bce4 to 000007feea129a1d
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
FAULTING_THREAD: ffffffffffffffff
DEFAULT_BUCKET_ID: INVALID_POINTER_READ_CALL
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ_CALL
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_WRONG_SYMBOLS_CALL__SYSTEM.EXECUTIONENGINEEXCEPTION
UPDATED AGAIN
Here is the WinDbg stack of the exception after I enabled paged heap:
(1480.e84): Access violation - code c0000005 (first chance)
ntdll!ZwTerminateProcess+0xa:
00000000`77c415da c3 ret
0:023> !clrstack
OS Thread Id: 0xe84 (23)
Child SP IP Call Site
0000000037ded848 0000000077c415da [HelperMethodFrame: 0000000037ded848]
0000000037dedab0 000007fee9effd17 System.Text.StringBuilder.ToString()*** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\8f7f691aa155c11216387cf3420d9d1b\mscorlib.ni.dll
0000000037dedb00 000007ff00cceae9 Sgml.Entity.ScanToken(System.Text.StringBuilder, System.String, Boolean)
0000000037dedb70 000007ff00cd19b2 Sgml.SgmlDtd.ParseAttDefault(Char, Sgml.AttDef)
0000000037dedbc0 000007ff00cd120b Sgml.SgmlDtd.ParseAttDef(Char)
0000000037dedc00 000007ff00cd1057 Sgml.SgmlDtd.ParseAttList(System.Collections.Generic.Dictionary`2<System.String,Sgml.AttDef>, Char)
0000000037dedc50 000007ff00cd10cd Sgml.SgmlDtd.ParseAttList(System.Collections.Generic.Dictionary`2<System.String,Sgml.AttDef>, Char)
0000000037dedca0 000007ff00cd0e9a Sgml.SgmlDtd.ParseAttList()
0000000037dedd10 000007ff00cce1f1 Sgml.SgmlDtd.Parse()
0000000037dedd60 000007ff00ccd945 Sgml.SgmlDtd.Parse(System.Uri, System.String, System.IO.TextReader, System.String, System.String, System.Xml.XmlNameTable)
0000000037dede00 000007ff00ccd582 Sgml.SgmlReader.LazyLoadDtd(System.Uri)
0000000037dede70 000007ff00ccd1f9 Sgml.SgmlReader.OpenInput()
0000000037deded0 000007ff00cccd8c Sgml.SgmlReader.Read()
0000000037dedf20 000007fee67b3bfc System.Xml.XmlLoader.Load(System.Xml.XmlDocument, System.Xml.XmlReader, Boolean)*** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xml\8e4323f5bfb90be4621456033d8b404b\System.Xml.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xml\8e4323f5bfb90be4621456033d8b404b\System.Xml.ni.dll
0000000037dedf80 000007fee67b36e0 System.Xml.XmlDocument.Load(System.Xml.XmlReader)
[deleted]
0000000037deea90 000007feea61d432 System.Threading.Tasks.Task.Execute()
0000000037deeaf0 000007fee9ed17ec System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0000000037deeb50 000007feea617c35 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
0000000037deebd0 000007feea617a30 System.Threading.Tasks.Task.ExecuteEntry(Boolean)
0000000037deec10 000007fee9f1b953 System.Threading.ThreadPoolWorkQueue.Dispatch()
0000000037deecb0 000007fee9f1b7a5 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
0000000037def310 000007feeae4dc54 [DebuggerU2MCatchHandlerFrame: 0000000037def310]
0:023> !verifyheap
-verify will only produce output if there are errors in the heap
The garbage collector data structures are not in a valid state for traversal.
It is either in the "plan phase," where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
object 000000000e34caf8: bad member 000000001024b9a0 at 000000000e34cb08
curr_object: 000000000e34caf8
Last good object: 000000000e34cab0
----------------
0:023> !analyze
Last event: 1480.e84: Exit process 0:1480, code 80131506
debugger time: Sun Sep 18 14:22:42.592 2011 (UTC + 1:00)
0:023> !analyze -v
Last event: 1480.e84: Exit process 0:1480, code 80131506
debugger time: Sun Sep 18 14:22:42.592 2011 (UTC + 1:00)
0:023> .do e34cab0
^ Syntax error in '.do e34cab0'
0:023> !do e34cab0
Name: System.String
MethodTable: 000007feea026870
EEClass: 000007fee9baed58
Size: 72(0x48) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: appliedFiltersContainer
Fields:
MT Field Offset Type VT Attr Value Name
000007feea02c758 4000103 8 System.Int32 1 instance 23 m_stringLength
000007feea02b298 4000104 c System.Char 1 instance 61 m_firstChar
000007feea026870 4000105 10 System.String 0 shared static Empty
>> Domain:Value 00000000021343a0:000000000db21420 <<
0:023> !do e34caf8
<Note: this object has an invalid CLASS field>
Name: System.Reflection.RuntimeAssembly
MethodTable: 000007feea02a128
EEClass: 000007fee9baf968
Size: 48(0x30) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
000007feea9ef7f0 4000e14 8 ...solveEventHandler 0 instance 0000000000000000 _ModuleResolve
000007feea036338 4000e15 10 ...che.InternalCache 0 instance 000000001024b9a0 m_cachedData
000007feea0259c8 4000e16 18 System.Object 0 instance 000000000e3abd18 m_syncRoot
000007feea033450 4000e17 20 System.IntPtr 1 instance 37a95f10 m_assembly
What can it be?
Recently, I was faced with a managed heap corruption which was something new to me. I was very frustrated with it and had to learn many things to be able to debug it. I want to thank Seva Titov who gave me right direction to start. His answer was concise and very helpful. I want to log the actions I have taken to debug the problem for my own reference. Probably this will be helpful for others who are new to this.
Debug Heap Corruption in .NET 4:
How to suspect the heap corruption?
Briefly:
The application crashes randomly with no regards to the applied exception catching and even goes through blankets like catch(Exception) which are supposed to catch all exceptions.
Examining the CLR stack in the application crash dumps shows the garbage collector on the top of the stack:
000000001dabd8c8 000007feea129a1d [**HelperMethodFrame**: 000000001dabd8c8]
000000001dabda00 000007fee90cfce8 System.Text.StringBuilder.ExpandByABlock(Int32)
000000001dabda40 000007fee90cfba4 System.Text.StringBuilder.Append(Char*, Int32)
...
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000007feea129a1d (**clr!WKS::gc_heap**::find_first_object+0x0000000000000092)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000003d80
...
The CLR stack always shows different points. Whether the crash occurred or the code which is shown is clearly irrelevant, like StringBuilder's method which is shown to cause the exception.
For more details refer to .NET Crash: Managed Heap Corruption calling unmanaged code.
Going step by step. Each next step is used if the previous one doesn't help.
Step 1. Check the code.
Check the code for unsafe or native code usages:
Review the code for unsafe, DllImport statements.
Download .NET Reflector and use it to analyze the application assemblies for PInvoke. In the same way, analyze the third-party assemblies which are used by the application.
If unsafe or native code usage is found, direct extra attention to those. The most common cause of the heap corruption in such cases is a buffer overflow or an argument type mismatch. Ensure that the buffer supplied to the native code to fill is big enough and that all arguments passed to the native code are of the expected type.
Step 2. Check if this corrupted state exception can be caught.
To handle such exceptions, one need to decorate the method which contains the catch(Exception) statement with the [HandleProcessCorruptedStateExceptions] attribute or apply the following in the app.config file:
<configuration>
<runtime>
<legacyCorruptedStateExceptionsPolicy enabled="true" />
</runtime>
</configuration>
In the case the exception was caught successfully, you can log and examine it. This means this is not a corrupted heap issue.
Corrupted heap exceptions cannot be handled at all: HandleProcessCorruptedStateExceptions doesn't seem to work.
More information on corrupted state exceptions, see All about Corrupted State Exceptions in .NET4.
Step 3. Live debugging.
In this step, we debug the crashing application live in the production environment (or where we can reproduce the crash).
Download Debugging Tools for Windows from Microsoft Windows SDK for Windows 7 and .NET Framework 4 (a web installer will be downloaded which will allow selecting the required components to install - mark all components). It will install both 32 and 64 bit (if your system is x64) versions of the required debugging tools.
Here one needs to know how to attach WinDbg to a live process, how to take crash dumps and examine them, how to load SOS extension in WinDbg (google for details).
Enable debugging helpers:
Launch Application Verifier (C:\Program Files\Application Verifier - use the required edition, either x86 or x64, depending on your executable compilation mode), add your executable there in the left pane and in the right pane check one node "Basics / Heaps". Save the changes.
Launch Global Flags helper (C:\Program Files\Debugging Tools for Windows\gflags.exe - again select the correct edition, x86 or x64). Once Global Flags is started, go to the "Image File" tab and at the top text box enter the name of your executable file without any paths (for example, "MyProgram.exe"). Then press the Tab key and set the following boxes:
Enable heap tail checking
Enable heap free checking
Enable heap parameter checking
Enable heap validation on call
Disable heap coalesce on free
Enable page heap
Enable heap tagging
Enable application verifier
Debugger (type the path to the installed WinDbg in the text box to the right, for example, C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe -g).
For more details, refer to Heap Corruption, Part 2.
Go to "Control Panel/System and Security/System" (or right-click "Computer" in the Start menu and select "Properties". There click "Advanced system settings", in the displayed dialog, go to "Advanced" tab and click the "Environment Variables" button. In the displayed dialog, add a new System variable (if you are an system administrator - a User variable otherwise - you need need to logout/login in this case). The required variable is "COMPLUS_HeapVerify" with a value of "1". More details can be found in Stack Overflow question .NET/C#: How to set debugging environment variable COMPLUS_HeapVerify?.
Now we are ready to start debugging. Start the application. WinDbg should start automatically for it. Leave the application running until it crashes into WinDgb and then examine the dump.
TIP: To quickly remove Global Flags, Application Verifier and the debugger attachment settings, delete the following key in the registry:
x64 - HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\*YourAppName*
Step 4. Enable MDAs.
Try to use the Managed Debugging Assistants. Details are in Stack Overflow question What MDAs are useful to track a heap corruption?.
MDAs must be used along with WinDbg. I used them even along with Global Flags and Application Verifier.
Step 5. Enable GCStress.
Using GCStress is an extreme option, because the application becomes almost unusable, but it is still a way to go. More details are in GCStress: How to turn on in Windows 7?.
Step 6. Compile for x86.
If your application is currently being compiled for "Any CPU" or "x64" platform, try to compile it for "x86" if there is no difference for you which platform to use. I saw this reported to solve the problem for somebody.
Step 7. Disable concurrent GC - this is what worked for me
There is a reported known issue in .NET 4 reported in the thread Access Violation in .NET 4 Runtime in gc_heap::garbage_collect with no unmanaged modules. The problem can be solved by disabling the concurrent GC in the app.config file:
<?xml version="1.0"?>
<configuration>
<runtime>
<gcConcurrent enabled="false" />
</runtime>
</configuration>
You have managed heap corruption. It is not easy to find the root cause of the problem for managed heap corruption, because the problem usually demonstrates itself long after the heap is corrupted. In your case, the StringBuilder is a red herring. Corruption happened sometime before.
What I would do is the following:
Check if you have any unsafe C# code. If you have any, double check the logic there.
Enable paged heap for your application. Running it with paged heap will help uncover problems with unmanaged code -- in case unmanaged code is corrupting the managed heap.
Run !VerifyHeap in different places. This way you might be able to localize the place in your code where corruption happens.
If you have the server type of garbage collection enabled for your application, temporarily change that to workstation garbage collection -- you will get more predictable behavior this way.
Read through Tesses' blog post .NET Crash: Managed Heap Corruption calling unmanaged code. It demonstrates some examples of managed heap corruption.
Note that when you will be running your code under WinDbg, you will come across occasional first chance AV. It is safe to ingore that, just type sxd av once you attach WinDbg to the process, and investigate only second chance AVs.