I have a long running i/o heavy (network/disk) multithreaded c# desktop application that sporadically crashes with "Stack cookie instrumentation code detected a stack-based buffer overrun."
A typical stack trace of the crashing thread is shown below.
The program runs for anywhere between 30 minutes and 6 hours before it typically crashes with a similar stack.
I have inspected the stack traces of the other application threads during a crash and haven't noticed any patterns.
I have considered that there may be a hardware issue, but there is much activity on the machine and no other signs of corruption (application crashes, event log errors, etc).
The C# application uses no unsafe blocks and no libraries that use unsafe blocks as far as I have determined.
Two questions:
Is it possible to get symbols for the System.ni.dll calls in the trace below?
How can I go about determining the cause of the crash? (My strongest suspicion is a framework bug.)
My system is i5-4790k (not overclocked), Windows 2012R2, targeting .net 4.6 x64, fully patched.
Visual studio 2015, fully patched.
Running the Debug build.
clr.dll!__report_gsfailure() Unknown
clr.dll!SafeHandle::Release(bool) Unknown
clr.dll!SafeHandle::Dispose(void) Unknown
clr.dll!SafeHandle::DisposeNative(class SafeHandle *) Unknown
System.ni.dll!00007ffa765ddc4b() Unknown
System.ni.dll!00007ffa765dda8a() Unknown
System.ni.dll!00007ffa765e7e0e() Unknown
System.ni.dll!00007ffa765c420f() Unknown
System.ni.dll!00007ffa765c41d4() Unknown
mscorlib.ni.dll!00007ffa7740f18a() Unknown
System.ni.dll!00007ffa76b19bbf() Unknown
System.ni.dll!00007ffa765e0d0b() Unknown
System.ni.dll!00007ffa7657ac4f() Unknown
System.ni.dll!00007ffa765e0508() Unknown
System.ni.dll!00007ffa765e0319() Unknown
System.ni.dll!00007ffa765e714d() Unknown
System.ni.dll!00007ffa765e6cdd() Unknown
System.ni.dll!00007ffa765e0157() Unknown
System.ni.dll!00007ffa765dfe48() Unknown
System.ni.dll!00007ffa765dfdb5() Unknown
System.ni.dll!00007ffa765dfa0d() Unknown
System.ni.dll!00007ffa765de83d() Unknown
System.ni.dll!00007ffa765de401() Unknown
System.ni.dll!00007ffa765de2e0() Unknown
System.ni.dll!00007ffa765dacac() Unknown
System.ni.dll!00007ffa765da9f0() Unknown
System.ni.dll!00007ffa765da22e() Unknown
System.ni.dll!00007ffa765d7fc7() Unknown
System.ni.dll!00007ffa765d743d() Unknown
System.ni.dll!00007ffa76579af6() Unknown
00007ffa19832eb7() Unknown
00007ffa1978eb0f() Unknown
00007ffa19844cfa() Unknown
00007ffa199e83d7() Unknown
00007ffa1978ce48() Unknown
00007ffa1978ccb5() Unknown
00007ffa199e810f() Unknown
mscorlib.ni.dll!00007ffa77d37116() Unknown
mscorlib.ni.dll!00007ffa77d22a90() Unknown
mscorlib.ni.dll!00007ffa77dfa795() Unknown
mscorlib.ni.dll!00007ffa773d2c97() Unknown
mscorlib.ni.dll!00007ffa773fa77e() Unknown
mscorlib.ni.dll!00007ffa773fa617() Unknown
mscorlib.ni.dll!00007ffa773d2f3d() Unknown
mscorlib.ni.dll!00007ffa773d2608() Unknown
mscorlib.ni.dll!00007ffa77386b50() Unknown
clr.dll!CallDescrWorkerInternal() Unknown
clr.dll!CallDescrWorkerWithHandler(struct CallDescrData *,int) Unknown
clr.dll!MethodDescCallSite::CallTargetWorker(unsigned __int64 const *) Unknown
clr.dll!QueueUserWorkItemManagedCallback(void *) Unknown
clr.dll!Frame::Push(void) Unknown
clr.dll!Frame::Push(void) Unknown
clr.dll!Frame::Push(void) Unknown
clr.dll!ManagedPerAppDomainTPCount::DispatchWorkItem(bool *,bool *) Unknown
clr.dll!ManagedPerAppDomainTPCount::DispatchWorkItem(bool *,bool *) Unknown
clr.dll!ThreadpoolMgr::ExecuteWorkRequest(bool *,bool *) Unknown
clr.dll!ThreadpoolMgr::WorkerThreadStart(void *) Unknown
clr.dll!Thread::intermediateThreadProc(void *) Unknown
kernel32.dll!BaseThreadInitThunk() Unknown
ntdll.dll!RtlUserThreadStart() Unknown
I faced a similar behavior. I suppose it is a bug in RyuJIT compiler. To avoid this you need to fallback to JIT64 using <useLegacyJit enabled="1" /> in app.config. Like this:
<runtime>
<useLegacyJit enabled="1" />
</runtime>
Related
I'm trying to swap database providers out in .NET code to MariaDB. As part of that, I'm trying to open a MySqlConnection. This in turn results in an argument null execption with a message of: "Value cannot be null. Parameter name: element" and the following call stack:
mscorlib.dll!System.Attribute.GetCustomAttributes(System.Reflection.Assembly element, System.Type attributeType, bool inherit) Line 756 C#
mscorlib.dll!System.Attribute.GetCustomAttribute(System.Reflection.Assembly element, System.Type attributeType, bool inherit) Line 814 C#
mscorlib.dll!System.Reflection.CustomAttributeExtensions.GetCustomAttribute<System.Runtime.Versioning.TargetFrameworkAttribute>(System.Reflection.Assembly element) Line 27 C#
> MySql.Data.dll!MySql.Data.MySqlClient.MySqlConnectAttrs.InitFramework() Unknown
MySql.Data.dll!MySql.Data.MySqlClient.MySqlConnectAttrs.MySqlConnectAttrs() Unknown
[Native to Managed Transition]
[Managed to Native Transition]
MySql.Data.dll!MySql.Data.MySqlClient.NativeDriver.SetConnectAttrs() Unknown
MySql.Data.dll!MySql.Data.MySqlClient.Authentication.MySqlAuthenticationPlugin.Authenticate(bool reset) Unknown
MySql.Data.dll!MySql.Data.MySqlClient.NativeDriver.Authenticate(string authMethod, bool reset) Unknown
MySql.Data.dll!MySql.Data.MySqlClient.NativeDriver.Open() Unknown
MySql.Data.dll!MySql.Data.MySqlClient.Driver.Open() Unknown
MySql.Data.dll!MySql.Data.MySqlClient.Driver.Create(MySql.Data.MySqlClient.MySqlConnectionStringBuilder settings) Unknown
MySql.Data.dll!MySql.Data.Failover.FailoverManager.AttemptConnection(MySql.Data.MySqlClient.MySqlConnection connection, string originalConnectionString, out string connectionString, bool mySqlPoolManager) Unknown
MySql.Data.dll!MySql.Data.MySqlClient.MySqlConnection.Open() Unknown
(My code below this point)
I'm using a connection string similar to:
<add name="MyMariaDatabase" connectionString="server=myservername.rds.amazonaws.com;port=3306; database=mydb;uid=myuser; pwd=mypassword" providerName="MySql.Data.MySqlClient" />
Sample connection code:
using (var client = new MySql.Data.MySqlClient.MySqlConnection(connStr))
{
client.Open();
var result = client.ExecuteScalar(someSqlStatement);
}
This is all running in .NET Framework 4.7.2 and MySQL.Data 8.0.19.
Any idea what's going wrong? The error seems deep inside the framework and it's not giving me helpful information.
You're encountering bug 95242, a known problem in Oracle's MySQL Connector/NET (aka MySql.Data).
I would recommend switching to MySqlConnector, an OSS MySQL and MariaDB client library. As well as fixing many bugs in Connector/NET and adding true async I/O support, it is independent of Oracle so it has support for MariaDB-specific features such as the GSSAPI authentication plugin and batch support.
Your Port must be seperated by semicolon.
Instead of
"server=myservername.rds.amazonaws.com,3306; database=mydb;uid=myuser; pwd=mypassword"
use this Connenction string
"Server=myservername.rds.amazonaws.com;Port=3306;Database=mydb;Uid=myuser;Pwd=mypassword;"
I have a Xamarin.Forms UWP app (Xamarin.Forms v3.4.0.1008975, UWP target/min version 16299, VS 2017 15.9.11). When compiled without the .net native tool chain it builds and runs correctly. When compiled with the .net native tool chain it builds fine but at run time it fails. This is an issue because I cannot release the app to the windows store. This app has previously (~10 months ago) been released to the store and worked correctly. There has been an extensive overhaul since then with updated versions of XF, Prism, sqlite packages and the addition of a couple of other packages amongst many other changes so it's nearly impossible to work out what change might have caused it.
I managed to get past some initial problems with the Rg.Plugins.Popup and Xam.Plugin.Iconize.FontAwesome packages by getting a list of assemblies for them and passing them into Xamarin.Forms.Forms.Init(e, assemblies); in App.xaml.cs in the UWP project. This fixed crashes/display issues with popups and font icons in my app.
When the app starts up I am seeing several handled FileNotFoundException on the Xamarin.Forms.Forms.Init(...) call of the form Cannot load assembly 'clrcompression'. No metadata found for this assembly.. The missing assemblies are:
clrcompression
e_sqlite3
libEGL
libGLESv2
libSkiaSharp
SkiaSharp.Views.Interop.UWP
sqlite3
As said these errors are handled and only seen when debugging. I've not seen any definite evidence that these errors are a problem as the app is definitely reading some data from Sqlite without error and I'm seeing images drawn correctly with SkiaSharp.
However I am getting other run time errors when I perform a certain actions that are crashing the app. For example Unhandled exception at 0x05F8F74C (SharedLibrary.dll) in MyApp.UWP.exe: 0x00001007. occurred. If I look in the threads window I get a bit more information:
Not Flagged > 13884 0 Worker Thread <No Name>
System.Private.SharedLibrary.Interop.Generated.dll!__Interop.api_ms_win_core_kernel32_legacy_l1_1_0_dll.PInvoke_RaiseFailFastException
System.Private.SharedLibrary.Interop.Generated.dll!__Interop.api_ms_win_core_kernel32_legacy_l1_1_0_dll.PInvoke_RaiseFailFastException(Interop._EXCEPTION_RECORD* pExceptionRecord, System.IntPtr pContextRecord, uint dwFlags)
System.Private.CoreLib.dll!Interop.mincore.PInvoke_RaiseFailFastException(Interop._EXCEPTION_RECORD* pExceptionRecord, System.IntPtr pContextRecord, uint dwFlags)
System.Private.CoreLib.dll!Interop.mincore.RaiseFailFastException(uint faultCode, System.IntPtr pExAddress, System.IntPtr pExContext) Line 122
System.Private.CoreLib.dll!System.RuntimeExceptionHelpers.FailFast(string message, System.Exception exception, System.RuntimeExceptionHelpers.RhFailFastReason reason, System.IntPtr pExAddress, System.IntPtr pExContext) Line 237
System.Private.CoreLib.dll!System.RuntimeExceptionHelpers.RuntimeFailFast(System.RuntimeExceptionHelpers.RhFailFastReason reason, System.Exception exception, System.IntPtr pExAddress, System.IntPtr pExContext) Line 200
[External Code]
System.Private.CoreLib.dll!Internal.Runtime.Augments.RuntimeAugments.RunFunctionWithConservativelyReportedBufferInternal<Internal.Runtime.TypeLoader.CallConversionParameters>(int cbBuffer, System.IntPtr pfnTargetToInvoke, ref Internal.Runtime.TypeLoader.CallConversionParameters context, ref System.Runtime.RuntimeImports.ConservativelyReportedRegionDesc regionDesc)
System.Private.CoreLib.dll!Internal.Runtime.Augments.RuntimeAugments.RunFunctionWithConservativelyReportedBuffer<Internal.Runtime.TypeLoader.CallConversionParameters>(int cbBuffer, System.IntPtr pfnTargetToInvoke, ref Internal.Runtime.TypeLoader.CallConversionParameters context)
System.Private.TypeLoader.dll!Internal.Runtime.TypeLoader.CallConverterThunk.CallConversionThunk(System.IntPtr callerTransitionBlockParam, System.IntPtr callConversionId)
[External Code]
Prism.dll!Prism.Mvvm.BindableBase.SetProperty<int?>(ref int? storage, int? value, string propertyName)
[External Code]
System.Private.CoreLib.dll!Internal.Runtime.Augments.RuntimeAugments.CallDescrWorker(System.IntPtr callDescr) Line 970
System.Private.TypeLoader.dll!Internal.Runtime.TypeLoader.CallConverterThunk.InvokeTarget(void* allocatedStackBuffer, ref Internal.Runtime.TypeLoader.CallConversionParameters conversionParams)
System.Private.CoreLib.dll!Internal.Runtime.Augments.RuntimeAugments.RunFunctionWithConservativelyReportedBufferInternal<Internal.Runtime.TypeLoader.CallConversionParameters>(int cbBuffer, System.IntPtr pfnTargetToInvoke, ref Internal.Runtime.TypeLoader.CallConversionParameters context, ref System.Runtime.RuntimeImports.ConservativelyReportedRegionDesc regionDesc)
System.Private.CoreLib.dll!Internal.Runtime.Augments.RuntimeAugments.RunFunctionWithConservativelyReportedBuffer<Internal.Runtime.TypeLoader.CallConversionParameters>(int cbBuffer, System.IntPtr pfnTargetToInvoke, ref Internal.Runtime.TypeLoader.CallConversionParameters context)
System.Private.TypeLoader.dll!Internal.Runtime.TypeLoader.CallConverterThunk.CallConversionThunk(System.IntPtr callerTransitionBlockParam, System.IntPtr callConversionId)
[External Code]
MyApp.dll!MyApp.ViewModels.AppLevel.AppStatusViewModel.CurrentSynchroniseJobTasksCount.set(int? value) Line 415
MyApp.dll!MyApp.ViewModels.AppLevel.AppStatusViewModel.HandleDataSyncStartEvent(string payload) Line 287
[External Code]
MyApp.dll!MyApp.Services.Synchronisation.DataSyncCoordinator.DoSynchronisationJob() Line 84
MyApp.dll!MyApp.Services.Synchronisation.DataSyncCoordinator.AttemptSynchronisationTask(MyApp.Services.Synchronisation.IDatabaseSyncJob job) Line 74
MyApp.dll!MyApp.Services.Synchronisation.DataSyncService.DeleteLocalData(string dbResetKey) Line 29
MyApp.dll!MyApp.ViewModels.ClearLocalDataPopupViewModel.Submit() Line 66
[Resuming Async Method]
[External Code]
Unfortunately while this tells me where the problem is happening it doesn't give me any clues as to why it's happening and how I might fix it. In this case it seems to be failing to set a property on a viewmodel but to even get to this point in the code it must have done this successfully in other places. I guess I'm really looking for guidance on how to investigate this/get more information as it's going to be hard to solve specific problems without some source code (which I cannot really provide). I have tried debugging with optimizations turned off which is often suggested but it's not provided any more information than I've posted here.
I'm also worried that even if I find out why this particular exception is happening and fix it that there might be many more issues in the app that only appear when .net native tool chain is used. Are there any tools I can use to help with this?
So this turned out to be an issue with Prism (7.0.0.396) and the .net native. It looks like binding properties with a nullable type (int? in this case) using BindableBase.SetProperty fail when compiled with the .net native tool chain. The following appears to fix this:
public class BindableBaseWithFix : BindableBase
{
protected virtual bool SetProperty<T>(ref T? storage, T? value, [CallerMemberName] string propertyName = null)
where T : struct
{
if (EqualityComparer<T?>.Default.Equals(storage, value))
return false;
storage = value;
RaisePropertyChanged(propertyName);
return true;
}
}
I have an error and this is shown:
[OverflowException: Value was either too large or too small for a Decimal.]
System.Decimal..ctor(Double value) +0
System.Decimal.op_Explicit(Double value) +30
MyMethod(int myParameter) +5174
AnotherMethod(int myParameter) ....
What does +5174 mean?
It's not a line number.
It will be the IL offset, this happens when no PDB files / debug information are present.
You can have PDB data generated for release assemblies, which makes tracking down issues much more straight forward.
My program throws an error which it cannot handle by a catch(Exception e) block and then it crashes:
Access Violation Corrupted State Exception.
This is the weird thing, because, as I know, corrupted state exceptions are thrown from unmanaged code, while here I get this exception while calling a StringBuilder method.
The code runs in a background thread and crashes from time to time which cannot be easily reproduced. So I attached WinDbg to the process and have the following stack of the exception:
000000001dabd8c8 000007feea129a1d [HelperMethodFrame: 000000001dabd8c8]
000000001dabda00 000007fee90cfce8 System.Text.StringBuilder.ExpandByABlock(Int32)
000000001dabda40 000007fee90cfba4 System.Text.StringBuilder.Append(Char*, Int32)
000000001dabdaa0 000007fee9102955 System.Text.StringBuilder.Append(System.String, Int32, Int32)
000000001dabdaf0 000007ff00bf5ce3 MineUtils.Common.Strings.Strings.Replace(System.String, System.String, System.String, Boolean, Boolean)
000000001dabdb90 000007ff00bf5a59 MineUtils.Common.Strings.Strings.RemoveSubstrings(System.String, System.String, System.String, Boolean) [D:\Programs\Visual Studio 2005 Projects\MineUtils.Common\Strings\Strings.Common-Main.cs # 1481
WinDbg shows this exception occurred:
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000007feea129a1d (clr!WKS::gc_heap::find_first_object+0x0000000000000092)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000003d80
Attempt to read from address 0000000000003d80
I read such exceptions can be handled with a method attribute [HandleProcessCorruptedStateExceptions], but why does this exception ever occur if I only use StringBuilder?
This is the previous WinDbg analysis (StringBuilder.ToString() causes the exception):
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
clr!WKS::gc_heap::find_first_object+92
000007fe`ea129a1d f70100000080 test dword ptr [rcx],80000000h
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000007feea129a1d (clr!WKS::gc_heap::find_first_object+0x0000000000000092)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000001
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000001c98
Attempt to read from address 0000000000001c98
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: 0000000000001c98
READ_ADDRESS: 0000000000001c98
FOLLOWUP_IP:
clr!WKS::gc_heap::find_first_object+92
000007fe`ea129a1d f70100000080 test dword ptr [rcx],80000000h
MOD_LIST: <ANALYSIS/>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
MANAGED_STACK:
(TransitionMU)
000000001AB7DFC0 000007FEE90CFE07 mscorlib_ni!System.Text.StringBuilder.ToString()+0x27
000000001AB7E010 000007FF00C750A9 SgmlReaderDll!Sgml.Entity.ScanToken(System.Text.StringBuilder, System.String, Boolean)+0x169
000000001AB7E080 000007FF00C760E6 SgmlReaderDll!Sgml.SgmlDtd.ParseParameterEntity(System.String)+0xc6
000000001AB7E0F0 000007FF00C76FD8 SgmlReaderDll!Sgml.SgmlDtd.ParseModel(Char, Sgml.ContentModel)+0x298
000000001AB7E160 000007FF00C7701C SgmlReaderDll!Sgml.SgmlDtd.ParseModel(Char, Sgml.ContentModel)+0x2dc
000000001AB7E1D0 000007FF00C7701C SgmlReaderDll!Sgml.SgmlDtd.ParseModel(Char, Sgml.ContentModel)+0x2dc
000000001AB7E240 000007FF00C76BA5 SgmlReaderDll!Sgml.SgmlDtd.ParseContentModel(Char)+0x65
000000001AB7E290 000007FF00C763D7 SgmlReaderDll!Sgml.SgmlDtd.ParseElementDecl()+0xe7
000000001AB7E320 000007FF00C747A1 SgmlReaderDll!Sgml.SgmlDtd.Parse()+0xc1
000000001AB7E370 000007FF00C73EF5 SgmlReaderDll!Sgml.SgmlDtd.Parse(System.Uri, System.String, System.IO.TextReader, System.String, System.String, System.Xml.XmlNameTable)+0x175
000000001AB7E410 000007FF00C73B33 SgmlReaderDll!Sgml.SgmlReader.LazyLoadDtd(System.Uri)+0x163
000000001AB7E480 000007FF00C737B9 SgmlReaderDll!Sgml.SgmlReader.OpenInput()+0x19
000000001AB7E4E0 000007FF00C7334C SgmlReaderDll!Sgml.SgmlReader.Read()+0x1c
000000001AB7E530 000007FEE5983C4C System_Xml_ni!System.Xml.XmlLoader.Load(System.Xml.XmlDocument, System.Xml.XmlReader, Boolean)+0xac
000000001AB7E590 000007FEE5983730 System_Xml_ni!System.Xml.XmlDocument.Load(System.Xml.XmlReader)+0x90
...
000000001AB7F0A0 000007FEE97ED792 mscorlib_ni!System.Threading.Tasks.Task.Execute()+0x82
000000001AB7F100 000007FEE90A181C mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0xdc
000000001AB7F160 000007FEE97E7F95 mscorlib_ni!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)+0x1b5
000000001AB7F1E0 000007FEE97E7D90 mscorlib_ni!System.Threading.Tasks.Task.ExecuteEntry(Boolean)+0xb0
000000001AB7F220 000007FEE90EBA83 mscorlib_ni!System.Threading.ThreadPoolWorkQueue.Dispatch()+0x193
000000001AB7F2C0 000007FEE90EB8D5 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()+0x35
(TransitionUM)
EXCEPTION_OBJECT: !pe 2a61228
Exception object: 0000000002a61228
Exception type: System.ExecutionEngineException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131506
MANAGED_OBJECT_NAME: System.ExecutionEngineException
MANAGED_STACK_COMMAND: _EFN_StackTrace
LAST_CONTROL_TRANSFER: from 000007feea12bce4 to 000007feea129a1d
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[PSEUDO_THREAD]
FAULTING_THREAD: ffffffffffffffff
DEFAULT_BUCKET_ID: INVALID_POINTER_READ_CALL
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ_CALL
BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_WRONG_SYMBOLS_CALL__SYSTEM.EXECUTIONENGINEEXCEPTION
UPDATED AGAIN
Here is the WinDbg stack of the exception after I enabled paged heap:
(1480.e84): Access violation - code c0000005 (first chance)
ntdll!ZwTerminateProcess+0xa:
00000000`77c415da c3 ret
0:023> !clrstack
OS Thread Id: 0xe84 (23)
Child SP IP Call Site
0000000037ded848 0000000077c415da [HelperMethodFrame: 0000000037ded848]
0000000037dedab0 000007fee9effd17 System.Text.StringBuilder.ToString()*** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\8f7f691aa155c11216387cf3420d9d1b\mscorlib.ni.dll
0000000037dedb00 000007ff00cceae9 Sgml.Entity.ScanToken(System.Text.StringBuilder, System.String, Boolean)
0000000037dedb70 000007ff00cd19b2 Sgml.SgmlDtd.ParseAttDefault(Char, Sgml.AttDef)
0000000037dedbc0 000007ff00cd120b Sgml.SgmlDtd.ParseAttDef(Char)
0000000037dedc00 000007ff00cd1057 Sgml.SgmlDtd.ParseAttList(System.Collections.Generic.Dictionary`2<System.String,Sgml.AttDef>, Char)
0000000037dedc50 000007ff00cd10cd Sgml.SgmlDtd.ParseAttList(System.Collections.Generic.Dictionary`2<System.String,Sgml.AttDef>, Char)
0000000037dedca0 000007ff00cd0e9a Sgml.SgmlDtd.ParseAttList()
0000000037dedd10 000007ff00cce1f1 Sgml.SgmlDtd.Parse()
0000000037dedd60 000007ff00ccd945 Sgml.SgmlDtd.Parse(System.Uri, System.String, System.IO.TextReader, System.String, System.String, System.Xml.XmlNameTable)
0000000037dede00 000007ff00ccd582 Sgml.SgmlReader.LazyLoadDtd(System.Uri)
0000000037dede70 000007ff00ccd1f9 Sgml.SgmlReader.OpenInput()
0000000037deded0 000007ff00cccd8c Sgml.SgmlReader.Read()
0000000037dedf20 000007fee67b3bfc System.Xml.XmlLoader.Load(System.Xml.XmlDocument, System.Xml.XmlReader, Boolean)*** WARNING: Unable to verify checksum for C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xml\8e4323f5bfb90be4621456033d8b404b\System.Xml.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xml\8e4323f5bfb90be4621456033d8b404b\System.Xml.ni.dll
0000000037dedf80 000007fee67b36e0 System.Xml.XmlDocument.Load(System.Xml.XmlReader)
[deleted]
0000000037deea90 000007feea61d432 System.Threading.Tasks.Task.Execute()
0000000037deeaf0 000007fee9ed17ec System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
0000000037deeb50 000007feea617c35 System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
0000000037deebd0 000007feea617a30 System.Threading.Tasks.Task.ExecuteEntry(Boolean)
0000000037deec10 000007fee9f1b953 System.Threading.ThreadPoolWorkQueue.Dispatch()
0000000037deecb0 000007fee9f1b7a5 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
0000000037def310 000007feeae4dc54 [DebuggerU2MCatchHandlerFrame: 0000000037def310]
0:023> !verifyheap
-verify will only produce output if there are errors in the heap
The garbage collector data structures are not in a valid state for traversal.
It is either in the "plan phase," where objects are being moved around, or
we are at the initialization or shutdown of the gc heap. Commands related to
displaying, finding or traversing objects as well as gc heap segments may not
work properly. !dumpheap and !verifyheap may incorrectly complain of heap
consistency errors.
object 000000000e34caf8: bad member 000000001024b9a0 at 000000000e34cb08
curr_object: 000000000e34caf8
Last good object: 000000000e34cab0
----------------
0:023> !analyze
Last event: 1480.e84: Exit process 0:1480, code 80131506
debugger time: Sun Sep 18 14:22:42.592 2011 (UTC + 1:00)
0:023> !analyze -v
Last event: 1480.e84: Exit process 0:1480, code 80131506
debugger time: Sun Sep 18 14:22:42.592 2011 (UTC + 1:00)
0:023> .do e34cab0
^ Syntax error in '.do e34cab0'
0:023> !do e34cab0
Name: System.String
MethodTable: 000007feea026870
EEClass: 000007fee9baed58
Size: 72(0x48) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: appliedFiltersContainer
Fields:
MT Field Offset Type VT Attr Value Name
000007feea02c758 4000103 8 System.Int32 1 instance 23 m_stringLength
000007feea02b298 4000104 c System.Char 1 instance 61 m_firstChar
000007feea026870 4000105 10 System.String 0 shared static Empty
>> Domain:Value 00000000021343a0:000000000db21420 <<
0:023> !do e34caf8
<Note: this object has an invalid CLASS field>
Name: System.Reflection.RuntimeAssembly
MethodTable: 000007feea02a128
EEClass: 000007fee9baf968
Size: 48(0x30) bytes
File: C:\Windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
Fields:
MT Field Offset Type VT Attr Value Name
000007feea9ef7f0 4000e14 8 ...solveEventHandler 0 instance 0000000000000000 _ModuleResolve
000007feea036338 4000e15 10 ...che.InternalCache 0 instance 000000001024b9a0 m_cachedData
000007feea0259c8 4000e16 18 System.Object 0 instance 000000000e3abd18 m_syncRoot
000007feea033450 4000e17 20 System.IntPtr 1 instance 37a95f10 m_assembly
What can it be?
Recently, I was faced with a managed heap corruption which was something new to me. I was very frustrated with it and had to learn many things to be able to debug it. I want to thank Seva Titov who gave me right direction to start. His answer was concise and very helpful. I want to log the actions I have taken to debug the problem for my own reference. Probably this will be helpful for others who are new to this.
Debug Heap Corruption in .NET 4:
How to suspect the heap corruption?
Briefly:
The application crashes randomly with no regards to the applied exception catching and even goes through blankets like catch(Exception) which are supposed to catch all exceptions.
Examining the CLR stack in the application crash dumps shows the garbage collector on the top of the stack:
000000001dabd8c8 000007feea129a1d [**HelperMethodFrame**: 000000001dabd8c8]
000000001dabda00 000007fee90cfce8 System.Text.StringBuilder.ExpandByABlock(Int32)
000000001dabda40 000007fee90cfba4 System.Text.StringBuilder.Append(Char*, Int32)
...
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 000007feea129a1d (**clr!WKS::gc_heap**::find_first_object+0x0000000000000092)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000003d80
...
The CLR stack always shows different points. Whether the crash occurred or the code which is shown is clearly irrelevant, like StringBuilder's method which is shown to cause the exception.
For more details refer to .NET Crash: Managed Heap Corruption calling unmanaged code.
Going step by step. Each next step is used if the previous one doesn't help.
Step 1. Check the code.
Check the code for unsafe or native code usages:
Review the code for unsafe, DllImport statements.
Download .NET Reflector and use it to analyze the application assemblies for PInvoke. In the same way, analyze the third-party assemblies which are used by the application.
If unsafe or native code usage is found, direct extra attention to those. The most common cause of the heap corruption in such cases is a buffer overflow or an argument type mismatch. Ensure that the buffer supplied to the native code to fill is big enough and that all arguments passed to the native code are of the expected type.
Step 2. Check if this corrupted state exception can be caught.
To handle such exceptions, one need to decorate the method which contains the catch(Exception) statement with the [HandleProcessCorruptedStateExceptions] attribute or apply the following in the app.config file:
<configuration>
<runtime>
<legacyCorruptedStateExceptionsPolicy enabled="true" />
</runtime>
</configuration>
In the case the exception was caught successfully, you can log and examine it. This means this is not a corrupted heap issue.
Corrupted heap exceptions cannot be handled at all: HandleProcessCorruptedStateExceptions doesn't seem to work.
More information on corrupted state exceptions, see All about Corrupted State Exceptions in .NET4.
Step 3. Live debugging.
In this step, we debug the crashing application live in the production environment (or where we can reproduce the crash).
Download Debugging Tools for Windows from Microsoft Windows SDK for Windows 7 and .NET Framework 4 (a web installer will be downloaded which will allow selecting the required components to install - mark all components). It will install both 32 and 64 bit (if your system is x64) versions of the required debugging tools.
Here one needs to know how to attach WinDbg to a live process, how to take crash dumps and examine them, how to load SOS extension in WinDbg (google for details).
Enable debugging helpers:
Launch Application Verifier (C:\Program Files\Application Verifier - use the required edition, either x86 or x64, depending on your executable compilation mode), add your executable there in the left pane and in the right pane check one node "Basics / Heaps". Save the changes.
Launch Global Flags helper (C:\Program Files\Debugging Tools for Windows\gflags.exe - again select the correct edition, x86 or x64). Once Global Flags is started, go to the "Image File" tab and at the top text box enter the name of your executable file without any paths (for example, "MyProgram.exe"). Then press the Tab key and set the following boxes:
Enable heap tail checking
Enable heap free checking
Enable heap parameter checking
Enable heap validation on call
Disable heap coalesce on free
Enable page heap
Enable heap tagging
Enable application verifier
Debugger (type the path to the installed WinDbg in the text box to the right, for example, C:\Program Files\Debugging Tools for Windows (x64)\windbg.exe -g).
For more details, refer to Heap Corruption, Part 2.
Go to "Control Panel/System and Security/System" (or right-click "Computer" in the Start menu and select "Properties". There click "Advanced system settings", in the displayed dialog, go to "Advanced" tab and click the "Environment Variables" button. In the displayed dialog, add a new System variable (if you are an system administrator - a User variable otherwise - you need need to logout/login in this case). The required variable is "COMPLUS_HeapVerify" with a value of "1". More details can be found in Stack Overflow question .NET/C#: How to set debugging environment variable COMPLUS_HeapVerify?.
Now we are ready to start debugging. Start the application. WinDbg should start automatically for it. Leave the application running until it crashes into WinDgb and then examine the dump.
TIP: To quickly remove Global Flags, Application Verifier and the debugger attachment settings, delete the following key in the registry:
x64 - HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\*YourAppName*
Step 4. Enable MDAs.
Try to use the Managed Debugging Assistants. Details are in Stack Overflow question What MDAs are useful to track a heap corruption?.
MDAs must be used along with WinDbg. I used them even along with Global Flags and Application Verifier.
Step 5. Enable GCStress.
Using GCStress is an extreme option, because the application becomes almost unusable, but it is still a way to go. More details are in GCStress: How to turn on in Windows 7?.
Step 6. Compile for x86.
If your application is currently being compiled for "Any CPU" or "x64" platform, try to compile it for "x86" if there is no difference for you which platform to use. I saw this reported to solve the problem for somebody.
Step 7. Disable concurrent GC - this is what worked for me
There is a reported known issue in .NET 4 reported in the thread Access Violation in .NET 4 Runtime in gc_heap::garbage_collect with no unmanaged modules. The problem can be solved by disabling the concurrent GC in the app.config file:
<?xml version="1.0"?>
<configuration>
<runtime>
<gcConcurrent enabled="false" />
</runtime>
</configuration>
You have managed heap corruption. It is not easy to find the root cause of the problem for managed heap corruption, because the problem usually demonstrates itself long after the heap is corrupted. In your case, the StringBuilder is a red herring. Corruption happened sometime before.
What I would do is the following:
Check if you have any unsafe C# code. If you have any, double check the logic there.
Enable paged heap for your application. Running it with paged heap will help uncover problems with unmanaged code -- in case unmanaged code is corrupting the managed heap.
Run !VerifyHeap in different places. This way you might be able to localize the place in your code where corruption happens.
If you have the server type of garbage collection enabled for your application, temporarily change that to workstation garbage collection -- you will get more predictable behavior this way.
Read through Tesses' blog post .NET Crash: Managed Heap Corruption calling unmanaged code. It demonstrates some examples of managed heap corruption.
Note that when you will be running your code under WinDbg, you will come across occasional first chance AV. It is safe to ingore that, just type sxd av once you attach WinDbg to the process, and investigate only second chance AVs.
I have an application written in C# I believe and it adds images to a SQL Server 2005 Database. It requires .NET 3.5 to be installed on my computer. I installed .NET 3.5 and setup a database. It runs fine but then once it gets to image 100 when running on one computer, It stops and gives me this error: Can't open image(s) with error: External component has thrown an exception....
When I run the program on my own computer I am able to reach 300 images but then it stops after 300 images and gives me Can't open image(s) with error: External component has thrown an exception.... error once again.
please help!
Heres a stack:
See the end of this message for details on invoking
just-in-time (JIT) debugging instead of this dialog box.
************** Exception Text **************
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity)
at System.Text.StringBuilder..ctor(String value, Int32 startIndex, Int32 length, Int32 capacity)
at System.Windows.Forms.Control.get_WindowText()
at System.Windows.Forms.TextBoxBase.get_WindowText()
at System.Windows.Forms.Control.set_CacheTextInternal(Boolean value)
at System.Windows.Forms.Control.PerformLayout(LayoutEventArgs args)
at System.Windows.Forms.Control.System.Windows.Forms.Layout.IArrangedElement.PerformLayout(IArrangedElement affectedElement, String affectedProperty)
at System.Windows.Forms.Layout.LayoutTransaction.DoLayout(IArrangedElement elementToLayout, IArrangedElement elementCausingLayout, String property)
at System.Windows.Forms.Control.OnResize(EventArgs e)
at System.Windows.Forms.Control.OnSizeChanged(EventArgs e)
at System.Windows.Forms.Control.UpdateBounds(Int32 x, Int32 y, Int32 width, Int32 height, Int32 clientWidth, Int32 clientHeight)
at System.Windows.Forms.Control.UpdateBounds()
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.TextBoxBase.WndProc(Message& m)
at System.Windows.Forms.TextBox.WndProc(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
************** Loaded Assemblies **************
mscorlib
Assembly Version: 2.0.0.0
Win32 Version:
CodeBase: file:///C:/Windows/Microsoft.NET/Framework/v2.0.50727/mscorlib.dll
Do you have more information about the stack trace when the exception is thrown? From your description it sounds like a resource is not getting closed or disposed properly and is eventually running out of resources.
Edit: After seeing your stack trace it confirmed my suspicion that a resource was not being handled properly and it's running the system low on memory. As suggested in the comments you should contact the builder of the program and suggest that they debug it using a profiler and properly dispose of the objects, or to let them go out of scope so the garbage collector can dispose of them. Also you might suggest that if the error is occurring when a file is being loaded that they load the file via a stream using buffers rather than loading the whole file into memory at one time, and then release it after it is no longer needed.