.NET Runtime version 2.0.50727.3082 - Fatal Execution Engine Error (7A0979C6) (80131506)

Question

We get random crashes on XP.  The application is a service running .NET 2.0 (but the systems have 3.5 SP1 on them), and it invokes an ActiveX control from Eltima to create a virtual serial port.  The service acts as a man-in-the-middle between the virtual serial port and one of the physical (or another virtual, in the case of NetModem) serial ports on the system.  So, a user's application talks to us via the virtual serial port, and, if we see a dial string of interest, we take certain actions, otherwise we just pass the data between the two sides.  Ultimately, the user connects to some device and it's generally during this communication that we'll get a spontaneous crash deep in the bowels of .NET.

We've captured it twice today using ADPlus.  The first time it was a tad useful, as I discovered that I'd made a release build with tracing still turned on.  Peculiarly, however, Windbg showed the crash as occurring... well, you can see the output here:

0:016> .load C:\Windows\Microsoft.NET\Framework\v2.0.50727\sos
0:016> !clrstack
OS Thread Id: 0xb60 (16)
ESP       EIP    
0374efb4 7c90120e [HelperMethodFrame: 0374efb4]
0374f00c 792de014 System.DateTimeFormat.FormatCustomized(System.DateTime, System.String, System.Globalization.DateTimeFormatInfo, System.TimeSpan)
0374f134 792dde04 System.DateTimeFormat.Format(System.DateTime, System.String, System.Globalization.DateTimeFormatInfo, System.TimeSpan)
0374f164 792ddc67 System.DateTimeFormat.Format(System.DateTime, System.String, System.Globalization.DateTimeFormatInfo)
0374f178 79763dc5 System.DateTime.ToLongTimeString()
0374f184 00c6ef49 LsAcc.CPhysicalPort.ReceiveThread()
0374f5c0 792d6d66 System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
0374f5cc 792e01ef System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
0374f5e4 792d6ce4 System.Threading.ThreadHelper.ThreadStart()
0374f80c 79e71b4c [GCFrame: 0374f80c]
0:016> kb
ChildEBP RetAddr  Args to Child             
0374dce8 7a096467 0374e18c 79f7762b f35255ef ntdll!DbgBreakPoint
0374e198 7a096796 80131506 7a0979c6 00000000 mscorwks!EEPolicy::LogFatalError+0x2b5
0374e1b0 7a0979d1 80131506 7a0979c6 00000000 mscorwks!EEPolicy::HandleFatalError+0x4d
0374e1d4 79f5f503 0374e254 f352567f 00184f20 mscorwks!CLRVectoredExceptionHandlerPhase3+0xbc
0374e208 79f5f3a5 0374e254 f352564b 0374e394 mscorwks!CLRVectoredExceptionHandlerPhase2+0x20
0374e23c 79edc2f8 0374e254 00004000 00184f20 mscorwks!CLRVectoredExceptionHandler+0x10a
0374e268 79edc4f5 0374e378 0374f61c 0374e34c mscorwks!CPFH_RealFirstPassHandler+0x607
0374e28c 7c9032a8 0374e378 0374f61c 0374e394 mscorwks!COMPlusFrameHandler+0x15a
0374e2b0 7c90327a 0374e378 0374f61c 0374e394 ntdll!ExecuteHandler2+0x26
0374e360 7c90e48a 00000000 0374e394 0374e378 ntdll!ExecuteHandler+0x24
0374e360 79f93cc9 00000000 0374e394 0374e378 ntdll!KiUserExceptionDispatcher+0xe
0374e680 79f93ee5 011c5a64 0374f1d8 012b5de8 mscorwks!WKS::gc_heap::mark_object_simple+0x16c
0374e694 79f8ecb6 0374f1d8 0374edc4 00000001 mscorwks!WKS::GCHeap::Promote+0x8d
0374e6ac 79f8ec88 79f9330f 0374f1d8 0374edc4 mscorwks!PromoteCarefully+0x29
0374e6c8 79f8e300 0374ed90 0374f1d8 00000001 mscorwks!GcEnumObject+0x29
0374e7dc 79f8d8ea 0374ea64 0374e808 00000000 mscorwks!EECodeManager::EnumGcRefs+0x65c
0374e838 79e8efd7 009168c8 0374ed90 00000000 mscorwks!GcStackCrawlCallBack+0x1b7
0374e84c 79e8d7d5 0374e8d8 79f8d7e2 0374ed90 mscorwks!Thread::MakeStackwalkerCallback+0x15
0374ea30 79e8dc47 00184f20 0374ed90 79e8dc47 mscorwks!Thread::StackWalkFramesEx+0x382
0374ed6c 79f8d6e0 79f8d7e2 0374ed90 00000500 mscorwks!Thread::StackWalkFrames+0xb8

The actual call in my ReceiveThread() function would have been System.DateTime.Now.ToLongTimeString().  Since this is part of the tracing functionality that shouldn't have been in a release build, anyway, I built the DLL without those calls and tried again.  Within a few minuntes, we got a second failure:

0:006> ~#
.  6  Id: a8.1e0 Suspend: 1 Teb: 7ffd4000 Unfrozen
      Start: ole32!CRpcThreadCache::RpcWorkerThreadEntry (774fe43b)
      Priority: 0  Priority class: 32  Affinity: 3
0:006> !clrstack
OS Thread Id: 0x1e0 (6)
ESP       EIP    
0394f264 7c90120e [HelperMethodFrame: 0394f264]
0394f2d0 792cc853 System.Reflection.MethodBase.CheckArguments(System.Object[], System.Reflection.Binder, System.Reflection.BindingFlags, System.Globalization.CultureInfo, System.Signature)
0394f30c 792d50d5 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, Boolean)
0394f348 792d4fde System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
0394f9d8 79e71b4c [GCFrame: 0394f9d8]
0394f9bc 79e71b4c [GCFrame: 0394f9bc]
0:006> kb
ChildEBP RetAddr  Args to Child             
0394df4c 7a096467 0394e3f0 79f7762b 463771a5 ntdll!DbgBreakPoint
0394e3fc 7a096796 80131506 7a0979c6 00000000 mscorwks!EEPolicy::LogFatalError+0x2b5
0394e414 7a0979d1 80131506 7a0979c6 00000000 mscorwks!EEPolicy::HandleFatalError+0x4d
0394e438 79f5f503 0394e4b8 46377635 001b2618 mscorwks!CLRVectoredExceptionHandlerPhase3+0xbc
0394e46c 79f5f3a5 0394e4b8 463776f9 0394e5f8 mscorwks!CLRVectoredExceptionHandlerPhase2+0x20
0394e4a0 79edc2f8 0394e4b8 00004000 001b2618 mscorwks!CLRVectoredExceptionHandler+0x10a
0394e4cc 79edc4f5 0394e5dc 0394f38c 0394e5b0 mscorwks!CPFH_RealFirstPassHandler+0x607
0394e4f0 7c9032a8 0394e5dc 0394f38c 0394e5f8 mscorwks!COMPlusFrameHandler+0x15a
0394e514 7c90327a 0394e5dc 0394f38c 0394e5f8 ntdll!ExecuteHandler2+0x26
0394e5c4 7c90e48a 00000000 0394e5f8 0394e5dc ntdll!ExecuteHandler+0x24
0394e5c4 79f95f16 00000000 0394e5f8 0394e5dc ntdll!KiUserExceptionDispatcher+0xe
0394e8f0 79f95dda 012f9d78 0394f00c 012ab730 mscorwks!WKS::gc_heap::mark_object_simple1+0x148
0394e91c 79f93ee5 009171fc 0000002a 012ab730 mscorwks!WKS::gc_heap::mark_object_simple+0x19d
0394e930 79f8d77a 0417f39c 0394f040 00000000 mscorwks!WKS::GCHeap::Promote+0x8d
0394e948 79f8e298 0394f00c 0417f39c 00000000 mscorwks!GcEnumObject+0x2d
0394ea5c 79f8d8ea 0394ece0 0394ea88 00000000 mscorwks!EECodeManager::EnumGcRefs+0x78e
0394eab8 79e8efd7 009168c0 0394f00c 00000000 mscorwks!GcStackCrawlCallBack+0x1b7
0394eacc 79e8d7d5 0394eb58 79f8d7e2 0394f00c mscorwks!Thread::MakeStackwalkerCallback+0x15
0394ecb0 79e8dc47 0394f00c 79e8dc47 0394ece0 mscorwks!Thread::StackWalkFramesEx+0x382
0394efe8 79f8d6e0 79f8d7e2 0394f00c 00000500 mscorwks!Thread::StackWalkFrames+0xb8

I am a bit concerned that in both cases, the non-managed stack shows that we're ultimately arriving at DebugBreak().  I have triple-checked my code and I'm not invoking it.  I'm not sure if the managed stack trace is valid (or, for that matter, whether the non-managed stack trace is, either).  At least in the first of these captures, the managed stack trace actually represented something in my code that I could find.  This second one (somewhere in System.Reflection?!?) has nothing to do with anything we have coded directly.  On the other hand, the non-managed stack trace is the same for both failures, and in each case, it looks like the garbage collector is unhappy, but that doesn't help (me) at all.

This does not seem to be a problem under Vista with mscorwks 2.0.50727.3053, but we've had it occur on two systems running XP with mscorwks 2.0.50727.3082.

I'm entertaining advice and suggestions at this point!  Do you have any you could share?

Thanks,

=-John-=

Answer 1

Yes.  In both case, the garbage collector detects that the garbage collected heap is corrupted and pulls the plug on the program.  Your first suspicion should go to any kind of unmanaged code you might be using, including P/Invoke, COM components and ActiveX controls.  A buffer overrun is the classic source of corruption.  Your second should go to anti-virus software, Symantec's has been troublesome.

---

Hans Passant.