none
Help with analyzing crash minidump RRS feed

  • Question

  • Hi

    Some of our customers are experiencing random crashes with our .net 3.5 app. The crash can happen after running the app for 1 minute, 5 hours or 5 days. The application just disappears without a trace and no entries are added to the eventlog. All affected users are using Windows Xp SP3. And the problem is increasing as more and more Xp users are upgrading from Sp2 to SP3.  Users on Vista and Windows 7 do not have this problem.  To try and find the cause of the problem I deployed Debugging tools for windows to a couple of the affected customers and setup adplus.exe to monitor our application for crashes.

    Here is the exception, threads and the stacktrace from the minidump.

    Windows XP Version 2600 (Service Pack 3) MP (2 procs) Free x86 compatible
    Product: WinNt, suite: SingleUserTS
    Machine Name:
    Debug session time: Wed Sep  1 16:02:22.000 2010 (UTC + 2:00)
    System Uptime: not available
    Process Uptime: 0 days 3:43:25.000
    ................................................................
    ................................................................
    ................................................................
    ......................
    This dump file has an exception of interest stored in it.
    The stored exception information can be accessed via .ecxr.
    (1320.132c): Access violation - code c0000005 (first/second chance not available)
    eax=00000000 ebx=00000000 ecx=21313394 edx=7c9032bc esi=00000000 edi=00000000
    eip=21313394 esp=2382e054 ebp=2382e074 iopl=0         nv up ei pl zr na pe nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246
    21313394 0100            add     dword ptr [eax],eax  ds:0023:00000000=????????
    0:027> .sympath srv*http://msdl.microsoft.com/download/symbols
    Symbol search path is: srv*http://msdl.microsoft.com/download/symbols
    Expanded Symbol search path is: srv*http://msdl.microsoft.com/download/symbols
    0:027> .reload
    ................................................................
    ................................................................
    ................................................................
    ......................
    0:027> .load c:\windows\microsoft.net\framework\v2.0.50727\sos.dll
    ------------------------------------------------------------
    sos.dll needs a full memory dump for complete functionality.
    You can create one with .dump /ma <filename>
    ------------------------------------------------------------
    0:027> !threads
    ThreadCount: 12
    UnstartedThread: 0
    BackgroundThread: 9
    PendingThread: 0
    DeadThread: 2
    Hosted Runtime: no
                                          PreEmptive   GC Alloc           Lock
           ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception
       0    1 112c 00171c88      6020 Enabled  2758c660:2758c660 0016e798     0 STA
       2    2  9e0 0017dee0      b220 Enabled  00000000:00000000 0016e798     0 Ukn (Finalizer)
       4    3  a28 001cdf08    80a220 Enabled  00000000:00000000 0016e798     0 Ukn (Threadpool Completion Port)
       5    4 177c 0022b878    80a220 Enabled  00000000:00000000 0016e798     0 Ukn (Threadpool Completion Port)
       6    6  a84 066bfbd8   880b220 Enabled  2757c7f0:2757e660 0016e798     0 Ukn (Threadpool Completion Port)
       9    9 1054 066d3508   180b220 Enabled  275788ec:2757a660 0016e798     0 Ukn (Threadpool Worker)
      10    c  228 066d6c30   180b220 Enabled  00000000:00000000 0016e798     0 Ukn (Threadpool Worker)
      20    8  f60 10d678b8   200b220 Enabled  2757316c:27574660 0016e798     0 Ukn
      22    b  c90 10d67ca0   200b220 Enabled  2757516c:27576660 0016e798     0 Ukn
      23    5 16c8 10d69028   200b220 Enabled  27571244:27572660 0016e798     0 Ukn
    XXXX    a    0 10d68858      9820 Enabled  00000000:00000000 0016e798     0 Ukn
    XXXX    7    0 10d670e8      9820 Enabled  00000000:00000000 0016e798     0 Ukn
    0:027> !dumpstack
    OS Thread Id: 0x132c (27)
    Current frame: 21313394
    ChildEBP RetAddr  Caller,Callee
    2382e050 7c9032a8 ntdll!ExecuteHandler2+0x26
    2382e074 7c90327a ntdll!ExecuteHandler+0x24, calling ntdll!ExecuteHandler2
    2382e098 7c92aa0f ntdll!RtlDispatchException+0xb1, calling ntdll!RtlpExecuteHandlerForException
    2382e0c8 773db171 comctl32!SendNotifyEx+0x57, calling comctl32!CCSendNotify
    2382e104 7742afbf comctl32!ListView_ForwardHeaderNotify+0x20, calling comctl32!SendNotifyEx
    2382e124 7c90e48a ntdll!KiUserExceptionDispatcher+0xe, calling ntdll!RtlDispatchException
    2382e424 21313394 21313394 ====> Exception Code 2382ffa4 cxr@2382e158 exr@2382ff74
    2382e388 7e4dfe1a hhctrl!xHtmlHelpA+0x85e, calling hhctrl!InitializeSession
    2382e3a8 7e4df5c6 hhctrl!xHtmlHelpA+0xa, calling hhctrl!_EH_prolog
    2382e3bc 7e4dbcf4 hhctrl!HtmlHelpA+0x144, calling hhctrl!xHtmlHelpA
    2382e420 7c9032a8 ntdll!ExecuteHandler2+0x26
    2382e444 7c90327a ntdll!ExecuteHandler+0x24, calling ntdll!ExecuteHandler2
    2382e468 7c92aa0f ntdll!RtlDispatchException+0xb1, calling ntdll!RtlpExecuteHandlerForException
    2382e4c8 7e41882a user32!UserCallWinProcCheckWow+0x116, calling user32!_SEH_epilog
    2382e4cc 7e42927b user32!SendMessageWorker+0x4a5, calling user32!UserCallWinProcCheckWow
    2382e4f4 7c90e48a ntdll!KiUserExceptionDispatcher+0xe, calling ntdll!RtlDispatchException
    2382e7f4 21313394 21313394 ====> Exception Code 2382ffa4 cxr@2382e528 exr@2382ff74
    2382e758 7e4dfe1a hhctrl!xHtmlHelpA+0x85e, calling hhctrl!InitializeSession
    2382e778 7e4df5c6 hhctrl!xHtmlHelpA+0xa, calling hhctrl!_EH_prolog
    2382e78c 7e4dbcf4 hhctrl!HtmlHelpA+0x144, calling hhctrl!xHtmlHelpA
    2382e7f0 7c9032a8 ntdll!ExecuteHandler2+0x26
    2382e814 7c90327a ntdll!ExecuteHandler+0x24, calling ntdll!ExecuteHandler2
    2382e838 7c92aa0f ntdll!RtlDispatchException+0xb1, calling ntdll!RtlpExecuteHandlerForException
    2382e860 7e418734 user32!InternalCallWinProc+0x28
    2382e88c 7e418bd9 user32!_EndUserApiHook+0x11, calling kernel32!InterlockedIncrement+0x4
    2382e898 7e41885a user32!UserCallWinProcCheckWow+0x170, calling ntdll!RtlDeactivateActivationContextUnsafeFast
    2382e8a0 7e41882a user32!UserCallWinProcCheckWow+0x116, calling user32!_SEH_epilog
    2382e8c4 7c90e48a ntdll!KiUserExceptionDispatcher+0xe, calling ntdll!RtlDispatchException
    2382ebc4 21313394 21313394 ====> Exception Code 2382ffa4 cxr@2382e8f8 exr@2382ff74
    2382eb28 7e4dfe1a hhctrl!xHtmlHelpA+0x85e, calling hhctrl!InitializeSession
    2382eb48 7e4df5c6 hhctrl!xHtmlHelpA+0xa, calling hhctrl!_EH_prolog
    2382eb5c 7e4dbcf4 hhctrl!HtmlHelpA+0x144, calling hhctrl!xHtmlHelpA
    2382ebc0 7c9032a8 ntdll!ExecuteHandler2+0x26
    2382ebe4 7c90327a ntdll!ExecuteHandler+0x24, calling ntdll!ExecuteHandler2
    2382ec08 7c92aa0f ntdll!RtlDispatchException+0xb1, calling ntdll!RtlpExecuteHandlerForException
    2382ec40 7e42c228 user32!RealDefWindowProcA+0x47, calling user32!RealDefWindowProcWorker
    2382ec58 7e418bd9 user32!_EndUserApiHook+0x11, calling kernel32!InterlockedIncrement+0x4
    2382ec68 7e42c23c user32!DefWindowProcA+0x94, calling user32!_EndUserApiHook
    2382ec6c 7e42c1e9 user32!DefWindowProcA+0x86, calling user32!_SEH_epilog
    2382ec74 7e42c17e user32!DefWindowProcA
    2382ec80 7e42c17e user32!DefWindowProcA
    2382ec94 7c90e48a ntdll!KiUserExceptionDispatcher+0xe, calling ntdll!RtlDispatchException
    2382ef94 21313394 21313394 ====> Exception Code 2382ffa4 cxr@2382ecc8 exr@2382ff74
    2382eef8 7e4dfe1a hhctrl!xHtmlHelpA+0x85e, calling hhctrl!InitializeSession
    2382ef18 7e4df5c6 hhctrl!xHtmlHelpA+0xa, calling hhctrl!_EH_prolog
    2382ef2c 7e4dbcf4 hhctrl!HtmlHelpA+0x144, calling hhctrl!xHtmlHelpA
    2382ef90 7c9032a8 ntdll!ExecuteHandler2+0x26
    2382efb4 7c90327a ntdll!ExecuteHandler+0x24, calling ntdll!ExecuteHandler2
    2382efd8 7c92aa0f ntdll!RtlDispatchException+0xb1, calling ntdll!RtlpExecuteHandlerForException
    2382f000 774eeea1 ole32!CoUninitialize+0x64, calling ole32!NotifyInitializeSpies
    2382f018 7c910a36 ntdll!RtlpCoalesceFreeBlocks+0x383, calling ntdll!RtlpUpdateIndexRemoveBlock
    2382f038 7c9116a6 ntdll!RtlpCoalesceFreeBlocks+0x13d, calling ntdll!RtlpUpdateIndexRemoveBlock
    2382f064 7c90e48a ntdll!KiUserExceptionDispatcher+0xe, calling ntdll!RtlDispatchException
    2382f364 21313394 21313394 ====> Exception Code 2382ffa4 cxr@2382f098 exr@2382ff74
    2382f2c8 7e4dfe1a hhctrl!xHtmlHelpA+0x85e, calling hhctrl!InitializeSession
    2382f2e8 7e4df5c6 hhctrl!xHtmlHelpA+0xa, calling hhctrl!_EH_prolog
    2382f2fc 7e4dbcf4 hhctrl!HtmlHelpA+0x144, calling hhctrl!xHtmlHelpA
    2382f360 7c9032a8 ntdll!ExecuteHandler2+0x26
    2382f384 7c90327a ntdll!ExecuteHandler+0x24, calling ntdll!ExecuteHandler2
    2382f3a8 7c92aa0f ntdll!RtlDispatchException+0xb1, calling ntdll!RtlpExecuteHandlerForException
    2382f3ec 7e428eab user32!DispatchClientMessage+0xae, calling user32!_SEH_epilog
    2382f434 7c90e48a ntdll!KiUserExceptionDispatcher+0xe, calling ntdll!RtlDispatchException
    2382f734 21313394 21313394 ====> Exception Code 2382ffa4 cxr@2382f468 exr@2382ff74
    2382f698 7e4dfe1a hhctrl!xHtmlHelpA+0x85e, calling hhctrl!InitializeSession
    2382f6b8 7e4df5c6 hhctrl!xHtmlHelpA+0xa, calling hhctrl!_EH_prolog
    2382f6cc 7e4dbcf4 hhctrl!HtmlHelpA+0x144, calling hhctrl!xHtmlHelpA
    2382f730 7c9032a8 ntdll!ExecuteHandler2+0x26
    2382f754 7c90327a ntdll!ExecuteHandler+0x24, calling ntdll!ExecuteHandler2
    2382f778 7c92aa0f ntdll!RtlDispatchException+0xb1, calling ntdll!RtlpExecuteHandlerForException
    2382f7a0 5b291adb uxtheme!_ThemeDefWindowProc+0x14e, calling uxtheme!CThemeWnd::Release
    2382f7a8 7e42c17e user32!DefWindowProcA
    2382f7e8 5b2ac2b1 uxtheme!ThemeDefWindowProcA+0x18, calling uxtheme!_ThemeDefWindowProc
    2382f804 7c90e48a ntdll!KiUserExceptionDispatcher+0xe, calling ntdll!RtlDispatchException
    2382fb04 21313394 21313394 ====> Exception Code 2382ffa4 cxr@2382f838 exr@2382ff74
    2382fa68 7e4dfe1a hhctrl!xHtmlHelpA+0x85e, calling hhctrl!InitializeSession
    2382fa88 7e4df5c6 hhctrl!xHtmlHelpA+0xa, calling hhctrl!_EH_prolog
    2382fa9c 7e4dbcf4 hhctrl!HtmlHelpA+0x144, calling hhctrl!xHtmlHelpA
    2382fb00 7c9032a8 ntdll!ExecuteHandler2+0x26
    2382fb24 7c90327a ntdll!ExecuteHandler+0x24, calling ntdll!ExecuteHandler2
    2382fb48 7c92aa0f ntdll!RtlDispatchException+0xb1, calling ntdll!RtlpExecuteHandlerForException
    2382fb80 77c1c2e3 msvcrt!free+0xc8, calling msvcrt!_SEH_epilog
    2382fb84 5b294e43 uxtheme!CThemeWnd::`scalar deleting destructor'+0x19, calling uxtheme!operator delete
    2382fb94 5b294e20 uxtheme!CThemeWnd::Release+0x23, calling uxtheme!CThemeWnd::`scalar deleting destructor'
    2382fba4 5b291adb uxtheme!_ThemeDefWindowProc+0x14e, calling uxtheme!CThemeWnd::Release
    2382fbac 7e42c17e user32!DefWindowProcA
    2382fbd4 7c90e48a ntdll!KiUserExceptionDispatcher+0xe, calling ntdll!RtlDispatchException
    2382fed4 7e42776f user32!GetMessageA+0x91 ====> Exception cxr@2382fc08
    2382fe38 7e4dfe1a hhctrl!xHtmlHelpA+0x85e, calling hhctrl!InitializeSession
    2382fe58 7e4df5c6 hhctrl!xHtmlHelpA+0xa, calling hhctrl!_EH_prolog
    2382fe6c 7e4dbcf4 hhctrl!HtmlHelpA+0x144, calling hhctrl!xHtmlHelpA
    2382ffb4 7c80b729 kernel32!BaseThreadStart+0x37
    0:027> !pe
    The current thread is unmanaged
    0:027> !pe
    The current thread is unmanaged
    0:027> .ecxr
    eax=00000000 ebx=00000000 ecx=21313394 edx=7c9032bc esi=00000000 edi=00000000
    eip=21313394 esp=2382e054 ebp=2382e074 iopl=0         nv up ei pl zr na pe nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246
    21313394 0100            add     dword ptr [eax],eax  ds:0023:00000000=????????


    From the stacktrace I can see that the error is probably related to HtmlHelp (HelpProvider component?). I have checked with our users and they have not used the help system in the instance of the app that crashed.

    I would appreciate help in analyzing the above stacktrace and hopefully finding the cause of the problem.

    Friday, September 3, 2010 8:30 AM

Answers

  • It could be that some managed control does not properly pin objects passed to unmanaged code which can cause such behaviour when a GC cycle has happened or is happening. To further pinpoint the root cause you should get several dumps and check for communalities. If all are hanging around in the GC then the probabilities are high that an object that just has been finalized (your actally posted the finalizer thread call stack there) which did trigger a GC. My first guess would be that some window handle was not pinned and this was passed to unmanaged code which now gets UI events although the real object is gone.

     

    Yours,

       Alois Kraus

     

    • Marked as answer by a_Xandu_ Thursday, September 9, 2010 6:53 AM
    Monday, September 6, 2010 8:33 PM

All replies

  • Did you try !analyze -v?

    You should have a look at the Exception record to check under which circumstance this did happen. From the call stack I do not see any managed code involved there.

    It would help to print the call Stacks of the others as well.

    ~*e!DumpStack

    should do the trick.

    You can display the contents of the first exception record with.

    .exr 2382ff74

    that should give some hints.

    Yours,

       Alois Kraus

     

    Friday, September 3, 2010 7:18 PM
  • Hi and thanks for the hints. After executing the .Analyze command I found the EXCEPTION_RECORD and the .exr command on that gave me this:

    .exr 0x2382fbec
    ExceptionAddress: 7e42776f (user32!GetMessageA+0x00000091)
       ExceptionCode: c0000005 (Access violation)
      ExceptionFlags: 00000000
    NumberParameters: 2
       Parameter[0]: 00000000
       Parameter[1]: 0fa57414
    Attempt to read from address 0fa57414

    So it seems like hhctrl!InitializeSession calls GetMEssage with a invalid parameter and that starts the exception chain that can be seen in the stack trace above.

    When looking at the stacks for the other threads I found that it seems like garbagecollection is running when the app crashes:

    Current frame: mscorwks!WKS::gc_heap::relocate_survivors_in_brick+0x35
    ChildEBP RetAddr  Caller,Callee
    00d6fbac 79f79f56 mscorwks!WKS::gc_heap::relocate_survivors+0xc0, calling mscorwks!WKS::gc_heap::relocate_survivors_in_brick
    00d6fbdc 79f79cbc mscorwks!WKS::gc_heap::relocate_phase+0x60, calling mscorwks!WKS::gc_heap::relocate_survivors
    00d6fc10 79f76f61 mscorwks!WKS::gc_heap::plan_phase+0x7f3, calling mscorwks!WKS::gc_heap::relocate_phase
    00d6fc28 79f70bde mscorwks!CheckPromoted+0x1a
    00d6fc2c 7c90dc1a ntdll!NtSetEvent+0xc
    00d6fc44 79e8c5c9 mscorwks!CLREvent::Set+0xf6, calling mscorwks!_EH_epilog3
    00d6fc64 79f71cda mscorwks!SyncBlockCache::GCWeakPtrScan+0x18e, calling mscorwks!SyncBlockCache::GCWeakPtrScanElement
    00d6fc74 79e8c5c9 mscorwks!CLREvent::Set+0xf6, calling mscorwks!_EH_epilog3
    00d6fc78 79f80f58 mscorwks!SyncBlockCache::GCWeakPtrScan+0x292
    00d6fd00 79f763cb mscorwks!WKS::gc_heap::gc1+0x6e, calling mscorwks!WKS::gc_heap::plan_phase
    00d6fd20 79f76945 mscorwks!WKS::gc_heap::garbage_collect+0x253, calling mscorwks!WKS::gc_heap::gc1
    00d6fd34 79f7665a mscorwks!WKS::GCHeap::GarbageCollectGeneration+0x1a9, calling mscorwks!WKS::gc_heap::garbage_collect
    00d6fd60 79f55049 mscorwks!WKS::GCHeap::GarbageCollectTry+0x33, calling mscorwks!WKS::GCHeap::GarbageCollectGeneration
    00d6fd70 79f55085 mscorwks!WKS::GCHeap::GarbageCollect+0x67, calling mscorwks!WKS::GCHeap::GarbageCollectTry
    00d6fd90 79f76308 mscorwks!WKS::WaitForFinalizerEvent+0xb8
    00d6fdac 79f7a728 mscorwks!WKS::GCHeap::FinalizerThreadWorker+0x49, calling mscorwks!WKS::WaitForFinalizerEvent
    00d6fdc0 79e9848f mscorwks!Thread::DoADCallBack+0x32a
    00d6fdd4 79e9842b mscorwks!Thread::ShouldChangeAbortToUnload+0xe3, calling mscorwks!Thread::DoADCallBack+0x2db
    00d6fdfc 79f6efe4 mscorwks!ThreadStore::UnlockThreadStore+0x4c, calling mscorwks!DecCantStopCount
    00d6fe08 79f04c82 mscorwks!ThreadStore::TransferStartedThread+0xaa, calling mscorwks!ThreadStore::UnlockThreadStore
    00d6fe68 79e98351 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a, calling mscorwks!Thread::ShouldChangeAbortToUnload+0x32
    00d6fea4 79f074d4 mscorwks!ManagedThreadBase_NoADTransition+0x32, calling mscorwks!Thread::ShouldChangeAbortToUnload+0x29d
    00d6fecc 79f074e5 mscorwks!ManagedThreadBase::FinalizerBase+0xd, calling mscorwks!ManagedThreadBase_NoADTransition
    00d6fedc 79f5445c mscorwks!WKS::GCHeap::FinalizerThreadStart+0xbb, calling mscorwks!ManagedThreadBase::FinalizerBase
    00d6ff14 79f75715 mscorwks!Thread::intermediateThreadProc+0x49
    00d6ffa0 79f75703 mscorwks!Thread::intermediateThreadProc+0x37, calling mscorwks!_alloca_probe_16
    00d6ffb4 7c80b729 kernel32!BaseThreadStart+0x37

    All other threads (except for the unmanaged one) has the last call on the stack as 0013ea4c 7c90df5a ntdll!ZwWaitForSingleObject+0xc. I also see calls to mscorwks!WKS::GCHeap::WaitUntilGCComplete+0x34, calling mscorwks!CLREvent::Wait on these stacks which supports my assumption about the carbage collection.

    What I'm then wondering is why a unmanaged thread is trying to initialize the HTML Active X control when the garbage collection is running? I was suspecting the HelpProvider class but after checking out the source it seems that the HelpProvider is not even loaded when the crash happens. We do have a WebBrowser control in our app and AFAIK this is a wrapped activeX control which I guess could be running on a unmanaged thread. So Im starting to think the problem might be caused by the webbrowser control.

    Monday, September 6, 2010 7:09 AM
  • It could be that some managed control does not properly pin objects passed to unmanaged code which can cause such behaviour when a GC cycle has happened or is happening. To further pinpoint the root cause you should get several dumps and check for communalities. If all are hanging around in the GC then the probabilities are high that an object that just has been finalized (your actally posted the finalizer thread call stack there) which did trigger a GC. My first guess would be that some window handle was not pinned and this was passed to unmanaged code which now gets UI events although the real object is gone.

     

    Yours,

       Alois Kraus

     

    • Marked as answer by a_Xandu_ Thursday, September 9, 2010 6:53 AM
    Monday, September 6, 2010 8:33 PM
  • Thanks, I have managed to obtain 5 similar minidumps and was finally able to find the module that crashed the application. It was actually a COM module that was only loaded in a spesific scenario. This was the COM module that used the HTMLHelp activeX (that can be seen in the stacktrace) and when the COM module was released by the GC then it crashed because of a access violation in  the HTML Help uninitialize method. I have not put any resources in to finding out why this happened because I was lucky enough to have the source code for the com module and could remove all references to the HTML Help Active X.
    Thursday, September 9, 2010 6:53 AM