none
With W3WP Abnormal Termination Dialog being attempted, how to determine whether this is IIS bug or just a detected deadlock RRS feed

  • Question

  • We had an ASP.Net process which seemed to be hung.   In the dump I was provided, a thread is trying to pop a MessageBox that says what W3WP.exe has abnormally terminated.  The rest of the dump looks like there’s an out-of-memory condition with an ongoing GC which is blocked by the thread trying to show the MessageBox.

    My questions:  How can I determine whether this represents an IIS bug of some kind or whether it is just a case of IIS detecting that this worker process is deadlocked and has forced a restart?   And why would an actual MessageBox be shown by an individual thread like this?  (The server was, of course, effectively “headless”.  No one’s watching that console).

    Here’s the situation:

    A GC has been triggered by thread 12.  

    In addition to 12, Threads 19, 20, 22, 25, and 38 area all simply stalled waiting on the GC.

    That GC cannot proceed because thread 11 has disabled pre-emptive GC.  

    Thread 11 is currently trying to use an InternalDialogBox to notify that the W3WP process has abnormally terminated.

    Threads 14 (the Suspend EE thread) and 15 are GC threads.

    Thread 14 is also holding a critical section which is wanted by thread #39

    Thread 38 has hit a previous exception and is trying to do an HttpApplication.RecordError() to log it.

    0:038> !threads -special

    ThreadCount: 19

    UnstartedThread: 1

    BackgroundThread: 18

    PendingThread: 1

    DeadThread: 0

    Hosted Runtime: no

                                          PreEmptive   GC Alloc           Lock

           ID OSID ThreadOBJ    State     GC       Context       Domain   Count APT Exception

      11    1  8e4 000cce80   1808222 Disabled 00000000:00000000 0013e040     0 MTA (Threadpool Worker)

      16    2  fb0 001461f0      b220 Enabled  00000000:00000000 0013e040     0 MTA (Finalizer)

      17    3  73c 0015fcc0    80a220 Enabled  00000000:00000000 0013e040     0 MTA (Threadpool Completion Port)

      18    4  fc0 00168708      1220 Enabled  00000000:00000000 0013e040     0 Ukn

      19    5 11f4 025c2230   180b220 Enabled  00000000:00000000 00168d38     1 MTA (Threadpool Worker)

      20    6 1694 025d2998   180b220 Enabled  00000000:00000000 00168d38     1 MTA (Threadpool Worker)

      12    7 14b0 025d0e98   180a220 Enabled  52812c40:52813460 00168d38     1 MTA (Threadpool Worker)

      22    9  20c 02630550   180b220 Enabled  54e347a0:54e34d50 00168d38     1 MTA (Threadpool Worker)

       2   11  f94 026b6a40       220 Enabled  00000000:00000000 0013e040     0 Ukn

       4   12 10e0 137e8818       220 Enabled  00000000:00000000 0013e040     0 Ukn

       3   13  21c 02655188       220 Enabled  00000000:00000000 0013e040     0 Ukn

       9   14 14a4 000d4770   880a220 Enabled  00000000:00000000 0013e040     0 MTA (Threadpool Completion Port)

      25   15 13ac 138a2008      b220 Enabled  54e2d020:54e2ed50 00168d38     0 MTA

      28    d 11f0 000c3ab8       220 Enabled  00000000:00000000 0013e040     0 STA

      29    e 1500 138a7ff0       220 Enabled  00000000:00000000 0013e040     0 STA

      30    a  110 000e8ea8       220 Enabled  00000000:00000000 0013e040     0 STA

       5   16 15dc 02665050       220 Enabled  00000000:00000000 0013e040     0 Ukn

      38   10  22c 0266cb60   180b220 Enabled  5280d9d0:5280f460 00168d38     1 MTA (Threadpool Worker)

      39    8 15bc 138e9118      1400 Enabled  00000000:00000000 0013e040     0 Ukn

           OSID     Special thread type

        9   14a4    IOCompletion

       10   16cc    Gate

       11    8e4    ThreadpoolWorker

       12   14b0    ThreadpoolWorker

       13   1450    DbgHelper

       14    3d8    GC SuspendEE

       15   11a4    GC

       16    fb0    Finalizer

       17    73c    Timer

       18    fc0    ADUnloadHelper

       19   11f4    ThreadpoolWorker

       20   1694    ThreadpoolWorker

       22    20c    ThreadpoolWorker

       38    22c    ThreadpoolWorker

       39   15bc    ThreadpoolWorker

    Thread 11, which is the one who has GC enabled has the following current call stack.   Note that he’s the thread which is trying to pop a message saying “…w3wp.exe …abnormal program termination” via an InternalDialogBox.  He also has called the UnhandledExceptionFilter

    0:011> kvn 200

    # ChildEBP RetAddr  Args to Child             

    00 01d4ebe8 7739bf53 7739610a 00000000 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    01 01d4ec20 7738965e 14c40086 00000000 00000001 user32!NtUserWaitMessage+0xc

    02 01d4ec48 7739f762 77380000 0012e1b8 00000000 user32!InternalDialogBox+0xd0 (FPO: [Non-Fpo])

    03 01d4ef08 7739f047 01d4f064 00000000 ffffffff user32!SoftModalMessageBox+0x94b (FPO: [Non-Fpo])

    04 01d4f058 7739eec9 01d4f064 00000028 00000000 user32!MessageBoxWorker+0x2ba (FPO: [Non-Fpo])

    05 01d4f0b0 773d7d0d 00000000 000e9a50 138cc280 user32!MessageBoxTimeoutW+0x7a (FPO: [Non-Fpo])

    06 01d4f0e4 773c42c8 00000000 01d4f280 1f2d463c user32!MessageBoxTimeoutA+0x9c (FPO: [Non-Fpo])

    07 01d4f104 773c42a4 00000000 01d4f280 1f2d463c user32!MessageBoxExA+0x1b (FPO: [Non-Fpo])

    08 01d4f120 1f2d2546 00000000 01d4f280 1f2d463c user32!MessageBoxA+0x45 (FPO: [Non-Fpo])

    WARNING: Stack unwind information not available. Following frames may be wrong.

    09 01d4f320 1f2d0d4b 0000000a 1f2ce99d 00000000 LTAUT14N!L_AutEditVectorObject+0xb826

    0a 01d4f358 77e76157 01d4f5d8 00000000 00000000 LTAUT14N!L_AutEditVectorObject+0xa02b

    0b 01d4f5b0 77e79243 01d4f5d8 77e61aa9 01d4f5e0 kernel32!UnhandledExceptionFilter+0x12a (FPO: [Non-Fpo])

    0c 01d4f5b8 77e61aa9 01d4f5e0 00000000 01d4f5e0 kernel32!BaseThreadStart+0x4a (FPO: [SEH])

    0d 01d4f5e0 7c8285c2 01d4f99c 01d4ffdc 01d4f6bc kernel32!_except_handler3+0x61 (FPO: [Uses EBP] [3,0,7])

    0e 01d4f604 7c828593 01d4f99c 01d4ffdc 01d4f6bc ntdll!ExecuteHandler2+0x26

    0f 01d4f6ac 7c8284ac 01d46000 01d4f6bc 00010007 ntdll!ExecuteHandler+0x24

    10 01d4f98c 77e4bef7 01d4f99c 00000002 e06d7363 ntdll!RtlRaiseException+0x3d

    11 01d4f9ec 78158e89 e06d7363 00000001 00000003 kernel32!RaiseException+0x53 (FPO: [Non-Fpo])

    12 01d4fa24 7a175ff8 01d4fa34 7a396c84 7a3ddab0 msvcr80!_CxxThrowException+0x46 (FPO: [Non-Fpo]) (CONV: stdcall) [f:\dd\vctools\crt_bld\self_x86\crt\prebuild\eh\throw.cpp @ 161]

    13 01d4fa38 7a12debb 8b6a3d68 79fbc2b8 01d4fc0c mscorwks!ThrowOutOfMemory+0x24 (FPO: [Non-Fpo])

    14 01d4fb64 7a12fe25 00000000 01d4fb9c 01d4fc0c mscorwks!Thread::RaiseCrossContextException+0x408 (FPO: [Non-Fpo])

    15 01d4fc18 79ee797b 00000002 79ee7946 01d4fd10 mscorwks!Thread::DoADCallBack+0x2a2 (FPO: [Non-Fpo])

    16 01d4fc34 79e9afeb 01d4fd10 01d4fcbc 79f4ac97 mscorwks!Thread::DoADCallBack+0x310 (FPO: [Non-Fpo])

    17 01d4fcc8 79e9af11 01d4fd10 8b6a3b08 7a3ce24c mscorwks!Thread::ShouldChangeAbortToUnload+0xe3 (FPO: [Non-Fpo])

    18 01d4fd04 79e9b09d 01d4fd10 00000002 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a (FPO: [Non-Fpo])

    19 01d4fd2c 79ee80bc 00000002 79ee86e0 01d4fd70 mscorwks!Thread::ShouldChangeAbortToUnload+0x33e (FPO: [Non-Fpo])

    1a 01d4fd44 79ee86ad 00000002 79ee86e0 01d4fd70 mscorwks!ManagedThreadBase::ThreadPool+0x13 (FPO: [Non-Fpo])

    1b 01d4fd8c 79ee86dc 0269a838 00000001 00000001 mscorwks!AddTimerCallbackEx+0x83 (FPO: [Non-Fpo])

    1c 01d4fda0 79ee8782 0269a838 00000001 8b6a3bd0 mscorwks!AddTimerCallback+0x10 (FPO: [Non-Fpo])

    1d 01d4fddc 79ee8444 138e5848 8b6a3824 79ee7d5c mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64 (FPO: [Non-Fpo])

    1e 01d4fe28 79ee7d56 00000000 00000000 77e619b9 mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x9a (FPO: [Non-Fpo])

    1f 01d4fe3c 79ee7eca 8b6a3898 00000000 79ee7da9 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf (FPO: [0,0,4])

    20 01d4fe94 79f94203 00000000 00000000 00000000 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x20b (FPO: [Non-Fpo])

    21 01d4ffb8 77e6481f 000f1638 00000000 00000000 mscorwks!ThreadpoolMgr::intermediateThreadProc+0x49 (FPO: [Non-Fpo])

    22 01d4ffec 00000000 79f941bd 000f1638 00000000 kernel32!BaseThreadStart+0x34 (FPO: [Non-Fpo])

    Looking at the msvcr80!_CxxThrowException call and checking the symbols near the exception at the address in Parm2:

    0:011> ln 7a396c84

    (7a396c84)   mscorwks!_TI2PAVException   |  (7a396c94)   mscorwks!_CTA2PAVException

    Exact matches:

        mscorwks!_TI2PAVException = <no type information>

    And attempting to dump the exception address out of Parm1 as that type:

    0:011> dt -r mscorwks!_TI2PAVException 01d4fa34

    *************************************************************************

    ***                                                                   ***

    ***                                                                   ***

    ***    Your debugger is not using the correct symbols                 ***

    ***                                                                   ***

    ***    In order for this command to work properly, your symbol path   ***

    ***    must point to .pdb files that have full type information.      ***

    ***                                                                   ***

    ***    Certain .pdb files (such as the public OS symbols) do not      ***

    ***    contain the required information.  Contact the group that      ***

    ***    provided you with these symbols if you need this command to    ***

    ***    work.                                                          ***

    ***                                                                   ***

    ***    Type referenced: mscorwks!_TI2PAVException                     ***

    ***                                                                   ***

    *************************************************************************

    Symbol mscorwks!_TI2PAVException not found.

    0:011> dt -r _TI2PAVException 01d4fa34

    Symbol _TI2PAVException not found.

    So we have no definition for that type.

    Ok.  Trying to get to this another way:   Looking at the unhandled exception filter params:

    0:011> dd 01d4f5d8

    01d4f5d8  01d4f99c 01d4f6bc 01d4f604 7c8285c2

    01d4f5e8  01d4f99c 01d4ffdc 01d4f6bc 01d4f6a0

    01d4f5f8  01d4fcbc 7c8285d6 01d4ffdc 01d4f6ac

    01d4f608  7c828593 01d4f99c 01d4ffdc 01d4f6bc

    01d4f618  01d4f6a0 77e61a48 00000000 01d4f99c

    01d4f628  01d4ffdc 7c831441 01d4f99c 01d4ffdc

    01d4f638  01d4f6bc 01d4f6a0 77e61a48 000cce80

    01d4f648  00000000 01d4fa24 bfba1002 79e74928

    Looking at the corresponding exception address:

    0:011> .exr 01d4f99c

    ExceptionAddress: 77e4bef7 (kernel32!RaiseException+0x00000053)

       ExceptionCode: e06d7363 (C++ EH exception)

      ExceptionFlags: 00000001

    NumberParameters: 3

       Parameter[0]: 19930520

       Parameter[1]: 01d4fa34

       Parameter[2]: 7a396c84

      pExceptionObject: 01d4fa34

      _s_ThrowInfo    : 7a396c84

      Type            : class Exception *

      Type            : void *

    Looking at Parm2 of the Exception Record (This is a 32-bit app on WinServer 2003)

    0:011> dd 7a396c84 l4

    7a396c84  00000000 00000000 00000000 7a396c94

    0:011> dd 7a396c94 l2

    7a396c94  00000002 79f4ad2c

    0:011> dd 79f4ad2c l2

    79f4ad2c  00000001 7a3d0b6c

    0:011> da 7a3d0b6c+8

    7a3d0b74  ".PAVException@@"

    Looking at the corresponding Context information:

    0:011> .cxr 01d4f6bc

    eax=01d4f99c ebx=00000000 ecx=00000000 edx=ffffffff esi=01d4fa24 edi=000cce80

    eip=77e4bef7 esp=01d4f998 ebp=01d4f9ec iopl=0         nv up ei pl nz na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00200206

    kernel32!RaiseException+0x53:

    77e4bef7 5e              pop     esi

    0:011> kv

      *** Stack trace for last set context - .thread/.cxr resets it

    ChildEBP RetAddr  Args to Child             

    01d4f9ec 78158e89 e06d7363 00000001 00000003 kernel32!RaiseException+0x53 (FPO: [Non-Fpo])

    01d4fa24 7a175ff8 01d4fa34 7a396c84 7a3ddab0 msvcr80!_CxxThrowException+0x46 (FPO: [Non-Fpo]) (CONV: stdcall) [f:\dd\vctools\crt_bld\self_x86\crt\prebuild\eh\throw.cpp @ 161]

    01d4fa38 7a12debb 8b6a3d68 79fbc2b8 01d4fc0c mscorwks!ThrowOutOfMemory+0x24 (FPO: [Non-Fpo])

    01d4fb64 7a12fe25 00000000 01d4fb9c 01d4fc0c mscorwks!Thread::RaiseCrossContextException+0x408 (FPO: [Non-Fpo])

    01d4fc18 79ee797b 00000002 79ee7946 01d4fd10 mscorwks!Thread::DoADCallBack+0x2a2 (FPO: [Non-Fpo])

    01d4fc34 79e9afeb 01d4fd10 01d4fcbc 79f4ac97 mscorwks!Thread::DoADCallBack+0x310 (FPO: [Non-Fpo])

    01d4fcc8 79e9af11 01d4fd10 8b6a3b08 7a3ce24c mscorwks!Thread::ShouldChangeAbortToUnload+0xe3 (FPO: [Non-Fpo])

    01d4fd04 79e9b09d 01d4fd10 00000002 00000000 mscorwks!Thread::ShouldChangeAbortToUnload+0x30a (FPO: [Non-Fpo])

    01d4fd2c 79ee80bc 00000002 79ee86e0 01d4fd70 mscorwks!Thread::ShouldChangeAbortToUnload+0x33e (FPO: [Non-Fpo])

    01d4fd44 79ee86ad 00000002 79ee86e0 01d4fd70 mscorwks!ManagedThreadBase::ThreadPool+0x13 (FPO: [Non-Fpo])

    01d4fd8c 79ee86dc 0269a838 00000001 00000001 mscorwks!AddTimerCallbackEx+0x83 (FPO: [Non-Fpo])

    01d4fda0 79ee8782 0269a838 00000001 8b6a3bd0 mscorwks!AddTimerCallback+0x10 (FPO: [Non-Fpo])

    01d4fddc 79ee8444 138e5848 8b6a3824 79ee7d5c mscorwks!ThreadpoolMgr::AsyncTimerCallbackCompletion+0x64 (FPO: [Non-Fpo])

    01d4fe28 79ee7d56 00000000 00000000 77e619b9 mscorwks!UnManagedPerAppDomainTPCount::DispatchWorkItem+0x9a (FPO: [Non-Fpo])

    01d4fe3c 79ee7eca 8b6a3898 00000000 79ee7da9 mscorwks!ThreadpoolMgr::ExecuteWorkRequest+0xaf (FPO: [0,0,4])

    01d4fe94 79f94203 00000000 00000000 00000000 mscorwks!ThreadpoolMgr::WorkerThreadStart+0x20b (FPO: [Non-Fpo])

    01d4ffb8 77e6481f 000f1638 00000000 00000000 mscorwks!ThreadpoolMgr::intermediateThreadProc+0x49 (FPO: [Non-Fpo])

    01d4ffec 00000000 79f941bd 000f1638 00000000 kernel32!BaseThreadStart+0x34 (FPO: [Non-Fpo])

    Still no clue as to what caused the unhandled exception filter to fire and how that resulted in an attempt to display a message box about W3WP abnormal termination.  

    So I displayed the stack around the kernel32!RaiseException() call in frame 11.   I’m curious why the R0PAVEEArgumentException reference occurs in the stack prior to the OOM.  That would seem to fit with the TI2PAVException I found via the other path, but I can find no information on this exception or what it means.  

    0:038> dds 01d4f9ec l60

    01d4f9ec  01d4fa24

    01d4f9f0  78158e89 msvcr80!_CxxThrowException+0x46 [f:\dd\vctools\crt_bld\self_x86\crt\prebuild\eh\throw.cpp @ 161]

    01d4f9f4  e06d7363

    01d4f9f8  00000001

    01d4f9fc  00000003

    01d4fa00  01d4fa18

    01d4fa04  e06d7363

    01d4fa08  00000001

    01d4fa0c  00000000

    01d4fa10  00000000

    01d4fa14  00000003

    01d4fa18  19930520

    01d4fa1c  01d4fa34

    01d4fa20  7a396c84 mscorwks!_TI2PAVException

    01d4fa24  01d4fa38

    01d4fa28  7a175ff8 mscorwks!ThrowOutOfMemory+0x24

    01d4fa2c  01d4fa34

    01d4fa30  7a396c84 mscorwks!_TI2PAVException

    01d4fa34  7a3ddab0 mscorwks!DBCS_MAXWID+0x18

    01d4fa38  01d4fb64

    01d4fa3c  7a12debb mscorwks!Thread::RaiseCrossContextException+0x408

    01d4fa40  8b6a3d68

    01d4fa44  79fbc2b8 mscorwks!_CT??_R0PAVEEArgumentException+0x16658

    01d4fa48  01d4fc0c

    01d4fa4c  01d4fc0c

    01d4fa50  bfba1002

    01d4fa54  79e74928 mscorwks!GCFrame::`vftable'

    01d4fa58  01d4fb9c

    01d4fa5c  01d4fad8

    01d4fa60  00000002

    01d4fa64  000cce80

    01d4fa68  00000000

    01d4fa6c  bfba1002

    01d4fa70  79e74928 mscorwks!GCFrame::`vftable'

    01d4fa74  ffffffff

    01d4fa78  01d4faf4

    01d4fa7c  00000003

    01d4fa80  000cce80

    01d4fa84  00000000

    01d4fa88  000cce80

    01d4fa8c  00000001

    01d4fa90  01d4fd10

    01d4fa94  01d4fb10

    01d4fa98  79f4ac97 mscorwks!CLRVectoredExceptionHandlerShimX86

    01d4fa9c  8b6a3d10

    01d4faa0  000cce80

    01d4faa4  01d4fd10

    01d4faa8  00000000

    01d4faac  01d4fadc

    01d4fab0  79e7ce75 mscorwks!CLRException::HandlerState::CleanupTry+0x16

    01d4fab4  79f0a91c mscorwks!Exception::`vftable'

    01d4fab8  00000000

    01d4fabc  00000000

    01d4fac0  01d4fb38

    01d4fac4  00000001

    01d4fac8  00000001

    01d4facc  000cce80

    01d4fad0  8b6a3d34

    01d4fad4  01d4fb9c

    01d4fad8  00000000

    01d4fadc  00000000

    01d4fae0  ffffffff

    01d4fae4  01d44000

    01d4fae8  00000000

    01d4faec  01d4fb28

    01d4faf0  00000052

    01d4faf4  00000000

    01d4faf8  00000000

    01d4fafc  00000000

    01d4fb00  00000000

    01d4fb04  00000000

    01d4fb08  01d4fb28

    01d4fb0c  00000000

    01d4fb10  01d4fb44

    01d4fb14  79efe744 mscorwks!Thread::DoesRegionContainGuardPage+0x19

    01d4fb18  01d44000

    01d4fb1c  01d4fb28

    01d4fb20  0000001c

    01d4fb24  01d11000

    01d4fb28  01d44000

    01d4fb2c  01d10000

    01d4fb30  00000004

    01d4fb34  00002000

    01d4fb38  00001000

    01d4fb3c  00000104

    01d4fb40  00020000

    01d4fb44  01d4fb58

    01d4fb48  79efe71f mscorwks!Thread::DetermineIfGuardPagePresent+0x33

    01d4fb4c  01d11000

    01d4fb50  8b6a3d68

    01d4fb54  01d4fa40

    01d4fb58  01d4fc0c

    01d4fb5c  7a36a733 mscorwks! ?? ::FNODOBFM::`string'+0x22b32

    01d4fb60  ffffffff

    01d4fb64  01d4fc18

    01d4fb68  7a12fe25 mscorwks!Thread::DoADCallBack+0x2a2

    One last issue:  Thread 38 was trying to log an error when this dump was taken.   It did call RareDisablePreemptiveGC, and WaitUntilGCComplete, but it’s not listed in the dump’s !Threads output as still having GC disabled, so I can only guess that it got through that situation and re-enabled GC and is waiting for something else.  So I’m thinking it’s just a victim of the overall OOM situation.  (Someone please correct me if that’s wrong). 

    Just in case that’s part of the problem, here are his unmanaged and managed callstacks:

    0:011> ~38s

    eax=00000218 ebx=0266cb60 ecx=00000210 edx=03ef6434 esi=000003a4 edi=00000000

    eip=7c82845c esp=1fbfe3c4 ebp=1fbfe434 iopl=0         nv up ei pl zr na pe nc

    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000246

    ntdll!KiFastSystemCallRet:

    7c82845c c3              ret

    0:038> kvn 200

     # ChildEBP RetAddr  Args to Child              

    00 1fbfe3c0 7c827b79 77e61d06 000003a4 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

    01 1fbfe3c4 77e61d06 000003a4 00000000 00000000 ntdll!ZwWaitForSingleObject+0xc (FPO: [3,0,0])

    02 1fbfe434 79e7551f 000003a4 ffffffff 00000000 kernel32!WaitForSingleObjectEx+0xac (FPO: [Non-Fpo])

    03 1fbfe478 79e7546b 000003a4 ffffffff 00000000 mscorwks!PEImage::LoadImage+0x1af (FPO: [Non-Fpo])

    04 1fbfe4c8 79e7548a ffffffff 00000000 00000000 mscorwks!CLREvent::WaitEx+0x117 (FPO: [Non-Fpo])

    05 1fbfe4dc 79f6f74f ffffffff 00000000 00000000 mscorwks!CLREvent::Wait+0x17 (FPO: [Non-Fpo])

    06 1fbfe4f0 79f6ead0 00000000 95012320 00000002 mscorwks!SVR::GCHeap::WaitUntilGCComplete+0x34 (FPO: [Non-Fpo])

    07 1fbfe52c 79fb7777 95012ea4 5280d710 00000000 mscorwks!Thread::RareDisablePreemptiveGC+0x1b4 (FPO: [Non-Fpo])

    08 1fbfe8a8 0e977a77 527f550c 5280d6fc 00000000 mscorwks!DebugStackTrace::GetStackFramesInternal+0x8f9 (FPO: [Non-Fpo])

    09 1fbfe8c4 0e977a27 527f550c 00000000 00000001 mscorlib_ni+0x1e7a77

    0a 1fbfe8e0 0e978ae7 00000001 0266cb60 527f550c mscorlib_ni+0x1e7a27

    0b 1fbfe8f8 0e969662 1fbfea3c 1fbfea08 1f3f7992 mscorlib_ni+0x1e8ae7

    0c 1fbfe910 660a0d8e 660a0d93 527f683c 527f6018 mscorlib_ni+0x1d9662

    0d 1fbfea24 66083178 02ae97c8 02aea818 527f5d3c System_Web_ni+0x180d8e

    0e 1fbfea44 66083178 067ea548 02ae9dfc 02ae97c8 System_Web_ni+0x163178

    0f 1fbfea64 660a04d6 527f5d3c 02ae97c8 00000001 System_Web_ni+0x163178

    10 00000000 00000000 00000000 00000000 00000000 System_Web_ni+0x1804d6

    0:038> !clrstack

    OS Thread Id: 0x22c (38)

    ESP       EIP    

    1fbfe57c 7c82845c [HelperMethodFrame_2OBJ: 1fbfe57c] System.Diagnostics.StackTrace.GetStackFramesInternal(System.Diagnostics.StackFrameHelper, Int32, System.Exception)

    1fbfe8b4 0e977a77 System.Diagnostics.StackTrace.CaptureStackTrace(Int32, Boolean, System.Threading.Thread, System.Exception)

    1fbfe8d8 0e977a27 System.Diagnostics.StackTrace..ctor(System.Exception, Boolean)

    1fbfe8ec 0e978ae7 System.Environment.GetStackTrace(System.Exception, Boolean)

    1fbfe900 0e969662 System.Exception.get_StackTrace()

    1fbfe90c 1f3f7992 ApolloWebSite.Global_asax.Application_Error(System.Object, System.EventArgs)

    1fbfea3c 01efb2b5 [MulticastFrame: 1fbfea3c] System.EventHandler.Invoke(System.Object, System.EventArgs)

    1fbfea50 66083178 System.Web.HttpApplication.RaiseOnError()

    1fbfea80 66084662 System.Web.HttpApplication.RecordError(System.Exception)

    1fbfea94 6608fd35 System.Web.HttpApplication+ApplicationStepManager.ResumeSteps(System.Exception)

    1fbfeae4 660839dc System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(System.Web.HttpContext, System.AsyncCallback, System.Object)

    1fbfeb00 66086f4c System.Web.HttpRuntime.ProcessRequestInternal(System.Web.HttpWorkerRequest)

    1fbfeb34 66086bf3 System.Web.HttpRuntime.ProcessRequestNoDemand(System.Web.HttpWorkerRequest)

    1fbfeb44 66085d8c System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr, Int32)

    1fbfed58 79f3c33d [ContextTransitionFrame: 1fbfed58]

    1fbfed8c 79f3c33d [GCFrame: 1fbfed8c]

    1fbfeee8 79f3c33d [ComMethodFrame: 1fbfeee8]

    0:038>

    Dumping Thread 38’s stack via !dso, there are exceptions on its stack, but all are either OOM exceptions, are the application’s own wrapper exception around the OOM exception, or are a System.Web.HttpUnhandledException with an inner OOM exception.  So that doesn’t seem helpful.

    I’ve run out of ideas as to what to check next.   So I still have no clue as to the nature or cause of any UnhandledException other than the ones reacting either directly to the OOM condition or indirectly to the System.Web.HttpUnhandledException wrapping an inner OOM exception.  

    And I still am no closer to ultimately figuring out whether the W3WP process crashed on its own or if it was simply reacting to an OOM/disabled GC deadlock plus why a dialogbox display was being attempted in a headless process.

    Can anyone suggest how to proceed here?

    Thanks!

    -Bob

    Tuesday, October 8, 2013 5:58 PM

All replies

  • Hi Bob, any clues?

    We've run in almost the same issues here and any help would be very appreciated!

    Thanks in advance,

    - Orlando Selenu

    Monday, April 7, 2014 2:15 PM