locked
Deadlock in dbghelp.dll version 6.3.9600.16384 (Windows 8.1 SDK) RRS feed

  • Question

  • Greetings!

    I was taking the latest greatest Debugging Tools for Windows from Windows 8.1 SDK for a QA spin when I encountered a deadlock in dbghelp.dll. I was running a script in cdb.exe with the commands below:

    .logopen <insert your path here>
    .echo "something"
    !analyze -v
    .echo "something else"
    kb 200


    CDB hung after partially displaying the stack from kb 200:

    ChildEBP RetAddr  Args to Child              
    04eafbb0 77ecf896 75c49595 00000000 00000000 ntdll!DbgBreakPoint
    04eafbe0 76d233aa 00000000 04eafc2c 77e69ef2 ntdll!DbgUiRemoteBreakin+0x3c


    Here is what I would expect to see (full stack)

    ChildEBP RetAddr  Args to Child              
    04eafbb0 77ecf896 75c49595 00000000 00000000 ntdll!DbgBreakPoint
    04eafbe0 76d233aa 00000000 04eafc2c 77e69ef2 ntdll!DbgUiRemoteBreakin+0x3c
    04eafbec 77e69ef2 00000000 75c49259 00000000 kernel32!BaseThreadInitThunk+0xe
    04eafc2c 77e69ec5 77ecf85a 00000000 00000000 ntdll!__RtlUserThreadStart+0x70
    04eafc44 00000000 77ecf85a 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b

    I’ve collected 3 hang dumps that show the deadlock condition. If you are a Microsoft employee, let me know and I’ll be happy to share the dumps with you.

    Here is what the deadlock looks like:

    0:000> ~*kb
    
    .  0  Id: 2848.2b94 Suspend: 1 Teb: fffdd000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    000da090 74e0149d 00000344 00000000 000da0d8 ntdll!ZwWaitForSingleObject+0x15
    000da0fc 75281194 00000344 000001f4 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
    000da114 75281148 00000344 000001f4 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
    000da128 6e039522 00000344 000001f4 cada38d2 kernel32!WaitForSingleObject+0x12
    000db05c 6e03aed6 070050dc 070050cc 00000000 dbghelp!DiaLocatePdbMultiThread+0x8d3
    000db4c8 6e056f4c 00000001 030ca4c0 00000000 dbghelp!diaGetPdb+0x1be
    000db6f8 6e055bdd cada3316 00000000 030ca4c0 dbghelp!GetDebugData+0x230
    000dbb98 6e055885 007328b8 030ca4c0 000dbd24 dbghelp!modload+0x285
    000dbbc4 6e058e08 00000000 cada348e 00000e9c dbghelp!LoadSymbols+0x355
    000dbc00 6e0416f6 76d233a9 00000000 cada34c6 dbghelp!GetModFromAddr+0x38
    000dbc48 6e04185a 76d233a9 00000000 cada3406 dbghelp!ldiaGetFrameData+0x29
    000dbc88 6e04b119 76d233a9 00000000 000dbee8 dbghelp!diaGetFrameData+0x32
    000dbca0 6e0783d0 76d10000 00000000 000133a9 dbghelp!DbhStackServices::GetUnwindInfoFromSymbols+0x29
    000dbccc 6e07ce40 76d10000 00000000 000133a9 dbghelp!DbsStackServices::UnwindInfoHolder::GetFromSymbols+0x26
    000dbd50 6e079e60 00000000 000dd6e8 000dbf10 dbghelp!DbsX86StackUnwinder::ApplyUnwindInfo+0x73
    000dbd80 6e078e13 000dcbd8 000dd6e8 000dbee8 dbghelp!DbsX86StackUnwinder::Unwind+0x13e
    000dbd94 6e062ed0 000dd6e8 0000000c 009c7c88 dbghelp!DbsStackUnwinder::DbhUnwind+0xcd
    000dbe98 6e06348c 000dd6e8 000dcbd8 0003bac0 dbghelp!PickX86Walk+0x13f
    000dcb98 5fd1d2e4 0000014c 00000e9c 00000005 dbghelp!StackWalkEx+0x38e
    000dd808 5fd1da02 002ce360 03500020 05a0106c dbgeng!TargetInfo::GetTargetStackFrames+0x44d
    000dd880 5fcd8cdd 0000000c 0000101d cada5032 dbgeng!DoStackTrace+0x123
    000dd8f8 5fcd9e84 00000000 00000000 00308aa8 dbgeng!WrapParseStackCmd+0x107
    000dd968 5fcda955 cada5102 0577a02a 00000001 dbgeng!ProcessCommands+0x91c
    000dd9c8 5fc4cdd4 00000003 cada5112 000373f0 dbgeng!ProcessCommandsAndCatch+0x91
    000dde34 5fcdaae5 00000006 00000003 cada56b6 dbgeng!Execute+0x226
    000dde7c 5fc71f90 000ddf08 5fc7b93d 5fde7b2c dbgeng!ProcessCurBraceBlock+0x67
    000dde84 5fc7b93d 5fde7b2c 000373f0 000373f0 dbgeng!DotBlock+0x10
    000dde98 5fcd9dd8 00000000 00000000 00308680 dbgeng!DotCommand+0x36
    000ddf08 5fcda955 cada57a2 0134d27a 00000001 dbgeng!ProcessCommands+0x870
    000ddf68 5fc4cdd4 00000000 cada57b2 00000002 dbgeng!ProcessCommandsAndCatch+0x91
    000de3d0 5fcd8ac4 00000002 00000000 cada6cf2 dbgeng!Execute+0x226
    000de438 5fcd9c8d 00000000 00000000 00000000 dbgeng!ParseDollar+0x20b
    000de4a8 5fcda955 cada6dc2 000de556 00000001 dbgeng!ProcessCommands+0x725
    000de508 5fc4cdd4 00000000 cada6dd2 00000000 dbgeng!ProcessCommandsAndCatch+0x91
    000de974 5fc4cfc5 00000002 00000000 cada6142 dbgeng!Execute+0x226
    000de9c0 5fc4cf18 000373f8 00000001 000dea00 dbgeng!DebugClient::ExecuteWide+0x8d
    000dec10 00188c92 000373f8 00000001 000dec50 dbgeng!DebugClient::Execute+0x74
    000dfc5c 0018ac98 00000000 00000001 00000000 cdb!MainLoop+0x407
    000dfe94 0018c5fe 00000005 002bedc8 000321f0 cdb!main+0x2ad
    000dfed4 7528336a fffde000 000dff20 77209f72 cdb!SetLastError+0x206
    000dfee0 77209f72 fffde000 6207da85 00000000 kernel32!BaseThreadInitThunk+0xe
    000dff20 77209f45 0018c694 fffde000 00000000 ntdll!__RtlUserThreadStart+0x70
    000dff38 00000000 0018c694 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       1  Id: 2848.2124 Suspend: 1 Teb: fffda000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    011ff9d4 77222f91 00000003 009c2068 00000001 ntdll!NtWaitForMultipleObjects+0x15
    011ffb68 7528336a 00000000 011ffbb4 77209f72 ntdll!TppWaiterpThread+0x33d
    011ffb74 77209f72 009c2038 6315de11 00000000 kernel32!BaseThreadInitThunk+0xe
    011ffbb4 77209f45 77222e65 009c2038 00000000 ntdll!__RtlUserThreadStart+0x70
    011ffbcc 00000000 77222e65 009c2038 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       2  Id: 2848.1c78 Suspend: 1 Teb: fffa9000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0178fbc8 77208e44 00000348 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
    0178fc2c 77208d28 00000000 00000000 0224e008 ntdll!RtlpWaitOnCriticalSection+0x13e
    0178fc54 6e038b14 6e11fa84 cbaf7412 00000000 ntdll!RtlEnterCriticalSection+0x150
    0178fc9c 7528336a 0224e008 0178fce8 77209f72 dbghelp!GetPdbThreadProc+0x52
    0178fca8 77209f72 0224e008 6372d94d 00000000 kernel32!BaseThreadInitThunk+0xe
    0178fce8 77209f45 6e038ac2 0224e008 00000000 ntdll!__RtlUserThreadStart+0x70
    0178fd00 00000000 6e038ac2 0224e008 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       3  Id: 2848.17f4 Suspend: 1 Teb: fffac000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    032afdd8 77223392 00000158 032afe8c 6120da9d ntdll!NtWaitForWorkViaWorkerFactory+0x12
    032aff38 7528336a 009c1218 032aff84 77209f72 ntdll!TppWorkerThread+0x216
    032aff44 77209f72 009c1218 6120da21 00000000 kernel32!BaseThreadInitThunk+0xe
    032aff84 77209f45 77223e85 009c1218 00000000 ntdll!__RtlUserThreadStart+0x70
    032aff9c 00000000 77223e85 009c1218 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       4  Id: 2848.19cc Suspend: 1 Teb: fffaf000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    01b5f830 77223392 00000150 01b5f8e4 63bfdc35 ntdll!NtWaitForWorkViaWorkerFactory+0x12
    01b5f990 7528336a 009c1218 01b5f9dc 77209f72 ntdll!TppWorkerThread+0x216
    01b5f99c 77209f72 009c1218 63bfdc79 00000000 kernel32!BaseThreadInitThunk+0xe
    01b5f9dc 77209f45 77223e85 009c1218 00000000 ntdll!__RtlUserThreadStart+0x70
    01b5f9f4 00000000 77223e85 009c1218 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       5  Id: 2848.2cf8 Suspend: 1 Teb: fffd7000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    01fafd34 77223392 00000158 01fafde8 63f0db31 ntdll!NtWaitForWorkViaWorkerFactory+0x12
    01fafe94 7528336a 009c1218 01fafee0 77209f72 ntdll!TppWorkerThread+0x216
    01fafea0 77209f72 009c1218 63f0db45 00000000 kernel32!BaseThreadInitThunk+0xe
    01fafee0 77209f45 77223e85 009c1218 00000000 ntdll!__RtlUserThreadStart+0x70
    01fafef8 00000000 77223e85 009c1218 00000000 ntdll!_RtlUserThreadStart+0x1b
    
    0:000> !handle 00000344 F
    Handle 00000344
      Type         	Thread
      Attributes   	0
      GrantedAccess	0x1fffff:
             Delete,ReadControl,WriteDac,WriteOwner,Synch
             Terminate,Suspend,Alert,GetContext,SetContext,SetInfo,QueryInfo,SetToken,Impersonate,DirectImpersonate
      HandleCount  	5
      PointerCount 	8
      Name         	<none>
      Object specific information
        Thread Id   2848.1c78
        Priority    10
        Base Priority 0
    
    0:000> !handle 00000348 F
    Handle 00000348
      Type         	Event
      Attributes   	0
      GrantedAccess	0x100003:
             Synch
             QueryState,ModifyState
      HandleCount  	2
      PointerCount 	4
      Name         	<none>
      Object specific information
        Event Type Auto Reset
        Event is Waiting
    
    0:000> !cs 6e11fa84 
    -----------------------------------------
    Critical section   = 0x6e11fa84 (dbghelp!g+0xB4)
    DebugInfo          = 0x0072c9b0
    LOCKED
    LockCount          = 0x1
    WaiterWoken        = No
    OwningThread       = 0x00002b94
    RecursionCount     = 0x1
    LockSemaphore      = 0x348
    SpinCount          = 0x00000000

    So, thread 2b94 owns critical section 6e11fa84 and is waiting for thread handle 00000344 (thread ID 1c78). Thread 1c78, in turn, is waiting to acquire access to critical section 6e11fa84.

    Here is the output from !analyze –v –hang

    0:000> !analyze -v -hang
    *******************************************************************************
    *                                                                             *
    *                        Exception Analysis                                   *
    *                                                                             *
    *******************************************************************************
    
    GetUrlPageData2 (WinHttp) failed: 12029.
    
    FAULTING_IP: 
    +0
    00000000 ??              ???
    
    EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
    ExceptionAddress: 00000000
       ExceptionCode: 80000007 (Wake debugger)
      ExceptionFlags: 00000000
    NumberParameters: 0
    
    CONTEXT:  00000000 -- (.cxr 0x0;r)
    eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000344 edi=000da0d8
    eip=771ef8d1 esp=000da090 ebp=000da0fc iopl=0         nv up ei pl zr na pe nc
    cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
    ntdll!ZwWaitForSingleObject+0x15:
    771ef8d1 83c404          add     esp,4
    
    BUGCHECK_STR:  HANG
    
    PROCESS_NAME:  cdb.exe
    
    OVERLAPPED_MODULE: Address regions for 'exts' and 'symsrv.dll' overlap
    
    ERROR_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
    
    EXCEPTION_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
    
    NTGLOBALFLAG:  80400
    
    APPLICATION_VERIFIER_FLAGS:  0
    
    APP:  cdb.exe
    
    ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) x86fre
    
    DERIVED_WAIT_CHAIN:  
    
    Dl Eid Cid     WaitType
    -- --- ------- --------------------------
    x  0   2848.2b94 Thread Handle          -->
    x  2   2848.1c78 Event                  --^
    
    WAIT_CHAIN_COMMAND:  ~0s;k;;~2s;k;;
    
    BLOCKING_THREAD:  00001c78
    
    DEFAULT_BUCKET_ID:  APPLICATION_HANG_DEADLOCK_BlockedOn_EventHandle
    
    PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_DEADLOCK_BlockedOn_EventHandle
    
    LAST_CONTROL_TRANSFER:  from 77208e44 to 771ef8d1
    
    FAULTING_THREAD:  00000002
    
    STACK_TEXT:  
    0178fbc8 77208e44 00000348 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
    0178fc2c 77208d28 00000000 00000000 0224e008 ntdll!RtlpWaitOnCriticalSection+0x13e
    0178fc54 6e038b14 6e11fa84 cbaf7412 00000000 ntdll!RtlEnterCriticalSection+0x150
    0178fc9c 7528336a 0224e008 0178fce8 77209f72 dbghelp!GetPdbThreadProc+0x52
    0178fca8 77209f72 0224e008 6372d94d 00000000 kernel32!BaseThreadInitThunk+0xe
    0178fce8 77209f45 6e038ac2 0224e008 00000000 ntdll!__RtlUserThreadStart+0x70
    0178fd00 00000000 6e038ac2 0224e008 00000000 ntdll!_RtlUserThreadStart+0x1b
    
    
    FOLLOWUP_IP: 
    dbghelp!GetPdbThreadProc+52
    6e038b14 66ff052e1d126e  inc     word ptr [dbghelp!g+0x235e (6e121d2e)]
    
    SYMBOL_STACK_INDEX:  3
    
    SYMBOL_NAME:  dbghelp!GetPdbThreadProc+52
    
    FOLLOWUP_NAME:  MachineOwner
    
    MODULE_NAME: dbghelp
    
    IMAGE_NAME:  dbghelp.dll
    
    DEBUG_FLR_IMAGE_TIMESTAMP:  52158c87
    
    STACK_COMMAND:  ~2s ; kb
    
    BUCKET_ID:  HANG_dbghelp!GetPdbThreadProc+52
    
    FAILURE_BUCKET_ID:  APPLICATION_HANG_DEADLOCK_BlockedOn_EventHandle_cfffffff_dbghelp.dll!GetPdbThreadProc
    
    ANALYSIS_SOURCE:  UM
    
    FAILURE_ID_HASH_STRING:  um:application_hang_deadlock_blockedon_eventhandle_cfffffff_dbghelp.dll!getpdbthreadproc
    
    FAILURE_ID_HASH:  {00cbcbb4-b83a-2c1c-f28e-6b4f93151309}
    
    Followup: MachineOwner
    ---------
    

    Version info:

    0:000> lmvm dbghelp
    start    end        module name
    6e010000 6e159000   dbghelp    (pdb symbols)          s:\symcache\dbghelp.pdb\1F1872F9B5FE4BDFA1C97A96BDD8076F1\dbghelp.pdb
        Loaded symbol image file: dbghelp.dll
        Image path: C:\DebugTools_x86_6.3.9600.16384\x86\dbghelp.dll
        Image name: dbghelp.dll
        Timestamp:        Wed Aug 21 22:59:03 2013 (52158C87)
        CheckSum:         00133556
        ImageSize:        00149000
        File version:     6.3.9600.16384
        Product version:  6.3.9600.16384
        File flags:       0 (Mask 3F)
        File OS:          40004 NT Win32
        File type:        2.0 Dll
        File date:        00000000.00000000
        Translations:     0409.04b0
        CompanyName:      Microsoft Corporation
        ProductName:      Microsoft® Windows® Operating System
        InternalName:     DBGHELP.DLL
        OriginalFilename: DBGHELP.DLL
        ProductVersion:   6.3.9600.16384
        FileVersion:      6.3.9600.16384 (debuggers(dbg).130821-1623)
        FileDescription:  Windows Image Helper
        LegalCopyright:   © Microsoft Corporation. All rights reserved.
    

    I’m sharing in hopes that Microsoft will fix this problem. I think I’ll be rolling back to older Debugging Tools.

    Thank you,
    Olegas


    Olegas

    Friday, February 28, 2014 7:30 PM

All replies

  • Here is another note: I left the hung process running to see what happens later and went about my business. About 2 hours later, the CDB process in question crashed due to heap corruption.

    Thread 1c78 is nowhere to be found. I’m guessing it encountered EXCEPTION_POSSIBLE_DEADLOCK.
    Heap corruption condition was reported from a new thread with ID 3008.
    Here are the full thread stacks for the crash:

    0:002> ~*kb
    
       0  Id: 2848.2b94 Suspend: 0 Teb: fffdd000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    000d9ad8 74e0149d 00000234 00000000 000d9b20 ntdll!ZwWaitForSingleObject+0x15
    000d9b44 75281194 00000234 000001f4 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
    000d9b5c 75281148 00000234 000001f4 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
    000d9b70 6e039522 00000234 000001f4 cada222a kernel32!WaitForSingleObject+0x12
    000daaa4 6e03aed6 070040b4 070040a4 00000000 dbghelp!DiaLocatePdbMultiThread+0x8d3
    000daf10 6e056f4c 00000001 0079e540 00000000 dbghelp!diaGetPdb+0x1be
    000db140 6e055bdd cada3d6e 00000000 0079e540 dbghelp!GetDebugData+0x230
    000db5e0 6e055885 007328b8 0079e540 000db76c dbghelp!modload+0x285
    000db60c 6e058e08 00000000 cada3ec6 00000e9c dbghelp!LoadSymbols+0x355
    000db648 6e0416f6 737755aa 00000000 cada3e1e dbghelp!GetModFromAddr+0x38
    000db690 6e04185a 737755aa 00000000 cada3e5e dbghelp!ldiaGetFrameData+0x29
    000db6d0 6e04b119 737755aa 00000000 000db930 dbghelp!diaGetFrameData+0x32
    000db6e8 6e0783d0 73770000 00000000 000055aa dbghelp!DbhStackServices::GetUnwindInfoFromSymbols+0x29
    000db714 6e07ce40 73770000 00000000 000055aa dbghelp!DbsStackServices::UnwindInfoHolder::GetFromSymbols+0x26
    000db798 6e079e60 00000000 000dd130 000db958 dbghelp!DbsX86StackUnwinder::ApplyUnwindInfo+0x73
    000db7c8 6e078e13 000dc620 000dd130 000db930 dbghelp!DbsX86StackUnwinder::Unwind+0x13e
    000db7dc 6e062ed0 000dd130 0000000c 009c7c88 dbghelp!DbsStackUnwinder::DbhUnwind+0xcd
    000db8e0 6e06348c 000dd130 000dc620 0003bac0 dbghelp!PickX86Walk+0x13f
    000dc5e0 5fd1d2e4 0000014c 00000e9c 00000001 dbghelp!StackWalkEx+0x38e
    000dd250 5fd1da02 000395c0 03500020 05a0106c dbgeng!TargetInfo::GetTargetStackFrames+0x44d
    000dd2c8 5fd3f2d4 0000000c 0000101d cada5b82 dbgeng!DoStackTrace+0x123
    000dd348 5fcd9ef0 00000000 00000000 01302f88 dbgeng!ParseThreadCmds+0x2d3
    000dd3b8 5fcda955 cada5cd2 04f6900a 00000001 dbgeng!ProcessCommands+0x988
    000dd418 5fc4cdd4 00000003 cada5ce2 000373f0 dbgeng!ProcessCommandsAndCatch+0x91
    000dd880 5fcdaae5 00000006 00000003 cada5002 dbgeng!Execute+0x226
    000dd8c8 5fc77235 000373f0 5fde8180 7562ad52 dbgeng!ProcessCurBraceBlock+0x67
    000dd8dc 5fc5f810 000dd968 5fc7b93d 5fde8180 dbgeng!DotIf+0xb0
    000dd8f8 5fcd9dd8 00000000 00000000 00308aa8 dbgeng!DbgSqmSession::AppendCmdNameStr+0x1a
    000dd968 5fcda955 cada5102 0577a02a 00000001 dbgeng!ProcessCommands+0x870
    000dd9c8 5fc4cdd4 00000003 cada5112 000373f0 dbgeng!ProcessCommandsAndCatch+0x91
    000dde34 5fcdaae5 00000006 00000003 cada56b6 dbgeng!Execute+0x226
    000dde7c 5fc71f90 000ddf08 5fc7b93d 5fde7b2c dbgeng!ProcessCurBraceBlock+0x67
    000dde84 5fc7b93d 5fde7b2c 000373f0 000373f0 dbgeng!DotBlock+0x10
    000dde98 5fcd9dd8 00000000 00000000 00308680 dbgeng!DotCommand+0x36
    000ddf08 5fcda955 cada57a2 0134d27a 00000001 dbgeng!ProcessCommands+0x870
    000ddf68 5fc4cdd4 00000000 cada57b2 00000002 dbgeng!ProcessCommandsAndCatch+0x91
    000de3d0 5fcd8ac4 00000002 00000000 cada6cf2 dbgeng!Execute+0x226
    000de438 5fcd9c8d 00000000 00000000 00000000 dbgeng!ParseDollar+0x20b
    000de4a8 5fcda955 cada6dc2 000de556 00000001 dbgeng!ProcessCommands+0x725
    000de508 5fc4cdd4 00000000 cada6dd2 00000000 dbgeng!ProcessCommandsAndCatch+0x91
    000de974 5fc4cfc5 00000002 00000000 cada6142 dbgeng!Execute+0x226
    000de9c0 5fc4cf18 000373f8 00000001 000dea00 dbgeng!DebugClient::ExecuteWide+0x8d
    000dec10 00188c92 000373f8 00000001 000dec50 dbgeng!DebugClient::Execute+0x74
    000dfc5c 0018ac98 00000000 00000001 00000000 cdb!MainLoop+0x407
    000dfe94 0018c5fe 00000005 002bedc8 000321f0 cdb!main+0x2ad
    000dfed4 7528336a fffde000 000dff20 77209f72 cdb!SetLastError+0x206
    000dfee0 77209f72 fffde000 6207da85 00000000 kernel32!BaseThreadInitThunk+0xe
    000dff20 77209f45 0018c694 fffde000 00000000 ntdll!__RtlUserThreadStart+0x70
    000dff38 00000000 0018c694 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       1  Id: 2848.2124 Suspend: 0 Teb: fffda000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    011ff9d4 77222f91 00000003 009c2068 00000001 ntdll!NtWaitForMultipleObjects+0x15
    011ffb68 7528336a 00000000 011ffbb4 77209f72 ntdll!TppWaiterpThread+0x33d
    011ffb74 77209f72 009c2038 6315de11 00000000 kernel32!BaseThreadInitThunk+0xe
    011ffbb4 77209f45 77222e65 009c2038 00000000 ntdll!__RtlUserThreadStart+0x70
    011ffbcc 00000000 77222e65 009c2038 00000000 ntdll!_RtlUserThreadStart+0x1b
    
    #  2  Id: 2848.3008 Suspend: 0 Teb: fffaf000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    01c3f4c8 7729f659 c0000374 772d4270 01c3f50c ntdll!RtlReportCriticalFailure+0x57
    01c3f4d8 7729f739 00000002 63c9d0a9 00000000 ntdll!RtlpReportHeapFailure+0x21
    01c3f50c 7724e045 00000008 00030000 0224e000 ntdll!RtlpLogHeapFailure+0xa1
    01c3f53c 756298cd 00030000 00000000 0224e008 ntdll!RtlFreeHeap+0x64
    01c3f588 6e038684 0224e008 00000000 02254ee8 msvcrt!free+0xcd
    01c3f598 6e038bf5 6e08fec2 6e08dc28 01c3fb08 dbghelp!DeallocateValidationArray+0x39
    01c3f59c 6e08fec2 6e08dc28 01c3fb08 6e08dc38 dbghelp!GetPdbThreadProc+0x133
    01c3f5a0 6e08dc28 01c3fb08 6e08dc38 cb147d26 dbghelp!_NLG_Return2
    01c3f5cc 6e08dcf4 6e11e008 01c3fb08 00000000 dbghelp!_local_unwind4+0x80
    01c3f5e0 6e08ddec 01c3fb18 6e11e008 00000000 dbghelp!_EH4_LocalUnwind+0x10
    01c3f614 6e08e1a2 6e11e008 6e08c3f7 01c3f720 dbghelp!_except_handler4_common+0xec
    01c3f634 7722b499 01c3f720 01c3fb08 01c3f770 dbghelp!_except_handler4+0x20
    01c3f658 7722b46b 01c3f720 01c3fb08 01c3f770 ntdll!ExecuteHandler2+0x26
    01c3f67c 7722b40e 01c3f720 01c3fb08 01c3f770 ntdll!ExecuteHandler+0x24
    01c3f708 771e0133 00c3f720 01c3f770 01c3f720 ntdll!RtlDispatchException+0x127
    01c3f708 77208e19 00c3f720 01c3f770 01c3f720 ntdll!KiUserExceptionDispatcher+0xf
    01c3faa8 77208d28 00000000 00000000 02254ee8 ntdll!RtlpWaitOnCriticalSection+0xbd
    01c3fad0 6e038b5a 02254ed0 cb147396 00000000 ntdll!RtlEnterCriticalSection+0x150
    01c3fb18 7528336a 02254ee8 01c3fb64 77209f72 dbghelp!GetPdbThreadProc+0x98
    01c3fb24 77209f72 02254ee8 63c9dec1 00000000 kernel32!BaseThreadInitThunk+0xe
    01c3fb64 77209f45 6e038ac2 02254ee8 00000000 ntdll!__RtlUserThreadStart+0x70
    01c3fb7c 00000000 6e038ac2 02254ee8 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       3  Id: 2848.3284 Suspend: 0 Teb: fffac000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0379fb50 77223392 00000150 0379fc04 6173d915 ntdll!NtWaitForWorkViaWorkerFactory+0x12
    0379fcb0 7528336a 009c1218 0379fcfc 77209f72 ntdll!TppWorkerThread+0x216
    0379fcbc 77209f72 009c1218 6173d959 00000000 kernel32!BaseThreadInitThunk+0xe
    0379fcfc 77209f45 77223e85 009c1218 00000000 ntdll!__RtlUserThreadStart+0x70
    0379fd14 00000000 77223e85 009c1218 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       4  Id: 2848.33d8 Suspend: 0 Teb: fffa6000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0368fc1c 77223392 00000150 0368fcd0 6162d8d9 ntdll!NtWaitForWorkViaWorkerFactory+0x12
    0368fd7c 7528336a 009c1218 0368fdc8 77209f72 ntdll!TppWorkerThread+0x216
    0368fd88 77209f72 009c1218 6162d86d 00000000 kernel32!BaseThreadInitThunk+0xe
    0368fdc8 77209f45 77223e85 009c1218 00000000 ntdll!__RtlUserThreadStart+0x70
    0368fde0 00000000 77223e85 009c1218 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       5  Id: 2848.1788 Suspend: 0 Teb: fffa9000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0179fde8 77208e44 00000348 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
    0179fe4c 77208d28 00000000 00000000 0521ba58 ntdll!RtlpWaitOnCriticalSection+0x13e
    0179fe74 6e038b14 6e11fa84 cbae7632 00000000 ntdll!RtlEnterCriticalSection+0x150
    0179febc 7528336a 0521ba58 0179ff08 77209f72 dbghelp!GetPdbThreadProc+0x52
    0179fec8 77209f72 0521ba58 6373daad 00000000 kernel32!BaseThreadInitThunk+0xe
    0179ff08 77209f45 6e038ac2 0521ba58 00000000 ntdll!__RtlUserThreadStart+0x70
    0179ff20 00000000 6e038ac2 0521ba58 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       6  Id: 2848.1b48 Suspend: 0 Teb: fffd7000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0343fbd8 77223392 00000150 0343fc8c 6149d89d ntdll!NtWaitForWorkViaWorkerFactory+0x12
    0343fd38 7528336a 009c1218 0343fd84 77209f72 ntdll!TppWorkerThread+0x216
    0343fd44 77209f72 009c1218 6149d821 00000000 kernel32!BaseThreadInitThunk+0xe
    0343fd84 77209f45 77223e85 009c1218 00000000 ntdll!__RtlUserThreadStart+0x70
    0343fd9c 00000000 77223e85 009c1218 00000000 ntdll!_RtlUserThreadStart+0x1b

    Since the deadlock appears to be the "first domino" in this chain of events, I’m not going to enable pageheap just yet, but I wanted to share the latest.

    Thank you


    Olegas

    Friday, February 28, 2014 9:07 PM
  • I must be lucky because I keep running into the same bug. Today, CDB hung again with the same deadlock after executing the commands below:

    .lines -e
    ~0s
    kb

    Debugger hung halfway through displaying a stack.
    CDB thread stacks:

    0:006> ~*kb
    
       0  Id: 2824.207c Suspend: 1 Teb: fffdd000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    000aaa70 773c149d 000002c0 00000000 000aaab8 ntdll!ZwWaitForSingleObject+0x15
    000aaadc 76c01194 000002c0 000001f4 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
    000aaaf4 76c01148 000002c0 000001f4 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
    000aab08 5afd9522 000002c0 000001f4 ffd56666 kernel32!WaitForSingleObject+0x12
    000aba3c 5afdaed6 02cdd364 02cdd354 00000000 dbghelp!DiaLocatePdbMultiThread+0x8d3
    000abea8 5aff6f4c 00000001 00407ea8 00000000 dbghelp!diaGetPdb+0x1be
    000ac0d8 5aff5bdd ffd51922 00000000 00407ea8 dbghelp!GetDebugData+0x230
    000ac578 5aff5885 006e3ad0 00407ea8 000ac704 dbghelp!modload+0x285
    000ac5a4 5aff8e08 00000000 ffd519ba 0000009c dbghelp!LoadSymbols+0x355
    000ac5e0 5afe16f6 704a0b63 00000000 ffd51a72 dbghelp!GetModFromAddr+0x38
    000ac628 5afe185a 704a0b63 00000000 ffd51a32 dbghelp!ldiaGetFrameData+0x29
    000ac668 5afeb119 704a0b63 00000000 000ac8c8 dbghelp!diaGetFrameData+0x32
    000ac680 5b0183d0 70490000 00000000 00010b63 dbghelp!DbhStackServices::GetUnwindInfoFromSymbols+0x29
    000ac6ac 5b01ce40 70490000 00000000 00010b63 dbghelp!DbsStackServices::UnwindInfoHolder::GetFromSymbols+0x26
    000ac730 5b019e60 00000000 000ae0c8 000ac8f0 dbghelp!DbsX86StackUnwinder::ApplyUnwindInfo+0x73
    000ac760 5b018e13 000ad5b8 000ae0c8 000ac8c8 dbghelp!DbsX86StackUnwinder::Unwind+0x13e
    000ac774 5b002ed0 000ae0c8 0000000c 007ad948 dbghelp!DbsStackUnwinder::DbhUnwind+0xcd
    000ac878 5b00348c 000ae0c8 000ad5b8 0098ba78 dbghelp!PickX86Walk+0x13f
    000ad578 0ff5d2e4 0000014c 0000009c 00000098 dbghelp!StackWalkEx+0x38e
    000ae1e8 0ff5da02 001fdb38 00550020 002c2c94 dbgeng!TargetInfo::GetTargetStackFrames+0x44d
    000ae260 0ff18cdd 0000000c 0000101d ffd53efc dbgeng!DoStackTrace+0x123
    000ae2d8 0ff19e84 00000000 00000000 00000000 dbgeng!WrapParseStackCmd+0x107
    000ae348 0ff1a955 ffd53f8c 000ae3f6 00000001 dbgeng!ProcessCommands+0x91c
    000ae3a8 0fe8cdd4 00000000 ffd53f9c 00000000 dbgeng!ProcessCommandsAndCatch+0x91
    000ae814 0fe8cfc5 00000002 00000000 ffd5340c dbgeng!Execute+0x226
    000ae860 0fe8cf18 009872f8 00000001 000ae8a0 dbgeng!DebugClient::ExecuteWide+0x8d
    000aeab0 00358c92 009872f8 00000001 000aeaf0 dbgeng!DebugClient::Execute+0x74
    000afafc 0035ac98 00000000 00000001 00000000 cdb!MainLoop+0x407
    000afd34 0035c5fe 00000004 001eed40 00982238 cdb!main+0x2ad
    000afd74 76c0336a fffde000 000afdc0 77a59f72 cdb!SetLastError+0x206
    000afd80 77a59f72 fffde000 62344c2f 00000000 kernel32!BaseThreadInitThunk+0xe
    000afdc0 77a59f45 0035c694 fffde000 00000000 ntdll!__RtlUserThreadStart+0x70
    000afdd8 00000000 0035c694 fffde000 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       1  Id: 2824.235c Suspend: 1 Teb: fffd7000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0097f884 77a72f91 00000004 02cde000 00000001 ntdll!NtWaitForMultipleObjects+0x15
    0097fa18 76c0336a 00000000 0097fa64 77a59f72 ntdll!TppWaiterpThread+0x33d
    0097fa24 77a59f72 02cddfd0 62a94b8b 00000000 kernel32!BaseThreadInitThunk+0xe
    0097fa64 77a59f45 77a72e65 02cddfd0 00000000 ntdll!__RtlUserThreadStart+0x70
    0097fa7c 00000000 77a72e65 02cddfd0 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       2  Id: 2824.2bf8 Suspend: 1 Teb: fffac000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0375fc84 773c149d 00000358 00000001 00000000 ntdll!ZwWaitForSingleObject+0x15
    0375fcf0 76c01194 00000358 ffffffff 00000001 KERNELBASE!WaitForSingleObjectEx+0x98
    0375fd08 6f4e33b7 00000358 ffffffff 00000001 kernel32!WaitForSingleObjectExImplementation+0x75
    0375fd6c 76c0336a 00000000 0375fdb8 77a59f72 rasman!RasmanServiceMonitorThread+0xe7
    0375fd78 77a59f72 00000000 614b4c57 00000000 kernel32!BaseThreadInitThunk+0xe
    0375fdb8 77a59f45 6f4e32fb 00000000 00000000 ntdll!__RtlUserThreadStart+0x70
    0375fdd0 00000000 6f4e32fb 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       3  Id: 2824.1cb0 Suspend: 1 Teb: fffa3000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    037af890 6f57635c 00000454 037af8c4 037af8b8 ntdll!ZwRemoveIoCompletion+0x15
    037af8bc 76c0336a 6f5764b3 037af908 77a59f72 mswsock!SockAsyncThread+0x83
    037af8c8 77a59f72 02d261f0 614448e7 00000000 kernel32!BaseThreadInitThunk+0xe
    037af908 77a59f45 6f5762ee 02d261f0 00000000 ntdll!__RtlUserThreadStart+0x70
    037af920 00000000 6f5762ee 02d261f0 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       4  Id: 2824.320 Suspend: 1 Teb: fffda000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    032afd10 77a58e44 0000045c 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
    032afd74 77a58d28 00000000 00000000 041a4000 ntdll!RtlpWaitOnCriticalSection+0x13e
    032afd9c 5afd8b14 5b0bfa84 fcf521be 00000000 ntdll!RtlEnterCriticalSection+0x150
    032afde4 76c0336a 041a4000 032afe30 77a59f72 dbghelp!GetPdbThreadProc+0x52
    032afdf0 77a59f72 041a4000 61144fdf 00000000 kernel32!BaseThreadInitThunk+0xe
    032afe30 77a59f45 5afd8ac2 041a4000 00000000 ntdll!__RtlUserThreadStart+0x70
    032afe48 00000000 5afd8ac2 041a4000 00000000 ntdll!_RtlUserThreadStart+0x1b
    
       5  Id: 2824.2538 Suspend: 1 Teb: fffaf000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    0240f948 77a73392 00000264 0240f9fc 607e4b47 ntdll!NtWaitForWorkViaWorkerFactory+0x12
    0240faa8 76c0336a 00466088 0240faf4 77a59f72 ntdll!TppWorkerThread+0x216
    0240fab4 77a59f72 00466088 607e4b1b 00000000 kernel32!BaseThreadInitThunk+0xe
    0240faf4 77a59f45 77a73e85 00466088 00000000 ntdll!__RtlUserThreadStart+0x70
    0240fb0c 00000000 77a73e85 00466088 00000000 ntdll!_RtlUserThreadStart+0x1b
    
    #  6  Id: 2824.2c50 Suspend: 1 Teb: fffa9000 Unfrozen
    ChildEBP RetAddr  Args to Child              
    036afbf4 77abf926 61544dcb 00000000 00000000 ntdll!DbgBreakPoint
    036afc24 76c0336a 00000000 036afc70 77a59f72 ntdll!DbgUiRemoteBreakin+0x3c
    036afc30 77a59f72 00000000 61544d9f 00000000 kernel32!BaseThreadInitThunk+0xe
    036afc70 77a59f45 77abf8ea 00000000 00000000 ntdll!__RtlUserThreadStart+0x70
    036afc88 00000000 77abf8ea 00000000 00000000 ntdll!_RtlUserThreadStart+0x1b

    Event handles and critical section:

    0:006> !handle 000002c0 F
    Handle 000002c0
      Type         	Thread
      Attributes   	0
      GrantedAccess	0x1fffff:
             Delete,ReadControl,WriteDac,WriteOwner,Synch
             Terminate,Suspend,Alert,GetContext,SetContext,SetInfo,QueryInfo,SetToken,Impersonate,DirectImpersonate
      HandleCount  	5
      PointerCount 	8
      Name         	<none>
      Object specific information
        Thread Id   2824.320
        Priority    15
        Base Priority 0
    
    0:006> !handle 0000045c F
    Handle 0000045c
      Type         	Event
      Attributes   	0
      GrantedAccess	0x100003:
             Synch
             QueryState,ModifyState
      HandleCount  	2
      PointerCount 	4
      Name         	<none>
      Object specific information
        Event Type Auto Reset
        Event is Waiting
    
    0:006> !cs 5b0bfa84 
    -----------------------------------------
    Critical section   = 0x5b0bfa84 (dbghelp!g+0xB4)
    DebugInfo          = 0x006dc978
    LOCKED
    LockCount          = 0x1
    WaiterWoken        = No
    OwningThread       = 0x0000207c
    RecursionCount     = 0x1
    LockSemaphore      = 0x45C
    SpinCount          = 0x00000000

    I sure hope Microsoft will fix it soon.


    Olegas

    Friday, March 7, 2014 7:52 PM