none
how to diagnostic a svchost hang RRS feed

  • Question

  • Hi All,

    there was issue issue we found on our system Win 2008 Server R2.

     Description: 

           in our code, there was a stack overrun issue will cause application crash in some case, and 100% reproduced.

           but, one day we found one client doesn't crash but behave not as expected.

    0:080:x86> kv fff
      Memory  ChildEBP RetAddr  Args to Child              
              15b6f870 15268882 15b6f970 15f02d68 fffffffe msvcr90!memcpy+0x5a
    WARNING: Stack unwind information not available. Following frames may be wrong.
           88 15b6f8f8 15268d79 04d07470 15b6f970 00000008 ExcepApp+0x78882
           34 15b6f92c 1526cb07 04d07470 15b6f970 00000008 ExcepApp+0x78d79
          270 15b6fb9c 00000000 1636f520 5c3bd1f5 8000003a ExcepApp+0x7cb07
    
    0:080:x86> r
    
    eax=15f02d66 ebx=04cc2560 ecx=3ffffe5b edx=00000002 esi=15f033f8
    edi=15b70000
    
    eip=74cbae7a esp=15b6f868 ebp=15b6f870 iopl=0      
      nv up ei pl nz na po nc
    
    cs=0023  ss=002b  ds=002b  es=002b  fs=0053
     gs=002b             efl=00010202
    
    msvcr90!memcpy+0x5a:
    
    74cbae7a f3a5            rep movs
    dword ptr es:[edi],dword ptr [esi]
    
    0:080:x86> dd edi L4
    
    15b70000  ???????? ???????? ???????? ????????
    
    0:080:x86> dd edi-4 L4
    
    15b6fffc  00000000 ???????? ???????? ????????

     then we captured a kernel dump with LiveKD and found:

    0: kd> !thread fffffa800694fb60 THREAD fffffa800694fb60  Cid 0a0c.14dc  Teb: 000000007ee81000 Win32Thread: 0000000000000000 WAIT: (Suspended) KernelMode Non-Alertable SuspendCount 1 FreezeCount 1     fffffa800694fe38  Semaphore Limit 0x2 Waiting for reply to ALPC Message fffff8a00d7492c0 : queued at port fffffa8007e79090 : owned by process fffffa8007b82b30 Not impersonating DeviceMap                 fffff8a000006110 Owning Process            fffffa8005e05b30       Image:         RSLinxNG.exe Attached Process          N/A            Image:         N/A Wait Start TickCount      16200644       Ticks: 80631 (0:00:20:59.859) Context Switch Count      273333              UserTime                  00:00:56.593 KernelTime                00:00:04.593 Win32 Start Address 0x0000000074ca345e Stack Init fffff8800540ddb0 Current fffff8800540c200 Base fffff8800540e000 Limit fffff88005408000 Call 0 Priority 10 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5 Child-SP          RetAddr           : Args to Child                                                           : Call Site fffff880`0540c240 fffff800`01688f92 : fffffa80`0694fb60 fffffa80`0694fb60 00000000`00000000 fffffa80`0000000a : nt!KiSwapContext+0x7a fffff880`0540c380 fffff800`0168b7af : 00000000`00000000 00000002`00000000 fffffa80`00000000 fffff800`016864da : nt!KiCommitThreadWait+0x1d2

    fffff880`0540c410 fffff800`01676734 : 00000000`00000000 00000000`00000005 00000000`00000000 00000000`00000000 : nt!KeWaitForSingleObject+0x19f

    fffff880`0540c4b0 fffff800`016773b1 : fffffa80`0694fb60 fffffa80`0694fbb0 00000000`001fffff fffffa80`00000000 : nt!KiSuspendThread+0x54

    fffff880`0540c4f0 fffff800`0168919d : fffffa80`0694fb60 00000000`00000000 fffff800`016766e0 00000000`00000000 : nt!KiDeliverApc+0x201

    fffff880`0540c570 fffff800`0168b7af : 00000000`00000000 fffffa80`0694fb60 fffffa80`00000000 fffff800`0168a03a : nt!KiCommitThreadWait+0x3dd

    fffff880`0540c600 fffff800`016a5d4f : 00000000`00000000 00000000`00000011 00000000`00000001 00000000`00000000 : nt!KeWaitForSingleObject+0x19f

    fffff880`0540c6a0 fffff800`01993506 : fffffa80`06b938c8 fffffa80`0694ff20 fffff8a0`0c91f901 00000000`00000000 : nt!AlpcpSignalAndWait+0x8f

    fffff880`0540c750 fffff800`01992c00 : 00000000`00000000 fffff880`0540ccf0 00000000`00000000 fffffa80`03da3000 : nt!AlpcpReceiveSynchronousReply+0x46

    fffff880`0540c7b0 fffff800`019909fb : fffffa80`08d3bbb0 fffff800`00120000 fffff880`0540ccf0 fffff880`0540cc00 : nt!AlpcpProcessSynchronousRequest+0x33d

    fffff880`0540c8f0 fffff800`01682f93 : fffffa80`0694fb60 fffff880`0540ca90 00000000`00000000 00000000`00000000 : nt!NtAlpcSendWaitReceivePort+0x1ab

    fffff880`0540c9a0 fffff800`0167f530 : fffff800`01a717b6 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0540ca10)

    fffff880`0540cba8 fffff800`01a717b6 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiServiceLinkage

    fffff880`0540cbb0 fffff800`019c5981 : 00000000`80010001 fffffa80`04cbc780 00000000`00000000 00000000`00000000 : nt!DbgkpSendErrorMessage+0x266

    fffff880`0540ccd0 fffff800`016cf6bc : fffff880`0540da00 fffff880`0540d340 fffff880`0540dc20 00000000`00000002 : nt! ?? ::NNGAKEGL::`string'+0x35d28

    fffff880`0540ce10 fffff800`016d19ac : fffff880`0540da00 fffff880`0540da00 fffff880`0540dc20 fffff880`0540d4e0 : nt! ?? ::FNODOBFM::`string'+0x49961

    fffff880`0540d4b0 fffff800`016859fb : fffff880`0540da00 fffffa80`0694fb60 00000000`754f867b 00000000`00000000 : nt!KiRaiseException+0x1b4

    fffff880`0540dae0 fffff800`01682f93 : 00000000`00000001 fffffa80`0694fb60 00000000`00369e01 00000000`7ee81000 : nt!NtRaiseException+0x7b

    fffff880`0540dc20 00000000`754fc9f1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0540dc20)

    00000000`14c0e120 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : 0x754fc9f1


    0: kd> !alpc /m fffff8a00d7492c0
    
    Message @ fffff8a00d7492c0
      MessageID             : 0x0584 (1412)
      CallbackID            : 0x240B6D1 (37795537)
      SequenceNumber        : 0x00000001 (1)
      Type                  : LPC_EXCEPTION
      DataLength            : 0x00E8 (232)
      TotalLength           : 0x0110 (272)
      Canceled              : No
      Release               : No
      ReplyWaitReply        : No
      Continuation          : Yes
      OwnerPort             : fffffa8008d3bbb0 [ALPC_CLIENT_COMMUNICATION_PORT]
      WaitingThread         : fffffa800694fb60
      QueueType             : ALPC_MSGQUEUE_PENDING
      QueuePort             : fffffa8007e79090 [ALPC_CONNECTION_PORT]
      QueuePortOwnerProcess : fffffa8007b82b30 (svchost.exe)
      ServerThread          : fffffa8006b937c0
      QuotaCharged          : No
      CancelQueuePort       : 0000000000000000
      CancelSequencePort    : 0000000000000000
      CancelSequenceNumber  : 0x00000000 (0)
      ClientContext         : 0000000000000000
      ServerContext         : 0000000000000000
      PortContext           : d8000000000f5c76
      CancelPortContext     : 0000000000000000
      SecurityData          : 0000000000000000
      View                  : 0000000000000000
    


    0: kd> .thread fffffa8006b937c0
    Implicit thread is now fffffa80`06b937c0
    0: kd> !thread fffffa8006b937c0
    THREAD fffffa8006b937c0  Cid 1e8c.220c  Teb: 000007fffffae000 Win32Thread: 0000000000000000 WAIT: (WrLpcReceive) UserMode Non-Alertable
        fffffa8006b93b80  Semaphore Limit 0x1
    Not impersonating
    DeviceMap                 fffff8a000006110
    Owning Process            fffffa8007b82b30       Image:         svchost.exe
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      16312516     
    Context Switch Count      1755             
    UserTime                  00:00:00.000
    KernelTime                00:00:00.000
    Win32 Start Address 0x000007feefe94c88
    Stack Init fffff88007af2db0 Current fffff88007af2750
    Base fffff88007af3000 Limit fffff88007aed000 Call 0
    Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr           : Args to Child                                                           : Call Site
    fffff880`07af2790 fffff800`01688f92 : fffffa80`06b937c0 fffffa80`06b937c0 00000000`00000000 fffff8a0`00000001 : nt!KiSwapContext+0x7a
    fffff880`07af28d0 fffff800`0168b7af : 00000000`00000000 fffff800`016cf8dc 00000000`000000c4 00000000`00000000 : nt!KiCommitThreadWait+0x1d2
    fffff880`07af2960 fffff800`0198fc19 : 00000000`00000000 00000000`00000010 00000000`00000001 00000000`00000000 : nt!KeWaitForSingleObject+0x19f
    fffff880`07af2a00 fffff800`0198f69c : 00000000`00000000 00000000`00000001 00000000`00000000 00000000`00000000 : nt!AlpcpReceiveMessagePort+0x189
    fffff880`07af2a60 fffff800`01990a36 : fffffa80`07e79090 00000000`00000000 00000000`00000000 fffffa80`07e79090 : nt!AlpcpReceiveMessage+0x2d9
    fffff880`07af2b00 fffff800`01682f93 : fffffa80`06b937c0 fffff880`07af2ca0 00000000`00c1f048 fffff880`07af2ca0 : nt!NtAlpcSendWaitReceivePort+0x1e6
    fffff880`07af2bb0 00000000`77bf1b6a : 00364bfb`002cf47b 00364bfc`002cf47c 00364c02`002cf482 00364c1a`002cf49a : nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`07af2c20)
    00000000`00c1f028 00364bfb`002cf47b : 00364bfc`002cf47c 00364c02`002cf482 00364c1a`002cf49a 00364c28`002cf4a8 : ntdll!ZwAlpcSendWaitReceivePort+0xa
    00000000`00c1f030 00364bfc`002cf47c : 00364c02`002cf482 00364c1a`002cf49a 00364c28`002cf4a8 00364c32`002cf4b2 : 0x364bfb`002cf47b
    00000000`00c1f038 00364c02`002cf482 : 00364c1a`002cf49a 00364c28`002cf4a8 00364c32`002cf4b2 00364afc`002cf4c4 : 0x364bfc`002cf47c
    00000000`00c1f040 00364c1a`002cf49a : 00364c28`002cf4a8 00364c32`002cf4b2 00364afc`002cf4c4 00364b06`002cf4ce : 0x364c02`002cf482
    00000000`00c1f048 00364c28`002cf4a8 : 00364c32`002cf4b2 00364afc`002cf4c4 00364b06`002cf4ce 00364b1a`002cf4e2 : 0x364c1a`002cf49a
    00000000`00c1f050 00364c32`002cf4b2 : 00364afc`002cf4c4 00364b06`002cf4ce 00364b1a`002cf4e2 00364b24`002cf4ec : 0x364c28`002cf4a8
    00000000`00c1f058 00364afc`002cf4c4 : 00364b06`002cf4ce 00364b1a`002cf4e2 00364b24`002cf4ec 00364b44`002cf50c : 0x364c32`002cf4b2
    00000000`00c1f060 00364b06`002cf4ce : 00364b1a`002cf4e2 00364b24`002cf4ec 00364b44`002cf50c 00364b4f`002cf517 : 0x364afc`002cf4c4
    00000000`00c1f068 00364b1a`002cf4e2 : 00364b24`002cf4ec 00364b44`002cf50c 00364b4f`002cf517 00364b56`002cf51e : 0x364b06`002cf4ce
    00000000`00c1f070 00364b24`002cf4ec : 00364b44`002cf50c 00364b4f`002cf517 00364b56`002cf51e 00364b5d`002cf525 : 0x364b1a`002cf4e2
    00000000`00c1f078 00364b44`002cf50c : 00364b4f`002cf517 00364b56`002cf51e 00364b5d`002cf525 00364b67`002cf52f : 0x364b24`002cf4ec
    00000000`00c1f080 00364b4f`002cf517 : 00364b56`002cf51e 00364b5d`002cf525 00364b67`002cf52f 00364b6c`002cf534 : 0x364b44`002cf50c
    00000000`00c1f088 00364b56`002cf51e : 00364b5d`002cf525 00364b67`002cf52f 00364b6c`002cf534 00364b72`002cf53a : 0x364b4f`002cf517
    00000000`00c1f090 00364b5d`002cf525 : 00364b67`002cf52f 00364b6c`002cf534 00364b72`002cf53a 0035797c`002cf548 : 0x364b56`002cf51e
    00000000`00c1f098 00364b67`002cf52f : 00364b6c`002cf534 00364b72`002cf53a 0035797c`002cf548 0039a8d0`002cf560 : 0x364b5d`002cf525
    00000000`00c1f0a0 00364b6c`002cf534 : 00364b72`002cf53a 0035797c`002cf548 0039a8d0`002cf560 0039a8d8`002cf568 : 0x364b67`002cf52f
    00000000`00c1f0a8 00364b72`002cf53a : 0035797c`002cf548 0039a8d0`002cf560 0039a8d8`002cf568 0039a8de`002cf56e : 0x364b6c`002cf534
    00000000`00c1f0b0 0035797c`002cf548 : 0039a8d0`002cf560 0039a8d8`002cf568 0039a8de`002cf56e 0039a8e6`002cf576 : 0x364b72`002cf53a
    00000000`00c1f0b8 0039a8d0`002cf560 : 0039a8d8`002cf568 0039a8de`002cf56e 0039a8e6`002cf576 0039a8ec`002cf57c : 0x35797c`002cf548
    00000000`00c1f0c0 0039a8d8`002cf568 : 0039a8de`002cf56e 0039a8e6`002cf576 0039a8ec`002cf57c 0039a8f4`002cf584 : 0x39a8d0`002cf560
    00000000`00c1f0c8 0039a8de`002cf56e : 0039a8e6`002cf576 0039a8ec`002cf57c 0039a8f4`002cf584 0039a8fa`002cf58a : 0x39a8d8`002cf568
    00000000`00c1f0d0 0039a8e6`002cf576 : 0039a8ec`002cf57c 0039a8f4`002cf584 0039a8fa`002cf58a 0039a902`002cf592 : 0x39a8de`002cf56e
    00000000`00c1f0d8 0039a8ec`002cf57c : 0039a8f4`002cf584 0039a8fa`002cf58a 0039a902`002cf592 0039a908`002cf598 : 0x39a8e6`002cf576
    00000000`00c1f0e0 0039a8f4`002cf584 : 0039a8fa`002cf58a 0039a902`002cf592 0039a908`002cf598 0039a911`002cf5a1 : 0x39a8ec`002cf57c
    00000000`00c1f0e8 0039a8fa`002cf58a : 0039a902`002cf592 0039a908`002cf598 0039a911`002cf5a1 0039a919`002cf5a9 : 0x39a8f4`002cf584
    00000000`00c1f0f0 0039a902`002cf592 : 0039a908`002cf598 0039a911`002cf5a1 0039a919`002cf5a9 0039a920`002cf5af : 0x39a8fa`002cf58a
    00000000`00c1f0f8 0039a908`002cf598 : 0039a911`002cf5a1 0039a919`002cf5a9 0039a920`002cf5af 0039a925`002cf5b4 : 0x39a902`002cf592
    00000000`00c1f100 0039a911`002cf5a1 : 0039a919`002cf5a9 0039a920`002cf5af 0039a925`002cf5b4 0039a92f`002cf5be : 0x39a908`002cf598
    00000000`00c1f108 0039a919`002cf5a9 : 0039a920`002cf5af 0039a925`002cf5b4 0039a92f`002cf5be 0039a93c`002cf5cb : 0x39a911`002cf5a1
    00000000`00c1f110 0039a920`002cf5af : 0039a925`002cf5b4 0039a92f`002cf5be 0039a93c`002cf5cb 0039a955`002cf5e4 : 0x39a919`002cf5a9
    00000000`00c1f118 0039a925`002cf5b4 : 0039a92f`002cf5be 0039a93c`002cf5cb 0039a955`002cf5e4 0039a95e`002cf5ed : 0x39a920`002cf5af
    00000000`00c1f120 0039a92f`002cf5be : 0039a93c`002cf5cb 0039a955`002cf5e4 0039a95e`002cf5ed 0039a969`002cf5f8 : 0x39a925`002cf5b4
    00000000`00c1f128 0039a93c`002cf5cb : 0039a955`002cf5e4 0039a95e`002cf5ed 0039a969`002cf5f8 0039a96d`002cf5fc : 0x39a92f`002cf5be
    

    There were 5 threads in this svchost process, and there was another thread looks like this above one.

    our issue is easy to fix and it's not my concern,  what I want to ask is how to do further investigation about why svchost not response the alpc message. 

    I'm not sure if post here is property, if not please help to move to the correct category, thanks a lot and sorry for inconvenience.

    Thanks 

    Levi


    One world, one dream

    Tuesday, November 20, 2012 2:28 AM