none
Windows Server 2012 stuck on boot when running on top of KVM with CPU grouping enabled RRS feed

  • Question

  • Hello All,

    We observe an interesting problem while running Windows Server 2012 on top of KVM with multiple CPUs.

    When CPU grouping is disabled, Windows boots and runs fine.

    When CPU grouping is enabled by:

    bcdedit /groupaware on
    bcdedit /groupsize 2

    Windows hangs on boot while showing black screen with logo and stalled round progress indicator.

    We produced crash dump by injection of NMI interrupt via QEMU/KVM debugging facilities, the call stacks look as following:

    0: kd> ~0
    0: kd> kv
    Child-SP          RetAddr           : Args to Child                                                           : Call Site
    fffff803`95645c08 fffff803`941828de : 00000000`00000080 00000000`004f4454 00000000`00000000 00000000`00000000 : nt!KeBugCheckEx
    fffff803`95645c10 fffff803`93b66c09 : 00000000`00000001 fffff803`94196030 00000000`00000000 fffffa80`05654968 : hal!HalBugCheckSystem+0x9a
    fffff803`95645c50 fffff803`94183204 : 00000000`000006c0 fffff803`95645e30 fffff803`93d03100 fffff803`94196030 : nt!WheaReportHwError+0x249
    fffff803`95645cb0 fffff803`93be17a7 : fffff803`95645e70 00000000`00000010 00000000`80000005 fffff803`93a2227d : hal!HalHandleNMI+0x150
    fffff803`95645ce0 fffff803`93a84d02 : 00000000`32f7fc10 fffff803`95645ef0 00000000`00000000 fffff803`93d03180 : nt! ?? ::FNODOBFM::`string'+0x13d6d
    fffff803`95645d30 fffff803`93a84b73 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KxNmiInterrupt+0x82
    fffff803`95645e70 fffff803`93abb0cb : 00000000`00000000 fffff880`0315efb8 00000000`00001000 00000000`00000000 : nt!KiNmiInterrupt+0x173 (TrapFrame @ fffff803`95645e70)
    fffff880`0315eab0 fffff803`93b0191b : 00000000`00000000 00000000`00000000 fffff880`0315f200 fffff803`9417b291 : nt!KeFlushMultipleRangeTb+0x3c0
    fffff880`0315ecb0 fffff880`01709b45 : fffffa80`018a7040 00000000`0f0049ed fffff880`0315f200 00000000`00000000 : nt!MmSetAddressRangeModified+0x14b
    fffff880`0315edb0 fffff880`0170a471 : fffff8a0`00ba0010 00000000`0f004b81 00000000`00000001 00000000`00000500 : Ntfs!LfsFlushLfcb+0xa55
    fffff880`0315efc0 fffff803`93ac8df5 : fffff880`0315f160 fffffa80`0187a6a8 00000000`00000000 fffffa80`05621b80 : Ntfs!LfsFlushLfcbCallout+0x25
    fffff880`0315eff0 fffff803`93ac9d85 : fffff880`0170a44c fffff880`0315f160 00000000`00000000 fffff803`93d03180 : nt!KeExpandKernelStackAndCalloutInternal+0xe5
    fffff880`0315f0f0 fffff880`0170a434 : 00000000`0000000d fffff803`93d03180 fffff803`93d03180 fffffa80`018a7040 : nt!KeExpandKernelStackAndCalloutEx+0x25
    fffff880`0315f130 fffff880`017cbe9b : 00000000`00000000 fffff8a0`00ba0010 fffff880`0315f200 fffff8a0`00b18901 : Ntfs!LfsFlushLfcbOnNewStack+0x50
    fffff880`0315f190 fffff880`017c8f75 : fffff8a0`00ba0010 00000000`0f004b81 fffff880`0315f480 fffff8a0`00b18950 : Ntfs!LfsFlushToLsnPriv+0x13f
    fffff880`0315f230 fffff880`017c851f : fffff8a0`00ba0010 fffff8a0`00ba0010 fffffa80`057c2480 00000000`00000062 : Ntfs!LfsWriteLfsRestart+0x145
    fffff880`0315f290 fffff880`017c6083 : fffff8a0`00b98c10 fffff880`00000070 fffff880`00002102 00000000`00000000 : Ntfs!LfsWriteRestartArea+0x15f
    fffff880`0315f3d0 fffff880`017c9889 : 00000000`00000000 fffff880`0315fa90 00000000`00000000 00000000`00000000 : Ntfs!NtfsCheckpointVolume+0x6f3
    fffff880`0315f920 fffff880`017c57eb : fffff880`0315fa90 fffff880`0315fa90 fffffa80`057c2188 00000000`00000000 : Ntfs!NtfsCheckpointAllVolumesWorker+0x61
    fffff880`0315f970 fffff880`017c954c : fffff880`0315fa90 00000000`00000000 fffff880`017c9828 fffff880`0315fa40 : Ntfs!NtfsForEachVcb+0x1bb
    fffff880`0315fa10 fffff803`93ac0391 : fffffa80`018a7040 fffff880`0175d6d8 fffff803`93c91080 fffff803`93c91000 : Ntfs!NtfsCheckpointAllVolumes+0x12c
    fffff880`0315fcc0 fffff803`93a2f521 : 00000000`00000000 00000000`00000080 fffff803`93ac0250 fffffa80`018a7040 : nt!ExpWorkerThread+0x142
    fffff880`0315fd50 fffff803`93a6ddd6 : fffff803`93d03180 fffffa80`018a7040 fffffa80`018bd040 fffffa80`01897040 : nt!PspSystemThreadStartup+0x59
    fffff880`0315fda0 00000000`00000000 : fffff880`03160000 fffff880`0315a000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16
    0: kd> ~1 
    1: kd> kv
    Child-SP          RetAddr           : Args to Child                                                           : Call Site
    fffff880`031ce940 fffff803`93a5319b : 00000000`00000000 fffff880`04171000 00000000`80000002 00000000`00000001 : nt!KiIpiSendRequestEx+0x166
    fffff880`031ce990 fffff803`93a4d364 : 00000000`00000040 00000000`00000040 fffff880`031cebb0 ffffffff`ffffffff : nt!KeFlushSingleTb+0x11b
    fffff880`031ceab0 fffff803`93a4e9fc : 00000000`20000000 fffffa80`0178d250 00000000`0007d9b7 2aaaaaaa`aaaaaaab : nt!MiStealPage+0x914
    fffff880`031cee60 fffff803`93a4ff87 : 00000000`ffffffff 00000000`0007d9b8 00000000`00000103 fffff803`20000000 : nt!MiClaimPhysicalRun+0x1fc
    fffff880`031ceee0 fffff803`93e8ea59 : fffff880`031cf2e9 00000000`00000005 fffff880`0198f000 fffff8a0`00015470 : nt!MiFindContiguousPages+0x337
    fffff880`031cf030 fffff803`93e8f0ec : 00000000`7d9b8963 00000000`0007d9b8 00000000`00000000 fffff8a0`00003000 : nt!MiAllocateDriverPage+0x49
    fffff880`031cf0a0 fffff803`93e9eafb : 00000000`00000000 fffff880`031cf2e9 00000000`00000000 fffff6fc`4000cc78 : nt!MiLoadImageSection+0x23c
    fffff880`031cf230 fffff803`93e9f43e : fffff880`031cf390 00000000`00000000 00000000`00000001 00000000`00000000 : nt!MmLoadSystemImage+0x1c3
    fffff880`031cf340 fffff803`93e9979c : 00000000`00000000 00000000`00000000 fffff880`031cf860 00000000`00000003 : nt!IopLoadDriver+0x2ca
    fffff880`031cf610 fffff803`93e97fb6 : fffff8a0`00d04e00 fffff880`0097b340 00000000`00000000 00000000`00000000 : nt!PipCallDriverAddDeviceQueryRoutine+0x22c
    fffff880`031cf730 fffff803`93e95bc0 : 00000000`00000000 fffff880`00000002 00000000`00000000 00000000`000007ff : nt!PnpCallDriverQueryServiceHelper+0x13e
    fffff880`031cf7b0 fffff803`93e9611e : fffffa80`01892010 fffff880`031cfa40 fffffa80`01887670 fffffa80`018d4700 : nt!PipCallDriverAddDevice+0x400
    fffff880`031cf940 fffff803`93f09c07 : fffff803`93a91700 00000000`00000001 00000000`00000000 fffff803`93d90a52 : nt!PipProcessDevNodeTree+0x1ca
    fffff880`031cfbc0 fffff803`93b1381f : 00000001`00000003 00000000`00000000 fffff803`93ce06e0 00000000`00000000 : nt!PiProcessStartSystemDevices+0x87
    fffff880`031cfc10 fffff803`93ac0391 : fffffa80`01898040 fffff803`93b134bc fffff803`93ce06e0 fffffa80`0189c600 : nt!PnpDeviceActionWorker+0x363
    fffff880`031cfcc0 fffff803`93a2f521 : 00000000`00000000 00000000`00000080 fffff803`93ac0250 fffffa80`01898040 : nt!ExpWorkerThread+0x142
    fffff880`031cfd50 fffff803`93a6ddd6 : fffff880`02a15180 fffffa80`01898040 fffffa80`0188d040 fffffa80`01897040 : nt!PspSystemThreadStartup+0x59
    fffff880`031cfda0 00000000`00000000 : fffff880`031d0000 fffff880`031ca000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x16
    7: kd> ~2
    2: kd> kv
    Child-SP          RetAddr           : Args to Child                                                           : Call Site
    fffff880`02aafa08 fffff803`93b4e12a : fffff880`00000006 fffff880`00000003 fffff880`02a92140 fffff803`93b4e9e4 : hal!HalProcessorIdle+0xf
    fffff880`02aafa10 fffff803`93ab69a0 : fffff880`02a8b5c8 00000000`05f5e100 00000000`00000000 00000000`00000000 : nt!PpmIdleDefaultExecute+0xa
    fffff880`02aafa40 fffff803`93ab53c0 : 00000000`00000000 0000040c`0000040c fffff880`02aafca8 fffff880`02aafcb0 : nt!PpmIdleExecuteTransition+0x47f
    fffff880`02aafc60 fffff803`93ab354c : fffff880`02a86180 fffff880`02a86180 00000000`00000000 fffff880`02a92140 : nt!PoIdle+0x1e0
    fffff880`02aafda0 00000000`00000000 : fffff880`02ab0000 fffff880`02aaa000 00000000`00000000 00000000`00000000 : nt!KiIdleLoop+0x2c
    2: kd> ~3

    <ALL OTHER CPU STACKS ARE THE SAME AS FOR CPU 2>

    There are always two or more CPUs stuck in KeFlush...() functions dealing with IPIs.

    The problem looks like interrupt loss during IPI-ALL-BUT-SELF dispatching, we tried to track possible problems in KVM code that may lead to an issue like this but found nothing so far.

    The issue looks even more interesting taking into consideration that as far as we know CPU grouping is SW-only feature that doesn't have real effect on hardware.

    We will be grateful for any idea that could help to resolve the problem.

    Thanks in advance!

    P.S. Kernel crash dump is available at https://www.dropbox.com/s/07qb6wz9nacwhxn/MEMORY.DMP.zip?dl=0

    Sunday, May 3, 2015 11:17 AM