Skip to main content

 none
Why Driver Verifier crashes with Force IRQL checking ? RRS feed

  • Question

  • Hello,
    I have driver verifier with activated standard settings including Force IRQL Checking and kernel thread from my driver calls NdisAcquireSpinLock. I receive bugcheck IRQL_NOT_LESS_OR_EQUAL with stack:

    00 96259c34 830e5a9e nt!MiEmptyWorkingSet+0x164
    01 96259c5c 833341f4 nt!MiTrimAllSystemPagableMemory+0x1ce
    02 96259c7c 8333dde8 nt!MmVerifierTrimMemory+0xdd
    03 96259c94 8333e33f nt!ViKeRaiseIrqlSanityChecks+0xb8
    04 96259cb0 8dc0c6a4 nt!VerifierKfAcquireSpinLock+0x50
    05 96259cdc 8dc19931 MyDriver!Function
    ...

    Function is called from kernel thread on passive level.
    Spin lock resides in memory allocated with NdisAllocateMemoryWithTag so it is in non-pageable memory. Why verifier crashes on NdisAcquireSpinLock ?  
    Wednesday, November 13, 2019 11:04 AM

Answers

  • Unfortunately this looks like system memory corruption, i.e. something has trashed a system table that then MiEmptyWorkingSet attempts to reference.   These are a pain to find.


    Don Burn Windows Driver Consulting Website: http://www.windrvr.com

    • Marked as answer by Peter Skvarka Thursday, November 14, 2019 8:04 PM
    Thursday, November 14, 2019 11:45 AM

All replies

  • Show us the !analyze -v from a system crash dump, right now we don't have enough data.


    Don Burn Windows Driver Consulting Website: http://www.windrvr.com

    Wednesday, November 13, 2019 11:46 AM
  • Running in pageable memory, eh? 

    -- pa

    Wednesday, November 13, 2019 9:42 PM
  • I received also second crash on the same machine so I am attaching two Bugcheck nalysis.

    In both cases verifier is activated for bmf.sys and ndis.sys. Bmf.sys is LWF filter.

    The second crash does not contain bmf.sys in the stack, so I have suspicion

    of memory related hardware problem, it is older testing machine Win7 x86.

    1: kd> !analyze -v ******************************************************************************* * * * Bugcheck Analysis * * * ******************************************************************************* IRQL_NOT_LESS_OR_EQUAL (a) An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high. This is usually caused by drivers using improper addresses. If a kernel debugger is available get the stack backtrace. Arguments: Arg1: c0055f24, memory referenced Arg2: 00000000, IRQL Arg3: 00000000, bitfield : bit 0 : value 0 = read operation, 1 = write operation bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status) Arg4: 83024999, address which referenced memory Debugging Details: ------------------ KEY_VALUES_STRING: 1 STACKHASH_ANALYSIS: 1 TIMELINE_ANALYSIS: 1 DUMP_CLASS: 1 DUMP_QUALIFIER: 401 BUILD_VERSION_STRING: 7601.23539.x86fre.win7sp1_ldr.160902-0600 SYSTEM_MANUFACTURER: System manufacturer SYSTEM_PRODUCT_NAME: System Product Name SYSTEM_VERSION: System Version BIOS_VENDOR: American Megatrends Inc. BIOS_VERSION: 0704 BIOS_DATE: 03/20/2006 BASEBOARD_MANUFACTURER: ASUSTeK Computer INC. BASEBOARD_PRODUCT: P5GPL-X BASEBOARD_VERSION: Rev 1.xx DUMP_TYPE: 1 BUGCHECK_P1: ffffffffc0055f24 BUGCHECK_P2: 0 BUGCHECK_P3: 0 BUGCHECK_P4: ffffffff83024999 READ_ADDRESS: c0055f24 CURRENT_IRQL: 0 FAULTING_IP: nt!MiEmptyWorkingSet+164 83024999 8b01 mov eax,dword ptr [ecx] CPU_COUNT: 2 CPU_MHZ: c8b CPU_VENDOR: GenuineIntel CPU_FAMILY: f CPU_MODEL: 4 CPU_STEPPING: 9 CPU_MICROCODE: f,4,9,0 (F,M,S,R) SIG: 3'00000000 (cache) 3'00000000 (init) DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT BUGCHECK_STR: 0xA PROCESS_NAME: System ANALYSIS_SESSION_HOST: LYZARD ANALYSIS_SESSION_TIME: 11-13-2019 20:56:32.0639 ANALYSIS_VERSION: 10.0.17763.1 x86fre TRAP_FRAME: 96259b18 -- (.trap 0xffffffff96259b18) ErrCode = 00000000 eax=864006a8 ebx=83169100 ecx=c0055f24 edx=00000000 esi=93c32d48 edi=00000027 eip=83024999 esp=96259b8c ebp=96259c34 iopl=0 nv up ei ng nz na pe cy cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010287 nt!MiEmptyWorkingSet+0x164: 83024999 8b01 mov eax,dword ptr [ecx] ds:0023:c0055f24=???????? Resetting default scope LAST_CONTROL_TRANSFER: from 83024999 to 8303dc3f STACK_TEXT: 96259b18 83024999 badb0d00 00000000 aa05e000 nt!KiTrap0E+0x1b3 96259c34 830e5a9e 864006a8 00000000 000171bf nt!MiEmptyWorkingSet+0x164 96259c5c 833341f4 00000000 00000000 000171bf nt!MiTrimAllSystemPagableMemory+0x1ce 96259c7c 8333dde8 913e1440 000171bf 00000000 nt!MmVerifierTrimMemory+0xdd 96259c94 8333e33f 00000002 000171bf 913e1440 nt!ViKeRaiseIrqlSanityChecks+0xb8 96259cb0 8dc0c6a4 00000000 9beb2078 9be82f58 nt!VerifierKfAcquireSpinLock+0x50 96259cdc 8dc19931 913e1440 9beb2158 9be82f70 Bmf!RemoveTimedOutDnsConnections+0x58 [f:\projects\bmf\trunk\bmf\dns.c @ 701] 96259d38 8dc19b8a 9be82f58 00000000 93c32d48 Bmf!GarbageRoutine+0x2ed [f:\projects\bmf\trunk\bmf\garbthread.c @ 949] 96259d50 831f1606 9be82f58 bd147a82 00000000 Bmf!GarbageThreadRoutine+0x4a [f:\projects\bmf\trunk\bmf\garbthread.c @ 237] 96259d90 8308f659 8dc19b40 9be82f58 00000000 nt!PspSystemThreadStartup+0x9e 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19 THREAD_SHA1_HASH_MOD_FUNC: dc1b9d6bb3d1992bba1fbd98ad52932f59f726f0 THREAD_SHA1_HASH_MOD_FUNC_OFFSET: d5651172b85bb8b2387b4936d5efec62174f5284 THREAD_SHA1_HASH_MOD: 1f503ed2d219890e857af75e6781eb739a9080c2 FOLLOWUP_IP: Bmf!RemoveTimedOutDnsConnections+58 [f:\projects\bmf\trunk\bmf\dns.c @ 701] 8dc0c6a4 8b5618 mov edx,dword ptr [esi+18h] FAULT_INSTR_CODE: 8818568b FAULTING_SOURCE_LINE: f:\projects\bmf\trunk\bmf\dns.c FAULTING_SOURCE_FILE: f:\projects\bmf\trunk\bmf\dns.c FAULTING_SOURCE_LINE_NUMBER: 701 SYMBOL_STACK_INDEX: 6 SYMBOL_NAME: Bmf!RemoveTimedOutDnsConnections+58 FOLLOWUP_NAME: MachineOwner MODULE_NAME: Bmf IMAGE_NAME: Bmf.sys DEBUG_FLR_IMAGE_TIMESTAMP: 5dca7a86 STACK_COMMAND: .thread ; .cxr ; kb FAILURE_BUCKET_ID: 0xA_VRF_Bmf!RemoveTimedOutDnsConnections+58 BUCKET_ID: 0xA_VRF_Bmf!RemoveTimedOutDnsConnections+58 PRIMARY_PROBLEM_CLASS: 0xA_VRF_Bmf!RemoveTimedOutDnsConnections+58 TARGET_TIME: 2019-11-13T10:04:04.000Z OSBUILD: 7601 OSSERVICEPACK: 1000 SERVICEPACK_NUMBER: 0 OS_REVISION: 0 SUITE_MASK: 272 PRODUCT_TYPE: 1 OSPLATFORM_TYPE: x86 OSNAME: Windows 7 OSEDITION: Windows 7 WinNt (Service Pack 1) TerminalServer SingleUserTS OS_LOCALE: USER_LCID: 0 OSBUILD_TIMESTAMP: 2016-09-02 16:50:49 BUILDDATESTAMP_STR: 160902-0600 BUILDLAB_STR: win7sp1_ldr BUILDOSVER_STR: 6.1.7601.23539.x86fre.win7sp1_ldr.160902-0600 ANALYSIS_SESSION_ELAPSED_TIME: 715 ANALYSIS_SOURCE: KM FAILURE_ID_HASH_STRING: km:0xa_vrf_bmf!removetimedoutdnsconnections+58 FAILURE_ID_HASH: {6f8a6272-2272-2875-c69c-d21837cdeeca} Followup: MachineOwner ---------

    ========= THE SECOND CRASH ===============================

    IRQL_NOT_LESS_OR_EQUAL (a)
    An attempt was made to access a pageable (or completely invalid) address at an
    interrupt request level (IRQL) that is too high.  This is usually
    caused by drivers using improper addresses.
    If a kernel debugger is available get the stack backtrace.
    Arguments:
    Arg1: c001ee58, memory referenced
    Arg2: 00000000, IRQL
    Arg3: 00000000, bitfield :
    bit 0 : value 0 = read operation, 1 = write operation
    bit 3 : value 0 = not an execute operation, 1 = execute operation (only on chips which support this level of status)
    Arg4: 8306c999, address which referenced memory

    Debugging Details:
    ------------------


    KEY_VALUES_STRING: 1


    STACKHASH_ANALYSIS: 1

    TIMELINE_ANALYSIS: 1


    DUMP_CLASS: 1

    DUMP_QUALIFIER: 401

    BUILD_VERSION_STRING:  7601.23539.x86fre.win7sp1_ldr.160902-0600

    SYSTEM_MANUFACTURER:  System manufacturer

    SYSTEM_PRODUCT_NAME:  System Product Name

    SYSTEM_VERSION:  System Version

    BIOS_VENDOR:  American Megatrends Inc.

    BIOS_VERSION:  0704    

    BIOS_DATE:  03/20/2006

    BASEBOARD_MANUFACTURER:  ASUSTeK Computer INC.

    BASEBOARD_PRODUCT:  P5GPL-X

    BASEBOARD_VERSION:  Rev 1.xx

    DUMP_TYPE:  1

    BUGCHECK_P1: ffffffffc001ee58

    BUGCHECK_P2: 0

    BUGCHECK_P3: 0

    BUGCHECK_P4: ffffffff8306c999

    READ_ADDRESS:  c001ee58 

    CURRENT_IRQL:  0

    FAULTING_IP: 
    nt!MiEmptyWorkingSet+164
    8306c999 8b01            mov     eax,dword ptr [ecx]

    CPU_COUNT: 2

    CPU_MHZ: c8b

    CPU_VENDOR:  GenuineIntel

    CPU_FAMILY: f

    CPU_MODEL: 4

    CPU_STEPPING: 9

    CPU_MICROCODE: f,4,9,0 (F,M,S,R)  SIG: 3'00000000 (cache) 3'00000000 (init)

    DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

    BUGCHECK_STR:  0xA

    PROCESS_NAME:  System

    ANALYSIS_SESSION_HOST:  LYZARD

    ANALYSIS_SESSION_TIME:  11-14-2019 07:05:15.0136

    ANALYSIS_VERSION: 10.0.17763.1 x86fre

    TRAP_FRAME:  8885aae4 -- (.trap 0xffffffff8885aae4)
    ErrCode = 00000000
    eax=86c006a8 ebx=831b1280 ecx=c001ee58 edx=00000000 esi=84bb1798 edi=0000002c
    eip=8306c999 esp=8885ab58 ebp=8885ac00 iopl=0         nv up ei ng nz na po cy
    cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010283
    nt!MiEmptyWorkingSet+0x164:
    8306c999 8b01            mov     eax,dword ptr [ecx]  ds:0023:c001ee58=????????
    Resetting default scope

    LAST_CONTROL_TRANSFER:  from 8306c999 to 83085c3f

    STACK_TEXT:  
    8885aae4 8306c999 badb0d00 00000000 181b5d42 nt!KiTrap0E+0x1b3
    8885ac00 8312da9e 86c006a8 00000000 8885acd4 nt!MiEmptyWorkingSet+0x164
    8885ac28 8337c1f4 00000000 00000000 8885acd4 nt!MiTrimAllSystemPagableMemory+0x1ce
    8885ac48 83385de8 85d0e508 0001a36d 00000000 nt!MmVerifierTrimMemory+0xdd
    8885ac60 8338633f 00000002 8885acd4 85d0e508 nt!ViKeRaiseIrqlSanityChecks+0xb8
    8885ac7c 85a2b160 85d0e718 85d0e508 00000000 nt!VerifierKfAcquireSpinLock+0x50
    8885ac94 85a2b0f8 85d0e508 00000001 8885acd4 ndis!ndisAcquireReadWriteLockX+0x5f
    8885acac 85ae21fe 85d0e508 00000001 8885acd4 ndis!NdisAcquireReadWriteLock+0x15
    8885acdc 85ae2a20 85d0e508 8c263a70 8885ad00 NETIO!WfpTimerWheelTimeoutHandler+0x3d
    8885acec 8323e83c 8c263a70 8c27d410 84bb1798 NETIO!WfpSysTimerPassiveCallback+0x20
    8885ad00 830a8f3b a98a4fe0 00000000 84bb1798 nt!IopProcessWorkItem+0x23
    8885ad50 83239606 00000001 a3be9189 00000000 nt!ExpWorkerThread+0x10d
    8885ad90 830d7659 830a8e2e 00000001 00000000 nt!PspSystemThreadStartup+0x9e
    00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19


    THREAD_SHA1_HASH_MOD_FUNC:  67757f82f4dce36c77ef2b61e9fd8e9b3c50a860

    THREAD_SHA1_HASH_MOD_FUNC_OFFSET:  d379825f80595c486878ba47e19eed4b2872bdcb

    THREAD_SHA1_HASH_MOD:  6fcc69005d312f9671ca2358a5cc461fa0d12c5a

    FOLLOWUP_IP: 
    NETIO!WfpTimerWheelTimeoutHandler+3d
    85ae21fe 8b4708          mov     eax,dword ptr [edi+8]

    FAULT_INSTR_CODE:  8d08478b

    SYMBOL_STACK_INDEX:  8

    SYMBOL_NAME:  NETIO!WfpTimerWheelTimeoutHandler+3d

    FOLLOWUP_NAME:  MachineOwner

    MODULE_NAME: NETIO

    IMAGE_NAME:  NETIO.SYS

    DEBUG_FLR_IMAGE_TIMESTAMP:  577e6c0b

    STACK_COMMAND:  .thread ; .cxr ; kb

    FAILURE_BUCKET_ID:  0xA_VRF_NETIO!WfpTimerWheelTimeoutHandler+3d

    BUCKET_ID:  0xA_VRF_NETIO!WfpTimerWheelTimeoutHandler+3d

    PRIMARY_PROBLEM_CLASS:  0xA_VRF_NETIO!WfpTimerWheelTimeoutHandler+3d

    TARGET_TIME:  2019-11-14T05:58:23.000Z

    OSBUILD:  7601

    OSSERVICEPACK:  1000

    SERVICEPACK_NUMBER: 0

    OS_REVISION: 0

    SUITE_MASK:  272

    PRODUCT_TYPE:  1

    OSPLATFORM_TYPE:  x86

    OSNAME:  Windows 7

    OSEDITION:  Windows 7 WinNt (Service Pack 1) TerminalServer SingleUserTS

    OS_LOCALE:  

    USER_LCID:  0

    OSBUILD_TIMESTAMP:  2016-09-02 16:50:49

    BUILDDATESTAMP_STR:  160902-0600

    BUILDLAB_STR:  win7sp1_ldr

    BUILDOSVER_STR:  6.1.7601.23539.x86fre.win7sp1_ldr.160902-0600

    ANALYSIS_SESSION_ELAPSED_TIME:  6f7

    ANALYSIS_SOURCE:  KM

    FAILURE_ID_HASH_STRING:  km:0xa_vrf_netio!wfptimerwheeltimeouthandler+3d

    FAILURE_ID_HASH:  {8b272ba9-64d5-2fad-9230-9df5ad5d43aa}

    Followup:     MachineOwner
    ---------



    • Edited by Peter Skvarka Thursday, November 14, 2019 6:15 AM Additional important info
    Thursday, November 14, 2019 5:41 AM
  • Running in pageable memory, eh? 

    No, spin lock resides in memory allocated with NdisAllocateMemoryWithTag()
    Thursday, November 14, 2019 5:45 AM
  • Unfortunately this looks like system memory corruption, i.e. something has trashed a system table that then MiEmptyWorkingSet attempts to reference.   These are a pain to find.


    Don Burn Windows Driver Consulting Website: http://www.windrvr.com

    • Marked as answer by Peter Skvarka Thursday, November 14, 2019 8:04 PM
    Thursday, November 14, 2019 11:45 AM