none
BAD_POOL_HEADER (19) RRS feed

  • Question

  • I trying to track down an issue with an ancient device driver. I have source code and can step in with WinDbg. We have never seen this issue with thousands of cards out there so it might be a mix of other hardware in the computer that is creating the issue that I'll go into now.  On an XP system after communicating with the card eventually there will be a BSOD and a BAD_HEADER_POOL that is probably coming from our driver.

    *******************************************************************************
    *                        Bugcheck Analysis                                    *
    *******************************************************************************
    BAD_POOL_HEADER (19)
    The pool is already corrupt at the time of the current request.
    This may or may not be due to the caller.
    The internal pool links must be walked to figure out a possible cause of
    the problem, and then special pool applied to the suspect tags or the driver
    verifier to a suspect driver.
    Arguments:
    Arg1: 00000020, a pool block header size is corrupt.
    Arg2: 89935358, The pool entry we were looking for within the page.
    Arg3: 89935370, The next pool entry.
    Arg4: 1a030001, (reserved)

    BUGCHECK_STR:  0x19_20
    POOL_ADDRESS:  89935358 Nonpaged pool
    DEFAULT_BUCKET_ID:  INTEL_CPU_MICROCODE_ZERO
    PROCESS_NAME:  623XP.exe
    IRP_ADDRESS:  8a664f48
    DEVICE_OBJECT: 89900298
    DRIVER_OBJECT: 899007c0
    IMAGE_NAME:  OurDrv.sys
    DEBUG_FLR_IMAGE_TIMESTAMP:  50ab85f0
    MODULE_NAME: OurDrv
    FAULTING_MODULE: f7757000 OurDrv
    MANAGED_STACK: !dumpstack -EE
    OS Thread Id: 0x0 (0)
    TEB information is not available so a stack size of 0xFFFF is assumed
    Current frame: 
    ChildEBP RetAddr  Caller,Callee
    LAST_CONTROL_TRANSFER:  from 80536753 to 804e30d9
    STACK_TEXT:  
    b2161570 80536753 00000003 b21618cc 00000000 nt!RtlpBreakWithStatusInstruction
    b21615bc 8053721e 00000003 00000000 89935358 nt!KiBugCheckDebugBreak+0x19
    b216199c 80537832 00000019 00000020 89935358 nt!KeBugCheck2+0x574
    b21619bc 80552fc8 00000019 00000020 89935358 nt!KeBugCheckEx+0x1b
    b2161a0c 804ed43f 89935360 00000000 8a664f88 nt!ExFreePoolWithTag+0x2c1
    b2161a64 804ed49a 8a664f88 b2161ab0 b2161aa4 nt!IopCompleteRequest+0xf4
    b2161ab4 80703ef2 00000000 00000000 b2161acc nt!KiDeliverApc+0xb3
    b2161ab4 80703ae4 00000000 00000000 b2161acc hal!HalpApcInterrupt+0xc6
    b2161b3c 804e62de 8a664f88 8a664f48 00000000 hal!KeReleaseQueuedSpinLock+0x3c
    b2161b5c 804ed4b4 8a664f88 89409740 00000000 nt!KeInsertQueueApc+0x6d
    b2161b90 80674849 899007c0 89900298 8a664f00 nt!IopfCompleteRequest+0x1d8
    b2161bfc f7757ff1 01bb7950 00000000 89900350 nt!IovCompleteRequest+0x9a
    b2161c1c 804e19ee 89900298 8a664f48 80703428 OurDrv!OurDrvDispatch+0xb1 [c:\winddk\7600.16385.1\src\OurDrv\OurDrv.c @ 1031]
    b2161c2c 80674145 8940a510 80703410 8a664f48 nt!IopfCallDriver+0x31
    b2161c50 8057184c 8a664fdc 89409740 8a664f48 nt!IovCallDriver+0xa0
    b2161c64 80582cef 89900298 8a664f48 89409740 nt!IopSynchronousServiceTail+0x60
    b2161d00 8058ecc3 00000168 00000000 00000000 nt!IopXxxControlFile+0x5ef
    b2161d34 804ddf0f 00000168 00000000 00000000 nt!NtDeviceIoControlFile+0x2a
    b2161d34 7c90eb94 00000168 00000000 00000000 nt!KiFastCallEntry+0xfc
    0123f750 7c90d8ef 7c801671 00000168 00000000 ntdll!KiFastSystemCallRet
    0123f754 7c801671 00000168 00000000 00000000 ntdll!ZwDeviceIoControlFile+0xc
    0123f7b4 003b13b2 00000168 9c406494 003e2ac8 KERNEL32!DeviceIoControl+0xdd
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0123f8ac 793d7a7b 01643160 793683dd 0123f8e0 rsdvc70f!dsp_close+0x72
    0123f8b4 793683dd 0123f8e0 0123f90c 01643160 mscorlib_ni+0x317a7b
    0123f8c8 793d7b5c 0164312c 00000000 00193938 mscorlib_ni+0x2a83dd
    0123f8e0 79e88f63 00c90000 00c93e78 0123f970 mscorlib_ni+0x317b5c
    0123f8f0 79e88ee4 0123f9c0 00000000 0123f990 mscorwks!CallDescrWorker+0x33
    0123f970 79e88e31 0123f9c0 00000000 0123f990 mscorwks!CallDescrWorkerWithHandler+0xa3
    0123faac 79e88d19 796a3000 0123fc28 0123fb40 mscorwks!MethodDesc::CallDescr+0x19c
    0123fac4 79e88cf6 796a3000 0123fc28 0123fb40 mscorwks!MethodDesc::CallTargetWorker+0x20
    0123fad8 7a07d585 0123fb40 99fb3b7c 00193938 mscorwks!MethodDescCallSite::Call+0x18
    0123fcdc 79ecb4a4 0123fe50 00000001 00193938 mscorwks!ThreadNative::KickOffThread_Worker+0x15d
    0123fcec 79ecb442 0123fdc8 0123fd74 79f93fe6 mscorwks!Thread::UserResumeThread+0xfb
    0123fd80 79ecb364 0123fdc8 99fb3da8 00000001 mscorwks!Thread::DoADCallBack+0x355
    0123fdbc 7a0e1b7e 0123fdc8 00000001 00000000 mscorwks!Thread::DoADCallBack+0x541
    0123fde4 7a0e1b95 00000001 7a07d42f 0123fe50 mscorwks!Thread::DoADCallBack+0x575
    0123fdf8 7a07f7b3 00000001 7a07d42f 0123fe50 mscorwks!ManagedThreadBase::KickOff+0x13
    0123fe94 79ecb00b 0017fa08 00000005 e2d86bd8 mscorwks!ThreadNative::KickOffThread+0x230
    0123ffb4 7c80b50b 00192f50 0109d688 7c80a417 mscorwks!Thread::intermediateThreadProc+0x49
    0123ffec 00000000 79ecafc5 00192f50 00000000 KERNEL32!BaseThreadStart+0x37

    STACK_COMMAND:  kb

    FOLLOWUP_IP: 
    OurDrv!OurDrvDispatch+b1 [c:\winddk\7600.16385.1\src\OurDrv\OurDrv.c @ 1031]
    f7757ff1 eb2f            jmp     OurDrv!OurDrvDispatch+0xe2 (f7758022)

    FAULTING_SOURCE_CODE:  
      1027: {
      1028: returnValue = pIrp->IoStatus.Status;
      1029: IoCompleteRequest(pIrp, IO_NO_INCREMENT);
      1030: }
    > 1031: else
      1032: {
      1033: returnValue = pIrp->IoStatus.Status;
      1034: IoAcquireCancelSpinLock(&kCancelSpin);
      1035: IoSetCancelRoutine(pIrp, OurDrvCancel);
      1036: IoReleaseCancelSpinLock(kCancelSpin);


    SYMBOL_STACK_INDEX:  c
    SYMBOL_NAME:  OurDrv!OurDrvDispatch+b1
    FOLLOWUP_NAME:  MachineOwner
    FAILURE_BUCKET_ID:  0x19_20_VRF_OurDrv!OurDrvDispatch+b1
    BUCKET_ID:  0x19_20_VRF_OurDrv!OurDrvDispatch+b1

    Not really sure why this shows that the fault is with an 'else' statement.  Looking for advise on how best to track this down. I've tried running verifier on the target computer but didn't seem to give anything different in regards to the crash.

    Thanks in advance! You guys have definitely saved me time in the past.

    Tuesday, November 20, 2012 2:35 PM

Answers

  • It is possible since memory corruption does have the problem of being detected well after the fact.  Check if there is a newer version of the PCI9030 driver and see if the problem goes away.


    Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

    Tuesday, November 20, 2012 3:41 PM

All replies

  • The reason it is showing the else is it is really in the IoCompleteRequest just above it.  If this driver will work on Win7 you might want to see if you can repoduce the problem there, since with a somewhat different memory manager it may point something out.  Also, with Driver Verifier try running with with the VerifyStart option as well as the VerifyEnd see http://msdn.microsoft.com/en-us/library/ff551832.aspx  Other than that, the only choice is to use !poolval and the other !pool calls in Windbg to try to find where your driver is corrupting the pool.


    Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

    Tuesday, November 20, 2012 2:53 PM
  • Hi Don,

      I can try Win7 shortly. I have some more info that maybe you can help confirm my thoughts.  I turned on the verifier for not only our driver but the other one that might be suspect since we've never had problems with our driver except on the system where another piece of hardware is included. This is what I get.

    OurDrv: IRP_MJ_DEVICE_CONTROL


    *** Fatal System Error: 0x000000c4
                           (0x00000081,0x89861E40,0x0000000A,0x00000000)

    Break instruction exception - code 80000003 (first chance)

    A fatal system error has occurred.
    Debugger entered on first try; Bugcheck callbacks have not been invoked.

    A fatal system error has occurred.

    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************

    Use !analyze -v to get detailed debugging information.

    BugCheck C4, {81, 89861e40, a, 0}

    *** ERROR: Module load completed but symbols could not be loaded for Pci9030.sys
    *** WARNING: Unable to verify checksum for PlxApi.dll
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for PlxApi.dll - 
    *** WARNING: Unable to verify checksum for pif30032.dll
    *** ERROR: Symbol file could not be found.  Defaulted to export symbols for pif30032.dll - 
    *** WARNING: Unable to verify checksum for mscorlib.ni.dll
    *** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll
    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Your debugger is not using the correct symbols                 ***
    ***                                                                   ***
    ***    In order for this command to work properly, your symbol path   ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: kernel32!pNlsUserInfo                         ***
    ***                                                                   ***
    *************************************************************************
    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Your debugger is not using the correct symbols                 ***
    ***                                                                   ***
    ***    In order for this command to work properly, your symbol path   ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: kernel32!pNlsUserInfo                         ***
    ***                                                                   ***
    *************************************************************************
    Probably caused by : Pci9030.sys ( Pci9030+d60 )

    Followup: MachineOwner
    ---------

    nt!RtlpBreakWithStatusInstruction:
    804e30d9 cc              int     3
    kd> g
    Break instruction exception - code 80000003 (first chance)

    A fatal system error has occurred.

    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************

    Use !analyze -v to get detailed debugging information.

    BugCheck C4, {81, 89861e40, a, 0}

    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Your debugger is not using the correct symbols                 ***
    ***                                                                   ***
    ***    In order for this command to work properly, your symbol path   ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: kernel32!pNlsUserInfo                         ***
    ***                                                                   ***
    *************************************************************************
    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Your debugger is not using the correct symbols                 ***
    ***                                                                   ***
    ***    In order for this command to work properly, your symbol path   ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: kernel32!pNlsUserInfo                         ***
    ***                                                                   ***
    *************************************************************************
    Probably caused by : Pci9030.sys ( Pci9030+d60 )

    Followup: MachineOwner
    ---------

    nt!RtlpBreakWithStatusInstruction:
    804e30d9 cc              int     3
    kd> g
    Break instruction exception - code 80000003 (first chance)

    A fatal system error has occurred.

    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************

    Use !analyze -v to get detailed debugging information.

    BugCheck C4, {81, 89861e40, a, 0}

    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Your debugger is not using the correct symbols                 ***
    ***                                                                   ***
    ***    In order for this command to work properly, your symbol path   ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: kernel32!pNlsUserInfo                         ***
    ***                                                                   ***
    *************************************************************************
    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Your debugger is not using the correct symbols                 ***
    ***                                                                   ***
    ***    In order for this command to work properly, your symbol path   ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: kernel32!pNlsUserInfo                         ***
    ***                                                                   ***
    *************************************************************************
    Probably caused by : Pci9030.sys ( Pci9030+d60 )

    Followup: MachineOwner
    ---------

    nt!RtlpBreakWithStatusInstruction:
    804e30d9 cc              int     3
    kd> !analyze -v
    *******************************************************************************
    *                                                                             *
    *                        Bugcheck Analysis                                    *
    *                                                                             *
    *******************************************************************************

    DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
    A device driver attempting to corrupt the system has been caught.  This is
    because the driver was specified in the registry as being suspect (by the
    administrator) and the kernel has enabled substantial checking of this driver.
    If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
    be among the most commonly seen crashes.
    Arguments:
    Arg1: 00000081, MmMapLockedPages called without MDL_MAPPING_CAN_FAIL
    Arg2: 89861e40, MDL address.
    Arg3: 0000000a, MDL flags.
    Arg4: 00000000, 0.

    Debugging Details:
    ------------------

    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Your debugger is not using the correct symbols                 ***
    ***                                                                   ***
    ***    In order for this command to work properly, your symbol path   ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: kernel32!pNlsUserInfo                         ***
    ***                                                                   ***
    *************************************************************************
    *************************************************************************
    ***                                                                   ***
    ***                                                                   ***
    ***    Your debugger is not using the correct symbols                 ***
    ***                                                                   ***
    ***    In order for this command to work properly, your symbol path   ***
    ***    must point to .pdb files that have full type information.      ***
    ***                                                                   ***
    ***    Certain .pdb files (such as the public OS symbols) do not      ***
    ***    contain the required information.  Contact the group that      ***
    ***    provided you with these symbols if you need this command to    ***
    ***    work.                                                          ***
    ***                                                                   ***
    ***    Type referenced: kernel32!pNlsUserInfo                         ***
    ***                                                                   ***
    *************************************************************************

    BUGCHECK_STR:  0xc4_81

    DEFAULT_BUCKET_ID:  INTEL_CPU_MICROCODE_ZERO

    PROCESS_NAME:  623XP.exe

    MANAGED_STACK: !dumpstack -EE
    OS Thread Id: 0x0 (0)
    TEB information is not available so a stack size of 0xFFFF is assumed
    Current frame: 
    ChildEBP RetAddr  Caller,Callee

    LAST_CONTROL_TRANSFER:  from 80536753 to 804e30d9

    STACK_TEXT:  
    b198a708 80536753 00000004 00001000 00000000 nt!RtlpBreakWithStatusInstruction
    b198a754 8053771f 00000004 bad0e000 89861e40 nt!KiBugCheckDebugBreak+0x19
    b198ab34 80537832 000000c4 00000081 89861e40 nt!KeBugCheck2+0xa75
    b198ab54 80674c38 000000c4 00000081 89861e40 nt!KeBugCheckEx+0x1b
    b198ab80 f776fd60 89861e40 00000000 89840b08 nt!VerifierMapLockedPages+0xbe
    WARNING: Stack unwind information not available. Following frames may be wrong.
    b198abe4 f777099e 899a78f8 00000000 00000004 Pci9030+0xd60
    b198ac1c 804e19ee 00000000 8a596f48 80703428 Pci9030+0x199e
    b198ac2c 80674145 8935e748 80703410 8a596f48 nt!IopfCallDriver+0x31
    b198ac50 8057184c 8a596fdc 894e6f90 8a596f48 nt!IovCallDriver+0xa0
    b198ac64 80582cef 899a7840 8a596f48 894e6f90 nt!IopSynchronousServiceTail+0x60
    b198ad00 8058ecc3 00000158 0000015c 00000000 nt!IopXxxControlFile+0x5ef
    b198ad34 804ddf0f 00000158 0000015c 00000000 nt!NtDeviceIoControlFile+0x2a
    b198ad34 7c90eb94 00000158 0000015c 00000000 nt!KiFastCallEntry+0xfc
    0123f4b8 7c90d8ef 7c8016be 00000158 0000015c ntdll!KiFastSystemCallRet
    0123f4bc 7c8016be 00000158 0000015c 00000000 ntdll!ZwDeviceIoControlFile+0xc
    0123f51c 003a2651 00000158 0022206c 00b60f30 KERNEL32!DeviceIoControl+0x78
    0123f624 100036a4 00000004 00000055 0123f6d4 PlxApi!PlxBusIopWrite+0x71
    0123f680 1000381d 0123f728 0123f820 00193938 pif30032!set_watchdog+0x25f0
    0123f6d4 1000145d 0123f77c 0123f820 00193938 pif30032!set_watchdog+0x2769
    0123f728 1000164a 0123f7e0 0123f820 00193938 pif30032!set_watchdog+0x3a9
    0123f77c 10004fb7 0048bb40 00000001 00000000 pif30032!set_watchdog+0x596
    0123f8ac 793d7a7b 01642428 793683dd 0123f8e0 pif30032!set_watchdog+0x3f03
    0123f8b4 793683dd 0123f8e0 0123f90c 01642428 mscorlib_ni+0x317a7b
    0123f8c8 793d7b5c 016423f4 00000000 00193938 mscorlib_ni+0x2a83dd
    0123f8e0 79e88f63 00c90000 00c93e38 0123f970 mscorlib_ni+0x317b5c
    0123f8f0 79e88ee4 0123f9c0 00000000 0123f990 mscorwks!CallDescrWorker+0x33
    0123f970 79e88e31 0123f9c0 00000000 0123f990 mscorwks!CallDescrWorkerWithHandler+0xa3
    0123faac 79e88d19 796a3000 0123fc28 0123fb40 mscorwks!MethodDesc::CallDescr+0x19c
    0123fac4 79e88cf6 796a3000 0123fc28 0123fb40 mscorwks!MethodDesc::CallTargetWorker+0x20
    0123fad8 7a07d585 0123fb40 150ffcd5 00193938 mscorwks!MethodDescCallSite::Call+0x18
    0123fcdc 79ecb4a4 0123fe50 00000001 00193938 mscorwks!ThreadNative::KickOffThread_Worker+0x15d
    0123fcec 79ecb442 0123fdc8 0123fd74 79f93fe6 mscorwks!Thread::UserResumeThread+0xfb
    0123fd80 79ecb364 0123fdc8 150ffa01 00000001 mscorwks!Thread::DoADCallBack+0x355
    0123fdbc 7a0e1b7e 0123fdc8 00000001 00000000 mscorwks!Thread::DoADCallBack+0x541
    0123fde4 7a0e1b95 00000001 7a07d42f 0123fe50 mscorwks!Thread::DoADCallBack+0x575
    0123fdf8 7a07f7b3 00000001 7a07d42f 0123fe50 mscorwks!ManagedThreadBase::KickOff+0x13
    0123fe94 79ecb00b 0017fa28 00000000 00000000 mscorwks!ThreadNative::KickOffThread+0x230
    0123ffb4 7c80b50b 00192f50 0109d688 7c80a417 mscorwks!Thread::intermediateThreadProc+0x49
    0123ffec 00000000 79ecafc5 00192f50 00000000 KERNEL32!BaseThreadStart+0x37


    STACK_COMMAND:  kb

    FOLLOWUP_IP: 
    Pci9030+d60
    f776fd60 8bd8            mov     ebx,eax

    SYMBOL_STACK_INDEX:  5

    SYMBOL_NAME:  Pci9030+d60

    FOLLOWUP_NAME:  MachineOwner

    MODULE_NAME: Pci9030

    IMAGE_NAME:  Pci9030.sys

    DEBUG_FLR_IMAGE_TIMESTAMP:  3e08b9c4

    FAILURE_BUCKET_ID:  0xc4_81_VRF_Pci9030+d60

    BUCKET_ID:  0xc4_81_VRF_Pci9030+d60

    Followup: MachineOwner
    ---------

    Is it possible that this is truly the culprit that is corrupting the memory and then since our card is being hit more often since it's memory is being polled that we report the failure during an ioctl IODispatch routine?

    Thanks.

    Tuesday, November 20, 2012 3:25 PM
  • It is possible since memory corruption does have the problem of being detected well after the fact.  Check if there is a newer version of the PCI9030 driver and see if the problem goes away.


    Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

    Tuesday, November 20, 2012 3:41 PM