How to analyze crash dump when your driver name does not show in call stack ? RRS feed

  • General discussion

  • I used !analyze -v and I got :


    The IO manager has caught a misbehaving driver.
    Arg1: 00000012, IRQL not equal across call to driver dispatch routine
    Arg2: 84c049bc, Driver dispatch routine address.
    Arg3: 00000000, IRQL before calling driver dispatch routine.
    Arg4: 00000002, Current IRQL.

    Debugging Details:

    BUGCHECK_STR:  0xc9_12


    84c049bc 8bff            mov     edi,edi

    84d309ae 8bf8            mov     edi,eax




    PROCESS_NAME:  System

    LAST_CONTROL_TRANSFER:  from 82af7e71 to 82a86394

    ae82f58c 82af7e71 00000003 4c983bba 00000065 nt!RtlpBreakWithStatusInstruction
    ae82f5dc 82af896d 00000003 8a5f8498 8a5f84fc nt!KiBugCheckDebugBreak+0x1c
    ae82f9a0 82af7d10 000000c9 00000012 84c049bc nt!KeBugCheck2+0x68b
    ae82f9c4 82b2a842 000000c9 00000012 84c049bc nt!KeBugCheckEx+0x1e
    ae82f9dc 82d4fdab 8a5f8498 00000000 ae82fa10 nt!VfBugCheckNoStackUsage+0x23
    ae82f9ec 82d4a6d4 8a5f84fc 00000000 8a856ef8 nt!VfAfterCallDriver+0x66
    ae82fa10 82a57473 00000000 8a856f1c 8a1d23b8 nt!IovCallDriver+0x269
    ae82fa24 82d5c3d0 8a5f83d0 8a856db8 8a1d6910 nt!IofCallDriver+0x1b
    ae82fa3c 82d4a6c3 8a1d69c8 8a856db8 a07d6000 nt!ViFilterDispatchGeneric+0x5e
    ae82fa60 82a57473 00000000 8a1e0ed8 8a1d6910 nt!IovCallDriver+0x258
    ae82fa74 84d309ae 8a5e0c70 8a856db8 8a1e0e20 nt!IofCallDriver+0x1b
    ae82fa90 82d4a6c3 8a1e0e20 8a856db8 8a1e60d8 volmgr!VmReadWrite+0x1a8
    ae82fab4 82a57473 00000000 8a856db8 8a1e0e20 nt!IovCallDriver+0x258
    ae82fac8 8515c475 8a856db8 ae82faf0 8515c547 nt!IofCallDriver+0x1b
    ae82fad4 8515c547 8a1e60d8 8a856db8 8a856db8 fvevol!FveRequestPassThrough+0x31
    ae82faf0 8515c759 8a1e60d8 8a856d01 8a59a0d8 fvevol!FveReadWrite+0x4d
    ae82fb20 8515c7a9 8a1e6020 8a856db8 ae82fb54 fvevol!FveFilterRundownReadWrite+0x197
    ae82fb30 82d4a6c3 8a1e6020 8a856db8 8a856f64 fvevol!FveFilterRundownWrite+0x33
    ae82fb54 82a57473 00000000 8a1e5474 8a1e6020 nt!IovCallDriver+0x258
    ae82fb68 853d676e 8a1e5418 8a856db8 00000003 nt!IofCallDriver+0x1b
    ae82fc48 853d68a5 8a1e5418 8a856db8 8b1c94c8 rdyboost!SmdProcessReadWrite+0xa14
    ae82fc68 82d4a6c3 8a1e5418 8a856db8 8a856f88 rdyboost!SmdDispatchReadWrite+0xcb
    ae82fc8c 82a57473 00000000 8a856fac 8a1e5360 nt!IovCallDriver+0x258
    ae82fca0 85386fd9 8aa24a88 8a856db8 8a1e6660 nt!IofCallDriver+0x1b
    ae82fcc8 853872fd 001e6660 8a856db8 ae82fcfc volsnap!VolsnapWriteFilter+0x265
    ae82fcd8 82d4a6c3 8a1e6660 8a856db8 ae82fd28 volsnap!VolSnapWrite+0x21
    ae82fcfc 82a57473 00000000 874f269c 8a1e6660 nt!IovCallDriver+0x258
    ae82fd10 84e3591c 874f25fc ae82fd34 82a8b10e nt!IofCallDriver+0x1b
    ae82fd1c 82a8b10e 874f269c 00000000 ffffffff Ntfs!NtfsStorageDriverCallout+0x14
    ae82fd1c 82a8b205 874f269c 00000000 ffffffff nt!KiSwapKernelStackAndExit+0x15a
    874f260c 82aab0fd 874f269c 84e35908 ae830000 nt!KiSwitchKernelStackAndCallout+0x31
    874f2680 84e34939 84e35908 874f269c 00000000 nt!KeExpandKernelStackAndCalloutEx+0x29d
    874f26ac 84e355a6 8a1e6660 8a856db8 8551f618 Ntfs!NtfsCallStorageDriver+0x2d
    874f26f0 84e340a0 8b866bb8 8a1e6660 8a856db8 Ntfs!NtfsMultipleAsync+0x4d
    874f2820 84e330a6 8b866bb8 8a856db8 a67a7500 Ntfs!NtfsNonCachedIo+0x413
    874f2938 84e3485f 8b866bb8 8a856db8 03a932ff Ntfs!NtfsCommonWrite+0x1ebd
    874f29b0 82d4a6c3 8a498020 8a856db8 00000000 Ntfs!NtfsFsdWrite+0x2e1
    874f29d4 82a57473 00000000 8a856db8 8a498020 nt!IovCallDriver+0x258
    874f29e8 84b9d20c 8a555e30 8a856db8 00000000 nt!IofCallDriver+0x1b
    874f2a0c 84b9d3cb 874f2a2c 8a555e30 00000000 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x2aa
    874f2a44 82d4a6c3 8a555e30 8a856db8 8b858570 fltmgr!FltpDispatch+0xc5
    874f2a68 82a57473 00000000 8a856db8 8a555e30 nt!IovCallDriver+0x258
    874f2a7c 82c58eee 8b858570 8a856db8 8a856fd8 nt!IofCallDriver+0x1b
    874f2a9c 82c597a2 8a555e30 8b858570 00000001 nt!IopSynchronousServiceTail+0x1f8
    874f2b38 82a5e42a 8a555e30 00000000 00000000 nt!NtWriteFile+0x6e8
    874f2b38 82a5d895 8a555e30 00000000 00000000 nt!KiFastCallEntry+0x12a
    874f2bd4 82c0a652 800002d0 00000000 00000000 nt!ZwWriteFile+0x11
    874f2c1c 82c5bf4c 8a0c0000 8a0c0000 00000000 nt!EtwpFlushBufferToLogfile+0x81
    874f2c3c 82a8dbc6 00000000 00000001 855546c0 nt!EtwpFlushBuffer+0xc3
    874f2d08 82c4d9da 855546c0 00000000 00000000 nt!EtwpFlushActiveBuffers+0x2c0
    874f2d50 82c2966d 855546c0 6555e3f6 00000000 nt!EtwpLogger+0x2a1
    874f2d90 82adb0d9 82c4d739 855546c0 00000000 nt!PspSystemThreadStartup+0x9e
    00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19



    SYMBOL_NAME:  volmgr!VmReadWrite+1a8

    FOLLOWUP_NAME:  MachineOwner

    MODULE_NAME: volmgr

    IMAGE_NAME:  volmgr.sys


    FAILURE_BUCKET_ID:  0xc9_12_VRF_volmgr!VmReadWrite+1a8

    BUCKET_ID:  0xc9_12_VRF_volmgr!VmReadWrite+1a8

    Followup: MachineOwner


    Here I know that my driver did something wrong what if I did not know which driver cause this bug , in that case how do I analyze such type of dump?

    Thursday, February 27, 2014 3:34 PM

All replies

  • First depending on the type of driver have you run Static Driver Verifier over it?  This tool can find a lot of the problems that lead to this sort of problem.

    I see you are running with Driver Verifier, you might want to run it without your driver present but all other drivers enabled, just to be sure that something else is not misbehaving?

    Now what sort of driver do you have?  Can it have an impact on a WriteFile call which is what is producing this failure?  If yes, then that is the first thing to look at, if not then things are going to get messier quickly since a likely scenario is a bad pointer dereference.

    The other approach people have used is slowly disabling functionality in their driver till the problem goes away.  Then desk checking that part of the driver like crazy.

    These can be hard bugs to find, since your driver and the rest of the kernel are sharing memory space and many other things.


    Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

    Thursday, February 27, 2014 3:46 PM
  • I am writing a disk cache filter driver. I got this bug when through ioctl I disable cache policy.

    Thanks, I got to know the problem is in my write callback.

    After disabling cache, whenever a write request comes I simply pass the request down but at this point I was not releasing a spin lock thats why the bug occurred.

    I have another problem that I want to discuss with you.

    Driver Verifier detected violation:

    Previously set IRP_MJ_POWER status has been converted to
    STATUS_NOT_SUPPORTED. This failure status is reserved for use of the OS
    - drivers cannot fail a Power IRP with this value.

    CulpritAddress = 82A45049, Irp = 86B7AE90.

    Break instruction exception - code 80000003 (first chance)
    82d6e24c cc              int     3

    Whenever I use driver verifier on, sometime I got this error. I do not know why I got break in debugger. I tried to trace it but did not find anything. 

    When I do not use verifier, I do not get this error

    Is this might be due to my driver or not ? 

    Do you have some information on it?

    Thursday, February 27, 2014 4:55 PM
  • In the future, you can use “!verifier 0x8” to dump IRQL log. There is also a description in debugger help and MSDN.


    Here is an example:

    0: kd> !verifier 0x8

    Displaying most recent 0x0000000000000004 entries from the IRQL transition log.
    There are up to 0x100 entries in the log.

    Thread:             ffffe001bec74080
    Old irql:           0000000000000000
    New irql:           0000000000000002
    Processor:          0000000000000001
    Time stamp:         0000000011432e48

        fffff80162373a55 fltmgr!FltFreeCallbackData+0x10f5
        fffff801623733fd fltmgr!FltFreeCallbackData+0xa9d
        fffff80162375f76 fltmgr!FltIsCallbackDataDirty+0x966
        fffff8016239e772 fltmgr!FltQueryInformationFile+0x72e
        fffff80161eef978 VerifierExt!xdv_IRP_MJ_CREATE_wrapper+0xfc

    Thread:             ffffe001bec74080
    Old irql:           0000000000000001
    New irql:           0000000000000000
    Processor:          0000000000000001
    Time stamp:         0000000011432e48

        fffff80162717243 Ntfs+0xc8243
        fffff801627152bb Ntfs+0xc62bb
        fffff801627147da Ntfs+0xc57da
        fffff80162713115 Ntfs+0xc4115
        fffff8016271144a Ntfs+0xc244a

    Thread:             ffffe001bec74080
    Old irql:           0000000000000000
    New irql:           0000000000000001
    Processor:          0000000000000001
    Time stamp:         0000000011432e48

        fffff80162717205 Ntfs+0xc8205
        fffff801627152bb Ntfs+0xc62bb
        fffff801627147da Ntfs+0xc57da
        fffff80162713115 Ntfs+0xc4115
        fffff8016271144a Ntfs+0xc244a

    Thread:             ffffe001bec74080
    Old irql:           0000000000000001
    New irql:           0000000000000000
    Processor:          0000000000000001
    Time stamp:         0000000011432e48

        fffff801627171b6 Ntfs+0xc81b6
        fffff801627152bb Ntfs+0xc62bb
        fffff801627147da Ntfs+0xc57da
        fffff80162713115 Ntfs+0xc4115
        fffff8016271144a Ntfs+0xc244a

    Friday, February 28, 2014 4:13 PM
  • This check is performed for a power IRP in both directions downward and upward (also for each device in the device stack). Can you check your IoCallDriver and IoComplete paths in debugger?

    Friday, February 28, 2014 5:30 PM
  • I am writing a kmdf based disk cache filter driver. I dont see any misbehavior in my driver. I handle only read and write requests wNS queue without power managed. Here is the dump if you could analyze :

    The IO manager has caught a misbehaving driver.
    Arg1: 0000021a, The previously-set IRP_MJ_POWER status has been converted to STATUS_NOT_SUPPORTED.
    Arg2: 82a14049, The address in the driver's code where the error was detected.
    Arg3: ae18ae90, IRP address.
    Arg4: 00000000

    Debugging Details:

    ADDITIONAL_DEBUG_TEXT:  Bugcheck data extracted fron VfErrorBugcheckData.

    BUGCHECK_STR:  0xc9_21a


    82a14049 5d              pop     ebp

    82a14049 5d              pop     ebp

    IRP_ADDRESS:  ae18ae90

    DEVICE_OBJECT: 8a1d9ac8

    DRIVER_OBJECT: 8a1d0f38

    IMAGE_NAME:  disk.sys


    MODULE_NAME: disk

    FAULTING_MODULE: 85198000 disk


    PROCESS_NAME:  System


    LAST_CONTROL_TRANSFER:  from 82d3d32a to 82d3d24c

    807c061c 82d3d32a 0000021a 82a14049 00000000 nt!ViErrorFinishReport+0x4c
    807c0670 82d441e9 0000021a ae18ae90 ae18af00 nt!VfErrorReport1+0x4d
    807c0694 82d3cedb ae18af24 ae18af00 00000001 nt!VfPowerVerifyIrpStackDownward+0xbe
    807c06b0 82d3b426 95e563a0 877fe6c8 ae18af24 nt!VfMajorVerifyIrpStackDownward+0x3e
    807c0714 82d3ad33 8c8efa48 ae18ae90 807c0744 nt!IovpCallDriver1+0x468
    807c0724 82d35670 877fe6c8 ae18af1c 877fe6c8 nt!VfBeforeCallDriver+0xe7
    807c0744 82a42473 ae18af00 8a1d9b80 877fe6c8 nt!IovCallDriver+0x206
    807c0758 82a14049 807c079c 851b272b 877fe6c8 nt!IofCallDriver+0x1b
    807c0760 851b272b 877fe6c8 ae18ae90 851b253f nt!IoCallDriver+0x10
    807c079c 82d35cd4 00000000 ae18af00 8a1d9dd0 CLASSPNP!ClasspPowerDownCompletion+0x1ec
    807c07cc 82a6eb33 00000000 8a1e0830 807c0840 nt!IovpLocalCompletionRoutine+0x14b
    807c0810 82d35b64 c0000010 8a1e0830 8a1e0830 nt!IopfCompleteRequest+0x128
    807c0878 84c06d97 8a1d9df4 877fe780 807c08ac nt!IovCompleteRequest+0x133
    807c0888 84c09fb2 877fe780 8a1e0830 8a1d9df4 ataport!IdeCompleteScsiIrp+0x31
    807c08ac 84c0646b c0000185 8a1e0830 807c08e0 ataport!IdePortPdoDispatch+0xa4
    807c08bc 82d356c3 877fe6c8 8a1e0830 8a1d9b80 ataport!IdePortDispatch+0x1d
    807c08e0 82a42473 00000000 8a1d9dd0 877fe6c8 nt!IovCallDriver+0x258
    807c08f4 851b1d85 8a1d9b80 ae18ae90 0000000e nt!IofCallDriver+0x1b
    807c0914 851b1975 8a1d9ac8 ae18ae90 0000000e CLASSPNP!ClasspPowerHandler+0x406
    807c092c 851b18dd 8a1d9ac8 ae18ae90 939c13d0 CLASSPNP!ClassSpinDownPowerHandler+0x8c
    807c0948 851ad3bf 8a1d9ac8 ae18ae90 ae18ae90 CLASSPNP!ClassDispatchPower+0x6c
    807c095c 82a11d16 8a1d9ac8 ae18ae90 95e56316 CLASSPNP!ClassGlobalDispatch+0x20
    807c0974 82d356b3 ae18af40 8a1d9ac8 ae18ae90 nt!IopPoHandleIrp+0x28
    807c0990 82a42473 00000000 8a1d61a0 8a1d9ac8 nt!IovCallDriver+0x248
    807c09a4 82a14049 807c09d0 84d44cfb 8a1d9ac8 nt!IofCallDriver+0x1b
    807c09ac 84d44cfb 8a1d9ac8 ae18ae90 8a4f6180 nt!IoCallDriver+0x10
    807c09d0 84d432e5 001d60e8 00000000 ae18ae90 partmgr!PmPower+0xa8
    807c09e4 82a11d16 8a1d60e8 ae18ae90 95e56316 partmgr!PmGlobalDispatch+0x1d
    807c09fc 82d356b3 ae18af64 8a1d60e8 ae18ae90 nt!IopPoHandleIrp+0x28
    807c0a18 82a42473 00000000 ae18af88 8a1d60e8 nt!IovCallDriver+0x248
    807c0a2c 82a14049 807c0a54 82d4734b 8a1d60e8 nt!IofCallDriver+0x1b
    807c0a34 82d4734b 8a1d60e8 ae18ae90 92c60a60 nt!IoCallDriver+0x10
    807c0a54 82a11d16 8a1d9a30 ae18ae90 95e56316 nt!ViFilterDispatchPower+0x5e
    807c0a6c 82d356b3 ae18af88 8a1d9978 8a1da048 nt!IopPoHandleIrp+0x28
    807c0a88 82a42473 00000000 ae18afac 8a1d9978 nt!IovCallDriver+0x248
    807c0a9c 82a14049 807c0ac4 84c36d94 8a1d9978 nt!IofCallDriver+0x1b
    807c0aa4 84c36d94 8a1d9978 ae18ae90 00000004 nt!IoCallDriver+0x10
    807c0ac4 84c36f45 8a1dab18 807c0ad8 8a1da048 Wdf01000!FxPkgFdo::_PowerPassDown+0x49
    807c0adc 84c38b4e 00000001 84c90d88 00000001 Wdf01000!FxPkgFdo::PowerReleasePendingDeviceIrp+0x3e
    807c0afc 84c38d68 807c0ba0 84c373bf 8a1da048 Wdf01000!FxPkgPnp::PowerGotoDxIoStopped+0x120
    807c0b04 84c373bf 8a1da048 00001000 84c90d60 Wdf01000!FxPkgPnp::PowerGotoDNotZeroIoStopped+0xd
    807c0ba0 84c3561e 8a1da048 0000031c 8a1da168 Wdf01000!FxPkgPnp::PowerEnterNewState+0x15c
    807c0bc8 84c365a1 8a1da048 807c0bf4 807c0c38 Wdf01000!FxPkgPnp::PowerProcessEventInner+0x167
    807c0c00 84c36efc 8a1da048 84c36bcc 8a1cec48 Wdf01000!FxPkgPnp::PowerProcessEvent+0x185
    807c0c08 84c36bcc 8a1cec48 8a1da048 807c0c44 Wdf01000!FxPkgFdo::DispatchDeviceSetPower+0x89
    807c0c18 84c31b07 8a1da048 807c0c38 8a1d9874 Wdf01000!FxPkgFdo::_DispatchSetPower+0x28
    807c0c44 84c28bc2 ae18ae90 8a1dab18 ae18ae90 Wdf01000!FxPkgPnp::Dispatch+0x1ad
    807c0c6c 84c28a33 8a1dab18 ae18ae90 8b2213d0 Wdf01000!FxDevice::Dispatch+0x155
    807c0c88 82a11d16 8a1dab18 ae18ae90 95e56316 Wdf01000!FxDevice::DispatchWithLock+0x77
    807c0ca0 82d356b3 ae18afac 8a1dab18 ae18ae90 nt!IopPoHandleIrp+0x28
    807c0cbc 82a42473 00000000 ae18afd0 8a1dab18 nt!IovCallDriver+0x248
    807c0cd0 82a14049 807c0cf8 82d4734b 8a1dab18 nt!IofCallDriver+0x1b
    807c0cd8 82d4734b 8a1dab18 ae18ae90 8a1dbe58 nt!IoCallDriver+0x10
    807c0cf8 82a11823 8a1dbf10 ae18ae90 00000000 nt!ViFilterDispatchPower+0x5e
    807c0d50 82c1466d 00000000 133ca796 00000000 nt!PopIrpWorker+0x351
    807c0d90 82ac60d9 82a114d2 00000000 00000000 nt!PspSystemThreadStartup+0x9e
    00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19


    FOLLOWUP_NAME:  MachineOwner

    FAILURE_BUCKET_ID:  0xc9_21a_VRF_IMAGE_disk.sys

    BUCKET_ID:  0xc9_21a_VRF_IMAGE_disk.sys

    Followup: MachineOwner

    And the info I get by using (!irp AE18AE90 )

    Irp is active with 7 stacks 2 is current (= 0xae18af24)
     No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  
         cmd  flg cl Device   File     Completion-Context
     [ 16, 2]   0 e0 8a1d9ac8 00000000 851b183f-8a1d9dd0 Success Error Cancel 
          \Driver\Disk CLASSPNP!ClasspStartNextPowerIrpCompletion
    Args: 00000000 00000001 00000004 00000000
    >[ 16, 2]   0 e1 8a1d9ac8 00000000 84d44d1e-00000000 Success Error Cancel pending
          \Driver\Disk partmgr!PmPowerCompletion
    Args: 00000000 00000001 00000004 00000000
     [ 16, 2]   0 e0 8a1d60e8 00000000 82d3bd3a-ae18af6c Success Error Cancel 
          \Driver\partmgr nt!IovpInternalCompletionTrap
    Args: 00000000 00000001 00000004 00000000
     [ 16, 2]   0 e0 8a1d9978 00000000 82d3bd3a-ae18af90 Success Error Cancel 
          \DRIVER\VERIFIER_FILTER nt!IovpInternalCompletionTrap
    Args: 00000000 00000001 00000004 00000000
     [ 16, 2]   0 e1 8a1dab18 00000000 82d3bd3a-ae18afb4 Success Error Cancel pending
          \Driver\diskfilter nt!IovpInternalCompletionTrap
    Args: 00000000 00000001 00000004 00000000
     [ 16, 2]   0 e1 8a1dbe58 00000000 00000000-00000000    pending
    Args: 00000000 00000001 00000004 00000000
     [  0, 0]   0  0 00000000 00000000 00000000-877fa830    

    Args: 00000000 00000000 00000000 00000000

    When my driver is loaded, sometime I get this bug. But after removing my driver this bug is not occurring. I did verifier on for every driver thought it might get reproduced. I am not handling Power irps thats why I am not able to understand why it is happening.
    Monday, March 3, 2014 9:29 AM
  • This could be an issue in CLASSPNP that is probably aggravated by your driver. Unfortunately it is difficult to do debugging over a forum discussion. Can you follow up with us over driverqualitytools@microsoft.com? Thanks.

    Monday, March 3, 2014 10:49 PM