How to analyze crash dump when your driver name does not show in call stack ?

Question

I used !analyze -v and I got :

-----------------------------------------------------------

DRIVER_VERIFIER_IOMANAGER_VIOLATION (c9)
The IO manager has caught a misbehaving driver.
Arguments:
Arg1: 00000012, IRQL not equal across call to driver dispatch routine
Arg2: 84c049bc, Driver dispatch routine address.
Arg3: 00000000, IRQL before calling driver dispatch routine.
Arg4: 00000002, Current IRQL.

Debugging Details:
------------------


BUGCHECK_STR:  0xc9_12

DRIVER_VERIFIER_IO_VIOLATION_TYPE:  12

FAULTING_IP: 
Wdf01000!FxDevice::DispatchWithLock+0
84c049bc 8bff            mov     edi,edi

FOLLOWUP_IP: 
volmgr!VmReadWrite+1a8
84d309ae 8bf8            mov     edi,eax

PREVIOUS_IRQL:  0

CURRENT_IRQL:  2

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

PROCESS_NAME:  System

LAST_CONTROL_TRANSFER:  from 82af7e71 to 82a86394

STACK_TEXT:  
ae82f58c 82af7e71 00000003 4c983bba 00000065 nt!RtlpBreakWithStatusInstruction
ae82f5dc 82af896d 00000003 8a5f8498 8a5f84fc nt!KiBugCheckDebugBreak+0x1c
ae82f9a0 82af7d10 000000c9 00000012 84c049bc nt!KeBugCheck2+0x68b
ae82f9c4 82b2a842 000000c9 00000012 84c049bc nt!KeBugCheckEx+0x1e
ae82f9dc 82d4fdab 8a5f8498 00000000 ae82fa10 nt!VfBugCheckNoStackUsage+0x23
ae82f9ec 82d4a6d4 8a5f84fc 00000000 8a856ef8 nt!VfAfterCallDriver+0x66
ae82fa10 82a57473 00000000 8a856f1c 8a1d23b8 nt!IovCallDriver+0x269
ae82fa24 82d5c3d0 8a5f83d0 8a856db8 8a1d6910 nt!IofCallDriver+0x1b
ae82fa3c 82d4a6c3 8a1d69c8 8a856db8 a07d6000 nt!ViFilterDispatchGeneric+0x5e
ae82fa60 82a57473 00000000 8a1e0ed8 8a1d6910 nt!IovCallDriver+0x258
ae82fa74 84d309ae 8a5e0c70 8a856db8 8a1e0e20 nt!IofCallDriver+0x1b
ae82fa90 82d4a6c3 8a1e0e20 8a856db8 8a1e60d8 volmgr!VmReadWrite+0x1a8
ae82fab4 82a57473 00000000 8a856db8 8a1e0e20 nt!IovCallDriver+0x258
ae82fac8 8515c475 8a856db8 ae82faf0 8515c547 nt!IofCallDriver+0x1b
ae82fad4 8515c547 8a1e60d8 8a856db8 8a856db8 fvevol!FveRequestPassThrough+0x31
ae82faf0 8515c759 8a1e60d8 8a856d01 8a59a0d8 fvevol!FveReadWrite+0x4d
ae82fb20 8515c7a9 8a1e6020 8a856db8 ae82fb54 fvevol!FveFilterRundownReadWrite+0x197
ae82fb30 82d4a6c3 8a1e6020 8a856db8 8a856f64 fvevol!FveFilterRundownWrite+0x33
ae82fb54 82a57473 00000000 8a1e5474 8a1e6020 nt!IovCallDriver+0x258
ae82fb68 853d676e 8a1e5418 8a856db8 00000003 nt!IofCallDriver+0x1b
ae82fc48 853d68a5 8a1e5418 8a856db8 8b1c94c8 rdyboost!SmdProcessReadWrite+0xa14
ae82fc68 82d4a6c3 8a1e5418 8a856db8 8a856f88 rdyboost!SmdDispatchReadWrite+0xcb
ae82fc8c 82a57473 00000000 8a856fac 8a1e5360 nt!IovCallDriver+0x258
ae82fca0 85386fd9 8aa24a88 8a856db8 8a1e6660 nt!IofCallDriver+0x1b
ae82fcc8 853872fd 001e6660 8a856db8 ae82fcfc volsnap!VolsnapWriteFilter+0x265
ae82fcd8 82d4a6c3 8a1e6660 8a856db8 ae82fd28 volsnap!VolSnapWrite+0x21
ae82fcfc 82a57473 00000000 874f269c 8a1e6660 nt!IovCallDriver+0x258
ae82fd10 84e3591c 874f25fc ae82fd34 82a8b10e nt!IofCallDriver+0x1b
ae82fd1c 82a8b10e 874f269c 00000000 ffffffff Ntfs!NtfsStorageDriverCallout+0x14
ae82fd1c 82a8b205 874f269c 00000000 ffffffff nt!KiSwapKernelStackAndExit+0x15a
874f260c 82aab0fd 874f269c 84e35908 ae830000 nt!KiSwitchKernelStackAndCallout+0x31
874f2680 84e34939 84e35908 874f269c 00000000 nt!KeExpandKernelStackAndCalloutEx+0x29d
874f26ac 84e355a6 8a1e6660 8a856db8 8551f618 Ntfs!NtfsCallStorageDriver+0x2d
874f26f0 84e340a0 8b866bb8 8a1e6660 8a856db8 Ntfs!NtfsMultipleAsync+0x4d
874f2820 84e330a6 8b866bb8 8a856db8 a67a7500 Ntfs!NtfsNonCachedIo+0x413
874f2938 84e3485f 8b866bb8 8a856db8 03a932ff Ntfs!NtfsCommonWrite+0x1ebd
874f29b0 82d4a6c3 8a498020 8a856db8 00000000 Ntfs!NtfsFsdWrite+0x2e1
874f29d4 82a57473 00000000 8a856db8 8a498020 nt!IovCallDriver+0x258
874f29e8 84b9d20c 8a555e30 8a856db8 00000000 nt!IofCallDriver+0x1b
874f2a0c 84b9d3cb 874f2a2c 8a555e30 00000000 fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x2aa
874f2a44 82d4a6c3 8a555e30 8a856db8 8b858570 fltmgr!FltpDispatch+0xc5
874f2a68 82a57473 00000000 8a856db8 8a555e30 nt!IovCallDriver+0x258
874f2a7c 82c58eee 8b858570 8a856db8 8a856fd8 nt!IofCallDriver+0x1b
874f2a9c 82c597a2 8a555e30 8b858570 00000001 nt!IopSynchronousServiceTail+0x1f8
874f2b38 82a5e42a 8a555e30 00000000 00000000 nt!NtWriteFile+0x6e8
874f2b38 82a5d895 8a555e30 00000000 00000000 nt!KiFastCallEntry+0x12a
874f2bd4 82c0a652 800002d0 00000000 00000000 nt!ZwWriteFile+0x11
874f2c1c 82c5bf4c 8a0c0000 8a0c0000 00000000 nt!EtwpFlushBufferToLogfile+0x81
874f2c3c 82a8dbc6 00000000 00000001 855546c0 nt!EtwpFlushBuffer+0xc3
874f2d08 82c4d9da 855546c0 00000000 00000000 nt!EtwpFlushActiveBuffers+0x2c0
874f2d50 82c2966d 855546c0 6555e3f6 00000000 nt!EtwpLogger+0x2a1
874f2d90 82adb0d9 82c4d739 855546c0 00000000 nt!PspSystemThreadStartup+0x9e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19


STACK_COMMAND:  kb

SYMBOL_STACK_INDEX:  b

SYMBOL_NAME:  volmgr!VmReadWrite+1a8

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: volmgr

IMAGE_NAME:  volmgr.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  4a5bbf1d

FAILURE_BUCKET_ID:  0xc9_12_VRF_volmgr!VmReadWrite+1a8

BUCKET_ID:  0xc9_12_VRF_volmgr!VmReadWrite+1a8

Followup: MachineOwner

-----------------------------------------------------------

Here I know that my driver did something wrong what if I did not know which driver cause this bug , in that case how do I analyze such type of dump?

Answer 1

First depending on the type of driver have you run Static Driver Verifier over it?  This tool can find a lot of the problems that lead to this sort of problem.

I see you are running with Driver Verifier, you might want to run it without your driver present but all other drivers enabled, just to be sure that something else is not misbehaving?

Now what sort of driver do you have?  Can it have an impact on a WriteFile call which is what is producing this failure?  If yes, then that is the first thing to look at, if not then things are going to get messier quickly since a likely scenario is a bad pointer dereference.

The other approach people have used is slowly disabling functionality in their driver till the problem goes away.  Then desk checking that part of the driver like crazy.

These can be hard bugs to find, since your driver and the rest of the kernel are sharing memory space and many other things.

 

---

Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

Answer 2

I am writing a disk cache filter driver. I got this bug when through ioctl I disable cache policy.

Thanks, I got to know the problem is in my write callback.

After disabling cache, whenever a write request comes I simply pass the request down but at this point I was not releasing a spin lock thats why the bug occurred.

I have another problem that I want to discuss with you.

Driver Verifier detected violation:

Previously set IRP_MJ_POWER status has been converted to
STATUS_NOT_SUPPORTED. This failure status is reserved for use of the OS
- drivers cannot fail a Power IRP with this value.

CulpritAddress = 82A45049, Irp = 86B7AE90.
************************************************************

Break instruction exception - code 80000003 (first chance)
nt!ViErrorFinishReport+0x4c:
82d6e24c cc              int     3

Whenever I use driver verifier on, sometime I got this error. I do not know why I got break in debugger. I tried to trace it but did not find anything. 

When I do not use verifier, I do not get this error

Is this might be due to my driver or not ? 

Do you have some information on it?

Answer 3

In the future, you can use “!verifier 0x8” to dump IRQL log. There is also a description in debugger help and MSDN.

http://msdn.microsoft.com/en-us/library/windows/hardware/ff565591(v=vs.85).aspx

Here is an example:

0: kd> !verifier 0x8

Displaying most recent 0x0000000000000004 entries from the IRQL transition log.
There are up to 0x100 entries in the log.

Thread:             ffffe001bec74080
Old irql:           0000000000000000
New irql:           0000000000000002
Processor:          0000000000000001
Time stamp:         0000000011432e48

    fffff80162373a55 fltmgr!FltFreeCallbackData+0x10f5
    fffff801623733fd fltmgr!FltFreeCallbackData+0xa9d
    fffff80162375f76 fltmgr!FltIsCallbackDataDirty+0x966
    fffff8016239e772 fltmgr!FltQueryInformationFile+0x72e
    fffff80161eef978 VerifierExt!xdv_IRP_MJ_CREATE_wrapper+0xfc

Thread:             ffffe001bec74080
Old irql:           0000000000000001
New irql:           0000000000000000
Processor:          0000000000000001
Time stamp:         0000000011432e48

    fffff80162717243 Ntfs+0xc8243
    fffff801627152bb Ntfs+0xc62bb
    fffff801627147da Ntfs+0xc57da
    fffff80162713115 Ntfs+0xc4115
    fffff8016271144a Ntfs+0xc244a

Thread:             ffffe001bec74080
Old irql:           0000000000000000
New irql:           0000000000000001
Processor:          0000000000000001
Time stamp:         0000000011432e48

    fffff80162717205 Ntfs+0xc8205
    fffff801627152bb Ntfs+0xc62bb
    fffff801627147da Ntfs+0xc57da
    fffff80162713115 Ntfs+0xc4115
    fffff8016271144a Ntfs+0xc244a

Thread:             ffffe001bec74080
Old irql:           0000000000000001
New irql:           0000000000000000
Processor:          0000000000000001
Time stamp:         0000000011432e48

    fffff801627171b6 Ntfs+0xc81b6
    fffff801627152bb Ntfs+0xc62bb
    fffff801627147da Ntfs+0xc57da
    fffff80162713115 Ntfs+0xc4115
    fffff8016271144a Ntfs+0xc244a

Answer 4

This check is performed for a power IRP in both directions downward and upward (also for each device in the device stack). Can you check your IoCallDriver and IoComplete paths in debugger?

Answer 5

I am writing a kmdf based disk cache filter driver. I dont see any misbehavior in my driver. I handle only read and write requests wNS queue without power managed. Here is the dump if you could analyze :

DRIVER_VERIFIER_IOMANAGER_VIOLATION (c9)
The IO manager has caught a misbehaving driver.
Arguments:
Arg1: 0000021a, The previously-set IRP_MJ_POWER status has been converted to STATUS_NOT_SUPPORTED.
Arg2: 82a14049, The address in the driver's code where the error was detected.
Arg3: ae18ae90, IRP address.
Arg4: 00000000

Debugging Details:
------------------


ADDITIONAL_DEBUG_TEXT:  Bugcheck data extracted fron VfErrorBugcheckData.

BUGCHECK_STR:  0xc9_21a

DRIVER_VERIFIER_IO_VIOLATION_TYPE:  21a

FAULTING_IP: 
nt!IoCallDriver+10
82a14049 5d              pop     ebp

FOLLOWUP_IP: 
nt!IoCallDriver+10
82a14049 5d              pop     ebp

IRP_ADDRESS:  ae18ae90

DEVICE_OBJECT: 8a1d9ac8

DRIVER_OBJECT: 8a1d0f38

IMAGE_NAME:  disk.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  4a5bbf20

MODULE_NAME: disk

FAULTING_MODULE: 85198000 disk

DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

PROCESS_NAME:  System

CURRENT_IRQL:  2

LAST_CONTROL_TRANSFER:  from 82d3d32a to 82d3d24c

STACK_TEXT:  
807c061c 82d3d32a 0000021a 82a14049 00000000 nt!ViErrorFinishReport+0x4c
807c0670 82d441e9 0000021a ae18ae90 ae18af00 nt!VfErrorReport1+0x4d
807c0694 82d3cedb ae18af24 ae18af00 00000001 nt!VfPowerVerifyIrpStackDownward+0xbe
807c06b0 82d3b426 95e563a0 877fe6c8 ae18af24 nt!VfMajorVerifyIrpStackDownward+0x3e
807c0714 82d3ad33 8c8efa48 ae18ae90 807c0744 nt!IovpCallDriver1+0x468
807c0724 82d35670 877fe6c8 ae18af1c 877fe6c8 nt!VfBeforeCallDriver+0xe7
807c0744 82a42473 ae18af00 8a1d9b80 877fe6c8 nt!IovCallDriver+0x206
807c0758 82a14049 807c079c 851b272b 877fe6c8 nt!IofCallDriver+0x1b
807c0760 851b272b 877fe6c8 ae18ae90 851b253f nt!IoCallDriver+0x10
807c079c 82d35cd4 00000000 ae18af00 8a1d9dd0 CLASSPNP!ClasspPowerDownCompletion+0x1ec
807c07cc 82a6eb33 00000000 8a1e0830 807c0840 nt!IovpLocalCompletionRoutine+0x14b
807c0810 82d35b64 c0000010 8a1e0830 8a1e0830 nt!IopfCompleteRequest+0x128
807c0878 84c06d97 8a1d9df4 877fe780 807c08ac nt!IovCompleteRequest+0x133
807c0888 84c09fb2 877fe780 8a1e0830 8a1d9df4 ataport!IdeCompleteScsiIrp+0x31
807c08ac 84c0646b c0000185 8a1e0830 807c08e0 ataport!IdePortPdoDispatch+0xa4
807c08bc 82d356c3 877fe6c8 8a1e0830 8a1d9b80 ataport!IdePortDispatch+0x1d
807c08e0 82a42473 00000000 8a1d9dd0 877fe6c8 nt!IovCallDriver+0x258
807c08f4 851b1d85 8a1d9b80 ae18ae90 0000000e nt!IofCallDriver+0x1b
807c0914 851b1975 8a1d9ac8 ae18ae90 0000000e CLASSPNP!ClasspPowerHandler+0x406
807c092c 851b18dd 8a1d9ac8 ae18ae90 939c13d0 CLASSPNP!ClassSpinDownPowerHandler+0x8c
807c0948 851ad3bf 8a1d9ac8 ae18ae90 ae18ae90 CLASSPNP!ClassDispatchPower+0x6c
807c095c 82a11d16 8a1d9ac8 ae18ae90 95e56316 CLASSPNP!ClassGlobalDispatch+0x20
807c0974 82d356b3 ae18af40 8a1d9ac8 ae18ae90 nt!IopPoHandleIrp+0x28
807c0990 82a42473 00000000 8a1d61a0 8a1d9ac8 nt!IovCallDriver+0x248
807c09a4 82a14049 807c09d0 84d44cfb 8a1d9ac8 nt!IofCallDriver+0x1b
807c09ac 84d44cfb 8a1d9ac8 ae18ae90 8a4f6180 nt!IoCallDriver+0x10
807c09d0 84d432e5 001d60e8 00000000 ae18ae90 partmgr!PmPower+0xa8
807c09e4 82a11d16 8a1d60e8 ae18ae90 95e56316 partmgr!PmGlobalDispatch+0x1d
807c09fc 82d356b3 ae18af64 8a1d60e8 ae18ae90 nt!IopPoHandleIrp+0x28
807c0a18 82a42473 00000000 ae18af88 8a1d60e8 nt!IovCallDriver+0x248
807c0a2c 82a14049 807c0a54 82d4734b 8a1d60e8 nt!IofCallDriver+0x1b
807c0a34 82d4734b 8a1d60e8 ae18ae90 92c60a60 nt!IoCallDriver+0x10
807c0a54 82a11d16 8a1d9a30 ae18ae90 95e56316 nt!ViFilterDispatchPower+0x5e
807c0a6c 82d356b3 ae18af88 8a1d9978 8a1da048 nt!IopPoHandleIrp+0x28
807c0a88 82a42473 00000000 ae18afac 8a1d9978 nt!IovCallDriver+0x248
807c0a9c 82a14049 807c0ac4 84c36d94 8a1d9978 nt!IofCallDriver+0x1b
807c0aa4 84c36d94 8a1d9978 ae18ae90 00000004 nt!IoCallDriver+0x10
807c0ac4 84c36f45 8a1dab18 807c0ad8 8a1da048 Wdf01000!FxPkgFdo::_PowerPassDown+0x49
807c0adc 84c38b4e 00000001 84c90d88 00000001 Wdf01000!FxPkgFdo::PowerReleasePendingDeviceIrp+0x3e
807c0afc 84c38d68 807c0ba0 84c373bf 8a1da048 Wdf01000!FxPkgPnp::PowerGotoDxIoStopped+0x120
807c0b04 84c373bf 8a1da048 00001000 84c90d60 Wdf01000!FxPkgPnp::PowerGotoDNotZeroIoStopped+0xd
807c0ba0 84c3561e 8a1da048 0000031c 8a1da168 Wdf01000!FxPkgPnp::PowerEnterNewState+0x15c
807c0bc8 84c365a1 8a1da048 807c0bf4 807c0c38 Wdf01000!FxPkgPnp::PowerProcessEventInner+0x167
807c0c00 84c36efc 8a1da048 84c36bcc 8a1cec48 Wdf01000!FxPkgPnp::PowerProcessEvent+0x185
807c0c08 84c36bcc 8a1cec48 8a1da048 807c0c44 Wdf01000!FxPkgFdo::DispatchDeviceSetPower+0x89
807c0c18 84c31b07 8a1da048 807c0c38 8a1d9874 Wdf01000!FxPkgFdo::_DispatchSetPower+0x28
807c0c44 84c28bc2 ae18ae90 8a1dab18 ae18ae90 Wdf01000!FxPkgPnp::Dispatch+0x1ad
807c0c6c 84c28a33 8a1dab18 ae18ae90 8b2213d0 Wdf01000!FxDevice::Dispatch+0x155
807c0c88 82a11d16 8a1dab18 ae18ae90 95e56316 Wdf01000!FxDevice::DispatchWithLock+0x77
807c0ca0 82d356b3 ae18afac 8a1dab18 ae18ae90 nt!IopPoHandleIrp+0x28
807c0cbc 82a42473 00000000 ae18afd0 8a1dab18 nt!IovCallDriver+0x248
807c0cd0 82a14049 807c0cf8 82d4734b 8a1dab18 nt!IofCallDriver+0x1b
807c0cd8 82d4734b 8a1dab18 ae18ae90 8a1dbe58 nt!IoCallDriver+0x10
807c0cf8 82a11823 8a1dbf10 ae18ae90 00000000 nt!ViFilterDispatchPower+0x5e
807c0d50 82c1466d 00000000 133ca796 00000000 nt!PopIrpWorker+0x351
807c0d90 82ac60d9 82a114d2 00000000 00000000 nt!PspSystemThreadStartup+0x9e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19


STACK_COMMAND:  kb

FOLLOWUP_NAME:  MachineOwner

FAILURE_BUCKET_ID:  0xc9_21a_VRF_IMAGE_disk.sys

BUCKET_ID:  0xc9_21a_VRF_IMAGE_disk.sys

Followup: MachineOwner
--------------------------------------------------------------

And the info I get by using (!irp AE18AE90 )

Irp is active with 7 stacks 2 is current (= 0xae18af24)
 No Mdl: No System Buffer: Thread 00000000:  Irp stack trace.  
     cmd  flg cl Device   File     Completion-Context
 [ 16, 2]   0 e0 8a1d9ac8 00000000 851b183f-8a1d9dd0 Success Error Cancel 
      \Driver\Disk CLASSPNP!ClasspStartNextPowerIrpCompletion
Args: 00000000 00000001 00000004 00000000
>[ 16, 2]   0 e1 8a1d9ac8 00000000 84d44d1e-00000000 Success Error Cancel pending
      \Driver\Disk partmgr!PmPowerCompletion
Args: 00000000 00000001 00000004 00000000
 [ 16, 2]   0 e0 8a1d60e8 00000000 82d3bd3a-ae18af6c Success Error Cancel 
      \Driver\partmgr nt!IovpInternalCompletionTrap
Args: 00000000 00000001 00000004 00000000
 [ 16, 2]   0 e0 8a1d9978 00000000 82d3bd3a-ae18af90 Success Error Cancel 
      \DRIVER\VERIFIER_FILTER nt!IovpInternalCompletionTrap
Args: 00000000 00000001 00000004 00000000
 [ 16, 2]   0 e1 8a1dab18 00000000 82d3bd3a-ae18afb4 Success Error Cancel pending
      \Driver\diskfilter nt!IovpInternalCompletionTrap
Args: 00000000 00000001 00000004 00000000
 [ 16, 2]   0 e1 8a1dbe58 00000000 00000000-00000000    pending
      \DRIVER\VERIFIER_FILTER
Args: 00000000 00000001 00000004 00000000
 [  0, 0]   0  0 00000000 00000000 00000000-877fa830    

Args: 00000000 00000000 00000000 00000000

When my driver is loaded, sometime I get this bug. But after removing my driver this bug is not occurring. I did verifier on for every driver thought it might get reproduced. I am not handling Power irps thats why I am not able to understand why it is happening.

Answer 6

This could be an issue in CLASSPNP that is probably aggravated by your driver. Unfortunately it is difficult to do debugging over a forum discussion. Can you follow up with us over [e-mail]? Thanks.