none
KMDF bus driver pcie child devices RRS feed

  • Question

  • Hello experts!

    We are in the planning phase for a device stack that needs to support aggregation of multiple PCIe devices.  Each PCIe device will have its own interrupt and common buffer for S/G dma.  Blocks of data passed from the upper layers will need to be *demuxed to the appropriate pciE device (one block contains data for multiple PCIe devices).  We control the firmware for the PCIe devices so the vendorID etc will be the same for all devices.

    I have studied the KMDF Toaster samples and child device lists, etc, but am still unclear how exactly to proceed in this case.

    My thinking thus far: 

    - Create a bus driver that is root-enumerated (always loaded) and somehow get notifications of PCIe devices (of interest) from PnP.

    - When a device of interest (our device vid/pid) comes along, create a child device for it which will invoke loading of a function driver for the newly discovered / created PCIe child device.

    - After the creation of at least one child device, the bus driver will create a single PDO for a (software enumerated) virtual device which will handle the logistics of exposing a larger virtual composite buffer to the upper layers and mapping the pages of this buffer to the appropriate S/G lists of the child devices.

    * Not actually demuxed, but instead have the pages of the buffer placed in the appropriate child device's S/G list. 

    Should be able to get that done by dinner time, right?


    Thanks.

    Wednesday, April 2, 2014 3:27 PM

Answers

  • So you are writing the functional driver for this device?  I had assumed that this was an existing device and you needed to agregate.  If the drivers for the PCIe are comming from you then make this a single driver, that can handle multiple PCIe devices.

    Basically, you would on the first PCIe device create the FDO for the set of devices, then as additional PCIe devices appear they are integrated.  There is only one instance of a device driver in Windows, the data for a particular device is maintained in the device extension or in the context for the device in a KMDF driver.


    Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

    Wednesday, April 2, 2014 4:00 PM
  • As Don said, you can do all of this in the FDO driver, you have the "fun" of managing these devices coming online (and offline) in random order. If the handle that abstracts the N devices belongs to a device that goes away, you have a problem making that handle work with the remaining devices. An alternative is to have a root enumerated FDO (that is not a bus driver) represent the device abstraction, apps opens the root device, the root device maps to the N PCIe devices underneath by opening a handle to each and sending io to each appropriately

    d -- This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, April 2, 2014 5:27 PM
  • Yes for each device you will get the sequence you describe.


    Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

    • Marked as answer by Wade Dawson Wednesday, April 2, 2014 9:08 PM
    Wednesday, April 2, 2014 7:44 PM
  • if you are transforming the raw data from N PCIe devices into more baked data, especially data that needs to be presented to other device stacks, then yes, you need to enumerate child stacks from the root device that represents the composition of the PCIe devices.  Note you don't need a 1:1 mapping for these baked composite devices. you can also enable multiple device interface GUIDs on the parent root FDO if those interfaces do not require another driver stack to be layered on top of it.

    d -- This posting is provided "AS IS" with no warranties, and confers no rights.

    • Marked as answer by Wade Dawson Thursday, April 3, 2014 12:28 AM
    Wednesday, April 2, 2014 8:31 PM

All replies

  • This does not sound like something you want a bus driver for at all.  This appears to be a filter driver, that blocks the I/O for the individual PCIe devices and instead creates an agregate device for all the devices.   I assume that all the devices are of the same class so a class filter with some smarts is what is needed.

    Bus drivers are for taking a device and creating N devices, your description implies you are looking for N devices to become one.


    Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

    Wednesday, April 2, 2014 3:32 PM
  • Thanks, Don.  I see your point, but I'm not exactly sure how you would go about it.  My PCIe FD's lower edge is just memory accesses, so you'd layer the class filter above it then have it pass S/G info or whatever down to my PCIe FDO?  Do you have to do anything special to ensure that you only end up with one instance of the class driver, regardless of the child count?

    Thanks.

    Wednesday, April 2, 2014 3:53 PM
  • So you are writing the functional driver for this device?  I had assumed that this was an existing device and you needed to agregate.  If the drivers for the PCIe are comming from you then make this a single driver, that can handle multiple PCIe devices.

    Basically, you would on the first PCIe device create the FDO for the set of devices, then as additional PCIe devices appear they are integrated.  There is only one instance of a device driver in Windows, the data for a particular device is maintained in the device extension or in the context for the device in a KMDF driver.


    Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

    Wednesday, April 2, 2014 4:00 PM
  • As Don said, you can do all of this in the FDO driver, you have the "fun" of managing these devices coming online (and offline) in random order. If the handle that abstracts the N devices belongs to a device that goes away, you have a problem making that handle work with the remaining devices. An alternative is to have a root enumerated FDO (that is not a bus driver) represent the device abstraction, apps opens the root device, the root device maps to the N PCIe devices underneath by opening a handle to each and sending io to each appropriately

    d -- This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, April 2, 2014 5:27 PM
  • Yes. We own the functional driver.  Your statement:

    "then as additional PCIe devices appear they are integrated..." is the crux of my question:

    So does that mean that in a given WDF driver (matched to the PCI vid/pid) , I'll get a evtDeviceAdd, followed by an evtPrepareHardware for each of our PCI devices? 

    -Wade

    Wednesday, April 2, 2014 6:27 PM
  • Hi Doron and thanks. So in the root-enumerated driver, I would use IoRegisterPlugPlayNotification or some variation to get notifications of my PCIe devices coming and going, manage a list of handles, etc, then do I/O to them using IOCTLs or DDI, etc.  Is it any problem for the root-enumerated device to be a bus driver?  The reason I believe it needs to be a bus driver is that it needs to create several virtual devices against which Kernel streaming miniports and other drivers will be loaded via INFs.

    Thanks again for your responses!

    Wednesday, April 2, 2014 6:58 PM
  • Yes for each device you will get the sequence you describe.


    Don Burn Windows Filesystem and Driver Consulting Website: http://www.windrvr.com Blog: http://msmvps.com/blogs/WinDrvr

    • Marked as answer by Wade Dawson Wednesday, April 2, 2014 9:08 PM
    Wednesday, April 2, 2014 7:44 PM
  • if you are transforming the raw data from N PCIe devices into more baked data, especially data that needs to be presented to other device stacks, then yes, you need to enumerate child stacks from the root device that represents the composition of the PCIe devices.  Note you don't need a 1:1 mapping for these baked composite devices. you can also enable multiple device interface GUIDs on the parent root FDO if those interfaces do not require another driver stack to be layered on top of it.

    d -- This posting is provided "AS IS" with no warranties, and confers no rights.

    • Marked as answer by Wade Dawson Thursday, April 3, 2014 12:28 AM
    Wednesday, April 2, 2014 8:31 PM
  • Hi Don, Doron,

    I also have a similar requirement to aggregate multiple physical devices with same VID into a single unified controller DO to which clients can get handle. I understand from your above suggestions that we need to have a separate DO which does not correspond to any particular physical device for this. I guess this can be a non-PnP CDO (control device object) or a virtual root-enumerated controller device, though I think latter is better to use because in that way you can support device interface notifications to clients.

    The follow-up question I have is that can I have the same driver expose the root enumerated virtual PnP controller device and also support multiple physical devices, instead of having different drivers for both ? Do you perceive such solution as non-standard with which we can get into any potential issues in future ? Also, is there any bad WHQL certification implications if we go for such approach ? Basically we are more inclined to have a solution without introducing an extra driver.

    Also, do we have any drivers in Windows world that follows similar approach which I outlined here ?

    Thanks & Regards,

    Harvijay


    • Edited by Harvijay Singh Thursday, October 15, 2015 12:40 AM seeking additional clarifications
    Thursday, October 15, 2015 12:00 AM
  • You should be able to support both devices with a single driver.  You do need to have smarts to handle the fact that physical devices have resources (registers, ports, interrupts) and virtual devices do not, but within those constraints you can do it.


    Don Burn Windows Driver Consulting Website: http://www.windrvr.com

    Friday, October 16, 2015 9:22 PM
  • Thanks Don for your inputs, really appreciated! And can you or Doron Holon please share some inputs on my other queries also ? That will help me to become more confident to go forward with this solution. 
    Friday, October 16, 2015 11:32 PM