none
PCIe-Bus and NUMA-Node Correlation RRS feed

  • Question

  • Hello,

    we need to handle NUMA setups and need to optimize I/O performance in our application. Therefore we have to identify if a PCIe device is [a] connected to a CPU's PCIe root complex or [b] to the chipset's PCIe root complex (e.g. via DMI). For case [a] we also need to identify the NUMA node number of the connected CPU.

    It is quite simple to enumerate all the PCI busses and devices, we can also detect that we are running on a NUMA system or not. But we could not find how to automatically distinguish the different types of PCIe ports and/or their hierarchy (i.e. CPU0 -> DMI -> PCH -> PCIe-root). PCIe root ports are all listed in a "flat hierarchy" as system devices per PCI bus and these busses have no specific parent. Is there any chance to find/detect the required information ([a] and [b]) programatically? Do the devices have some properties which allow detailed identification?

    We know it would be possible to identify all the devices by their PCI device id, but this would require to maintain a database of all past, current, and future device ids and provide an update service just for that - a huge effort. Using the devices' friendly names instead would be risky and unstable and not portable. Moreover, the solution should not only work for Intel systems but also for AMD and possible others.

    We read the related discussion CPU-socket and PCI-slot identification but this could not yet solve our problem.

    Can anyone help? Thanks!

    Friday, April 24, 2015 9:16 AM

Answers

  • I totally agree with Vegan that you need to check the assumption.  Depending on the target for the device (server versus desktop) plus the level of performance, you may find that your assumption is completely incorrect.  Unfortunately, ACPI will not indicate this to you since it below the level of the driver which is what you are assuming is NUMA aware.  As a correction also, Plug and Play is not a BIOS extension, for some legacy devices there were BIOS extensions, but in general Plug and Play is not related to BIOS.

    If the NDIS driver does not reference NdisGetRssProcessorInformation then you can pretty well say it is not NUMA aware, but a reference is not proof of being NUMA friendly.  The only way to determine for sure is to ask the vendor what they do.


    Don Burn Windows Driver Consulting Website: http://www.windrvr.com

    Monday, April 27, 2015 1:18 PM

All replies

  • Most servers have even numbers of DIMMs installed but not all, which makes it messy for applications as much as it does for lower level layers.

    https://msdn.microsoft.com/en-us/library/windows/desktop/ms724379%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

    Hope you are up on COM, I use that sometimes for lower level calls.

    So how low down the onion are you working at. Much lower and you are a BIOS developer.

    For a higher level, OpenMP allows me to ask the machine now many cores when on bare metal, hyper-v will show the allocated cores



    MSFT Signature

    Place your rig specifics into your signature like I have, makes it 100x easier!

    Hardcore Games Legendary is the Only Way to Play!
    Vegan Advocate How can you be an environmentalist and still eat meat?


    Friday, April 24, 2015 12:06 PM
  • Are you the one writing the device driver?  If so use IoGetNumaNode to get the devices node, i.e the one it is connected to.  Worst case you will need a way to query the device (such as an IOCTL) so you can set your application to the appropriate nodes.  

    If you are not writing the device driver, do you even know if the device driver is NUMA aware?  If it is not, then trying to optimize things at the at the application level is of questionable value, since the driver may be creating buffers without regards to NUMA, or scheduling actions without those considerations.

    Tell us more about the configuration, and goals, the forum then may be able to give you better guidance.


    Don Burn Windows Driver Consulting Website: http://www.windrvr.com

    Friday, April 24, 2015 1:16 PM
  • You want the CM_* APIs, such as CM_Get_Parent and CM_Get_DevNode_Property. These will allow you to traverse the device tree. I've use these (along with the SetupDI* APIs) to walk the PCIe tree to identify root complexes

     -Brian


    Azius Developer Training www.azius.com Windows device driver, internals, security, & forensics training and consulting. Blog at www.azius.com/blog


    Friday, April 24, 2015 8:07 PM
    Moderator
  • Thank you guys for your helpful answers. We are currently evaluating the ACPI approach...

    Some more details about our goals:

    We are not writing a device driver but an "normal" application running in user space. If required, we can execute our PCIe configuration/detection with administrator privileges. The device driver of our first use case (Intel Ethernet adapters) is NUMA-aware - at least if you configure RSS manually onto the correct nodes. But the same approach needs to work later for other PCIe devices, too. We can act on the assumption, that these device's drivers will also be NUMA-aware.

    Monday, April 27, 2015 8:20 AM
  • Thank you guys for your helpful answers. We are currently evaluating the ACPI approach...

    Some more details about our goals:

    We are not writing a device driver but an "normal" application running in user space. If required, we can execute our PCIe configuration/detection with administrator privileges. The device driver of our first use case (Intel Ethernet adapters) is NUMA-aware - at least if you configure RSS manually onto the correct nodes. But the same approach needs to work later for other PCIe devices, too. We can act on the assumption, that these device's drivers will also be NUMA-aware.

    You might want to enumerate to be sure, not a good idea to make an assumption

    https://msdn.microsoft.com/en-us/library/windows/hardware/Dn614028(v=vs.85).aspx

    ACPI is an extension to the BIOS along with Plug & Pray



    MSFT Signature

    Place your rig specifics into your signature like I have, makes it 100x easier!

    Hardcore Games Legendary is the Only Way to Play!
    Vegan Advocate How can you be an environmentalist and still eat meat?

    Monday, April 27, 2015 11:54 AM
  • I totally agree with Vegan that you need to check the assumption.  Depending on the target for the device (server versus desktop) plus the level of performance, you may find that your assumption is completely incorrect.  Unfortunately, ACPI will not indicate this to you since it below the level of the driver which is what you are assuming is NUMA aware.  As a correction also, Plug and Play is not a BIOS extension, for some legacy devices there were BIOS extensions, but in general Plug and Play is not related to BIOS.

    If the NDIS driver does not reference NdisGetRssProcessorInformation then you can pretty well say it is not NUMA aware, but a reference is not proof of being NUMA friendly.  The only way to determine for sure is to ask the vendor what they do.


    Don Burn Windows Driver Consulting Website: http://www.windrvr.com

    Monday, April 27, 2015 1:18 PM
  • Plug & Pray surfaced with Windows 95 and it arrived with the PCI bus. It was an add-on layer to enumerate the hardware so that the old jumper days would come to an end.

    It is loaded the same as the hard disk, but at a different layer. This is why the firmware now is so big, BIOS, PnP, ACPI and so on.

    I suggest as further reading, the kernal32 API

    http://www.inf.unideb.hu/~fazekasg/english/Operating_Systems_I_II/Win_kernelref.pdf

    some light reading to get your Monday off to a good start



    MSFT Signature

    Place your rig specifics into your signature like I have, makes it 100x easier!

    Hardcore Games Legendary is the Only Way to Play!
    Vegan Advocate How can you be an environmentalist and still eat meat?

    Monday, April 27, 2015 5:14 PM
  • Plug and Play is a component of the OS, it is not BIOS.  As far Plug and Play, when it was rolled out to developers at a one day session at the end of WinHEC 1997 it was explained it was driven by USB, where devices would dynamically be added and removed.   Windows worked fine with PCI well before that, I worked on several PCI drivers before that time.

    Why you would recommend a list of kernel API's (which has a number of known problems) when we are talking about Plug and Play is beyond me.


    Don Burn Windows Driver Consulting Website: http://www.windrvr.com

    Monday, April 27, 2015 5:28 PM
  • I provided the reference for the OP to read, I recommend everyone be more familiar with the lower levels in Windows.

    PnP is partially firmware and then handled by Windows at the device level.

    I had a PCI machine with my old Pentium MMX 200 which cost a mint. Came with Windows 95 on a CD.

    The USB was so new not even a mouse was available. 2 ports on the back, no front USB.

    I recall the BIOS called itself a PnP BIOS with that old EPA graphic that was around in the told days.



    MSFT Signature

    Place your rig specifics into your signature like I have, makes it 100x easier!

    Hardcore Games Legendary is the Only Way to Play!
    Vegan Advocate How can you be an environmentalist and still eat meat?

    Monday, April 27, 2015 6:03 PM
  • http://en.wikipedia.org/wiki/Legacy_Plug_and_Play


    MSFT Signature

    Place your rig specifics into your signature like I have, makes it 100x easier!

    Hardcore Games Legendary is the Only Way to Play!
    Vegan Advocate How can you be an environmentalist and still eat meat?

    Monday, April 27, 2015 6:05 PM
  • Plug and Play BIOS was a specification for older legacy hardware to allow identification of the device.  Modern device buses such as PCI, USB, 1394, etc have the identification built in.  ACPI BIOS does assist in modern systems with PnP, but Plug and Play is code in the OS. 


    Don Burn Windows Driver Consulting Website: http://www.windrvr.com

    Monday, April 27, 2015 6:15 PM
  • ACPI is now taking over from PnP but uPnP is still being used for networks etc.

    Which is why I first offered the ACPI reference earlier. Plug and Pray and now on the way out as ISA slots are long extinct and PCI are nearly so.

    PCI Express is the slot de jour until something else comes along.

    PCI and USB, Firewire etc are all part of the PnP legacy with ACPI now now being mature and the interface of choice.

    Hope the OP has enough background to be better able to move forward



    MSFT Signature

    Place your rig specifics into your signature like I have, makes it 100x easier!

    Hardcore Games Legendary is the Only Way to Play!
    Vegan Advocate How can you be an environmentalist and still eat meat?

    Monday, April 27, 2015 6:44 PM
  • News - we found a solution! My colleague posted in parallel at Stack Overflow and we got some hints that worked, at least on the machines we tested.

    Thank you for your help!

    Tuesday, May 5, 2015 4:50 PM