none
Posible cache issue in KMDF PCIe bus master DMA device driver RRS feed

  • Question

  • I have written a PCIe device driver for a bus master DMA device.  Once programmed and started, the bmdma device basically continually traverses a list of buffer descriptors that describe segments of physical pages that are part of a large audio ring buffer.  The device is programmed with a pointer to a DMA common buffer that consists of 64 bit entries (descriptors) that contain a physical address and several control bits for each buffer of data to be moved across the pcie bus.  The device can generate an MSI IRQ when it processes a descriptor with a specific control bit set.  To program the device the list of descriptors is created to describe a ring buffer allocated with the following code:

            SIZE_T RingSizeBytes = kRingMaxSamples*kRingMaxChans*sizeof(UINT32);
            PHYSICAL_ADDRESS low, high, skip;
            low.QuadPart = 0;
            high.QuadPart = 0xFFFFFFFFFFFFFFFF;
            skip.QuadPart = PAGE_SIZE;
            mRingBufferMDL = MmAllocatePagesForMdlEx(low, high, skip, RingSizeBytes, MmNonCached, 0);
            if (!mRingBufferMDL)
            {
                ExRaiseStatus(status);
            }
            // Map the buffer into kernel VA space
            mRingBufferVA = MmMapLockedPagesSpecifyCache(mRingBufferMDL, KernelMode, MmNonCached, NULL, FALSE, NormalPagePriority);

    The code then builds the descriptor list to describe the ring buffer using this function:

    UINT64 PcieAudioStream::GetRingBuffPhysicalPage(UINT32 offset)
    {
        UINT64 ppage=0;
        if (mRingBufferMDL == NULL)
        {
            return 0;
        }
    
        PUINT64 pfnList = MmGetMdlPfnArray(mRingBufferMDL);    // Init pointer to PFN array.  pfn = physical address >> 12
        UINT32 maxMdlIdx = ADDRESS_AND_SIZE_TO_SPAN_PAGES(MmGetMdlVirtualAddress(mRingBufferMDL), MmGetMdlByteCount(mRingBufferMDL));
        UINT32 mdlIdx = offset/PAGE_SIZE;
        if (mdlIdx >= maxMdlIdx)
        {
            return 0;
        }
        ppage = pfnList[mdlIdx] << PAGE_SHIFT;
        return ppage;
    }


    Additionally, the device accepts commands and returns responses in another common buffer, created :

    ...
            // Create Common Buffer for commands / responses
            WDF_COMMON_BUFFER_CONFIG  commonBufConfig;
            WDF_COMMON_BUFFER_CONFIG_INIT(&commonBufConfig, (kPageSize - 1));   // Align on page boundary
            status = WdfCommonBufferCreateWithConfig(mDmaEnabler, kPageSize, &commonBufConfig, WDF_NO_OBJECT_ATTRIBUTES, &mWdfDcpResponseCommonBuffer);
            if (!NT_SUCCESS(status))
            {
                ExRaiseStatus(status);
            }
            mDcpResponseBuffer = (PUINT8) WdfCommonBufferGetAlignedVirtualAddress(mWdfDcpResponseCommonBuffer);
    ...

    For commands / responses, the driver writes to the common buffer, then sets a bit in the device's BAR0 space indicating to the device that a command is waiting.  The device processes the command and provides a response in the common buffer, then signals an interrupt to the driver.   The driver's  ISR queues a dpc which reads the response from the common buffer and triggers processing either immediately  or in a later in a passive workItem.

    The problem is that in a release (optimized for speed) build, its appears as though the PCIe posted write to the common area has not completed even though the PCIe posted write to signal the MSI IRQ has.  The device DOES NOT enable the relaxed ordering bit in DEVSTAT and therefore should be requiring strictly ordered writes in all transactions.  The reason I suspect this is that I can hard code a conditional break point, i.e. if(dmaBuf.someResponse != kValid) KdBreakPoint();  The breakpoint gets hit and when I examine the conditional expression it is not true.  Additionally, inserting a KeStallExecutionProcessor(100), before reading the common buffer response fixes the issue, at least for the target machine in question; for others who knows... 

    The WDFDMAENABLER used by the common buffer creation code is created like this:

            WDF_DMA_ENABLER_CONFIG   dmaConfig;
            WDF_DMA_ENABLER_CONFIG_INIT(&dmaConfig,
                WdfDmaProfileScatterGather64Duplex,
                kDescListSize);
    
            status = WdfDmaEnablerCreate(device,
                &dmaConfig,
                WDF_NO_OBJECT_ATTRIBUTES,
                &mDmaEnabler);

    So finally to my question. ;)

    Because I am not using Wdf DMA transactions per se, and (also have continuous DMA running) is there anything special that I need to do I.e., KeFlushIoBuffers() etc... at any point in the DMA for my command or continuous DMA operations to ensure that caching is either disabled or flushed as needed?  For all read/write to the BAR0 area I use READ_REGISTER_xxx and WRITE_REGISTER_xxx. 


    • Edited by Wade_Dawson Thursday, December 18, 2014 8:55 AM typos, carify
    Thursday, December 18, 2014 8:42 AM

Answers

  • I have resolved the issue and it appears to be related to the pcie write that signals the MSI interrupt being issued by the hardware the actual write to populate the data they tha DPC will read.

     
    • Marked as answer by Wade_Dawson Thursday, January 8, 2015 3:35 PM
    Thursday, January 8, 2015 3:35 PM

All replies

  • Exactly.  For the common buffer used to hold the descriptor (written from the host side once and read from the device) all seems to be fine.  The issue occurs (of course) when memory is written by the device and read asynchronously by the host.  And as you have pointed out, there seems to be no way to control the caching using wdf common buffers.  I have, however, using the the !pte and !pfn commands examined the caching bits of the pages in question and they are displayed as non-cahced.  Im still unclear as to how many levels of cache may be involved here and if those bits are relevant to anything other than the windows memory / cache manager which I do not believe is involved here.

    Friday, December 19, 2014 1:06 PM
  • I have resolved the issue and it appears to be related to the pcie write that signals the MSI interrupt being issued by the hardware the actual write to populate the data they tha DPC will read.

     
    • Marked as answer by Wade_Dawson Thursday, January 8, 2015 3:35 PM
    Thursday, January 8, 2015 3:35 PM