locked
IMFMediaEngine performance issues on ARM tablet

    Question

  • I have implemented an in-game media player using Media Foundation's IMFMediaEngine following these two examples:

    http://code.msdn.microsoft.com/windowsapps/Media-Engine-Playback-ce1c82f0
    http://code.msdn.microsoft.com/wpapps/Media-engine-sample-0bd96b86

    I use the second example's method of rendering the video frames to geometry instead of transferring them directly to the swap chain. This is because the video player is running within a more complex app. The video doesn't take up the entire screen and there are UI elements that are displayed and responsive while the video is playing.

    Sharing a single Direct3D device between the IMFMediaEngine and the rest of the app via the MF_MEDIA_ENGINE_DXGI_MANAGER attribute causes severe instability in Windows Store apps (though it works fine on Windows Phone apps). To work around this, I create a second Direct3D device specifically for use by the Media Engine. This device must be created with the driver type set to D3D_DRIVER_TYPE_WARP. If the driver type is set to higher-performance D3D_DRIVER_TYPE_HARDWARE, the call to TransferVideoFrame fails silently-- the destination texture remains empty, but the method call returns S_OK.

    This setup works fine on x86 and x64 desktop environments and x86 tablets, but results in significant video latency on ARM tablets. Specifically, the call to TransferVideoFrame takes an extremely long time to complete. The same latency is observed when running the first example project on an ARM tablet and forcing it to use a WARP driver (though notably not when allowing it to use the HARDWARE driver).

    I have tested this problem with videos encoded with various codecs including mp4v, WVC1 and avc1 and observed no significant difference. Specific sample videos I've used can be found in the following locations:

    http://www.h264info.com/clips.html
    http://archive.org/details/Windows7WildlifeSampleVideo

    Ideally we'd be able to coerce the hardware driver into rendering to geometry, but another option would be to acquire and render to an independent swap chain. Unfortunately there is little to no information about how to create a second window (and its associated swap chain) in Windows Store apps. The only information I've found that suggests this is possible is two slides from Chas Boyd's //build/ presentation titled "Core technologies for Windows 8 games":

    http://channel9.msdn.com/Events/Build/2012/3-107

    His 36th slide mentions "Multiple CoreWindows in the same view - Each with its own SwapChain", which could work as a solution to our problem. Unfortunately, this point is not elaborated on in his talk.


    Any ideas or advice regarding this problem would be appreciated.
    Thanks!
    Tuesday, December 04, 2012 11:14 PM

Answers

  • "Sharing a single Direct3D device between the IMFMediaEngine and the rest of the app via the MF_MEDIA_ENGINE_DXGI_MANAGER attribute causes severe instability in Windows Store apps (though it works fine on Windows Phone apps)."

     

    Hi, you can try to enable some multithreaded optimizations.

    Do not set :

    D3D11_CREATE_DEVICE_SINGLETHREADED

    Turn multithreading on :

    ID3D10Multithread::SetMultithreadProtected

    And wait for vertical blank :

    IDXGISwapChain1::Present1




    • Edited by t-n-x Wednesday, December 05, 2012 12:47 AM
    • Marked as answer by Colin J. MacDougall Thursday, December 06, 2012 4:23 PM
    Wednesday, December 05, 2012 12:45 AM

All replies

  • "Sharing a single Direct3D device between the IMFMediaEngine and the rest of the app via the MF_MEDIA_ENGINE_DXGI_MANAGER attribute causes severe instability in Windows Store apps (though it works fine on Windows Phone apps)."

     

    Hi, you can try to enable some multithreaded optimizations.

    Do not set :

    D3D11_CREATE_DEVICE_SINGLETHREADED

    Turn multithreading on :

    ID3D10Multithread::SetMultithreadProtected

    And wait for vertical blank :

    IDXGISwapChain1::Present1




    • Edited by t-n-x Wednesday, December 05, 2012 12:47 AM
    • Marked as answer by Colin J. MacDougall Thursday, December 06, 2012 4:23 PM
    Wednesday, December 05, 2012 12:45 AM
  • That worked, after a fashion.

    The Direct3D device being used must be created with D3D11_CREATE_DEVICE_BGRA_SUPPORT (and possibly D3D11_CREATE_DEVICE_VIDEO_SUPPORT, but I haven't tested without this flag so I don't know for sure if it's necessary). Additionally, I get the best performance when I set the texture dimensions exactly equal to the video dimensions as given by IMFMediaEngine::GetNativeVideoSize. Codecs are somewhat limited, but WVC1 and avc1 work.

    Thanks for your help!

    Thursday, December 06, 2012 4:29 PM