locked
IMFSourceReader, IMFDXGIDeviceManager and texture2d formats

    Question

  • I am using a IMFSourceReader configured with a IMFDXGIDeviceManager to be able to support DXVA on video streams.  When I read a VC1, I can set the stream to MFVideoFormat_RGB32 and ReadSample gives me a 32bit RGB Texture2D (from IMFDXGIBuffer) as expected.  With an H264 stream, MFVideoFormat_RGB32 won't set on the IMFSourceReader video stream and must use MFVideoFormat_NV12.  This means IMFSourceReader::ReadSample gives me a Texture2D with the format of DXGI_FORMAT_NV12.

    Are there any built in methods to either get the source reader to deliver a DXGI_FORMAT_B8G8R8X8_UNORM (or similar) or built in MF methods to do the conversion on the GPU?  I did some some prototypes and it does look possible convert NV12 to via SRVs aliasing and a shader, but wanted to avoid if there's a built in way.

    Thanks!


    • Edited by jmorrill Friday, August 3, 2012 6:32 PM
    Friday, August 3, 2012 6:31 PM

Answers

  • Hi There,

    You're right, you don't need the hack above to make this work. You just need to configure the Source Reader (using IMFSourceReaderEx) with MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING attribute to achieve your goals. This should work with hardware that support D3D_FEATURE_LEVEL_9_3 and above provided the hardware supports NV12->XRGB32 conversion via VideoProcessBlt DDI. It's possible there are old hardware/display drivers that only support NV12->ARGB32 conversions in GPU, but even this should be very rare.

    Note that both IMFSourceReaderEx and MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING are new improvements for Windows 8 to enable GPU based video processing that should address your requirements.

    Hope this helps!

     


    Wednesday, August 8, 2012 11:11 PM

All replies

  • Hello Jmorrill,

    Please take a look at these two posts. If they don't answer your questions please let me know.

    http://social.msdn.microsoft.com/Forums/en-US/wingameswithdirectx/thread/e19906f4-afd5-455b-82ba-92c63addabec

    http://social.msdn.microsoft.com/Forums/en-US/wingameswithdirectx/thread/1cdbd573-13ad-4da9-ae61-96fc28f541b8

    I hope this helps,

    James


    Windows Media SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Friday, August 3, 2012 9:47 PM
    Moderator
  • Yeah I found those posts, which is where I got the hints.  I was just hoping I was either doing something incorrect to not get an rgb32 surface...and not have to write the conversion code.  In my prototype I was able to get the YUV color components, just have to write the code+shader to put them together in a rgb32 surface.

    If you are open to feedback, I see this as a common scenario and the solution is not very obvious.  I believe there is already a MFT to handle this, but it's "Desktop Only".  I'd be awesome if this functionality could be exposed to metro applications too!

    Thanks for the help!

    -Jer

    Friday, August 3, 2012 10:02 PM
  • I'm pretty close to converting the NV12 to an RGBA ID3D11Texture2D, using the hints from these forums.

    Here is a screenshot of  the issue I am having

    http://dl.dropbox.com/u/4165054/testYuv.jpg (image flipped with -height in bmp)

    The chroma (UV) is not lined up with the luminance (Y).  Also the luma isn't correctly lined up (green part at the bottom of the image).

    I can get everything correctly lined up by feeding magic numbers (eg adjusting the sampling location) to my pixel shader, but the magic numbers dont work for all video inputs of different resolutions).  For instance, 480p works great.  720p and 1080p have similar issues. Are there special consideration when using multiple ShaderResourceViews (R8 for Y and R8G8 for UV)?

    Thanks!



    • Edited by jmorrill Monday, August 6, 2012 9:01 PM
    Monday, August 6, 2012 8:54 PM
  • Hello Jer,

    Thanks for the feedback. Keep in mind that color space conversion can be extremely expensive. On ARM devices this could add up to decrease in battery life and possibly performance. Can you please give me details about why this conversion is necessary and the various scenarios that you expect for this functionality? If you can give me enough compelling reason for us to port this functionality I will certainly suggest it to the proper architects.

    -James 


    Windows Media SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Monday, August 6, 2012 9:32 PM
    Moderator
  • I would agree if this was a CPU operation, but the colorspace conversion will be happening on the GPU, where the cost is not much more than that of drawing a sprite (plus fill rate).

    The idea here, is to read video samples using IMFSourceReader (configured for DXVA decoding where available) and send those samples to the D3D/D2d rendering pipeline to do effects and compositing, then sends the result out to a decoder.  NV12 does not work directly with the D3D pipeline and instead need a DXGI_FORMAT_B8G8R8A8_UNORM.

    Anyone who wishes to do D2D/D3D processing with a DXVA enabled IMFSourceReader will have to do these types of conversions.

    If I understand correctly, IMFMediaEngine already does this for you internally but IMFMediaEngine + Video Effect API or TransferVideoFrame does not fit my use-case here because I need lower level API that IMFSourceReader provides to mix several video sources together.

    The end result would be something like this:

    (Multiple IMFSourceReaders) --> (Custom A/V Mixer) --> (Encoder Output)


    • Edited by jmorrill Monday, August 6, 2012 10:04 PM
    Monday, August 6, 2012 9:56 PM
  • Just an update:

    I was able to get IMFSourceReader to give me a DXGI_FORMAT_B8G8R8X8_UNORM surface(*).  I had to create an "inplace IMFTransform" that only took MediaFormat_RGB32.  It does no processing and just passes the input as the output sample.  Next I had to query IMFSourceReaderEx and run the IMFSourceReaderEx::AddTransformForStream, passing it the instantiated in-place IMFTransform following the instructions from MSDN (involving setting NV12, setting the mediaType, adding the transform, then setting the mediaType again with MediaFormat_RGB32).

    The IMFSourceReader now gets DXGI_FORMAT_B8G8R8X8_UNORM from ReadSample(...).  This seems like a weird "hack"...and, not totally obvious.

    *The main problem now, is this only works with D3D_FEATURE_LEVEL_10_1 or higher.  Way above the minimum requirements,  which are IIRC D3D_FEATURE_LEVEL_9_2, for Metro applications.  Since NV12 cannot be used as a ShaderResourceView in the D3D_FEATURE_LEVEL_9 feature levels, I feel like I will have to do the format conversions for these older GPUs by:

    a) Copy to system memory, do the conversions on the CPU to RGB then map back to a GPU surface

    b) Copy to system memory, copy each YUV plane to a surface (DXGI_FORMAT_R8_UNORM) and convert to RGB on the GPU

    Both methods have obvious performance concerns :(

    I hope there is something I am missing.  I do not want to give up DXVA decoding as it should perform better than the CPU required for colorspace conversions.  I was just hoping to get GPU support for both on 9.2 GPUs.  If you have any ideas, please let me know :)!

    -Jer


    • Edited by jmorrill Tuesday, August 7, 2012 9:20 PM
    Tuesday, August 7, 2012 9:07 PM
  • Hello Jer,

    That is quite the hack. Can you please tell me why you feel that you need to do this conversion? It is not clear to me why such a conversion should be necessary and how it can be useful to a large number of developers. If ypu can give me some business justification I can talk with the architects and see if we can get the behavior changed.

    Thanks,

    James


    Windows Media SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Wednesday, August 8, 2012 10:56 PM
    Moderator
  • Hi There,

    You're right, you don't need the hack above to make this work. You just need to configure the Source Reader (using IMFSourceReaderEx) with MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING attribute to achieve your goals. This should work with hardware that support D3D_FEATURE_LEVEL_9_3 and above provided the hardware supports NV12->XRGB32 conversion via VideoProcessBlt DDI. It's possible there are old hardware/display drivers that only support NV12->ARGB32 conversions in GPU, but even this should be very rare.

    Note that both IMFSourceReaderEx and MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING are new improvements for Windows 8 to enable GPU based video processing that should address your requirements.

    Hope this helps!

     


    Wednesday, August 8, 2012 11:11 PM
  • Kenny,

    That is EXACTLY what I was looking for!   Works great, even on Level9 feature levels! I don't know how I missed that.  I think I was confusing it with MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING which didn't work with the DXGI manager.

    Thank you and James for your help on this!  Now I have a bunch of code I can delete from my project!  :)

    -Jer


    • Edited by jmorrill Thursday, August 9, 2012 4:55 AM
    Thursday, August 9, 2012 4:52 AM
  • Glad the info was helpful. Would love to know if you have any feedback... what do like, what can be improved, what would like to see in future, etc.

    -Kenny

    Thursday, August 9, 2012 8:51 AM
  • What you just helped me with was my biggest request for a feature...but it's awesome it's already in there :).  So far, its been impressive how easy working with MF for Metro has been.  I was a little bit nervous at first when I noticed a lot of MF (eg, media session, topology, etc) was not available, but so far I don't see any huge hurdles in doing more advanced media processing in Metro.  I'm sure as I move along in this project, I'll have more feedback to give.

    For this Metro application, I am going to be starting with only Windows out-of-box codecs and container reader/writers.   But in the future, I may want to add my own.  Is this a scenario supported by IMFSourceReader and IMFSinkWriter?  I believe all the interfaces are there (IMFMediaSource, IMFStreamSink, IMFMediaSink), but unsure how IMFSourceReader and IMFSinkWriter would work with custom sources/sinks.

    For instance, if I wrote a custom MKV container reader that delivered XYZ samples to a custom XYZ decoder, how would you configure a IMFMediaSource to use both.

    On the other end, if I wrote a custom MKV container writer, that needed to encode samples using a custom XYZ encoder, how would I configure a IMFSinkWriter to use both?

    Thanks again!


    Thursday, August 9, 2012 6:34 PM
  • The docs for MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING

    http://msdn.microsoft.com/en-us/library/windows/desktop/hh162896%28v=vs.85%29.aspx/css


    specifies Window 8.  Is there a way to work for Windows 7 desktops?

    Monday, August 26, 2013 8:39 PM