none
Using a pixel shader on video stream to integrate in a C#/XAML application

    Question

  • I am trying to figure out how to play back a video in a C#/XAML application having parts of the video transparent. As far as I know - there is no built in support for that. I tried modifying the MFT from the "Media capture using webcam sample" - which seemed like it could just plug into a MediaElement.AddVideoEffect method, but I realized - the output pixel format does not include an alpha channel.

    My current theory is that I could use on of the DirectX and XAML methods to render parts of my application using DirectX and use a pixel shader to make parts of my video frame transparent based on transparency information provided in other parts of the frame and I have found some samples that do this ("Direct2D magazine app sample", "Direct3D tutorial sample", "Direct2D 3D transform effect sample" and "Direct2D lighting effects sample"), but I don't think they integrate video into the scene.

    I found a sample called "DirectX video rendering sample" which seems like it might be doing part of what I want, but this is in the Windows 8 CP desktop samples section, so I am worried it might include some APIs not approved for Metro Apps.

    Am I heading in the right direction? Is there a better sample or can you provide any tips of what steps I need to take to achieve what I describe?

    Thanks


    Filip Skakun

    Tuesday, March 20, 2012 10:00 PM

Answers

  • Hello Filip,

    Yes, you are on the right track. I'm not a D3D expert but I believe that D3D textures can be rendered using alpha transparency. You can use the Media Engine directly in frame server mode to get a D3D surface. You can then render the surface in your scene. We do allow you to mix D3D and XAML unfortunately the XAML is always rendered after your D3D scene. This is great for game developers because you can create your HUD in XAML and your game remains in D3D. Unfortunately I don't believe that this will work for you because I'm guessing that you are going to want to put your D3D rendered video over your XAML elements.

    The only other solution I can think of is to use the Media Engine in frame server mode and retrieve the raw video bits. You can then theoretically blit the bits to a Writable Bitmap in XAML. Assuming that a Writable Bitmap allows for alpha this should give you the functionality that you are looking for. Unfortunately this method also has a drawback. The video transfer and rendering is very processor intensive even though XAML is video accelerated you are still pushing a lot of bits around. If you do go this route I would highly recommend that you stay in C++ for maximum performance.

    IMFMediaEngine interface
    http://msdn.microsoft.com/en-us/library/windows/apps/hh447918.aspx

    IMFMediaEngine::TransferVideoFrame method
    http://msdn.microsoft.com/en-us/library/windows/apps/hh448021.aspx

    DirectX and XAML interop
    http://msdn.microsoft.com/library/windows/apps/hh825871.aspx

    I hope this helps,

    James


    Windows Media SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Tuesday, March 20, 2012 11:12 PM

All replies

  • Hello Filip,

    Yes, you are on the right track. I'm not a D3D expert but I believe that D3D textures can be rendered using alpha transparency. You can use the Media Engine directly in frame server mode to get a D3D surface. You can then render the surface in your scene. We do allow you to mix D3D and XAML unfortunately the XAML is always rendered after your D3D scene. This is great for game developers because you can create your HUD in XAML and your game remains in D3D. Unfortunately I don't believe that this will work for you because I'm guessing that you are going to want to put your D3D rendered video over your XAML elements.

    The only other solution I can think of is to use the Media Engine in frame server mode and retrieve the raw video bits. You can then theoretically blit the bits to a Writable Bitmap in XAML. Assuming that a Writable Bitmap allows for alpha this should give you the functionality that you are looking for. Unfortunately this method also has a drawback. The video transfer and rendering is very processor intensive even though XAML is video accelerated you are still pushing a lot of bits around. If you do go this route I would highly recommend that you stay in C++ for maximum performance.

    IMFMediaEngine interface
    http://msdn.microsoft.com/en-us/library/windows/apps/hh447918.aspx

    IMFMediaEngine::TransferVideoFrame method
    http://msdn.microsoft.com/en-us/library/windows/apps/hh448021.aspx

    DirectX and XAML interop
    http://msdn.microsoft.com/library/windows/apps/hh825871.aspx

    I hope this helps,

    James


    Windows Media SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Tuesday, March 20, 2012 11:12 PM
  • Thanks, I'll investigate my options.

    Filip Skakun

    Wednesday, March 21, 2012 4:55 AM