locked
How do you get webcam video into a Metro DirectX game

    Question

  • I can't find a way to get webcam video that doesn't involve XAML.

    You guys came up with a new API just for Windows 8 called IMFCaptureEngine, but for some reason this is Win32 only and can't be used in metro.

    Will IMFCaptureEngine ever become an approved Metro API?

    I wouldn't mind using Media Foundation's Media Session API, but that seems to be Win32 only as well.

    Will Media Foundation Media Session ever become an approved Metro API?

     
    Saturday, March 17, 2012 5:05 AM

All replies

  • You can use Windows::Media::Capture::MediaCapture. This is the same way you would capture video in a Xaml app: it is not restricted to Xaml or part of Windows::UI::Xaml.

    I see you're already involved in other threads about how to render a video in DirectX once you're captured it, so I'll let you continue that discussion on those threads.

    --Rob

    Saturday, March 17, 2012 5:56 AM
    Moderator
  • Hi Rob,

    I suppose you could say there are two ways to handle webcam video:

    1. Record Webcam video. Display recorded video later.

    2. Display live webcam video.

    The answer you provided is accurate for #1. I'm interested in #2 since I'd like to build a augmented reality app in metro.

    Upon further examination of IMFCaptureEngine, its not as useful as I thought. It only provides live video to a window handle or DirectComposition. I don't care for either so I rescind my vote for making IMFCaptureEngine an approved API.

    I was hoping there would be some accelerated path of getting live webcam frames into a Direct3D texture but I can't find any. Am I missing something?

    Saturday, March 17, 2012 1:03 PM
  • If you want to process the video to add augmented reality directly into it I believe you'll need to write a media filter. I've pointed our media expert at this thread and he'll follow up with more detail.

    --Rob

    Monday, March 19, 2012 5:25 PM
    Moderator
  • Hey Anon,

    Luckily the underlying WinRT Media Engine is very robust and can be easily configured to give you the functionality that you are looking for. I guess the answer to your architectural question just depends on how you plan to build your application.

    Personally if I was going to build a augmented reality application (such as the one we showed at Build) I would write an MFT that simply plugs into the high level "MediaCapture" object. Your MFT can process the raw capture data, and relay information to your app. Your app can then use this information to overlay your AR metadata using XAML or HTML5. One caveat here is that you want to keep the communication between your MFT and hosting application as quiet as possible for performance reasons.

    MediaCapture class:
    http://msdn.microsoft.com/en-us/library/windows/apps/windows.media.capture.mediacapture.aspx

    Extending the media platform using input, output and processing plug-ins:
    http://channel9.msdn.com/events/BUILD/BUILD2011/PLAT-783T

    The grayscale MFT sample is included with the "Media capture using webcam sample" found here: http://code.msdn.microsoft.com/windowsapps/Media-Capture-Sample-adf87622

    I hope this helps,

    James


    Windows Media SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Monday, March 19, 2012 6:15 PM
    Moderator
  • Hi James,

    Do you have any suggestions for getting the webcam image (pixel) data into Direct3D texture or surface?

    You guys wrote a sample of how to do this for Windows 7 called MFCaptureD3D:

    http://msdn.microsoft.com/en-us/library/windows/desktop/dd743690(v=vs.85).aspx

    However, it appears that the function "MFEnumDeviceSources" used in that demo is not on the approved metro COM API list. Odd because it seems like 99% of the MF functions in that sample are on the approved API list. Is there another way to pair a webcam with an IMFMediaSource?


    Monday, March 19, 2012 10:25 PM
  • Hello Anon,

    Have you looked at the "IAdvancedMediaCapture" interface? It looks like you can get a "IMFDXGIDeviceManager" from this interface. You should then be able to call "GetVideoService" to get a handle to a D3D device. I honestly haven't seen any sample code internally that shows how to do this (other than the MediaCapture object). So unfortunately I don't have anything to share with you. If you do run into problems let me know and I can compare your code to what we do in the MediaCapture object to try and help you get unstuck.

    I hope this helps,

    James 


    Windows Media SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Wednesday, March 21, 2012 12:41 AM
    Moderator