locked
How to dump raw video frame from webcam with JavaScript and C++

    Question

  • I am trying to retrieve raw video frame from webcam and blending it into another video stream(like picture in picture) with JavaScript and C++

    components.

     

    Solution 1 - Build MF component with custom sink to dump raw frame.

        I try to build MF topology but functions are not supported in metro ( MFEnumDeviceSource ).

        Is there other way to create capture source ?

    Solution 2 - Use Windows.Media.Capture.MediaCapture with custom sink.

        In JavaScript, startPreviewToCustomSinkAsync() always fail

        (Object doesn't support property or method startPreviewToCustomSinkAsync'),

        although it's available here : http://msdn.microsoft.com/en-us/library/windows/apps/br241124.aspx.

        Moreover, startPreviewToCustomSinkAsync need an encodingprofile, does it means I can not get raw frame ?


    That would be great help if anyone could provide some workable sample, thanks.

    Tuesday, December 6, 2011 10:34 AM

Answers

  • Hello SJ,

    It’s not clear from your post exactly why you are trying to combine two video data streams at the Media Foundation level. To do a simple picture in picture solution you should be able to simply use two “Media Elements” or “video” tags in XAML or HTML 5 respectfully. You should be able to just layer them in the “z” order and get the desired effect.

    I honestly can’t think of any reason why you would want to combine the streams at the Media Foundation layer. Are you looking to have frame accurate synchronization between the two streams? Keep in mind that since Windows is not a real time operating system you will never get perfect synchronization. Even if you are trying to do frame accurate sync you still can use two separate “media engine” instantiations and sync via a custom source.

    That said, in order to have your custom MF Source packaged and loaded from within your Metro application, you are going to need to add an interface or two to get things to work correctly. Unfortunately as of the “Build” developer preview release we don’t have a good bit of sample code to show you how to make your MFT callable from the Win RT environment.
     
    I know that Stan promised a sample when he spoke at “Build”. I’ve been working with Stan and others to make sure we get a good sample published as soon as we can. Once we have a clean sample that is up to date I will make sure to announce it on my blog (http://blogs.msdn.com/mediasdkstuff/). I’m not sure when we will get this published but I’m working to get it out as soon as possible.


    I hope this helps,

    James
    Windows Media SDK Technologies
    Microsoft Developer Services
    http://blogs.msdn.com/mediasdkstuff/

     

    Wednesday, December 14, 2011 1:24 AM
    Moderator
  • SJ,

    Sorry for the late reply. We have been on holiday in the United States.

    I think I understand your scenario but the Metro media engine was not designed with this scenario in mind. Using the “StartPreviewToCustomSynkAsync” should work. However the exact steps to make your custom sink callable from Metro have not been published yet. My guess is that your custom sink does not contain the proper “decoration” to enable it to be instantiated from the Metro environment. I will be releasing information on this as soon as I can on my blog. Unfortunately I don’t know when this sample will be available.

    That said I don’t think that there is a good solution to your problem in the Metro environment. If you have two separate custom sinks, what software layer are you going to use to mix the two streams? You will not be able to mix the streams in JavaScript or C#. You will need to have some type of communication between the two custom sinks. One of the two sinks will have to do the mixing.

    Keep in mind that manipulating two streams of HD video will be extremely processor intensive. You are going to need to use video acceleration to get a descent level of performance. Once you have the data mixed what are you going to do with the data? If you are doing to display it on the screen how are you going to do this? Are you going to wrap our frame server? You will not be able to hand the frames off to JavaScript or C# for display as a series of bitmap images, your performance will not be good enough.

    Using two custom sinks is a clever solution to a complex problem due to the limitations of the Metro Media Engine. Unfortunately due to the extreme complexity of your current solution I am skeptical that you will be able to get the level of performance that you are looking for without a large amount of effort and a deep knowledge of the video acceleration technologies available in the limited Metro environment. Please understand I think your solution is clever, I just want you to be aware of the extreme complexities that you will be facing if you pursue this solution.

    Thanks,

    James

    • Marked as answer by Bob_Bao Tuesday, January 3, 2012 7:13 AM
    Friday, December 30, 2011 10:47 PM
    Moderator

All replies

  • Hello SJ,

    It’s not clear from your post exactly why you are trying to combine two video data streams at the Media Foundation level. To do a simple picture in picture solution you should be able to simply use two “Media Elements” or “video” tags in XAML or HTML 5 respectfully. You should be able to just layer them in the “z” order and get the desired effect.

    I honestly can’t think of any reason why you would want to combine the streams at the Media Foundation layer. Are you looking to have frame accurate synchronization between the two streams? Keep in mind that since Windows is not a real time operating system you will never get perfect synchronization. Even if you are trying to do frame accurate sync you still can use two separate “media engine” instantiations and sync via a custom source.

    That said, in order to have your custom MF Source packaged and loaded from within your Metro application, you are going to need to add an interface or two to get things to work correctly. Unfortunately as of the “Build” developer preview release we don’t have a good bit of sample code to show you how to make your MFT callable from the Win RT environment.
     
    I know that Stan promised a sample when he spoke at “Build”. I’ve been working with Stan and others to make sure we get a good sample published as soon as we can. Once we have a clean sample that is up to date I will make sure to announce it on my blog (http://blogs.msdn.com/mediasdkstuff/). I’m not sure when we will get this published but I’m working to get it out as soon as possible.


    I hope this helps,

    James
    Windows Media SDK Technologies
    Microsoft Developer Services
    http://blogs.msdn.com/mediasdkstuff/

     

    Wednesday, December 14, 2011 1:24 AM
    Moderator
  • Hi James,

    Thanks for your reply.

    I would like to apply some effect to different webcam, merge them, apply more effects and

    display. :)

    Since create capture source directly is not available in Metro ,the plan change to -- create two

    MediaCapture  JavaScript, add effect and custom sink to each.

    But now I still blocked in how to use  startPreviewToCustomSinkAsync() with MediaCapture.

     

    Thanks,

    SJ

    Tuesday, December 20, 2011 3:33 AM
  • SJ,

    Sorry for the late reply. We have been on holiday in the United States.

    I think I understand your scenario but the Metro media engine was not designed with this scenario in mind. Using the “StartPreviewToCustomSynkAsync” should work. However the exact steps to make your custom sink callable from Metro have not been published yet. My guess is that your custom sink does not contain the proper “decoration” to enable it to be instantiated from the Metro environment. I will be releasing information on this as soon as I can on my blog. Unfortunately I don’t know when this sample will be available.

    That said I don’t think that there is a good solution to your problem in the Metro environment. If you have two separate custom sinks, what software layer are you going to use to mix the two streams? You will not be able to mix the streams in JavaScript or C#. You will need to have some type of communication between the two custom sinks. One of the two sinks will have to do the mixing.

    Keep in mind that manipulating two streams of HD video will be extremely processor intensive. You are going to need to use video acceleration to get a descent level of performance. Once you have the data mixed what are you going to do with the data? If you are doing to display it on the screen how are you going to do this? Are you going to wrap our frame server? You will not be able to hand the frames off to JavaScript or C# for display as a series of bitmap images, your performance will not be good enough.

    Using two custom sinks is a clever solution to a complex problem due to the limitations of the Metro Media Engine. Unfortunately due to the extreme complexity of your current solution I am skeptical that you will be able to get the level of performance that you are looking for without a large amount of effort and a deep knowledge of the video acceleration technologies available in the limited Metro environment. Please understand I think your solution is clever, I just want you to be aware of the extreme complexities that you will be facing if you pursue this solution.

    Thanks,

    James

    • Marked as answer by Bob_Bao Tuesday, January 3, 2012 7:13 AM
    Friday, December 30, 2011 10:47 PM
    Moderator
  • Hi SJ Weng,

    StartPreviewToCustomSinkAsync is webhosthidden. This means it is not available to Javascript projection but is available in C++ and C# projections. You could use startRecordToCustomSinkAsync to get what you need. You could need to create a custom encoding profile with the required video properties that your app needs.

    Thanks

    Radhika

    Wednesday, January 4, 2012 2:22 AM