How to get images/frames from recorded video with MediaCapture?

Question

I'm trying to make a Augmented Reality kind of Windows 8 Metro app. But i'm kinda having problems figuring out what is the best way to capture images/frames from a video captured with the MediaCapture class.

At the moment I use MediaCapture to record a video to a stream and use a dispatchtimer to take photo's every 300 milliseconds (it was less before, but since the Consumer Preview it won't work anymore if less than 300ms). The only use of the video recording is to keep the webcam turned on, or else it will continuously turn the webcam on/off

here is the sample of the code I have made:

MediaCapture mediaCapture = new MediaCapture();
                await mediaCapture.InitializeAsync(
                    new MediaCaptureInitializationSettings
                    {
                        StreamingCaptureMode = StreamingCaptureMode.Video
                    });

IRandomAccessStream picturestream = new InMemoryRandomAccessStream();
IRandomAccessStream videostream = new InMemoryRandomAccessStream();
await mediaCapture.StartRecordToStreamAsync(MediaEncodingProfile.CreateMp4(VideoEncodingQuality.Vga), videostream);

ImageEncodingProperties iep = new ImageEncodingProperties();
iep.Height = uint.Parse("600");
iep.Width = uint.Parse("800");
iep.Subtype = "JPEG";

DispatcherTimer dispatchTimer = new DispatcherTimer();
                dispatchTimer.Interval = new TimeSpan(0, 0, 0, 0, 300);
                dispatchTimer.Tick += async (s, e) =>
                {
                    await mediaCapture.CapturePhotoToStreamAsync(iep, picturestream);

                    //process image

                    BitmapImage bi = new BitmapImage();
                    bi.SetSource(picturestream);
                    picture.Source = bi;
                };
                dispatchTimer.Start();

Can anyone tell me if there is a better way to do this? is it possible to get the images/frames from the videostream?

Answer 1

Hello Mike,

At this time we do not allow access to the underlying media stream from managed languages. If you would like to get access to the underlying video stream you should write your own MFT plug in for the underlying media engine. This must be written in unmanaged C++ using the Metro COM framework. This component can then do all of the AR processing in real time and pass the results back to your managed UI application. We demoed a similar system at the "build" conference. Check out the links below for more info.

Media capture using webcam sample" found here: http://code.msdn.microsoft.com/windowsapps/Media-Capture-Sample-adf87622

The build conference "recognizer presentation": http://channel9.msdn.com/events/BUILD/BUILD2011/PLAT-776T

I hope this helps,

James

---

Windows Media SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

Answer 2

Will you simply not allow it, or is it possible but not recommended? I've been trying to read MediaCapture.StartRecordToStreamAsync as a stream and convert it into a byte array. I end up with a big byte array containing nothing but zeroes. Is this because it's not allowed, or does it sound like and error in my code. If error in code, will it be able to bit manipulate life video from camera?