locked
[UWP] Bug? mediaplayer not handle 3D stereoscopic video correctly. RRS feed

  • Question

  • Play this 3D video with windows media player, you will see left + right views.

    1. How can I get the same output  (left + right view) with Media​Player?

    I tried set StereoscopicVideoRenderMode, but seems not work.

    2. How can I know whether a video is 3D or not, if I can't how do I know when to use CopyFrameToStereoscopicVideoSurfaces when frame server mode is enabled?


    • Edited by BennyWang Thursday, June 1, 2017 8:59 AM
    Tuesday, May 16, 2017 9:54 AM

All replies

  • Hello,

    >>I tried set StereoscopicVideoRenderMode, but seems not work.

    Do you mean that you can not have the same output (left+right view) like in the windows media player? Or do you mean that when you wear the 3D glasses to watch this video, but it does not have any 3D effect?

    >>How can I know whether a video is 3D or not
    If you are using this MediaElement Class to play video, you can check whether this Video is 3D by using this MediaElement's IsStereo3Dvideo property, by the way using this method will only work with 3D metadata that is encoded in the H.264 SEI format. Or you may need to detect the encoding format by yourself, for more information, please check here.

    Besides, you can also try to use the third-party library to help you check whether a video is 3D or not.

    Thanks,
    Amy Peng


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Thursday, May 18, 2017 10:01 AM
  • Actually I need to get the video frame to do custom processing with shaders then present it.

    The only way I can find is to use MediaPlayer, put it into frame server mode, then use CopyFrameToVideoSurface or  CopyFrameToStereoscopicVideoSurfaces.

     
    But MediaPlayer seems has no way to tell whether a Video is 3D or not. So I don't know when to use  CopyFrameToVideoSurface and when to use CopyFrameToStereoscopicVideoSurfaces. 

    The StereoscopicVideoRenderMode is the only MediaPlayer's property looks a little bit related, I thought it maybe controls to output one view or side by side view, but it's not. What does it use for? 


    By the way is there any method to control MediaElement or MediaPlayerElement or IMFMediaEngineEx to show side by side picture for 3D video that 3D metadata is encoded in the H.264 SEI format?




    • Edited by BennyWang Monday, May 22, 2017 4:37 AM
    Friday, May 19, 2017 4:20 AM
  • set StereoscopicVideoRenderMode to StereoscopicVideoRenderMode.Stereo and use CopyFrameToStereoscopicVideoSurfaces , 

    when playing a video without 3d meta data, exception thrown. It's OK.

    But when playing a video with 3d meta data encoded,  like this one ,  the right surface is black, Why?  Is it a bug?

    How can I get the right view of this 3D video when writing UWP app? 

    Anybody can help?

    Tuesday, May 23, 2017 6:22 AM
  • Hi Benny,

    I don't know where you got the video or how you generated it but there is likely a problem with the encoding. h.264 supports the stereo profile and the MPEG 4 file source supports the stereo 3D atom. So as long as your content is encoded correctly stereo playback should be automatically enabled in the MediaPlayer.

    Also, please make sure you have permission from the copyright owner to use the video before posting it. Always attribute the video as well so the original copyright holder (if different from yourself) gets credit.

    -James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Friday, May 26, 2017 1:13 AM
  • Thanks James,

    It seems that the video isn't uploaded successfully.  I tried the original file again, the problem still exists.

    What do you mean "stereo playback should be automatically enabled"?

    I also tried use IMFMediaEngineEx directly, still got only one view.

    And someone asked similar question here . Seems I'm not the only one who has this problem.

    Haven't realized there may be a copyright issue, the file has been removed.

    But I can't find other MVC-3D videos. Do you know is there any standard test video can be used?









    • Edited by BennyWang Friday, May 26, 2017 9:43 AM
    Friday, May 26, 2017 4:26 AM
  • Hi James,

    I have tried several videos from YouTube. search "yt3d enable=true"

    The results are same, CopyFrameToStereoscopicVideoSurfaces can't get the right view surface.

    Could you please check it?

    Thanks!

    Saturday, May 27, 2017 2:05 AM
  • I use following code to try CopyFrameToStereoscopicVideoSurfaces. The right view's CanvasControl shows nothing.  What's wrong?

    <Page
        x:Class="App2.MainPage"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:local="using:App2"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:canvas="using:Microsoft.Graphics.Canvas.UI.Xaml"
        mc:Ignorable="d" Loaded="Page_Loaded">
    
        <Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
            <canvas:CanvasControl x:Name="lv" Margin="0,0,0,400" Width="400" Height="300" Draw="lv_Draw"></canvas:CanvasControl>
            <canvas:CanvasControl x:Name="rv" Margin="0,400,0,0" Width="400" Height="300" Draw="rv_Draw"></canvas:CanvasControl>
        </Grid>
    </Page>
    
    
    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using System.Runtime.InteropServices.WindowsRuntime;
    using Windows.Foundation;
    using Windows.Foundation.Collections;
    using Windows.UI.Xaml;
    using Windows.UI.Xaml.Controls;
    using Windows.UI.Xaml.Controls.Primitives;
    using Windows.UI.Xaml.Data;
    using Windows.UI.Xaml.Input;
    using Windows.UI.Xaml.Media;
    using Windows.UI.Xaml.Navigation;
    using Windows.Media.Playback;
    using Windows.Media.Core;
    using Microsoft.Graphics.Canvas;
    using System.Diagnostics;
    // The Blank Page item template is documented at https://go.microsoft.com/fwlink/?LinkId=402352&clcid=0x409
    
    namespace App2
    {
        /// <summary>
        /// An empty page that can be used on its own or navigated to within a Frame.
        /// </summary>
        public sealed partial class MainPage : Page
        {
            MediaPlayer mp;
            CanvasRenderTarget offscreenL;
            CanvasRenderTarget offscreenR;
            CanvasDevice device;
    
            public MainPage()
            {
                this.InitializeComponent();
            }
    
            private void Page_Loaded(object sender, RoutedEventArgs e)
            {
                mp = new MediaPlayer
                {
                    Source = MediaSource.CreateFromUri(new Uri("ms-appx:///Assets/rb.mp4")),
                                   
                    AutoPlay = true,
                    StereoscopicVideoRenderMode = StereoscopicVideoRenderMode.Stereo
    
                };
    
                
                mp.IsVideoFrameServerEnabled = true;
                mp.VideoFrameAvailable += Mp_VideoFrameAvailable;
    
                device = CanvasDevice.GetSharedDevice();
                offscreenL = new CanvasRenderTarget(device, 400, 300, 96);
                offscreenR = new CanvasRenderTarget(device, 400, 300, 96);
    
            }
    
            private async void Mp_VideoFrameAvailable(MediaPlayer sender, object args)
            {
                await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
                 {
    
                     mp.CopyFrameToStereoscopicVideoSurfaces(offscreenL, offscreenR);
                 });
                lv.Invalidate();
                rv.Invalidate();
                
            }
    
            private void lv_Draw(Microsoft.Graphics.Canvas.UI.Xaml.CanvasControl sender, Microsoft.Graphics.Canvas.UI.Xaml.CanvasDrawEventArgs args)
            {
                args.DrawingSession.DrawImage(offscreenL, 0,0);
    
            }
    
            private void rv_Draw(Microsoft.Graphics.Canvas.UI.Xaml.CanvasControl sender, Microsoft.Graphics.Canvas.UI.Xaml.CanvasDrawEventArgs args)
            {
                args.DrawingSession.DrawImage(offscreenR, 0, 0);
            }
        }
    }
    






    • Edited by BennyWang Wednesday, June 7, 2017 9:35 AM
    Wednesday, May 31, 2017 7:32 AM
  • Hi BennyWang,

    Sorry for the late reply I've been out of the office. I looked at the videos you suggested on YouTube. IANAL: I would recommend that you should talk with your legal representation about the proper use of YouTube content. That said, YouTube requires that the "st3d box" be present in the h.264 stream. YouTube appears to pack the 3d data Left-Right. Media Foundation does support single buffer 3D data packed Left-Right.

    You should be able to confirm that the file is 3D enabled and compatible with the Microsoft h.264 implementation by looking at the media type of the file in topo edit. You should see that the MF_MT_VIDEO_3D attribute is set. If the attribute is not set then the source parser cannot determine that the file is 3D because the appropriate header or box information is not present.

    If this attribute is set and the data is still not being delivered correctly it is possible that MF is not able to pull the necessary information from the file to configure the proper 3D format attributes. In the case of playing 3D data packed Left-Right the MFVideo3DSampleFormat_Packed_LeftRight attribute needs to be set.

    I hope this helps,

    James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Tuesday, June 6, 2017 12:55 AM
  • Hi James,

    Open the file in topoedit, the MF_MT_VIDEO_3D attribute and MF_MT_VIDEO_3D_FORMAT are set after play it. And if using XAML MediaElement control, MediaElement.IsStereo3DVideo also returns true.

    But playing the file in topoedit, the output is side by side. while using MediaElement or MediaPlayerElement the output is just left eye's content.

    What I want to know now is when writing UWP App

    1.how can I get the stereoscopic 3D video's right eye content?

    2.why CopyFrameToStereoscopicVideoSurfaces does not work in my test code above.

    Thanks!


    • Edited by BennyWang Wednesday, June 7, 2017 1:13 AM
    Tuesday, June 6, 2017 3:28 AM
  • Hi Benny,

    Okay, from the looks of it that should work. Do you have some 3D content that you own the copyright for that you can make available publically here in the forum? I want to make sure that I'm looking at content that you actually want to consume. In the meantime I'm looking through our library to see what 3D content we have available internally. I'll let you know what I find.

    Thanks,

    James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Wednesday, June 7, 2017 1:20 AM
  • James,

    thank you so much for taking the time to answer my question.

    With a friend help, I created a mp4 file. It just shows following picture.

    Please download it from here

     


    • Edited by BennyWang Wednesday, June 7, 2017 9:26 AM
    Wednesday, June 7, 2017 9:24 AM
  • Hi James,

    Is there any progress on this issue?

    Thanks!

    Monday, June 19, 2017 1:22 AM
  • Hi Benny,

    Sorry for the late reply. I have it on my schedule to look at this tomorrow. I'll get back to you by the end of the day and let you know what I find. Thanks much for providing the file.

    -James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Wednesday, June 21, 2017 12:46 AM
  • Hi Benny,

    I took a look at this yesterday but didn't get very far. Do you have a very small Visual Studio 2017 project that reproduces the issue? If so can you please zip it up, put it on your OneDrive and paste a link here. I want to make sure that we are approaching the problem from the same direction.

    Thanks,

    James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Thursday, June 22, 2017 10:42 PM
  • Hi James,

    I have uploaded the code.

    Thanks!

    Friday, June 23, 2017 2:25 AM
  • Hi Benny,

    Thanks for providing the code and thanks for your patience. I was able to dig into this for you today and debug into the WinRT source code.

    The issue that you are seeing is technically by design. The CopyFrameToStereoscopicVideoSurfaces API requires that the underlying device that was used to create the surfaces was itself created with the D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag. Without this flag the underlying MediaPlayer code is not able to lock both surfaces at the same time. Since it can't lock both surfaces at the same time it can only copy the texture data to a single surface. By default the surface chosen is the "left eye".

    The CanvasDevice.GetSharedDevice call returns a device that was created without the D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag set. Because of this you will not be able to use a standard CanvasDevice to render both the left and right view.

    It is possible for you to create a CanvasDevice using a Direct3D 11 device. So theoretically you could create a custom D3D 11 device with the D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag set and then get the shared device from there. Again, theoretically I'm honestly not that familiar with Win2D so I can't guarantee that this will work.

    Reference:

    CanvasDevice.CreateFromDirect3D11Device
    http://microsoft.github.io/Win2D/html/M_Microsoft_Graphics_Canvas_CanvasDevice_CreateFromDirect3D11Device.htm

    D3D11_RESOURCE_MISC_FLAGhttps://msdn.microsoft.com/en-us/library/windows/desktop/ff476203(v=vs.85).aspx

    I hope this helps,

    James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/


    Wednesday, June 28, 2017 1:49 AM
  • Hi James,

    Thanks! 

    But setting D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX  flag seems not work.

    I created a new project which does not use Win2D.

    It sets MiscFlags to D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX when creating Texture2D, but the right eye content is still not displayed.


    • Edited by BennyWang Friday, June 30, 2017 1:14 AM
    Thursday, June 29, 2017 1:49 AM
  • Do you mean when creating the device? Setting it on the texture won't have any effect.

    -James


    Windows SDK Technologies - Microsoft Developer Services - http://blogs.msdn.com/mediasdkstuff/

    Thursday, June 29, 2017 2:24 AM
  • How to create the device with the D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX flag?
    Thursday, June 29, 2017 4:53 AM
  • Anyone know how to do it?
    Tuesday, August 29, 2017 1:43 AM
  • Hi James,

    Would you please continue to help?

    Monday, September 11, 2017 1:51 AM
  • answers

    1. set StereoscopicVideoRenderMode to stereo when 3D is detected

    2. detect the video type with Windows::Media::MediaProperties::StereoscopicVideoPackingMode

    this thread helps  

    but I found a new problem, the frames get from CopyFrameToStereoscopicVideoSurfaces seems shifted,

    the left part seems not starts from (0,0) , there is a blue line at the bottom, and the right part has a black line 

    : (



    • Edited by BennyWang Thursday, October 19, 2017 6:30 AM
    Thursday, October 19, 2017 6:29 AM
  • answers

    1. set StereoscopicVideoRenderMode to stereo when 3D is detected

    2. detect the video type with Windows::Media::MediaProperties::StereoscopicVideoPackingMode

    this thread helps  

    but I found a new problem, the frames get from CopyFrameToStereoscopicVideoSurfaces seems shifted,

    the left part seems not starts from (0,0) , there is a blue line at the bottom, and the right part has a black line 

    : (



    Hello Benny,

    I'm working for 3D DirectX and got problem same as your. I have try many ways but still display in on eye (left || right).

    Could you please share your sourcode to me.

    I'm very happy to get your help.

    Thank you so much !

    Friday, June 29, 2018 3:40 AM