none
Audio capture example using MediaFrameReader (Windows.Media.Capture.Frame)? RRS feed

  • Question




  • We are using UWP API and the MediaFrameReader that goes with it:
    https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/process-media-frames-with-mediaframereader

    As mentionned in this article, we handled the frame arrived event, got the frame (frame?.VideoMediaFrame), put it in a SoftwareBitmap object, got the byte[] from it to deliver it to a navigator, got a Bitmap to save it in a video file if necessary ( Writer.WriteVideoFrame(bmp, new TimeSpan(date.Ticks - BeginDate.Ticks))).

    Everything worked fine until we decided to add the audio to the video.
    For that purpose we followed that article: https://docs.microsoft.com/en-us/windows/uwp/audio-video-camera/process-audio-frames-with-mediaframereader

    But whatever we code, the frame that arrives in the frame arrived event never gets an audio that we try to catch with frame?.AudioMediaFrame.

    Can you help us add the audio? Not a single example on MSDN seems to address this. All the code we find mention that they don't use the audio...
    Friday, October 4, 2019 7:20 AM

All replies

  • Hi,

    Do you mean when you try to get the reference.AudioMediaFrame, it returns null? When I used the code, it worked well. Can you please provide a simple sample that can be reproduced? In addition, if you want to add the audio to the video, you could try to use MediaComposition to achieve and here is an official demo for you to refer.

    Best Regards,

    Fay


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, October 7, 2019 3:10 AM
  • Hi,
    Thank you for answering and sorry for the delay.
    This is how we set up everything:
                   // Create a new media capture object.
                   _mediaCapture = new MediaCapture();
                   var settings = new MediaCaptureInitializationSettings()
                   {
                       // Select the source we will be reading from.
                       SourceGroup = groupModel,
                       // This media capture has exclusive control of the source.
                       SharingMode = MediaCaptureSharingMode.ExclusiveControl,
                       // Set to CPU to ensure frames always contain CPU SoftwareBitmap images,
                       // instead of preferring GPU D3DSurface images.
                       MemoryPreference = MediaCaptureMemoryPreference.Cpu,
                       // Capture audio and video
                       StreamingCaptureMode = StreamingCaptureMode.AudioAndVideo,
                   };
                   if (micFound)
                       settings.AudioDeviceId = micId;
                   try
                   {
                       // Initialize MediaCapture with the specified group.
                       // This can raise an exception if the source no longer exists,
                       // or if the source could not be initialized.
                       await _mediaCapture.InitializeAsync(settings);
                       var audioFrameSources = _mediaCapture.FrameSources.Where(x => x.Value.Info.MediaStreamType == MediaStreamType.Audio);
                       if (audioFrameSources.Count() == 0)
                       {
                           Console.WriteLine("No audio frame source was found."); //PLEASE NOTE WE NEVER GET THIS ....

                           return;
                       }
                       await UpdateFrameSource();
                       await CameraInit();
                   }
                   catch (Exception e)
                   {
                       Console.WriteLine(e.Message);
                       DisposeMediaCapture();
                   }
    And this is how we get the frames:
           private static async void _reader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
           {
               // TryAcquireLatestFrame will return the latest frame that has not yet been acquired.
               // This can return null if there is no such frame, or if the reader is not in the
               // “Started” state. The latter can occur if a FrameArrived event was in flight
               // when the reader was stopped.
               using (MediaFrameReference frame = sender.TryAcquireLatestFrame())
               {
                   AudioMediaFrame audioFrame = frame?.AudioMediaFrame; // THIS IS ALWAYS NULL
                   SoftwareBitmap softwareBitmap = FrameRenderer.ConvertToDisplayableImage(frame?.VideoMediaFrame);
                   BufferMediaFrame bufferMedia = frame?.BufferMediaFrame;
                   if (softwareBitmap != null)
                   {
                        softwareBitmap = putFrameInGoodOrientation(softwareBitmap);
                       _newFrame = await EncodedBytes(softwareBitmap, BitmapEncoder.JpegEncoderId);
                       if (IMustSaveTheFirstFrame)
                       {
                           saveTheFirstFrame(_newFrame);
                       }
                       if (Writer != null)
                           dealWithFrameForVideo(bufferMedia);
                       if (_newFrame == null)
                                Console.WriteLine("Frame nul apres encodedbytes");
                   }
               }
           }
    and yes I mean that audioFrame is null. Did we set up things badly? We followed the example to the letter :/ Thanks for your help

    Wednesday, October 9, 2019 2:38 PM
  • Hi,

    From this document, it says if the frame reference wraps a different type of frame, or if the audio frame data is in a format that is unsupported, then frame?.AudioMediaFrame is null. So it's better to use BufferMediaFrame which is always non-null.

    Best Regards,

    Fay


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Thursday, October 10, 2019 7:30 AM