locked
How to optimize MF Media Source for MKV playback (containing H264 and AC3)

    Question

  • I've built a working Media Foundation Media Source for matroska (MKV files) using the sample for an MPEG Source.  I've finally gotten it to a point where I can play the file and both video and audio are visible and audible.  I followed the MPEG1Source code quite closely actually.

    However, I can't seem to make the video playback smoothly.  Audio plays fine, but video just can't seem to keep up with the playback speed. It stutters and breaks up into blocks constantly.

    The only thing I have been able to try is to remove as much dynamic allocation of memory as I can.   While my parser parses the file and initially creates a bunch of objects on the heap (pertaining to track formatting, cue locations, etc) , when I start actually parsing video and audio frames, I'm no longer dynamically allocating memory on the heap.  I even removed a std::queue for frame lengths and created a fixed length circular buffer instead.

    What else do I need to try?

    I am simply parsing each of the video and audio frames in the order that they appear in the file.  The frames are not Annex B compatible (for H264), so in the DeliverPayload function I replace the length section of each NAL unit within the frame with a start code of 00 00 00 01.   Then I copy the frame into the stream's buffer.  

    One thing that is very different from the MPEG1Source example is that I am not creating structs that contain all of the video "initialization" data (i.e. MPEG1StreamHeader, MPEG1SystemHeader, MPEG1PacketHeader).  Instead, I am leaving all of it as public variables on my parser class.  So I am referring back to my parser quite a bit from the MKVSource class.  I don't know if this affects anything. 

    Anyways, I'd be happy to elaborate to get this working... Thanks!


    Lee McPherson




    Saturday, December 20, 2014 5:55 PM

Answers

  • Just a follow-up:

    I was indeed calculating the timecode / timestamp incorrectly.  MKV uses a technique called lacing to stitch frames together and save space.  While I was separating them and forwarding them correctly to the pipeline, the timecode in the header for each laced block only applies to the first frame.  I needed to add the default duration time to each successive frame laced in the block.

    While not specifying the timestamp using the SetSampleTime method caused smooth video, it was actually playing faster than the audio.  Each frame _needs_ a timestamp to sync correctly with the audio or other streams.


    Lee McPherson

    • Marked as answer by Lee McPherson Saturday, January 17, 2015 9:09 PM
    Saturday, January 17, 2015 9:08 PM

All replies

  • Ok, after playing around with everything, I got it to play without stuttering by removing the SetSampleTime method.  Either I am miscalculating the timestamp that needs to be sent or it shouldn't be set.  Either way, it seems to work now.

    Thanks anyways!


    Lee McPherson

    • Marked as answer by Lee McPherson Sunday, December 28, 2014 9:21 PM
    • Unmarked as answer by Lee McPherson Saturday, January 17, 2015 9:09 PM
    Saturday, December 20, 2014 9:24 PM
  • Hi Lee Mcpherson,

    Thank you for sharing this.

    Regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place. Click HERE to participate the survey.

    Monday, December 22, 2014 5:14 AM
    Moderator
  • Just a follow-up:

    I was indeed calculating the timecode / timestamp incorrectly.  MKV uses a technique called lacing to stitch frames together and save space.  While I was separating them and forwarding them correctly to the pipeline, the timecode in the header for each laced block only applies to the first frame.  I needed to add the default duration time to each successive frame laced in the block.

    While not specifying the timestamp using the SetSampleTime method caused smooth video, it was actually playing faster than the audio.  Each frame _needs_ a timestamp to sync correctly with the audio or other streams.


    Lee McPherson

    • Marked as answer by Lee McPherson Saturday, January 17, 2015 9:09 PM
    Saturday, January 17, 2015 9:08 PM