locked
Audio/Video Synchronization ; Deinterlacing RRS feed

  • Question

  • Hello community!

    I have some questions regarding Audio/Video synchronization and video deinterlacing.

    My scenario is the following (Windows Store App so the API is limited :) ):

    I made a custom scheme handler for RTP streams. I split the stream (MP2TS) channels and create for every audio/video channel a stram (IMFMediaStream). So far, (nearly) everything works fine. My issues are deinterlacing of the video (h264es) and audio/video synchronizing. I have also tried to set the properties of the media type to various MFVideoInterlaceMode values. Without effect.

    Regarding the synchronization: With the TS stream, I also get timestamps. But whenever I set the samples sampletime or duration, the video is stuttering...

    Any help is very welcome!

    Thanks in advance,

    Michael


    Monday, February 17, 2014 8:10 AM

Answers

  • Hi Michael,

    In MPEG-2 TS specification, PCR is used to synchronize audio, video, subtitle, etc. Audio and video have PTS that is related to PCR. PCR appears repeatedly in the bitstream. So you can synchronize audio and video from any position in the stream. Please check the specification because the above explanation is rough.

    Basic strategy for synchronization is as follows:

    * Fix a PCR in the stream.  Typically, you can choose the first PCR.

    * Subtract the PCR from each PTS for audio and video. It can be used as timestamp.

    PCR consists from PCR base (33bit) and PCR extension (9bit).  PTS is a 33bit field. Resolution of PCR base and PTS is 90KHz.  And PCR extension increases resolution of PCR to 27MHz.

    PCR and PTS are 33 bit field and their resolution is 90KHz.  So PCR and PTS wraparound about 26.5 hour.  You need to take care of this wraparound (You should not give negative huge value as timestamps around wraparound.).


    eijit

    • Marked as answer by Michael Sidy Monday, March 10, 2014 7:32 PM
    Wednesday, March 5, 2014 3:14 PM

All replies

  • Hello Michael Sidy,


    MFVideoInterlaceMode will be handled between video decoder and video renderer.  Typically, video decoder determines the appropriate MFVideoInterlaceMode value, then video renderer deinterlaces as specified.  So scheme handler/media source does not need to care about deinterlacing. The problem can be in video decoder and it's downstream component.  Or the h264es might not have appropriate interlace flags.


    How did you set sampletime for each sample?  Generally, decoding order and display order can be different in video codec.  When you set sampletime monotonically increasing in media source and decoding and display order are different, the video is stuttering.


    Thanks



    eijit


    • Edited by eijit Monday, February 17, 2014 5:07 PM add explanation
    Monday, February 17, 2014 5:06 PM
  • Hi eijit

    Thank you for your reply!

    I also thought, that the decoder has to decide, if the video has to be deinterlaced. Or at least the player has some method/property to activate deinterlacing. If I store the payload of the mpeg-ts packets as a file and use topoedit to render the file, all works. Even the video is deinterlaced. I tried to set the same properties, that I see in topoedit. But the video is still interlaced. I also found out, if I play the stored payload in windows mediaplayer, the video is deinterlaced. The same file in MediaElement is not deinterlaced.

    Regarding the sync:

    I get a timestamp value from the ts stream. I store the first timestamp and use the delta of the following timestamps as sample timestamp. But because I provide each elementary stream, I need some timestamp to sync the streams. I have no idea, how to sync the streams. Whenever I set some time to the sample time, the video is stuttering...

    Thank you again and maybe you have an idea how to sync and deinterlace the video :)

    Michael


    Tuesday, February 18, 2014 7:46 AM
  • any ideas regarding the issues?
    Monday, March 3, 2014 1:56 PM
  • Hello.

    Is your problem video stuttering, or audio/video snyc ?

    Monday, March 3, 2014 10:21 PM
  • Hi,

    If I don't set SampleTime and SampleDuration the Audio/Video is not sync. If I set SampleTime/Duration the video starts stuttering...

    Tuesday, March 4, 2014 9:20 AM
  • Hello.

    Mpeg2 and stuttering...

    I also face this problem with mpeg2 PS. First, because i don't really understand the specification about it. And second because some encoder do what they want... The specification are too permissive i think.

    The problem usually comes when B frames are presents.

    If you check how ffmpeg handles this with mpeg2 PS, they read some picture sequence, and calculate the timestamp by hand. It seems that some mpeg2 files, they lie for real timestamp.

    The problem with mpeg2 TS is to handle the timestamp correctly, because it is not a file, and it could change... I don't know much about TS.

    If you provide the mpeg2 file, check for an encoding that could be handle correctly.

    Tuesday, March 4, 2014 10:05 PM
  • Hi Michael,


    Sorry for the delayed response.


    Regarding the deinterlacing issue, I have no idea.


    As Miaou77 mentioned above, video codec uses different picture order between coded picture and display picture.  For example, coded pictures come from storage or network as follows:

    I2 B0 B1 P5 B3 B4 P8 B6 B7...

    where I, P, B mean picture type, number mean display order.

    Then video decoder decodes coded pictures in this order and outputs decoded pictures in the following order:

    B0 B1 I2 B3 B4 P5 B6 B7 P8...

    So, if you assign timestamp monotonically increasing for coded pictures, decoded pictures have inconsistent timestamp.  It will cause video stuttering.

    You can use PCR (Program Clock Reference) and PTS (Presentation Time Stamp) in MPEG-2 TS specification to assign timestamp correctly. I am going to explain the detail in the next post.


    eijit

    Wednesday, March 5, 2014 2:37 PM
  • Hi Michael,

    In MPEG-2 TS specification, PCR is used to synchronize audio, video, subtitle, etc. Audio and video have PTS that is related to PCR. PCR appears repeatedly in the bitstream. So you can synchronize audio and video from any position in the stream. Please check the specification because the above explanation is rough.

    Basic strategy for synchronization is as follows:

    * Fix a PCR in the stream.  Typically, you can choose the first PCR.

    * Subtract the PCR from each PTS for audio and video. It can be used as timestamp.

    PCR consists from PCR base (33bit) and PCR extension (9bit).  PTS is a 33bit field. Resolution of PCR base and PTS is 90KHz.  And PCR extension increases resolution of PCR to 27MHz.

    PCR and PTS are 33 bit field and their resolution is 90KHz.  So PCR and PTS wraparound about 26.5 hour.  You need to take care of this wraparound (You should not give negative huge value as timestamps around wraparound.).


    eijit

    • Marked as answer by Michael Sidy Monday, March 10, 2014 7:32 PM
    Wednesday, March 5, 2014 3:14 PM
  • Hello eijit.

    Interesting things you said.

    Do you have information about DTS ?

    For now I did not really read the specification correctly about PTS/DTS, but if you have more informations...

    Wednesday, March 5, 2014 8:42 PM
  • Hello Miaou77,


    I think DTS is useful to manage decode time for video decoders.  But I can't use DTS effectively. I mainly develop software for PCs. In PC platforms, video decoder hardly use CPU time constantly.  So such video decoder can't follow DTS exactly.  Instead, there is enough memory to avoid inconsistent decoding time by buffering. Therefore, video renderer totally follows PTS.

    P-STD (Program stream System Target Decoder) describes the usage of PTS, DTS and other related parameters.


    eijit

    Monday, March 10, 2014 3:21 PM
  • Hi eijit!

    Thank you very much for your explanaition. I will try the points you mentioned and will report back! This is my first project with media foundation and audio/video decoding at all. It's quite hard for me to understand and it's very time consuming but I didn't find another solution to bring an RTP-TS Live Stream to an Windows 8 App.

    I will mark your post as an answer, because I don't think, that a better one will come!

    Best regards,

    Michael

    Monday, March 10, 2014 7:31 PM