locked
Step Frame Backwards RRS feed

  • Question

  • Hi there,

     

    i´m currently trying to implement reverse frame stepping using a custom media source. Stepping forward works fine, but due to the nature of mpeg4 or h264, i cannot step back a single frame. Jumping back to the last I-Frame works fine. 

    However, could it be possible to create a custom MFT that buffers decompresses samples (say the last GOP) which replaces the decoded (i.e. invalid) sample during playback when rate is set to 0 and a scrub operation happens? i don't want to invest a lot of time just to realize this would not work.

    are there any other ideas how to achieve single frame stepping backwards?

    i´d appreciate any advice from the MF Pro´s over here ;)

    regards

    joachim

    Tuesday, February 22, 2011 10:07 AM

Answers

  • The only way to achieve frame stepping backwards in rate 0 is to seek backwards.  This does not really work well with standard Windows 7 components because there is a lot of latency in seeking.  Since you have your own source and MFT, however, you could minimize the latency and potentially get good performance.
    Saturday, June 16, 2012 12:03 AM

All replies

  • It is certainly feasible to create an MFT that buffers samples in order to handle reverse scrubbing, but there are some caveats.  Buffering decompressed samples could get quite expensive in terms of memory.  It also poses some latency problems, as when traversing a GOP boundary you potentially have to decode the entire previous GOP before you get the desired frame.  Additionally, it is possible for some formats to have open GOPs, where P or B frames have dependencies on frames in a previous GOP.  This would require parsing the compressed format to know when this is the case and then decoding the necessary previous GOP as well.

    Smooth reverse and reverse scrubbing is quite a difficult problem for arbitrary compressed formats with dependent frames.  Editing applications often make this problem easier by storing an intermediate format with known good editing parameters (closed GOP, small GOP size, no B frames) and then using optimized parameters when generating the final output file.

    Wednesday, March 9, 2011 10:05 PM
  • Hello Matt,

    Thanks for your reply. I´ve successfully managed to perform reverse playback using my custom media source and a custom decoder MFT (using h264 or mp4 video data).

    However, i have some problems with single frame step. It works as documented when i set the playback rate to 0 and calling IMFMediaSession.Start with VT_EMPTY. 

    however, i did not find a way to do the same in the other direction (e.g. single frame backwards). I tried setting the playback rate to -0 (which is a valid float value imho) but always receive 0 in my media source´s SetRate() method.

    is there any way to achieve single frame step backwards using a media session (like it works when doing forward playback)?

    i´d be grateful for any idea :)

    Monday, May 7, 2012 1:59 PM
  • Mathematically, zero is both positive and negative at the same time and also not a "proper" number. Even though it can be represented as a float/signed number via the sign bit, i believe it gets ignored in processor registers when the rest of the data are all zeros.

    I've never explored rate control before. Does providing a negative number to the media session for playback rate and issuing decreasing sample timestamps allow revers playback in the EVR? How does step framing work in forward playback with the rate set to zero? If the rate is zero, when does the pipeline request samples(i.e. what events)?

    Friday, May 11, 2012 1:53 AM
  • setting a negative playback rate lets the presentation clock run backwards (as far as i´ve read the documentation), that also means the media source has to support r ate control with negative playback rates, as well as MFT´s within the topology. 

    frame stepping with the media sessino can be performed by setting the playback rate to zero and calling IMFMediaSession.Start with VT_EMTPY as start time. Each time you call start, the pipeline requests the next frame from the media stream.

    however, doing the same the opposite way is either not possible at all using the media session (although used components, like media sources, MFT´s might support it), or at least not documented :-S

    Friday, May 11, 2012 7:12 AM
  • No chance of hardcoding your media source to deliver a single frame in reverse when the rate is -1 and the start time is VT_EMPTY? Or do all the other pipeline elements need to support that as well?
    Sunday, May 13, 2012 9:32 PM
  • well, hardcoding the behavior of the media source would not give the desired functionality here, as the media session would permanently request samples when playback rate is set to -1.

    scrubbing with media session (i.e. playback rate = 0) is treated somewhat special within the media session i guess.

    the only way i see for now is usinga hybrid approach with a source reader/media session combo (like the mfdub sample). 

    Monday, May 14, 2012 6:41 AM
  • The only way to achieve frame stepping backwards in rate 0 is to seek backwards.  This does not really work well with standard Windows 7 components because there is a lot of latency in seeking.  Since you have your own source and MFT, however, you could minimize the latency and potentially get good performance.
    Saturday, June 16, 2012 12:03 AM
  • Hey, Joachim. Did you ever finish your implementation? Any chance it's still supported and that you're sharing it? 
    Thursday, October 19, 2017 5:30 PM