locked
H264 Decoding using Source Reader and Custom Bytestream RRS feed

  • Question

  • I am trying to decode H264 data on Win 7.  My source of compressed H264 data is not a file or network, the source is the memory itself. I get the H264 compressed data in the memory frame by frame. I want to synchronously decompress each frame as I get it. I am trying to follow the example in the link http://msdn.microsoft.com/en-us/library/dd389281(VS.85).aspx. But the example uses a URL to create the source reader. But since my source of compressed data is memory I am thinking of writing custom byte stream which reads data from memory.

    1. Is it the right approach?

    2. If the data is read from file(as in a normal case) then there must be a memory copy required before the decompression happens. Can I avoid this memcpy since I already have the data in memory?
    • Edited by The March Hare Wednesday, November 4, 2009 1:55 PM use link tool, formatting
    Wednesday, November 4, 2009 1:27 PM

Answers

  • Your project is similar to mine. If you're using an abstract data source (i.e. anything that isn't a registered MIME extension or supported network protocol/byte stream) in conjunction with a source reader, you do two different things. Either create a custom media source or a custom byte stream.

    Custom byte stream is easier to implement but provides less flexibility than a custom media source. Since you are already getting individual frames, it's handier to write a custom media source because you'll have more flow control over the data and the state machine of the decoder. Either way is fine as long as you're only using a Source Reader. If you move onto Media Sessions, you'll probably need to create a Custom Media Source instead.

    Oh and to answer your part 2 question, instead of storing the data in memory first then copying the data, read the data straight into a Sample Buffer. Just make sure that your data source is ready and readable before you lock the IMFMediaBuffer object. You'll end up storing a collection/array of IMFMediaSamples instead of the raw frames you are storing at the moment.
    • Marked as answer by The March Hare Wednesday, December 2, 2009 4:03 PM
    Monday, November 9, 2009 1:25 AM

All replies

  • It sounds like a good plan. To avoid the memcpy, you could create a custom source generating samples with custom IMFMediaBuffer (or use MFCreateMemoryBuffer as memory allocator), but that would be much more work.
    Thursday, November 5, 2009 5:21 PM
  • Your project is similar to mine. If you're using an abstract data source (i.e. anything that isn't a registered MIME extension or supported network protocol/byte stream) in conjunction with a source reader, you do two different things. Either create a custom media source or a custom byte stream.

    Custom byte stream is easier to implement but provides less flexibility than a custom media source. Since you are already getting individual frames, it's handier to write a custom media source because you'll have more flow control over the data and the state machine of the decoder. Either way is fine as long as you're only using a Source Reader. If you move onto Media Sessions, you'll probably need to create a Custom Media Source instead.

    Oh and to answer your part 2 question, instead of storing the data in memory first then copying the data, read the data straight into a Sample Buffer. Just make sure that your data source is ready and readable before you lock the IMFMediaBuffer object. You'll end up storing a collection/array of IMFMediaSamples instead of the raw frames you are storing at the moment.
    • Marked as answer by The March Hare Wednesday, December 2, 2009 4:03 PM
    Monday, November 9, 2009 1:25 AM
  • @ _Nobby_

    Thanks for your reply. Good to know that you are working on a similar project. I am resuming this work after a month, so did not check the replies to my question.

    I went ahead to start looking at implementing a custom media source. But it looks like the source reader buffers a lot of data before it starts to decode. But I get only one frame at a time and I need to decode it before I get the next. The source readers ReadSample function is failing when I restrict the amount of data read through the bytestream.

    Do you have any Idea how to make it work? Or else does implementing the Media source is the (only) solution?

    And ya, how far your project has come in terms of implementation?

    @moderator:

    I did not get any email alert when I got replies to my query.


    Thursday, December 17, 2009 12:35 PM
  • The H264 decoder will buffer a lot of data. To be exact, the state-machine of the decoder will buffer the exact number of frames between each keyframe(or clean-point). For example, the H264 hardware I have delivers a keyframe every 30 frames so when I start my Media Session or use a Source Reader, there is no decoded output for 30 frames. If you have the ability to modify the keyframe interval for the data source, you can reduce the delay of the decoder by increasing the keyframe-rate. This comes at the cost of higher bandwidth though.

    At the moment, the only project I have finished which uses my custom media source is a playback application. I spent a lot of time tweaking and making my custom source smarter(in terms of buffering) so that live playback would have the least amount of delay possible(because of the decoder state-machine delay). My hardware delivers the data real-time so my real-time playback of the decoded data is always just over a second behind.

    If ReadSample is failing, check the HRESULT of the function. It will most likely be the Needs More Input value indicating that the decoder state machine requires more frames fed into it from the custom media source before you can get a decoded frame out of the Source Reader.

    Everything becomes really straight-forward once you get your head around the querks of the MF API. It really is a great engine for abstract media processing. I'm personally stoked about the Managed port of the API which has been put up on the team developer's blog. If you haven't been there, check out http://blogs.msdn.com/mf/. They're full application examples of typical uses for the MF pipeline.

    Thursday, December 17, 2009 2:39 PM
  • @moderator:

    I did not get any email alert when I got replies to my query.



    A couple of potential reasons:

    - there was a hiccup in the forums software

    - your email provider's spam filter rejected it

    In any case, it's a good idea to check every few days and update the thread with any additional information you have discovered.


    Please use Vote As Helpful (green up arrow at top-left of posts) and Mark As Answer where appropriate.
    My dshow site is http://tmhare.mvps.org.
    Thursday, December 17, 2009 2:46 PM
  • If you start with a key frame, does this mean it will produce data from the start?  It is 4 years later and there isn't much in the way of MS only online examples of using the h264 decoder for raw data.

    Does the custom Source have to get rid of non-image NALs?

    Friday, August 23, 2013 8:50 PM