Submitting raw H264 samples to SinkWriter RRS feed

  • Question

  • Hello,

    I can successfully encode H264 to avi file, using a IMFSinkWriter, by submitting YUV420 frames.

    Now, I would like to use the SinkWriter not as an encoder, but as a simple AVI writer. I encode the YUV420 frames with another encoder (Cuda NVENC), I get H264 samples, and I would like to send the H264 samples to SinkWriter so that the H264 stream is properly encapsulated in the AVI file.

    But I fail at finding the way to configure the input and output media types of the sinkwriter. None of the configurations I tried worked.

    Is it possible to use the sinkwriter that way ?

    I also tried to write the H264 samples directly to the imfbytestream behind the sinkwriter, but the resulting file does not work.


    Pierre Chatelier

    Tuesday, May 31, 2016 4:12 PM

All replies

  • Found the solution !

    By querying the streamsink from the mediasink of the sinkwriter, I can bypass the encoder.

    Wednesday, June 1, 2016 12:42 PM
  • in fact it does not really work. The file does not seem to be corrupted, but the images hardly look like what they should. VLC itself does not report any error, but the display is just terrible.

    There must be something else to do to configure the sinkwriter, even if I directly write to the mediaskink.

    Thursday, June 2, 2016 9:31 AM
  • You need to initialize Sink Writer so that it does not have embedded encoder MFT and accepts encoded data. This is actually simple and requires that you set it up with H.264 media type as input format.

    Next, you have to provide data to be written not just with good payload, which you supposedly already have, but also with proper order, time stamping, and key frame/splice point/clean point flags. Failure to do so typically results in file acceptable by players and incorrectly decoded image (esp. broken with seeking etc.).

    Thursday, June 2, 2016 9:41 AM
  • Thanks, but it is exactly what I have tried.

    More details :

    My outputmediatype

          hr = outputMediaType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);     
          hr = outputMediaType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264);   
          hr = outputMediaType->SetUINT32(MF_MT_AVG_BITRATE, bitRate);//bitrate is computed by a formula depending on frame size and fps
          hr = outputMediaType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);   
          hr = MFSetAttributeSize(outputMediaType, MF_MT_FRAME_SIZE, width, height);   //width and height 16-aligned
          hr = MFSetAttributeRatio(outputMediaType, MF_MT_FRAME_RATE, frameRate.first, frameRate.second);//typically (30, 1)
          hr = outputMediaType->SetUINT32(MF_MT_MPEG2_PROFILE, eAVEncH264VProfile_Main);
          hr = MFSetAttributeRatio(outputMediaType, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);   
          hr = outputMediaType->SetUINT32(MF_MT_MAX_KEYFRAME_SPACING, 16);
          hr = outputMediaType->SetUINT32(CODECAPI_AVEncCommonRateControlMode, eAVEncCommonRateControlMode_UnconstrainedVBR);
          hr = outputMediaType->SetUINT32(CODECAPI_AVEncCommonQuality, 100);
          hr = outputMediaType->SetUINT32(MF_MT_FIXED_SIZE_SAMPLES, FALSE);
          hr = outputMediaType->SetUINT32(MF_MT_ALL_SAMPLES_INDEPENDENT, FALSE);

    My inputMediaType when I use the MFT encoder (and it works)

            hr = inputMediaType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);
            hr = inputMediaType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12);//my frames are indeed in NV 12 
            hr = inputMediaType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);
            hr = MFSetAttributeSize(inputMediaType, MF_MT_FRAME_SIZE, width, height);
            unsigned __int32 fixedSampleSize = width*(16*((height+15)/16))+width*(height/2);//for Y, U and V
            hr = inputMediaType->SetUINT32(MF_MT_SAMPLE_SIZE, fixedSampleSize);
            hr = inputMediaType->SetUINT32(MF_MT_DEFAULT_STRIDE, width);
            hr = inputMediaType->SetUINT32(MF_MT_FIXED_SIZE_SAMPLES, TRUE);
            hr = inputMediaType->SetUINT32(MF_MT_ALL_SAMPLES_INDEPENDENT, TRUE);
            hr = MFSetAttributeRatio(inputMediaType, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);

    My InputMediatype when I try to bypass the MFT encoder and submit already encoded H264 samples (but does not work)

            hr = inputMediaType->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);

            hr = inputMediaType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264);//typically makes sinkWriter->SetInputMediaType(...) FAIL

            hr = inputMediaType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12);//nonsense but sinkWriter->SetInputMediaType(...) returns S_OK

            hr = inputMediaType->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);
            hr = MFSetAttributeSize(inputMediaType, MF_MT_FRAME_SIZE, width, height);//not sure it is needed
            hr = inputMediaType->SetUINT32(MF_MT_DEFAULT_STRIDE, width);//not sure it is needed
            hr = inputMediaType->SetUINT32(MF_MT_FIXED_SIZE_SAMPLES, FALSE);
            hr = inputMediaType->SetUINT32(MF_MT_ALL_SAMPLES_INDEPENDENT, FALSE);
            hr = MFSetAttributeRatio(inputMediaType, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);
            hr = inputMediaType->SetUINT32(MF_MT_COMPRESSED, TRUE);//I expected that attribute to be the most important

    Finally, when I receive the H264 samples, I encapsulate them in IMFSamples :

    MFCreateAlignedMemoryBuffer(sampleSize, MF_16_BYTE_ALIGNMENT, &mediaBuffer);
    hr = mediaBuffer->Lock(&pData, 0, 0);
    memcpy(pData, sampleBegin, sampleSize);//sampleBegin, sampleEnd are just char* pointers to raw H264 sample packets
    hr = mediaBuffer->Unlock();
    hr = mediaBuffer->SetCurrentLength(sampleSize);
    hr = MFCreateSample(&sample);
    hr = sample->AddBuffer(mediaBuffer);
    __int64 mediaTimeStamp = ...(relevant time stamps in WMF ticks)
    __int64 mediaDuration = ...(relevant duration in WMF ticks)
    hr = sample->SetSampleTime(mediaTimeStamp);
    hr = sample->SetSampleDuration(mediaDuration);

    And then I submit them :

    hr = streamSink->ProcessSample(sample); (I also tried sinkWriter->WriteSample(streamIndex, sample))

    Did I miss something ?

    A last info :

    The H264 raw samples are valid, because if I send them to a mov container through QuickTime API, the movie is perfect


    The situation is a little better if I set the MFSampleExtension_CleanPoint to TRUE for I-frames. I cannot use MFSampleExtension_VideoEncodePictureType since I am under Windows 7.
    The sequence will show images more often, but the images are still unexpectedly noisy, contrasted, with gray gaps...

    Thursday, June 2, 2016 12:31 PM
  • I also tried

    to set MF_READWRITE_DISABLE_CONVERTERS to TRUE on the sink writer

    to use outputMediaType->CopyAllItems(inputMediaType);

    The result is always the same.

    I wonder if there are undocumented flags that should be set on the IMFSample

    I also tried to catch the IMFTransform creation to replace it by a SampleCopierMFT, but still the same.
    (The following link explains how to create the class factory to override encoder creation)
    Friday, June 3, 2016 8:27 AM
  • Using MFVideoFormat_NV12 here doe snot make sense:

    hr = inputMediaType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_H264);//typically makes sinkWriter->SetInputMediaType(...) FAIL
    hr = inputMediaType->SetGUID(MF_MT_SUBTYPE, MFVideoFormat_NV12);//nonsense but sinkWriter->SetInputMediaType(...) returns S_OK

    If you set input format to MFVideoFormat_NV12 then even if you feed valid H.264 data, you are not going to produce valid MP4 file. You have to make MFVideoFormat_H264 work.

    Here is a fragment from my app which shows how building valid H.264 media type out of raw video media type:

    CComPtr<IMFSinkWriter> pSinkWriter;
        __C(MFCreateSinkWriterFromURL(Combine(GetPathDirectory(GetModulePath()), sFileName), NULL, NULL, &pSinkWriter));
        MF::CMediaType pWriterMediaType;
        // NOTE: H.264 Video Encoder
        pWriterMediaType[MF_MT_MAJOR_TYPE] = MFMediaType_Video;
        pWriterMediaType[MF_MT_SUBTYPE] = MFVideoFormat_H264;
        pWriterMediaType[MF_MT_AVG_BITRATE] = (UINT32) (10 << 10) * 1000; // 10 MBps
        pWriterMediaType[MF_MT_FRAME_RATE] = pMediaType.GetUINT64(MF_MT_FRAME_RATE);
        pWriterMediaType[MF_MT_FRAME_SIZE] = pMediaType.GetUINT64(MF_MT_FRAME_SIZE);
        pWriterMediaType[MF_MT_INTERLACE_MODE] = pMediaType.GetUINT32(MF_MT_INTERLACE_MODE);
        pWriterMediaType[MF_MT_PIXEL_ASPECT_RATIO] = pMediaType.GetUINT64(MF_MT_PIXEL_ASPECT_RATIO);
        DWORD nWriterStreamIndex;
        __C(pSinkWriter->AddStream(pWriterMediaType, &nWriterStreamIndex));
        __C(pSinkWriter->SetInputMediaType(nWriterStreamIndex, pMediaType, NULL));

    You are supposed to build repsective media type and set it up with the sink writer. Both AddStream and SetInputMediaType will have this media type and sinke writer won't do MFT magic, it will expect H.264 data whcih you get from hardware encoder, shape as a media sample and feed in...

    Friday, June 3, 2016 8:35 PM
  • I seem to do the same as you, but still no success.

    I tried to set MF_MT_MPEG2_LEVEL

    I tried to set the MF_MT_MPEG_SEQUENCE_HEADER as described in

    I tried to set the MF_MT_MPEG_SEQUENCE_HEADER  either before the SinkWriter->BeginWriting() and after the BeginWriting() (using a PlaceMarker callback in the later case), but still the same.

    Do you use Windows 7 too ?

    It seems that Windows 8 has many enhancements regarding Windows Media Foundation.

    Monday, June 6, 2016 2:36 PM
  • I tried different flavours on RateControl mode, both on the encoder size (NVENC or Intel media SDK in my case) and on the WMF configuration. I thought that there could have been some kind of discrepency.

    I still can't get a good-looking output file.

    If I raw write the samples to a file and use ffmpeg to "encapsulate" it in mp4 file (ffmpeg.exe -f h264 -i raw.264 -vcodec copy -r 30 output.mp4), it does work. This is just a new confirmation that the input stream is well formed.

    Do you really get no error when setting the inputMediaType to MFVideoFormat_H264 ? (i get INDALID_MEDIATYPE, though it is basically a clone of the outputMediaType previously set)
    I just ignore the error, but it is a hint that there may be a problem inside the MFT topology (but a real error woudl not generate any file ; in my case I do have an apparently "valid" mp4, that is displayed really improperly)


    Some news :

    I tried to binary compare what I produce with IMF and what I get from "raw H264 in a file+ffmpeg conversion to MP4". The files differs only by a few kb in the header and the footer.  The whole content is identical.
    This is really just a metadata problem regarding IMF MP4 container information.

    Monday, June 13, 2016 11:05 AM
  • [cries in the corner]
    Tuesday, June 28, 2016 2:41 PM
  • You could prepare a small project to demonstrate the problem, and put it online.

    Tuesday, June 28, 2016 3:14 PM
  • You are right, but it took me some time to make a standalone sample code.

    Here are the links : (the Visual project, with input data) (the expected output)

    The README explains what are the files, and explain how to perform the raw H264 in a file+ffmpeg conversion to MP4 mentioned earlier in that thread.

    Please note that I do not provide a link to ffmpeg.exe, since you would not trust it anyway.


    Pierre Chatelier

    Tuesday, July 5, 2016 11:45 AM
  • FYI. The code does not build with VS2015 but eventually fixed include/lib paths make it produce the binary. In Windows 10 the generated .h264 and .mp4 files look good. Specifically, MP4 is playable and displays the cup fine.

    Same code in Windows 7 produces incorrect MP4 file. Apparently the problem is that parameter sets are ignored and something else (defaults?) is used. Hence the artifacts - main profile payload with baseline parameter sets.

    Friday, July 8, 2016 2:14 PM
  • You might be not indicating profile/level, which the sink expects. Try this patch for your code:

        hr = outputMediaType->SetUINT32(MF_MT_VIDEO_PROFILE, 66);
        hr = outputMediaType->SetUINT32(MF_MT_VIDEO_LEVEL, 30);
    //    hr = outputMediaType->SetUINT32(MF_MT_MPEG2_PROFILE, eAVEncH264VProfile_Main);
    //    ::GetErrorMessage(hr);
    //    //hr = outputMediaType->SetUINT32(MF_MT_MPEG2_LEVEL, eAVEncH264VLevel4_2);
    //    //::GetErrorMessage(hr);

    Friday, July 8, 2016 4:17 PM
  • Many thanks.

    I do not have the opportunity to check under W10, but when I read the docs, I see so many changes in W10 that I was suspecting a large improvement.

    For W7, indeed, MF_MT_MPEG2_LEVEL seems to be a good lever. Even when keeping Profile_Main, lowering H264_Level from 4_2 to 3 will produce a correct sequence. It does looks like a bug on WMF side.

    I guess that my only resort is to adapt the code according to the detected Windows version at run-time.

    Monday, July 11, 2016 8:09 AM
  • I thought it was not level itself, but rather MF_MT_VIDEO_PROFILE vs MF_MT_MPEG2_PROFILE. And this should work well in all versions, you don't need to check OS version - just have it everywhere so that Win 7 implementation does not get confused.

    Monday, July 11, 2016 9:25 AM
  • MF_MT_VIDEO_PROFILE and MF_MT_VIDEO_LEVEL require W8.1. They cannot be used under W7.

    Tuesday, July 12, 2016 3:27 PM
  • I don't think so. Before posting previous two messages into this topic I verified this worked as a fix for me in Windows 7 (your project executed in Windows 7 generated proper video!).

    Even though MSDN indeed states it's MF_MT_MPEG2_PROFILE to be used.

    Tuesday, July 12, 2016 3:29 PM
  • I will check again, but I have a link problem when using MF_MT_VIDEO_PROFILE and MF_MT_VIDEO_LEVEL instead of MF_MT_MPEG2_PROFILE  and MF_MT_MPEG2_LEVEL.

    clearly states 

    Minimum supported client

    Windows 8.1 [desktop apps only]

    Wednesday, July 13, 2016 10:00 AM
  • It's just a GUID. The worst case is that it is ignored by the MFT as unknown. The documentation might be inaccurate. Or it is accurate - I am not sure, this is what I saw: encoder accepted it and produced good file. I thought you were trying to achieve exactly this.

    Wednesday, July 13, 2016 10:05 AM
  • Changing MF_MT_MPEG2_PROFILE to MF_MT_VIDEO_PROFILE or MF_MT_MPEG2_PROFILE  to MF_MT_VIDEO_PROFILE  does not change anything.

    Looking at mfapi.h it appears that it is normal : _VIDEO_ versions are exact aliases of _MPEG2_ version : the underlying guids are identical.

    I think that you fixed the bug only by setting 30 to the VIDEO/MPEG2_LEVEL, since indeed it lowers the H264 level to eAVEncH264VLevel3 instead of eAVEncH264VLevel4_2

    Monday, July 18, 2016 7:59 AM
  • And just for information, even in the case where it works, we can observe (at least under Windows 7) that
    sinkWriter->SetInputMediaType(0, inputMediaType, 0);
    returns an error code.

    We can just ignore that error code, but it is miselading at first (see the beginning of that thread when I claimed that setting the input type as MFVideoFormat_NV12 was a nonsense, but was used to get rid of an error)

    Thursday, July 28, 2016 11:51 AM