Skip to main content

 none
Video Decoding with custom MFT or MSFT H.264 Decoder MFT RRS feed

  • Question

  • Hi,

    I'm trying to decode an h.264 video stream in real-time, and I had 2 questions.

    1) There used to be a delay with the MSFT H.264 Decoder MFT of about 1 second. Has this been resolved?

    2) When I call IMFTransform::ProcessOutput, it works the first 8 times I call it, and after that it just hangs. What could be causing this?

    Thanks!


    Abhishek Bhargava

    Friday, August 9, 2019 8:36 AM

All replies

  • Hi

    1) There used to be a delay with the MSFT H.264 Decoder MFT of about 1 second. Has this been resolved?

    I don't see any fixes to this problem, not from Wikipedia or MSDN documents.

    2) When I call IMFTransform::ProcessOutput, it works the first 8 times I call it, and after that it just hangs. What could be causing this?

    This requires code analysis to determine the cause of the problem. It is recommended that you check the IMFTransform::ProcessOutput return value and further analyze it based on the return value.

    Best regards,

    Strive


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    Monday, August 12, 2019 8:32 AM
  • Hi, thanks for the response. It looks like the low latency mode option might have fixed it.

    I am doing that currently, but it hangs on ProcessOutput. Here's the code:

    I've tried many variations of releasing different things and removing buffers / not removing buffers and in all cases, either it hangs (if I don't release the output sample), or it gives me an output sample (outputBuffer.pSample) with no buffers, so no output. This always happens after exactly 8 successful decodes. Would really appreciate some help. 

    // Multithreading protection ID3D11Multithread* pMultithread; hr = pContext->QueryInterface(IID_PPV_ARGS(&pMultithread)); CHECK_HR(hr, "Failed to create Multithread object."); // Set multithread protected pMultithread->SetMultithreadProtected(true); // Iteration number int iteration_num = 0; // Decoding loop while (true) { mftStatus = 0; hr = pDecoderTransform->GetInputStatus(0, &mftStatus); CHECK_HR(hr, "Failed to GetInputStatus"); if (mftStatus != MFT_INPUT_STATUS_ACCEPT_DATA) { std::cout << "H.264 decoder MFT is not accepting data." << std::endl; exit(EXIT_FAILURE); } hr = MFCreateMemoryBuffer(file_size, &pMediaBuffer); CHECK_HR(hr, "Failed to MFCreateMemoryBuffer"); hr = MFCreateSample(&pVideoSample); CHECK_HR(hr, "Failed to MFCreateSample"); // Get pointer to MediaBuffer memory BYTE* pData; hr = pMediaBuffer->Lock(&pData, NULL, NULL); CHECK_HR(hr, "Failed to Lock MediaBuffer"); // Copy the video data into the media buffer memcpy_s(pData, file_size, video_bytes, file_size); // Set the new size of the buffer hr = pMediaBuffer->SetCurrentLength(file_size); CHECK_HR(hr, "Failed to SetCurrentLength for media buffer"); // Unlock the media buffer hr = pMediaBuffer->Unlock(); CHECK_HR(hr, "Failed to Unlock() MediaBuffer"); // Add buffer to the video sample hr = pVideoSample->AddBuffer(pMediaBuffer); CHECK_HR(hr, "Failed to AddBuffer to the video sample"); // Pass video sample to H.264 decoder hr = pDecoderTransform->ProcessInput(0, pVideoSample, 0); CHECK_HR(hr, "Failed to ProcessInput to video decoder."); while (true) { // Get output MFT_OUTPUT_DATA_BUFFER outputBuffer = { 0 }; memset(&outputBuffer, 0, sizeof(outputBuffer)); DWORD status = 0; outputBuffer.dwStreamID = 0; outputBuffer.dwStatus = 0; outputBuffer.pEvents = NULL; outputBuffer.pSample = NULL; hr = pDecoderTransform->ProcessOutput(MFT_PROCESS_OUTPUT_DISCARD_WHEN_NO_BUFFER, 1, &outputBuffer, &status); if (hr == MF_E_TRANSFORM_NEED_MORE_INPUT) { std::cout << "NEED MORE INPUT" << std::endl << std::endl; break; } else if (hr == MF_E_TRANSFORM_STREAM_CHANGE) { std::cout << "MF_E_TRANSFORM_STREAM_CHANGE" << std::endl; DWORD inputStreamCount; DWORD outputStreamCount; IMFMediaType* availableOutputType; CHECK_HR(pDecoderTransform->GetOutputAvailableType(0, 0, &availableOutputType), "GetOutputAvailableType failed."); CHECK_HR(pDecoderTransform->SetOutputType(0, availableOutputType, 0), "SetOutputType failed."); hr = pDecoderTransform->ProcessMessage(MFT_MESSAGE_COMMAND_FLUSH, NULL); CHECK_HR(hr, "Failed to ProcessMessage MFT_MESSAGE_COMMAND_FLUSH"); } else if (hr == S_OK) { DWORD bufLength; IMFMediaBuffer* mfOutputBuffer = NULL;

    IMFSample* mfOutputSample = outputBuffer.pSample; IMFDXGIBuffer* pDXGIBuffer = NULL; ID3D11Texture2D* pSurface = NULL; DWORD buf_count; hr = mfOutputSample->GetBufferCount(&buf_count); CHECK_HR(hr, "Failed to GetBufferCount."); hr = mfOutputSample->GetBufferByIndex(0, &mfOutputBuffer); if (FAILED(hr)) { if (outputBuffer.pEvents) { outputBuffer.pEvents->Release(); } mfOutputSample->Release(); handle_result(hr); pDecoderTransform->ProcessMessage(MFT_MESSAGE_COMMAND_FLUSH, NULL); break; } // Now we get Texture2D hr = mfOutputBuffer->QueryInterface(__uuidof(IMFDXGIBuffer), (void**)& pDXGIBuffer); CHECK_HR(hr, "Failed to get IMFDXGIBuffer from IMFMediaBuffer"); hr = pDXGIBuffer->GetResource(IID_PPV_ARGS(&pSurface)); CHECK_HR(hr, "Failed to GetResource, IMFDXGIBuffer to ID3D11Texture2D"); // RGBA tetxure ID3D11Texture2D* pRGBATex; D3D11_TEXTURE2D_DESC rgba_desc; pSurface->GetDesc(&rgba_desc); rgba_desc.MipLevels = 1; rgba_desc.ArraySize = 1; rgba_desc.Format = DXGI_FORMAT_B8G8R8A8_UNORM; rgba_desc.SampleDesc.Count = 1; rgba_desc.Usage = D3D11_USAGE_DEFAULT; rgba_desc.BindFlags = D3D11_BIND_RENDER_TARGET; rgba_desc.CPUAccessFlags = 0; hr = pDevice->CreateTexture2D(&rgba_desc, NULL, &pRGBATex); CHECK_HR(hr, "Failed to create RGBA texture"); std::cout << "Done with one iteration" << std::endl; /* ------------------ CLEANUP ----------------- */ if (outputBuffer.pEvents) { outputBuffer.pEvents->Release(); } hr = mfOutputSample->RemoveAllBuffers(); CHECK_HR(hr, "Failed to remove all buffers."); if (outputBuffer.pSample) { outputBuffer.pSample->Release(); } if (mfOutputBuffer) { mfOutputBuffer->Release(); } if (outputBuffer.dwStatus & MFT_OUTPUT_DATA_BUFFER_INCOMPLETE) { std::cout << "MFT_OUTPUT_DATA_BUFFER_INCOMPLETE" << std::endl; } iteration_num += 1; pDecoderTransform->ProcessMessage(MFT_MESSAGE_COMMAND_FLUSH, NULL); } else { std::cout << "Bad HRESULT in decoding step" << std::endl << std::endl; } } if (pVideoSample) { pMediaBuffer->Release(); pVideoSample->Release(); pVideoSample->RemoveAllBuffers(); } }



    Abhishek Bhargava

    Tuesday, August 13, 2019 6:30 AM
  • Hi

    Thank you for your reply.

    In view of the code you provided, I find some information that might be useful to you, as follows

    The behavior of ProcessOutput depends on the initial value of pSample and the value of the dwFlags parameter in the ProcessOutput method.

    • If pSample is NULL and dwFlags contains the MFT_PROCESS_OUTPUT_DISCARD_WHEN_NO_BUFFER flag, the MFT discards the output data.

      Restriction: This output stream must have the MFT_OUTPUT_STREAM_DISCARDABLE or MFT_OUTPUT_STREAM_LAZY_READ flag. (To get the flags for the output stream, call the IMFTransform::GetOutputStreamInfo method.)

    Refer: https://docs.microsoft.com/en-us/windows/win32/api/mftransform/ns-mftransform-mft_output_data_buffer#remarks

    This is an example of encoder processing:https://docs.microsoft.com/en-us/windows/win32/medfound/processing-data-in-the-encoder#encoder-processing-example

    In addition, I think what you should do is not just delete / add buffers, but check the values of variables in each line of code to determine which variable has an exception.

    Best regards,

    Strive


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    Tuesday, August 13, 2019 9:13 AM
  • Hmm even when I remove the MFT_PROCESS_OUTPUT_DISCARD_WHEN_NO_BUFFER flag, I see the same behavior after 8 iterations.

    When I call GetOutputStreamInfo, neither the MFT_OUTPUT_STREAM_DISCARDABLE flag or the MFT_OUTPUT_STREAM_LAZY_READ flags are set. Also, the MFT_OUTPUT_STREAM_PROVIDES_SAMPLES flag is set while the MFT_OUTPUT_STREAM_CAN_PROVIDE_SAMPLES flag is not set. 

    Also, none of the variables have an exception. The behavior is that after exactly 8 successful decodes, the outputBuffer.pSample has no valid buffers. I know this because outputBuffer.pSample->GetBufferCount(&buf_count) returns 0. So there's no decoded frame in the output sample.

    In the encoder example, the outputBuffer.pSample is set to a non-Null value. However, when I call GetOutputStreamInfo the MFT_OUTPUT_STREAM_PROVIDES_SAMPLES flag is set, which means I can't allocate and provide my own output samples. Should I try allocating and providing my own output samples anyway?

    Thanks so much for your help! Looking forward to a response.


    Abhishek Bhargava

    Tuesday, August 13, 2019 9:32 PM
  • Hi

    Would you mind sharing your complete case? 

    I will contact relevant engineers to discuss this issue in detail.

    Best regards,

    Strive


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    Thursday, August 15, 2019 2:33 AM
  • Strive Sun-MSFT, your signature indicates Microsoft affiliation and this leaves a false assumption that commenting here you are providing official support and you have good understanding of Media Foundation, which is clearly not true. For avoidance of this doubt maybe it's better to add some sort of disclaimer.

    http://alax.info/blog/tag/directshow

    Thursday, August 15, 2019 5:14 AM
  • See question Should I release the buffers within IMFSample before I release the IMFSample object? and especially the comment I gave under the answer.

    I can't run the code snippet you provided because it is out of context, but I am guessing the problem is exactly your unsupposed buffer removal.


    http://alax.info/blog/tag/directshow


    Thursday, August 15, 2019 6:29 AM
  • Thanks Roman and Strive, got this working by releasing pointers that Windows had internally allocated.

    Abhishek Bhargava


    Sunday, August 25, 2019 4:18 AM