locked
MFPutWorkItem requires main thread? RRS feed

  • Question

  • I've been getting an exception in my Windows 7 64-bit C++ application, when calling MFPutWorkItem() on a secondary thread. My IMFAsyncCallback::GetParameters() implementation gets called, but not my IMFAsyncCallback::Invoke() implementation. In my call stack I see_beginthreadex() at the top, CWorkQueue::CThread::Init() next down, and CWorkQueue::AddThread() next down again. My call to MFPutWorkItem() targets MFASYNC_CALLBACK_QUEUE_MULTITHREADED.

    When I call MFPutWorkItem() on the main/UI thread, it works fine. Is there a requirement for MFPutWorkItem() to be called on the main thread?

    To explain why I'm trying to use a secondary thread for the call:

    my application is using OpenGL to animate some geometry in my CWnd-based window,

    my OnPaint() message handler calls SwapBuffers() to display the current frame, copies the pixel data associated with the CWnd's area on the screen into a CImage, and releases a secondary thread to convert the CImage's pixel data from its RGB colour space into the YUV colour space for the 'H264 Encoder MFT'-configured IMFTransform, using an OpenGL shader program.

    When the shader program has finished, I was trying to copy the formatted YUV pixel data into an IMFBuffer, add that buffer to an IMFSample, and call MFPutWorkItem(). My Invoke() implementation calls IMFTransform::ProcessInput() on the IMFSample, to perform the H264 video encoding.

    I don't pretend to know much about Media Foundation, and there may well be much better ways of saving my animation into an AVI.

    My problem at this stage though is that MFPutWorkItem() doesn't seem to be happy being called on secondary thread. Is that behaviour expected, or is there something I'm not doing correctly?

    Thursday, October 15, 2015 12:22 PM

Answers

  • Hi David,

    what you trying to do there is i would say the wrong way of doing it.

    First to the MFPutWorkItem : I can tell you there is no special requirement, but it is expected to move work out from the calling thread ( note that its "calling" and not main ). In general, the IMFAsyncCallback should be created in the thread from which you launch the asynchronous method ( Begin ), but its not a requirement. Further more the thread should be CoInitialized with COINIT_MULTITHREADED.

    Now to your encoding and how it would be done right : I am a Direct3D coder myself and i understand that you are maybe doing more in rendering and cant read deeper into the API. In Media Foundation there are 2 Classes for beginners that simplify encoding and decoding. They are called IMFSourceReader and IMFSinkWriter. With the sink writer you can do all what you described in your scenario with one single class.

    The code is very straight forward and maybe not more than 50-100 lines. Whats more important is, you have to encode the backbuffer directly not the image data from the client area of the window, because that would be terribly slow, especialy with color converting on the CPU in between. The Sink Writer can do the color converting, so you dont have to bother with this. Its even using asynchronous calls if you set the MF_SINK_WRITER_DISABLE_THROTTLING flag on the creation of the Writer.

    Its done the following way : First create and setup the Sink Writer, then call IMFSinkWriter::BeginWriting. Then on every Present call you create a copy of your swapchains backbuffer ( IDirect3DSurface9 or ID3D11Texture2D ) and store it in an IMFSample. Call IMFSinkWriter::WriteSample ( its asynchronous and will not block your rendering thread ), and thats it. When you want to finish your encoding/recording all you need to do is call IMFSinkWriter::Finalize.

    The bonus of all this is that the color converting is done on the GPU. You could even use the Hardware Encoder of the GPU for the encoding, so that at the end your encoding will take nearly zero CPU power. There is the MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS flag for hardware encoding when you create the Sink Writer.

    The whole procedure of copying the backbuffer and call WriteSample doesnt cost you more than 2-3 ms on every Present call. So you can do encodings of even 144 FPS if you want to. You cant store your video to AVI container though, Media Foundation only works with ASF, WMV and MP4 prior to Windows 8.1.

    Regards,

    Francis




    • Marked as answer by DavidB5 Saturday, October 17, 2015 6:06 AM
    • Edited by Francis Grave Tuesday, May 31, 2016 3:23 PM Edit
    Friday, October 16, 2015 10:43 AM

All replies

  • Hi David,

    what you trying to do there is i would say the wrong way of doing it.

    First to the MFPutWorkItem : I can tell you there is no special requirement, but it is expected to move work out from the calling thread ( note that its "calling" and not main ). In general, the IMFAsyncCallback should be created in the thread from which you launch the asynchronous method ( Begin ), but its not a requirement. Further more the thread should be CoInitialized with COINIT_MULTITHREADED.

    Now to your encoding and how it would be done right : I am a Direct3D coder myself and i understand that you are maybe doing more in rendering and cant read deeper into the API. In Media Foundation there are 2 Classes for beginners that simplify encoding and decoding. They are called IMFSourceReader and IMFSinkWriter. With the sink writer you can do all what you described in your scenario with one single class.

    The code is very straight forward and maybe not more than 50-100 lines. Whats more important is, you have to encode the backbuffer directly not the image data from the client area of the window, because that would be terribly slow, especialy with color converting on the CPU in between. The Sink Writer can do the color converting, so you dont have to bother with this. Its even using asynchronous calls if you set the MF_SINK_WRITER_DISABLE_THROTTLING flag on the creation of the Writer.

    Its done the following way : First create and setup the Sink Writer, then call IMFSinkWriter::BeginWriting. Then on every Present call you create a copy of your swapchains backbuffer ( IDirect3DSurface9 or ID3D11Texture2D ) and store it in an IMFSample. Call IMFSinkWriter::WriteSample ( its asynchronous and will not block your rendering thread ), and thats it. When you want to finish your encoding/recording all you need to do is call IMFSinkWriter::Finalize.

    The bonus of all this is that the color converting is done on the GPU. You could even use the Hardware Encoder of the GPU for the encoding, so that at the end your encoding will take nearly zero CPU power. There is the MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS flag for hardware encoding when you create the Sink Writer.

    The whole procedure of copying the backbuffer and call WriteSample doesnt cost you more than 2-3 ms on every Present call. So you can do encodings of even 144 FPS if you want to. You cant store your video to AVI container though, Media Foundation only works with ASF, WMV and MP4 prior to Windows 8.1.

    Regards,

    Francis




    • Marked as answer by DavidB5 Saturday, October 17, 2015 6:06 AM
    • Edited by Francis Grave Tuesday, May 31, 2016 3:23 PM Edit
    Friday, October 16, 2015 10:43 AM
  • Sorry forgot that you are using OpenGL there not Direct3D. Media foundation can encode D3D surfaces, but for OpenGL i guess you either have to do some interop to get a Direct3D surface, or if you just want to encode the image data do this :

    Get the Image data and copy it into an IMFMediaBuffer. In Direct3D you would first copy the Backbuffer to a staging texture and then use Lock/Unlock to get your hands on the data. To copy the data to the IMFMediaBuffer use MFCopyImage. Then call WriteSample.

    Friday, October 16, 2015 11:06 AM
  • Thanks for your help. My code didn't specifically call CoInitialize() at all, so perhaps that was the problem (if it defaults to an STA model, maybe that explains why it wasn't happy starting the CWorkQueue thread). What's the role of MFStartup() in the environment initialisation then?

    By using an OpenGL shader program run on a secondary thread, the RGB->YUV colour space conversion occurs on the GPU (hardware?) without impeding the main thread as is. I'll certainly have a look at the ISinkWriter interface, but in the meantime I'd moved the code previously in my IMFAsyncCallback::Invoke() implementation, into a function that runs on (another) secondary thread and calls the IMFTransform::ProcessInput() and IMFTransform::ProcessOutput() methods to carry out the H264 encoding, meaning I didn't need to use MFPutWorkItem() (but had to create another thread of my own). Next puzzle if I pursue my initial approach is the AVI file format/calls. The performance of ISinkWriter sounds appealing, but then I'm doing most of this to learn new things anyway, and feeding stuff into a 'black-box' isn't always the best way to do that.

    Thanks for your time.


    Saturday, October 17, 2015 6:06 AM
  • The MFStartup routine is mainly for initializing the "Standard Work Queues" that the platform offers. It might do some extra initializations for objects that must be static available to the API, but as far as i know its solely for the Work Queues. You can create custom Work Queues and there is tons of stuff around Threading and Queues in Media Foundation.

    Encoding is a field that can be quite deep and you are right it would take some time to read in, but with the Sink Writer i would say is pretty easy and spares you all the thread creation and fiddeling.

    I am working on a Game Recorder, which is finished but still in the cleanup phase. The program comes to the market very soon and my latest performance tests show it might be even faster then NVIDIA's ShadowPlay or AMD's Raptor. Its quite a masterpiece, but it consists of half a million lines of code just for the Encoding ! And thats only whats left after 6 years of work and hundreds of times iterating over it. So i understand that you try working it out by yourself with some thread creation here and there and some "puzzling" :)

    If your Video Encoding is supposed to be just a tiny little side gadget and not the purpose of your main work, then my advise is : Dont read in deeper as necessary, as it could distract you from Rendering Work o/

    If you want to keep your Encoding part small but stable the Sink Writer is exactly what you are looking for. And you dont need any deep knowledge of Media Foundation to use it. If you want to work it out without the Sink Writer and still plan to use the MFPutWorkItem method, help me to understand the threading you doing there.

    So when you are done in the color conversion thread you want to call MFPutWorkItem from there ? Is your AsyncCallback created in the rendering thread or in the color conversion thread ? Is the Color conversion thread still alive after you call MFPutWorkItem ? If not that could be the problem.

    I would do it this way :

    Create and initialize your CImage. If your CImage class doesnt inherit from an IUnknown Interface, create a container and store the image, then call MFPutWorkItem. On the invoke call do your color conversion and Encoding.

    I will post you the appropriate container code in the next few hours. I am typing this from my gaming machine atm, not my workstation, and there is no Visual Studio on it 8)

    Regards,

    Francis



    Saturday, October 17, 2015 6:22 PM
  • If your CImage already inherits from IUnknown then simply call

    hr = MFPutWorkItem(MFASYNC_CALLBACK_QUEUE_STANDARD, pCallback, pImage); // pImage is your CImage

    Then in the Invoke call

    		HRESULT hr = S_OK;
    
    		IUnknown *pUnknown;
    
    		hr = pResult->GetState(&pUnknown);
    
    		CImage *pImage = static_cast<CImage*>(pUnknown);
    
    		// Now do your Async job here :
    
    		// 1. Color Conversion
    		// 2. Copy data in to a media buffer and store it in sample
    		// 3. Call ProcessInput and ProcessOutput methods

    If your CImage does not inherit from IUnknown you could either add the implementation or build yourself a Container like this :

    class AsyncImage : public IUnknown
    {
    	LONG m_cRef;
    
    	CImage *m_pImage;
    
    public:
    	AsyncImage() : m_cRef(1), m_pImage(NULL) { }
    	~AsyncImage()
    	{
    		if (m_pImage != NULL)
    		{
    			m_pImage->Release();
    			m_pImage = NULL;
    		}
    	}
    
    	STDMETHODIMP QueryInterface(REFIID riid, void **ppv)
    	{
    		static const QITAB qit[] = { QITABENT(AsyncImage, IUnknown), { 0 } };
    
    		return QISearch(this, qit, riid, ppv);
    	}
    
    	STDMETHODIMP_(ULONG) AddRef()
    	{
    		return InterlockedIncrement(&m_cRef);
    	}
    
    	STDMETHODIMP_(ULONG) Release()
    	{
    		LONG cRef = InterlockedDecrement(&m_cRef);
    
    		if (cRef == 0)
    		{
    			delete this;
    		}
    
    		return cRef;
    	}
    
    	HRESULT SetImage(CImage *pImage)
    	{
    		if (!pImage)
    		{
    			return E_INVALIDARG;
    		}
    
    		m_pImage = pImage;
    		m_pImage->AddRef();
    
    		return S_OK;
    	}
    
    	HRESULT GetImage(CImage **ppImage)
    	{
    		if (!m_pImage)
    		{
    			return MF_E_NOT_INITIALIZED;
    		}
    
    		*ppImage = m_pImage;
    		(*ppImage)->AddRef();
    
    		return S_OK;
    	}
    };

    Then call MFPutWorkItem like above and in the invoke get the container and from it the CImage.

    NOTE :

    Serialized Work Queues can be considered a Thread and therefore your Images are getting encoded after each other. A multithreaded Work Queue is dispatching items on the System thread pool, means the next thread available will do the work. If you want to use a synchronous working MFT with ProcessInput and ProcessOutput you have to do it serialized. With a multithreaded work Queue it could lead to errors.

    Regards,

    Francis







    • Proposed as answer by Francis Grave Sunday, October 18, 2015 12:34 PM
    • Edited by Francis Grave Tuesday, May 31, 2016 3:25 PM Edit
    Sunday, October 18, 2015 12:26 PM