locked
How to add audio data to a video file created by a SinkWriter ? RRS feed

  • Question

  • Hello,

     

    I am using Microsoft Visual Studio 2010 with Windows 7 Pro x64 for my work (C++). I have to mux an audio stream with a video stream. For that, I am using Media Foundation, in particular the SinkWriter, which seems to do this task...

    I used a tutorial, provided by Microsoft MSDN, to encode a video stream : in input of the SinkWriter, I give a buffer of uncompressed RGB 32bits pictures and in output I receive a video stream encoded H264. In the exemple, pictures are the same (static) and represent a solid green rectangle. That is not the most important and not what I am here. The video seems to be good, I can read it with Windows Media Player (no codec problem or something else). I used a .mp4 container.

    My problem is to add an audio stream (AAC, for example) to this video. How to do that ? Do I have to creat another SinkWriter, which will only treat audio, and after add this audio stream with the video ? Or, should I add directly to the created SinkWriter (video) the treatment of an audio stream ?

     

    Please, I am waiting for your helpful answers...

    Thank you so much !

     

    Julien

    Tuesday, June 7, 2011 7:20 AM

All replies

  • Is somebody interested about my problem?

    Please, can you help me? I really need your help!

     

    Best regards.

     

    Julien

    Friday, June 10, 2011 6:42 AM
  • hi there,

    althoug i haven't done something like this before, i guess you could just add another stream to the streamwriter.

    the IMFSinkWriter has a SetInputMediaType Method that allows to set a given media type for the stream. therefore, you would probably call this method twice (for your video stream and your audio stream).

    After that you just place the IMFSample´s (for audio and video) using the corresponding stream index

    the AddStream, SetInputMediatype or WriteSample method all allow to set a stream index identifier. 

    give it a try

    Friday, June 10, 2011 8:43 AM
  • Hi,

     

    At the moment, I just have a mean to generate 2 files with the SinkWriter, one which is video stream (encoding by H264) stored in a mp4 conteneur and another which is audio stream (encoding by Windows Media Audio Version 8) stored in a wma conteneur. I can not create a GOOD single file (mp4 format) which could associate the two streams...

    I tried this following code without any satisfying results :

    #include "wmcodecdsp.h"
    #include "mfapi.h"
    #include "mfidl.h"
    #include "Mfreadwrite.h"
    #include "mferror.h"
    
    #pragma comment(lib, "mfreadwrite")
    #pragma comment(lib, "mfplat")
    #pragma comment(lib, "mfuuid")
    
    
    const UINT32 VIDEO_WIDTH=720;
    const UINT32 VIDEO_HEIGHT=576;
    const UINT32 VIDEO_FPS=25;
    const UINT32 VIDEO_BIT_RATE=2000000;
    const GUID VIDEO_ENCODING_FORMAT=MFVideoFormat_H264;
    const GUID VIDEO_INPUT_FORMAT=MFVideoFormat_RGB32;
    const UINT32 VIDEO_PELS=VIDEO_WIDTH * VIDEO_HEIGHT;
    const UINT32 VIDEO_FRAME_COUNT 20 * VIDEO_FPS;
    
    const UINT32 SAMPLES_PER_SECOND=44100;
    const UINT32 AVG_BYTES_PER_SECOND=6003;
    const UINT32 NUM_CHANNELS=1;
    const UINT32 BITS_PER_SAMPLE=16;
    const UINT32 BLOCK_ALIGNMENT=2230;
    const UINT32 ONE_SECOND=10;
    const UINT32 BUFFER_LENGTH=BITS_PER_SAMPLE / 8 * NUM_CHANNELS * ( SAMPLES_PER_SECOND / ONE_SECOND );	
    const LONGLONG SAMPLE_DURATION=10000000 / (LONGLONG)ONE_SECOND;
    
    DWORD videoFrameBuffer[VIDEO_PELS];
    
    
    template <class T> void SafeRelease(T **ppT)
    {
      if (*ppT)
      {
        (*ppT)->Release();
        *ppT = NULL;
      }
    }
    
    
    //Initialisation of the SinkWriter...
    HRESULT InitializeSinkWriter(IMFSinkWriter **ppWriter, DWORD *pStreamIndex)
    {
      *ppWriter = NULL;
      *pStreamIndex = NULL;
    
      IMFSinkWriter  *pSinkWriter = NULL;
      IMFMediaType  *pMediaTypeOut = NULL;  
      IMFMediaType  *pMediaTypeIn = NULL; 
    	IMFAttributes	*pAttrib = NULL;
      DWORD      streamIndex;  
    	
      HRESULT hr = MFCreateSinkWriterFromURL(L"output.mp4", NULL, NULL, &pSinkWriter);
    
    	//output
      if (SUCCEEDED(hr))
      {
        hr = MFCreateMediaType(&pMediaTypeOut);  
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeOut->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);   
      }
    	if (SUCCEEDED(hr))
      {
        hr = pMediaTypeOut->SetGUID(MF_MT_SUBTYPE, VIDEO_ENCODING_FORMAT);
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeOut->SetUINT32(MF_MT_AVG_BITRATE, VIDEO_BIT_RATE); 
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeOut->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeSize(pMediaTypeOut, MF_MT_FRAME_SIZE, VIDEO_WIDTH, VIDEO_HEIGHT);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeOut, MF_MT_FRAME_RATE, VIDEO_FPS, 1);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeOut, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);  
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetGUID( MF_MT_MAJOR_TYPE, MFMediaType_Audio );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetGUID( MF_MT_SUBTYPE, MFAudioFormat_WMAudioV8 );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_SAMPLES_PER_SECOND, SAMPLES_PER_SECOND );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_AVG_BYTES_PER_SECOND, AVG_BYTES_PER_SECOND );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_NUM_CHANNELS, NUM_CHANNELS );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_BITS_PER_SAMPLE, BITS_PER_SAMPLE );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_PREFER_WAVEFORMATEX, 1 );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetUINT32( MF_MT_ALL_SAMPLES_INDEPENDENT, 1 );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetUINT32( MF_MT_FIXED_SIZE_SAMPLES, 1 );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_BLOCK_ALIGNMENT, BLOCK_ALIGNMENT );
      }
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->AddStream(pMediaTypeOut, &streamIndex);  
      }
    
      //input
      if (SUCCEEDED(hr))
      {
        hr = MFCreateMediaType(&pMediaTypeIn);  
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeIn->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);  
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeIn->SetGUID(MF_MT_SUBTYPE, VIDEO_INPUT_FORMAT);   
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeIn->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);  
      } 
    	if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeSize(pMediaTypeIn, MF_MT_FRAME_SIZE, VIDEO_WIDTH, VIDEO_HEIGHT);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeIn, MF_MT_FRAME_RATE, VIDEO_FPS, 1);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeIn, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);  
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetGUID( MF_MT_MAJOR_TYPE, MFMediaType_Audio );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetGUID( MF_MT_SUBTYPE, MFAudioFormat_PCM );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_BITS_PER_SAMPLE, BITS_PER_SAMPLE );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_SAMPLES_PER_SECOND, SAMPLES_PER_SECOND );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_NUM_CHANNELS, NUM_CHANNELS );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_PREFER_WAVEFORMATEX, 1 );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_BLOCK_ALIGNMENT, BITS_PER_SAMPLE / 8 * NUM_CHANNELS );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_AVG_BYTES_PER_SECOND, BITS_PER_SAMPLE / 8 * NUM_CHANNELS * SAMPLES_PER_SECOND );
      }
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->SetInputMediaType(streamIndex, pMediaTypeIn, NULL);  
      }
    
    	//Tell to the SinkWriter begin writing data...
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->BeginWriting();
      }
    	else
    	{
    		if(hr == MF_E_INVALIDMEDIATYPE)
    			UINT32 uiShutDown=0;
    
    		if(hr == MF_E_INVALIDSTREAMNUMBER)
    			UINT32 uiShutDown=1;
    
    		if(hr == MF_E_TOPO_CODEC_NOT_FOUND)
    			UINT32 uiShutDown=2;
    	}
    
      //Pointer of the calleur
      if (SUCCEEDED(hr))
      {
        *ppWriter = pSinkWriter;
        (*ppWriter)->AddRef();
        *pStreamIndex = streamIndex;
      }
    
      SafeRelease(&pSinkWriter);
      SafeRelease(&pMediaTypeOut);
      SafeRelease(&pMediaTypeIn);
    	SafeRelease(&pAttrib);
      return hr;
    }
    
    //Write a frame
    HRESULT WriteFrame(
      IMFSinkWriter *pWriter, 
      DWORD streamIndex, 
      const LONGLONG& rtStart,    //Time stamping
      const LONGLONG& rtDuration   //Duration of a frame
      )
    {
      IMFSample *pSample = NULL;
      IMFMediaBuffer *pBuffer = NULL;
    
      LONG cbWidth = 4 * VIDEO_WIDTH;
      DWORD cbBuffer = cbWidth * VIDEO_HEIGHT;
    	DWORD cbBufAudio = BUFFER_LENGTH / VIDEO_FPS;
    	DWORD cbMaxSizeBuf= cbBuffer + cbBufAudio;
    
      BYTE *pData = NULL;
    
      //Creation of the memory buffer
      HRESULT hr = MFCreateMemoryBuffer(cbMaxSizeBuf, &pBuffer);
    
      if (SUCCEEDED(hr))
      {
        hr = pBuffer->Lock(&pData, NULL, NULL);
      }
      if (SUCCEEDED(hr))
      {
        hr = MFCopyImage(
          pData,           //Destination buffer
         cbWidth,
          (BYTE*)videoFrameBuffer,
          cbWidth,
          cbWidth,
          VIDEO_HEIGHT
          );
    
    		for( DWORD n=cbBuffer; n < cbMaxSizeBuf; n++ )
    		{
    			pData[n] = (BYTE)( rand() & 0xFF );
    		}
      }
      if (pBuffer)
      {
        pBuffer->Unlock();
      }
    
      //Update buffer size
      if (SUCCEEDED(hr))
      {
        hr = pBuffer->SetCurrentLength(cbMaxSizeBuf);
      }
    
      if (SUCCEEDED(hr))
      {
        hr = MFCreateSample(&pSample);
      }
      if (SUCCEEDED(hr))
      {
        hr = pSample->AddBuffer(pBuffer);
      }
    
      //Update time stamping and duration
    	if (SUCCEEDED(hr))
      {
        hr = pSample->SetSampleTime(rtStart);
      }
      if (SUCCEEDED(hr))
      {
        hr = pSample->SetSampleDuration(rtDuration);
      }
    
    	if (SUCCEEDED(hr))
      {
        hr = pWriter->WriteSample(streamIndex, pSample);
      }
    
      SafeRelease(&pSample);
      SafeRelease(&pBuffer);
      return hr;
    }
    
    //main (this is the main window, into the OK Button)
    void CMP4_Sink_WriterDlg::OnBnClickedOk()
    {	
    	CFile			FAvi;
    	CFileException	FExcept;
    	DWORD		j=0;
    
      HRESULT hr = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
      if (SUCCEEDED(hr))
      {
        hr = MFStartup(MF_VERSION);
        if (SUCCEEDED(hr))
        {
          IMFSinkWriter *pSinkWriter = NULL;
          DWORD stream;
    
          hr = InitializeSinkWriter(&pSinkWriter, &stream);
          if (SUCCEEDED(hr))
          {
            //Send frames to the SinkWriter
            LONGLONG rtStart = 0;
            UINT64 rtDuration;
    
            MFFrameRateToAverageTimePerFrame(VIDEO_FPS, 1, &rtDuration);
    
    				//VIDEO AND AUDIO
            for (DWORD i = 0; i < VIDEO_FRAME_COUNT; ++i, ++j)
            {
    					//Pixels of the picture
    					for (DWORD k = 0 ; k < VIDEO_PELS; k++)
    					{
    						if(j>255)
    							j=0;
    
    						videoFrameBuffer[k] = ((j<<16) & 0x00FF0000) | ((j<<8) & 0x0000FF00) | (j & 0x000000FF);
    					}
    
              hr = WriteFrame(pSinkWriter, stream, rtStart, rtDuration);
              if (FAILED(hr))
              {
                break;
              }
              rtStart += rtDuration;
            }
          }
          if (SUCCEEDED(hr))
          {
            hr = pSinkWriter->Finalize();
          }
          SafeRelease(&pSinkWriter);
          MFShutdown();
        }
        CoUninitialize();
      }
    }
    
    

    Indeed, the "ouput.mp4" file generated can not be open by any player... Something must be wrong into my code!

    Please, can you tell me more ?

     

    Best regards.

     

    Julien

    Friday, June 10, 2011 2:57 PM
  • Hey Clad,

    1. The best way to take an existing video and add audio to it requires you to create a new media file which will contain both the video and audio. Read the video samples from the original file and write them into the new file while also writing the audio stream samples.

    2. If you read the Media Foundation SDK documentation about MP4 container, it supports exactly one video and one audio stream. The audio stream will not take WMA formats. It only supports AAC & MP3 audio and makes use of the built-in AAC encoder and decoder. For full information about using MP4 container with MF, see http://msdn.microsoft.com/en-us/library/dd757780(v=VS.85).aspx

    3. I didn't completely read your code snippet but one reason your MP4 files don't play back in players are that you haven't set the Sample Description Box for the video stream. In the case of H.264, the Sample Description box is an array of bytes which represent the SPS & PPS of the H.264 video. Players require the SPS & PPS to be embedded in the Sample Description box. The only exception I have seen is that the Microsoft H.264 decoder ignores this and requires that the SPS & PPS appears at least once in the data stream. This is because the Microsoft H.264 decoder supports media format changes within the stream.

    To set the Sample Description box, you need to set the Sink Writer output media type's Blob. Look at IMFAttributes::SetBlob() and read http://msdn.microsoft.com/en-us/library/dd319566(v=VS.85).aspx

    Hope this helps.

    Saturday, June 11, 2011 12:52 PM
  • Hi Nobby_

     

    Need mor help about :

    "1. The best way to take an existing video and add audio to it requires you to create a new media file which will contain both the video and audio. Read the video samples from the original file and write them into the new file while also writing the audio stream samples."

    How do you do that? I mean, how do you add audio stream samples to the video samples? Do you have an example of code for that? With mine, using a SinkWriter which treat both audio and video stream, and writing samples one by one (one image, one audio stream, one image, one audio stream etc... till the end), that is not working well, my final output file can not be played!


    "2. If you read the Media Foundation SDK documentation about MP4 container, it supports exactly one video and one audio stream. The audio stream will not take WMA formats. It only supports AAC & MP3 audio and makes use of the built-in AAC encoder and decoder. For full information about using MP4 container with MF, see http://msdn.microsoft.com/en-us/library/dd757780(v=VS.85).aspx."

    Yes, WMA can not be used in a MP4 container, I saw that. I have to use the AAC profile. MP3 could be used but you must import a codec by yourself, it can not use a built-in codec, such as the AAC do.
    I read these documents, I found some tutorials to encode H264 video with AAC audio, but it does not work too!


    "3. I didn't completely read your code snippet but one reason your MP4 files don't play back in players are that you haven't set the Sample Description Box for the video stream. In the case of H.264, the Sample Description box is an array of bytes which represent the SPS & PPS of the H.264 video. Players require the SPS & PPS to be embedded in the Sample Description box. The only exception I have seen is that the Microsoft H.264 decoder ignores this and requires that the SPS & PPS appears at least once in the data stream. This is because the Microsoft H.264 decoder supports media format changes within the stream."

    I did not use a sample description box... I will try to create one and test the code I wrote again.

     

    Thanks for helping, feel free to give me more help ^^

    Best regards,

     

    Julien


    Tuesday, June 14, 2011 3:36 PM
  • 1. It shouldn't matter if you alternate between writing audio samples and video samples or writing all the video samples first then all the audio samples second. All the chunk offsets within the file are managed by the MP4 container format as you write the samples. As long as the call to IMFSinkWriter->Finalize() returns S_OK, there aren't any critical problems with the file. As I said before, you need to insert the sample description box into the Media Type. IMFSinkWriter->Finalize() will still return S_OK if you forget the define the sample description box but the file will not play back in any media player if you don't.

    I don't have any example code for you as I don't personally work with audio yet. There is possibly even a way of augmenting the original MP4 file to add the audio to it but personally, I'd create a new MP4 file, copy the original video samples across and also add the audio data.

    2. If the tutorials aren't helping you, I'm not sure what the problem could be. I haven't used audio in MP4 yet myself.

    Wednesday, June 15, 2011 12:39 AM
  • Thank you very much Nobby_

     

    Ok, I will have to build this sample description box. I will tell you later whether it works or not... Let's go!

     

    Regards;

     

    Julien

    Wednesday, June 15, 2011 6:41 AM
  • Hi,

     

    I created the sample description box, as Nobby told me to do. The result is... not what I expected ^^.

    Indeed, it is still impossible for me to play a file I created (with WMP or VLC). I am really disappointed. I think I just do something wrong in my code (find it below my message). Please, someone could help me to do what I want => Create a file which includes both audio and video, using MPEG-4 format. I mean a file, playable with a player such as Media Player, encoded in H264 for video streams and AAC for audio streams.

    Tell me how to do this please! I am beginning crazy with this work :(

     

    Here is my code (something must be wrong, that is sure!!!) :

    //Include headers from SDK 7.1
    #include "wmcodecdsp.h"
    #include "mfapi.h"
    #include "mfidl.h"
    #include "Mfreadwrite.h"
    #include "mferror.h"
    
    
    //Include libs from SDK 7.1
    #pragma comment(lib, "mfreadwrite")
    #pragma comment(lib, "mfplat")
    #pragma comment(lib, "mfuuid")
    
    
    //video constantes
    const UINT32	VIDEO_WIDTH = 720;
    const UINT32	VIDEO_HEIGHT = 576;
    const UINT32	VIDEO_FPS = 25;
    const UINT32	VIDEO_BIT_RATE = 2000000;
    const GUID		VIDEO_ENCODING_FORMAT = MFVideoFormat_H264;
    const GUID		VIDEO_INPUT_FORMAT = MFVideoFormat_RGB32;
    const UINT32	VIDEO_PELS = VIDEO_WIDTH * VIDEO_HEIGHT;	
    const UINT32	VIDEO_FRAME_COUNT = 20 * VIDEO_FPS;
    
    //audio constantes
    const UINT32	SAMPLES_PER_SECOND = 44100;
    const UINT32	AVG_BYTES_PER_SECOND = 16000;
    const UINT32	NUM_CHANNELS = 1;
    const UINT32	BITS_PER_SAMPLE = 16;
    const UINT32	BLOCK_ALIGNMENT = 2230;
    const UINT32	ONE_SECOND = 10;
    const UINT32	BUFFER_LENGTH = BITS_PER_SAMPLE /
    	8 * NUM_CHANNELS * SAMPLES_PER_SECOND;
    const LONGLONG	SAMPLE_DURATION = 10000000;
    
    
    //Buffer for video streams
    DWORD videoFrameBuffer[VIDEO_PELS];
    
    
    //Creation a template
    template <class T> void SafeRelease(T **ppT)
    {
      if (*ppT)
      {
        (*ppT)->Release();
        *ppT = NULL;
      }
    }
    
    
    //Initialisation of the SinkWriter...
    HRESULT InitializeSinkWriter(IMFSinkWriter **ppWriter, DWORD *pStreamIndex)
    {
      *ppWriter = NULL;
      *pStreamIndex = NULL;
    
      IMFSinkWriter  *pSinkWriter = NULL;
      IMFMediaType  *pMediaTypeOut = NULL;  
      IMFMediaType  *pMediaTypeIn = NULL; 
    	IMFAttributes	*pAttrib = NULL;
      DWORD      streamIndex;  
    	UINT8			*pBuf8b = NULL;
    	UINT32			ArraySize = 0;
    	
      HRESULT hr = MFCreateSinkWriterFromURL(L"output.mp4", NULL, NULL, &pSinkWriter); //output file whose container will be MPEG-4 format (.mp4)
    
    	//output : video init
      if (SUCCEEDED(hr))
      {
        hr = MFCreateMediaType(&pMediaTypeOut);  
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeOut->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);   
      }
    	if (SUCCEEDED(hr))
      {
        hr = pMediaTypeOut->SetGUID(MF_MT_SUBTYPE, VIDEO_ENCODING_FORMAT);
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeOut->SetUINT32(MF_MT_AVG_BITRATE, VIDEO_BIT_RATE); 
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeOut->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeSize(pMediaTypeOut, MF_MT_FRAME_SIZE, VIDEO_WIDTH, VIDEO_HEIGHT);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeOut, MF_MT_FRAME_RATE, VIDEO_FPS, 1);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeOut, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);  
      }
    	if(SUCCEEDED(hr))
    	{
    		hr = pMediaTypeOut->SetBlob(MF_MT_MPEG4_SAMPLE_DESCRIPTION, pBuf8b, ArraySize);
    	}
    
       //output : audio init
    	if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeOut->SetGUID( MF_MT_MAJOR_TYPE, MFMediaType_Audio );
      }
      if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeOut->SetGUID( MF_MT_SUBTYPE, MFAudioFormat_AAC );
      }
    	if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_BITS_PER_SAMPLE, BITS_PER_SAMPLE );
      }
      if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_SAMPLES_PER_SECOND, SAMPLES_PER_SECOND );
      }
    	if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_NUM_CHANNELS, NUM_CHANNELS );
      }
      if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeOut->SetUINT32( MF_MT_AUDIO_AVG_BYTES_PER_SECOND, AVG_BYTES_PER_SECOND );
      }
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->AddStream(pMediaTypeOut, &streamIndex);  
      }
    
      //input : video init
      if (SUCCEEDED(hr))
      {
        hr = MFCreateMediaType(&pMediaTypeIn);  
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeIn->SetGUID(MF_MT_MAJOR_TYPE, MFMediaType_Video);  
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeIn->SetGUID(MF_MT_SUBTYPE, VIDEO_INPUT_FORMAT);   
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeIn->SetUINT32(MF_MT_INTERLACE_MODE, MFVideoInterlace_Progressive);  
      } 
    	if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeSize(pMediaTypeIn, MF_MT_FRAME_SIZE, VIDEO_WIDTH, VIDEO_HEIGHT);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeIn, MF_MT_FRAME_RATE, VIDEO_FPS, 1);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeIn, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);  
      }
    
      //input : audio init
    	if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeIn->SetGUID( MF_MT_MAJOR_TYPE, MFMediaType_Audio );
      }
      if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeIn->SetGUID( MF_MT_SUBTYPE, MFAudioFormat_PCM );
      }
      if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_BITS_PER_SAMPLE, BITS_PER_SAMPLE );
      }
      if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_SAMPLES_PER_SECOND, SAMPLES_PER_SECOND );
      }
      if (SUCCEEDED(hr))
      {
    		hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_NUM_CHANNELS, NUM_CHANNELS );
      }
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->SetInputMediaType(streamIndex, pMediaTypeIn, NULL);  
      }
    
      //Tell the SinkWriter to begin the treatment of data
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->BeginWriting();
      }
    	else //possible error codes for 'hr'...
    	{
    		if(hr == MF_E_INVALIDMEDIATYPE)
    			UINT32 uiShutDown=0;
    
    		if(hr == MF_E_INVALIDSTREAMNUMBER)
    			UINT32 uiShutDown=1;
    
    		if(hr == MF_E_TOPO_CODEC_NOT_FOUND)
    			UINT32 uiShutDown=2;
    	}
    
      //Returns the pointer of the caller
      if (SUCCEEDED(hr))
      {
        *ppWriter = pSinkWriter;
        (*ppWriter)->AddRef();
        *pStreamIndex = streamIndex;
      }
    
      SafeRelease(&pSinkWriter);
      SafeRelease(&pMediaTypeOut);
      SafeRelease(&pMediaTypeIn);
    	SafeRelease(&pAttrib);
    
      return hr;
    }
    
    
    //Use this function to write a video frame
    HRESULT WriteFrame(
      IMFSinkWriter *pWriter, 
      DWORD streamIndex, 
      const LONGLONG& rtStart,    // Time stamp.
      const LONGLONG& rtDuration   // Frame duration.
      )
    {
      IMFSample *pSample = NULL;
      IMFMediaBuffer *pBuffer = NULL;
    
    	const LONG cbWidth = 4 * VIDEO_WIDTH;
    	const DWORD cbBuffer = cbWidth * VIDEO_HEIGHT;
    	
    	BYTE *pData = NULL;
    
      // Create a new memory buffer.
      HRESULT hr = MFCreateMemoryBuffer(cbBuffer, &pBuffer);
    
      // Lock the buffer and copy the video frame to the buffer.
      if (SUCCEEDED(hr))
      {
        hr = pBuffer->Lock(&pData, NULL, NULL);
      }
      if (SUCCEEDED(hr))
      {
    		hr = MFCopyImage(
          pData,           // Destination buffer.
          cbWidth,          // Destination stride.
          (BYTE*)videoFrameBuffer,  // First row in source image.
          cbWidth,          // Source stride.
          cbWidth,          // Image width in bytes.
          VIDEO_HEIGHT        // Image height in pixels.
          );		
    	}
      if (pBuffer)
      {
        pBuffer->Unlock();
      }
    
      // Set the data length of the buffer.
      if (SUCCEEDED(hr))
      {
        hr = pBuffer->SetCurrentLength(cbBuffer);
      }
    
      // Create a media sample and add the buffer to the sample.
      if (SUCCEEDED(hr))
      {
        hr = MFCreateSample(&pSample);
      }
      if (SUCCEEDED(hr))
      {
        hr = pSample->AddBuffer(pBuffer);
      }
    
      // Set the time stamp and the duration.
      if (SUCCEEDED(hr))
      {
        hr = pSample->SetSampleTime(rtStart);
      }
      if (SUCCEEDED(hr))
      {
        hr = pSample->SetSampleDuration(rtDuration);
      }
    
      // Send the sample to the Sink Writer.
      if (SUCCEEDED(hr))
      {
        hr = pWriter->WriteSample(streamIndex, pSample);
      }
    
      SafeRelease(&pSample);
      SafeRelease(&pBuffer);
    
      return hr;
    }
    
    
    //MAIN FUNCTION (included into a DialogBox button)
    void CMP4_Sink_WriterDlg::OnBnClickedOk()
    {
    	DWORD			j=0;
    
      HRESULT hr = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
      if (SUCCEEDED(hr))
      {
        hr = MFStartup(MF_VERSION);
        if (SUCCEEDED(hr))
        {
          IMFSinkWriter *pSinkWriter = NULL;
          DWORD stream;
    
          //Launch SinkWriter init
          hr = InitializeSinkWriter(&pSinkWriter, &stream);
          if (SUCCEEDED(hr))
          {
            //Treatment of frames
            LONGLONG rtStart = 0;
            UINT64 rtDuration;
    
            MFFrameRateToAverageTimePerFrame(VIDEO_FPS, 1, &rtDuration);
    
    		//VIDEO FRAMES
            for (DWORD i = 0; i < VIDEO_FRAME_COUNT; ++i, ++j)
            {
    			//Pixels of the picture
    			for (DWORD k = 0 ; k < VIDEO_PELS; k++)
    			{
    				if(j>255)
    					j=0;
    
    				videoFrameBuffer[k] = ((j<<16) & 0x00FF0000) | ((j<<8) & 0x0000FF00) | (j & 0x000000FF);
    			}
    
              hr = WriteFrame(pSinkWriter, stream, rtStart, rtDuration);
              if (FAILED(hr))
              {
                break;
              }
    
    		   rtStart += rtDuration;
            }
    
    		//AUDIO FRAMES
    		for( DWORD nSample = 0; nSample < 20; nSample++ )
    		{
    			CComPtr<IMFSample> spSample;
    			CComPtr<IMFMediaBuffer> spBuffer;
    			BYTE *pbBuffer = NULL;
    			LONGLONG hnsSampleTime = 0;
    		
    			//Creation of a media sample
    			if (SUCCEEDED(hr))
    			{
    				hr=MFCreateSample( &spSample );
    			}
    			if (SUCCEEDED(hr))
    			{
                    hr=spSample->SetSampleDuration( SAMPLE_DURATION );	// 1 second
    			}
    			if (SUCCEEDED(hr))
    			{
                    hr=spSample->SetSampleTime( hnsSampleTime );		//every 1 second
    			}
    			hnsSampleTime += SAMPLE_DURATION;
    
    			//Add a media buffer
    			if (SUCCEEDED(hr))
    			{
                    hr=MFCreateMemoryBuffer( BUFFER_LENGTH, &spBuffer );
    			}
    			if (SUCCEEDED(hr))
    			{
                    hr=spBuffer->SetCurrentLength( BUFFER_LENGTH );
    			}
    			if (SUCCEEDED(hr))
    			{
    				hr=spSample->AddBuffer( spBuffer );
    			}
    			if (SUCCEEDED(hr))
    			{
    				hr=spBuffer->Lock( &pbBuffer, NULL, NULL );
    				if (SUCCEEDED(hr))
    				{
    					for( DWORD n=0; n < BUFFER_LENGTH; n++ )
    					{
    						pbBuffer[n] = (BYTE)( rand() & 0xFF ); //noise generated with a rand function
    					}
    				}
    			}
    			if (SUCCEEDED(hr))
    			{
    				hr=spBuffer->Unlock();
    			}
    
    			//Write the media sample
    			if (SUCCEEDED(hr))
    			{
    				hr=pSinkWriter->WriteSample( stream, spSample );
    			}
    		}
    		if (SUCCEEDED(hr))
    		{
    			hr = pSinkWriter->Finalize();
    		}
          }
          
          SafeRelease(&pSinkWriter);
          MFShutdown();
        }
        CoUninitialize();
      }
    }
    


    Hope you will help me!

     

    Best regards.

     

     

    Julien

    Tuesday, June 21, 2011 4:46 PM
  • Hi,

     

    I am working on both video and audio, and my aim is to get a file which includes these two different kinds of data, playable with WMP, VLC, Media Player Classic etc.

    My inputs are two buffers, one which contains video data (RGB 32bits - uncompressed) and one other which contains audio data (analog 16-bit samples, uncompressed).

     

    To do this, I want to use Windows Media Foundation. I am coding in VS2010, in C++ language, and my O.S. is Win7 Pro x64.

    I already tried to do some things with a SinkWriter, but I did not succeed in creating a mp4 file which include both audio and video. I just created two different files, one for video (using H264 encoder) and an other for audio (using AAC encoder). All of them seem OK, readable by players without any problem and contain all the data sent in input...

     

    - Should I use a SinkWriter which treats both audio and video at the same time (is it possible?) and automatically muxes the streams and stores them into a mp4 file?
    - Is it compulsory to use a MediaSink (such as MPEG-4 MediaSink) after using a SinkWriter which created the 2 streams?
    - Should I use two SinkWriters (one for audio and one for video), get two different files, for exemple a file video.mp4 and a file audio.mp4, and after that, create something which adds the 2 files and stores the result in a file output.mp4?

    ... Or, does it exist a different way to create the mp4 file I want?

     

    Please, bring me one or more solutions (in fact, just one, but a good one^^), I really need your help!

     

    Thank you very much for the help and the time you give me :)

     

    Best regards...

    Thursday, June 23, 2011 2:23 PM
  • Nobody to help me?

    All help you could provide me will be great!

     

    Thank you very much.

    Wednesday, June 29, 2011 12:31 PM
  • Check out MFCreateTranscodeProfile and MFCreateTranscodeTopology, providing your own IMFMediaSource implementation with an audio and video IMFMediaStream.
    This should get you started: http://msdn.microsoft.com/en-us/library/dd389290(VS.85).aspx
    Good luck!
    Wednesday, July 6, 2011 7:53 AM
  • Where did you get your audio type from?  The sample size you generate always needs to be a multiple of the block alignment, and it does not look like yours is a multiple of 2230.  Typically the safest way to generate an audio type for transcode is to use the MFTranscodeGetAudioOutputAvailableTypes function and choose the closest match to your desired audio type.  Audio codecs tend to be very picky about the formats they accept, so choosing an exact match to its available types ensures that it can generate the format.

    Thursday, July 28, 2011 11:50 PM
  • Hi Matt,

     

    Thank you for the interest you bring me, and the help too.

    In fact, my post is obsolet yet (I wrote it for a while, and I tried other things!). I did not succeed in the creation of a mpeg4 file, but I changed the way to do it.

    Indeed, I am now using the combination SinkWriter + MediaSink, but I still have some problems.

    I hope you can help me, indicating me what could be the problem, and the mean to solve it.

    Please, find my following code:

     

     

    //Including of headers from SDK 7.1
    #include "wmcodecdsp.h"
    #include "mfapi.h"
    #include "mfidl.h"
    #include "Mfreadwrite.h"
    #include "mferror.h"
    
    
    //Including of libraries from SDK 7.1
    #pragma comment(lib, "mf")
    #pragma comment(lib, "mfreadwrite")
    #pragma comment(lib, "mfplat")
    #pragma comment(lib, "mfuuid")
    
    
    //Video constants
    const GUID	    VIDEO_MAJOR_TYPE = MFMediaType_Video;			//for video treatment
    const GUID 	VIDEO_ENCODING_FORMAT = MFVideoFormat_H264;	//encoding format (output)
    const GUID 	VIDEO_INPUT_FORMAT = MFVideoFormat_RGB32;		//input format
    const UINT32	VIDEO_WIDTH = 720;							//picture size : width
    const UINT32	VIDEO_HEIGHT = 576;							//picture size : height
    const UINT32	VIDEO_FPS = 25;							//frames per second
    const UINT32	VIDEO_BIT_RATE = 2000000;					//encoding bitrate (in bps)
    const UINT32	VIDEO_MODE = MFVideoInterlace_Progressive;		//progressive or interlaced pictures
    const UINT32	VIDEO_PELS = VIDEO_WIDTH * VIDEO_HEIGHT;		//resolution (width * height)
    const UINT32	VIDEO_FRAME_COUNT = NUMBER_OF_SECONDS_FOR_FILE * VIDEO_FPS;	//frame counter (duration of the video)
    
    //Audio constants
    const GUID 	AUDIO_MAJOR_TYPE = MFMediaType_Audio;			//for audio treatment
    const GUID	    AUDIO_ENCODING_FORMAT = MFAudioFormat_AAC;		//encoding format (output)
    const GUID	    AUDIO_INPUT_FORMAT = MFAudioFormat_PCM;		//input format
    const UINT32	AUDIO_SAMPLES_PER_SECOND = 44100;			//samples per second
    const UINT32	AUDIO_AVG_BYTES_PER_SECOND = 16000;			//average bytes per second
    const UINT32	AUDIO_NUM_CHANNELS = 1;						//MONO or STEREO
    const UINT32	AUDIO_BITS_PER_SAMPLE = 16;					//bits per sample
    const UINT32	AUDIO_ONE_SECOND = 10;						//quantity of buffers per second
    const UINT32	AUDIO_BUFFER_LENGTH = AUDIO_BITS_PER_SAMPLE / 8 * AUDIO_NUM_CHANNELS * AUDIO_SAMPLES_PER_SECOND;<br/>                                                     //max. buffer size
    const LONGLONG	AUDIO_SAMPLE_DURATION = 10000000;			//sample duration
    
    
    //Buffer for one video frame
    DWORD videoFrameBuffer[VIDEO_PELS];
    
    
    //Creation of a template to release pointers
    template <class T> void SafeRelease(T **ppT)
    {
      if (*ppT)
      {
        (*ppT)->Release();
        *ppT = NULL;
      }
    }
    
    
    //Creation of the Byte Stream
    IMFByteStream* CreateFileByteStream(LPCWSTR FileName)
    {
    	//create file byte stream
    	IMFByteStream *pByteStream = NULL;
    
    	HRESULT hr = MFCreateFile(MF_ACCESSMODE_WRITE, MF_OPENMODE_DELETE_IF_EXIST, MF_FILEFLAGS_NONE, FileName, &pByteStream);
    
    	if(FAILED(hr))
    		pByteStream = NULL;
    
    	return pByteStream;
    }
    
    
    //Creation of the Video profile (H264)
    IMFMediaType* CreateVideoProfile()
    {
    	IMFMediaType *pMediaType = NULL;
    
    	HRESULT hr = MFCreateMediaType(&pMediaType);
    	if(SUCCEEDED(hr))
    	{
          hr = pMediaType->SetGUID(MF_MT_MAJOR_TYPE, VIDEO_MAJOR_TYPE);   
        }
    	if(SUCCEEDED(hr))
        {
          hr = pMediaType->SetGUID(MF_MT_SUBTYPE, VIDEO_ENCODING_FORMAT);
        }
        if(SUCCEEDED(hr))
        {
          hr = pMediaType->SetUINT32(MF_MT_AVG_BITRATE, VIDEO_BIT_RATE); 
        }
        if(SUCCEEDED(hr))
        {
          hr = pMediaType->SetUINT32(MF_MT_INTERLACE_MODE, VIDEO_MODE);  
        }
        if(SUCCEEDED(hr))
        {
          hr = MFSetAttributeSize(pMediaType, MF_MT_FRAME_SIZE, VIDEO_WIDTH, VIDEO_HEIGHT);  
        }
        if(SUCCEEDED(hr))
        {
          hr = MFSetAttributeRatio(pMediaType, MF_MT_FRAME_RATE, VIDEO_FPS, 1);  
        }
        if(SUCCEEDED(hr))
        {
          hr = MFSetAttributeRatio(pMediaType, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);  
    	}	
    
    	if(FAILED(hr))
    		pMediaType = NULL;
    
    	return pMediaType;
    }
    
    
    //Creation of the Audio profile (AAC)
    IMFMediaType* CreateAudioProfile()
    {
    	IMFMediaType *pMediaType = NULL;
    
    	HRESULT hr = MFCreateMediaType(&pMediaType);
    	if(SUCCEEDED(hr))
        {
    		hr=pMediaType->SetGUID( MF_MT_MAJOR_TYPE, AUDIO_MAJOR_TYPE );
        }
        if(SUCCEEDED(hr))
        {
    		hr=pMediaType->SetGUID( MF_MT_SUBTYPE, AUDIO_ENCODING_FORMAT );
        }
    	if(SUCCEEDED(hr))
        {
    		hr=pMediaType->SetUINT32( MF_MT_AUDIO_BITS_PER_SAMPLE, AUDIO_BITS_PER_SAMPLE );
        }
        if(SUCCEEDED(hr))
        {
    		hr=pMediaType->SetUINT32( MF_MT_AUDIO_SAMPLES_PER_SECOND, AUDIO_SAMPLES_PER_SECOND );
        }
    	if(SUCCEEDED(hr))
        {
    		hr=pMediaType->SetUINT32( MF_MT_AUDIO_NUM_CHANNELS, AUDIO_NUM_CHANNELS );
        }
        if(SUCCEEDED(hr))
        {
    		hr=pMediaType->SetUINT32( MF_MT_AUDIO_AVG_BYTES_PER_SECOND, AUDIO_AVG_BYTES_PER_SECOND );
        }
    
    	if(FAILED(hr))
    		pMediaType = NULL;
    
    	return pMediaType;
    }
    
    
    //Create an aggregate source (both audio and video)
    IMFMediaSource* CreateAggregatedSource(IMFMediaSource *pSource1, IMFMediaSource *pSource2, IMFMediaSource *pAggSource)
    {
      pAggSource = NULL;
      IMFCollection *pCollection = NULL;
    
      HRESULT hr = MFCreateCollection(&pCollection);
      if (SUCCEEDED(hr))
      {
        hr = pCollection->AddElement(pSource1);
      }
      if (SUCCEEDED(hr))
      {
        hr = pCollection->AddElement(pSource2);
      }
      if (SUCCEEDED(hr))
      {
        hr = MFCreateAggregateSource(pCollection, &pAggSource);
      }
    
      SafeRelease(&pCollection);
    
    	if(FAILED(hr))
    		pAggSource = NULL;
    
      return pAggSource;  
    }
    
    
    //Creation of the MPEG-4 MediaSink
    IMFMediaSink* CreateMediaSink(IMFByteStream *pByteStream, IMFMediaType *pVideoMediaType, IMFMediaType *pAudioMediaType)
    {
    	IMFMediaSink *pMediaSink = NULL;
    	DWORD pdwCharac = NULL;
    	DWORD pdwStreamCount = NULL;
    
    	HRESULT hr = MFCreateMPEG4MediaSink(pByteStream, pVideoMediaType, pAudioMediaType, &pMediaSink);
    
    	//// DEBUG ////
    	pMediaSink->GetCharacteristics(&pdwCharac);
    	pMediaSink->GetStreamSinkCount(&pdwStreamCount);
    		
    	if(FAILED(hr))
    		pMediaSink = NULL;
    
    	return pMediaSink;
    }
    
    
    IMFAttributes* CreateAttributesForSinkWriter()
    {
    	IMFAttributes *pMFAttributes = NULL;
    
    	HRESULT hr = MFCreateAttributes( &pMFAttributes, 100 );
    	
    	if(SUCCEEDED(hr))
    	{
    		hr = pMFAttributes->SetGUID( MF_TRANSCODE_CONTAINERTYPE, MFTranscodeContainerType_MPEG4 );
    	}
    	if(SUCCEEDED(hr))
    	{
    		hr = pMFAttributes->SetUINT32( MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS, FALSE ); //no hardware encoding
    	}
    	if(SUCCEEDED(hr))
    	{
    		hr = pMFAttributes->SetUINT32( MF_READWRITE_DISABLE_CONVERTERS , FALSE ); //enable converting formats
    	}
    
    	if(FAILED(hr))
    		pMFAttributes = NULL;
    
    	return pMFAttributes;
    }
    
    
    //Initialization of the Video SinkWriter...
    HRESULT InitializeSinkWriterVideo(IMFSinkWriter **ppWriter, DWORD *pStreamIndex, IMFMediaSink *pMediaSink)
    {
      *ppWriter = NULL;
      *pStreamIndex = NULL;
    
      IMFSinkWriter *pSinkWriter = NULL;
      IMFMediaType *pMediaTypeOut = NULL;  
      IMFMediaType *pMediaTypeIn = NULL; 
      IMFAttributes	 *pAttrib = NULL;
      DWORD     streamIndexV=0;
      DWORD		 streamIndexA=1;
    	
      HRESULT hr = MFCreateSinkWriterFromMediaSink(pMediaSink, NULL, &pSinkWriter);
    
      //input : video
      if (SUCCEEDED(hr))
      {
        hr = MFCreateMediaType(&pMediaTypeIn);  
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeIn->SetGUID(MF_MT_MAJOR_TYPE, VIDEO_MAJOR_TYPE);  
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeIn->SetGUID(MF_MT_SUBTYPE, VIDEO_INPUT_FORMAT);   
      }
      if (SUCCEEDED(hr))
      {
        hr = pMediaTypeIn->SetUINT32(MF_MT_INTERLACE_MODE, VIDEO_MODE);  
      } 
    	if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeSize(pMediaTypeIn, MF_MT_FRAME_SIZE, VIDEO_WIDTH, VIDEO_HEIGHT);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeIn, MF_MT_FRAME_RATE, VIDEO_FPS, 1);  
      }
      if (SUCCEEDED(hr))
      {
        hr = MFSetAttributeRatio(pMediaTypeIn, MF_MT_PIXEL_ASPECT_RATIO, 1, 1);  
      }
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->SetInputMediaType(streamIndexV, pMediaTypeIn, NULL);  
      }
    		
      //Tell to the Video SinkWriter to begin data treatment
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->BeginWriting();
      }
    	
      //Possible error codes
      if (FAILED(hr))
      {
    	UINT32 uiShutDown=9999;
    
    	if(hr == MF_E_INVALIDMEDIATYPE)
    		uiShutDown=0;
    
    	if(hr == MF_E_INVALIDSTREAMNUMBER)
    		uiShutDown=1;
    
    	if(hr == MF_E_TOPO_CODEC_NOT_FOUND)
    		uiShutDown=2;
    
    	if(hr == E_INVALIDARG)
    		uiShutDown=3;
      }
    
      //Returns the pointer of the caller
      if (SUCCEEDED(hr))
      {
        *ppWriter = pSinkWriter;
        (*ppWriter)->AddRef();
        *pStreamIndex = streamIndexV;
      }
    
      //Release pointers
      SafeRelease(&pSinkWriter);
      SafeRelease(&pMediaTypeOut);
      SafeRelease(&pMediaTypeIn);
      SafeRelease(&pAttrib);
    
      return hr;
    }
    
    
    //Initialization of the Audio SinkWriter...
    HRESULT InitializeSinkWriterAudio(IMFSinkWriter **ppWriter, DWORD *pStreamIndex, IMFMediaSink *pMediaSink)
    {
      *ppWriter = NULL;
      *pStreamIndex = NULL;
    
      IMFSinkWriter  *pSinkWriter = NULL;
      IMFMediaType *pMediaTypeOut = NULL;  
      IMFMediaType *pMediaTypeIn = NULL; 
      IMFAttributes	 *pAttrib = NULL;
      DWORD      streamIndex=1;  
    	
      HRESULT hr = MFCreateSinkWriterFromMediaSink(pMediaSink, NULL, &pSinkWriter);
    
      //input : audio
      if (SUCCEEDED(hr))
      {
         hr = MFCreateMediaType(&pMediaTypeIn);  
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetGUID( MF_MT_MAJOR_TYPE, AUDIO_MAJOR_TYPE );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetGUID( MF_MT_SUBTYPE, AUDIO_INPUT_FORMAT );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_BITS_PER_SAMPLE, AUDIO_BITS_PER_SAMPLE );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_SAMPLES_PER_SECOND, AUDIO_SAMPLES_PER_SECOND );
      }
      if (SUCCEEDED(hr))
      {
    	hr=pMediaTypeIn->SetUINT32( MF_MT_AUDIO_NUM_CHANNELS, AUDIO_NUM_CHANNELS );
      }
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->SetInputMediaType(streamIndex, pMediaTypeIn, NULL);  
      }
    
      //Tell the Audio SinkWriter to begin data treatment
      if (SUCCEEDED(hr))
      {
        hr = pSinkWriter->BeginWriting();
      }
    
      //Possible error codes
      if(FAILED(hr))
      {
    	if(hr == MF_E_INVALIDMEDIATYPE)
    		UINT32 uiShutDown=0;
    
    	if(hr == MF_E_INVALIDSTREAMNUMBER)
    		UINT32 uiShutDown=1;
    
    	if(hr == MF_E_TOPO_CODEC_NOT_FOUND)
    		UINT32 uiShutDown=2;
      }
    
      //Returns the pointer of the caller
      if (SUCCEEDED(hr))
      {
        *ppWriter = pSinkWriter;
        (*ppWriter)->AddRef();
        *pStreamIndex = streamIndex;
      }
    
      //Release pointers
      SafeRelease(&pSinkWriter);
      SafeRelease(&pMediaTypeOut);
      SafeRelease(&pMediaTypeIn);
      SafeRelease(&pAttrib);
    
      return hr;
    }
    
    
    //Write a video frame
    HRESULT WriteVideoFrame(IMFSinkWriter *pWriter, DWORD streamIndex, const LONGLONG& rtStart, const LONGLONG& rtDuration)
    {
      /*
      * rtStart : time stamp
      * rtDuration : frame duration
      */
    
      IMFSample		 *pSample = NULL;
      IMFMediaBuffer	 *pBuffer = NULL;
    
      const LONG cbWidth = 4 * VIDEO_WIDTH;
      const DWORD cbBuffer = 4 * VIDEO_PELS;
    	
      BYTE *pData = NULL;
    
      //Create a new memory buffer, whose max. size is cbBuffer (H*L*4 for one picture)
      HRESULT hr = MFCreateMemoryBuffer(cbBuffer, &pBuffer);
    
      //Lock the buffer and copy the video frame to the buffer
      if (SUCCEEDED(hr))
      {
        hr = pBuffer->Lock(&pData, NULL, NULL);
      }
      if (SUCCEEDED(hr))
      {
    	hr = MFCopyImage(pData, cbWidth, (BYTE*)videoFrameBuffer, cbWidth, cbWidth, VIDEO_HEIGHT);		
    		
    	/*
    	* pData : Destination buffer
    	* cbWidth : Destination stride
    	* (BYTE*)videoFrameBuffer : First row in source image
    	* cbWidth : Source stride
    	* cbWidth : Image width in bytes
    	* VIDEO_HEIGHT: Image height in pixels
    	*/
    
      }
      if (pBuffer)
      {
        pBuffer->Unlock();
      }
    
      //Set the data length of the buffer
      if (SUCCEEDED(hr))
      {
        hr = pBuffer->SetCurrentLength(cbBuffer);
      }
    
      //Create a media sample and add the buffer to the sample
      if (SUCCEEDED(hr))
      {
        hr = MFCreateSample(&pSample);
      }
      if (SUCCEEDED(hr))
      {
        hr = pSample->AddBuffer(pBuffer);
      }
    
      //Set the time stamp and the duration
      if (SUCCEEDED(hr))
      {
        hr = pSample->SetSampleTime(rtStart);
      }
      if (SUCCEEDED(hr))
      {
        hr = pSample->SetSampleDuration(rtDuration);
      }
    
      //Send the sample to the Sink Writer
      if (SUCCEEDED(hr))
      {
        hr = pWriter->WriteSample(streamIndex, pSample);
      }
    
      //Release pointers
      SafeRelease(&pSample);
      SafeRelease(&pBuffer);
    
      return hr;
    }
    
    
    //Write an audio packet
    HRESULT WriteAudioPacket(IMFSinkWriter *pWriter, DWORD streamIndex, const LONGLONG& rtStart, const LONGLONG& rtDuration, UINT32 Quantity)
    {
      IMFSample		*pSample = NULL;
      IMFMediaBuffer	*pBuffer = NULL;
    	
      const DWORD cbBuffer = Quantity * 2;
    	
      BYTE *pData = NULL;
    	
    
      //Create a new memory buffer, whose max. size is cbBuffer (QuantityOfSamplesPerVideoFrame * 2 Bytes)
      HRESULT hr = MFCreateMemoryBuffer(cbBuffer, &pBuffer);
    
      //Lock the buffer and copy the audio packet to the buffer
      if (SUCCEEDED(hr))
      {
    	hr=pBuffer->Lock( &pData, NULL, NULL );
      }
      if (SUCCEEDED(hr))
      {
    	for( DWORD n=0; n < cbBuffer; n++ )
    	{
    		pData[n] = (BYTE)( rand() & 0xFF );	//generation of random noise
    	}
      }
      if (SUCCEEDED(hr))
      {
    	hr=pBuffer->Unlock();
      }
    
      // Set the data length of the buffer
      if (SUCCEEDED(hr))
      {
    	hr=pBuffer->SetCurrentLength( cbBuffer );
      }
    
      //Create a media sample and add the buffer to the sample
      if (SUCCEEDED(hr))
      {
        hr = MFCreateSample(&pSample);
      }
      if (SUCCEEDED(hr))
      {
        hr = pSample->AddBuffer(pBuffer);
      }
    
      //Set the time stamp and the duration
      if (SUCCEEDED(hr))
      {
        hr = pSample->SetSampleTime(rtStart);
      }
      if (SUCCEEDED(hr))
      {
        hr = pSample->SetSampleDuration(rtDuration);
      }
    
      //Send the sample to the Sink Writer
      if (SUCCEEDED(hr))
      {
        hr = pWriter->WriteSample(streamIndex, pSample);
      }
    
      //Release pointers
      SafeRelease(&pSample);
      SafeRelease(&pBuffer);
    
      return hr;
    }
    
    
    // MAIN FUNCTION
    void CMediaSinkDlg::OnBnClickedOk()
    {	
    	HRESULT hr = S_OK;
    	IMFByteStream			*spByteStream = NULL;
    	IMFMediaSink			*pMediaSink = NULL;
        IMFSinkWriter			*spSinkWriterVid = NULL;
    	IMFSinkWriter			*spSinkWriterAud = NULL;
    	IMFMediaType			*spVideo = NULL;
    	IMFMediaType			*spAudio = NULL;
    	//IMFMediaEventGenerator	*spMFMediaEvtGene = NULL;
    	//IMFMediaEvent			*spMFMediaEvent = NULL;
    	IMFAttributes			*spAttrib = NULL;
    
        DWORD					sindexVid=0, sindexAud=0, j=0;
    
    	LPCWSTR	str = L"outputfile.mp4";
    
        hr = CoInitialize(NULL);
        if (SUCCEEDED(hr))
        {
    		hr = MFStartup(MF_VERSION);
    		if (SUCCEEDED(hr))
    		{
    			spByteStream = CreateFileByteStream(str);
    			if (spByteStream != NULL)
    			{
    				spVideo = CreateVideoProfile();
    			}
    			if (spVideo != NULL)
    			{
    				spAudio = CreateAudioProfile();
    			}
    			if (spAudio != NULL)
    			{
    				pMediaSink = CreateMediaSink(spByteStream, spVideo, spAudio);
    			}
    
    			if (pMediaSink != NULL)
    			{
    				hr=InitializeSinkWriterVideo(&spSinkWriterVid, &sindexVid, pMediaSink);
    				if(SUCCEEDED(hr))
    				{
    					LONGLONG rtStartVid = 0;
    					UINT64 rtDurationVid;
    
    					/********************************************************
    					*						VIDEO PART				 *
    					********************************************************/
    
    					//Calculate the average time per frame, for video<br/>                    MFFrameRateToAverageTimePerFrame(VIDEO_FPS, 1, &rtDurationVid);
    
    					//loop to treat all the pictures
    					for (DWORD i = 0; i < VIDEO_FRAME_COUNT; ++i, ++j)
    					{
    						//Picture pixels
    						for (DWORD k = 0 ; k < VIDEO_PELS; k++)
    						{
    							if(j>255)
    								j=0;
    
    							videoFrameBuffer[k] = ((j<<16) & 0x00FF0000) | ((j<<8) & 0x0000FF00) | (j & 0x000000FF);
    						}
    
    						hr = WriteVideoFrame(spSinkWriterVid, sindexVid, rtStartVid, rtDurationVid);
    						if (FAILED(hr))
    						{
    							break;
    						}
    
    						//Update the time stamp value
    						rtStartVid += rtDurationVid;
    					}
    
    					//Finalization of writing with the Video SinkWriter
    					if (SUCCEEDED(hr))
    					{
    						hr=spSinkWriterVid->Finalize();
    					}
    				}
    			}
    			
    			SafeRelease(&spVideo);
    			SafeRelease(&spSinkWriterVid);
    			
    			if (SUCCEEDED(hr))
    			{
    				hr=InitializeSinkWriterAudio(&spSinkWriterAud, &sindexAud, pMediaSink);				
    				if (SUCCEEDED(hr))
    				{
    					LONGLONG rtStartAud = 0;
    					UINT64 rtDurationAud;
    					double QtyAudioSamplesPerVideoFrame = 0;
    
    					//Calculate the approximate quantity of samples, according to a video frame duration
    					//44100 Hz -> 1 s
    					//????? Hz -> 0.04 s (= 40 ms = one video frame duration)
    					if(VIDEO_FPS != 0)
    						QtyAudioSamplesPerVideoFrame = ( (double) AUDIO_SAMPLES_PER_SECOND / (double) VIDEO_FPS );
    					else
    						QtyAudioSamplesPerVideoFrame = 0;
    
    					MFFrameRateToAverageTimePerFrame(VIDEO_FPS, 1, &rtDurationAud);	//we treat the same duration as the video
    					//it means that we will treat N audio packets for the last of one picture (=40 ms)
    					
    					//loop to treat all the audio packets
    					if(rtDurationAud != 0)
    					{
    						for (DWORD i = 0; i < VIDEO_FRAME_COUNT; ++i)
    						{
    							//Audio packets
    							hr = WriteAudioPacket(spSinkWriterAud, sindexAud, rtStartAud, rtDurationAud, (UINT32) QtyAudioSamplesPerVideoFrame);
    							if (FAILED(hr))
    							{
    								break;
    							}
    
    							//Update the time stamp value
    							rtStartAud += rtDurationAud;
    						}
    
    						//Finalization of writing with the Audio SinkWriter
    						if (SUCCEEDED(hr))
    						{
    							hr=spSinkWriterAud->Finalize();
    						}
    					}
    				}
    			}
    
    			//Release pointers
    			SafeRelease(&spByteStream);
    			SafeRelease(&spAudio);
    			SafeRelease(&spSinkWriterAud);
    			SafeRelease(&spAttrib);
    
    			//Shutdown the MediaSink (not done by the SinkWriter)
    			pMediaSink->Shutdown();
    			SafeRelease(&pMediaSink);
    		}
    		
    		//Shutdown MediaFoundation
        MFShutdown();
    		CoUninitialize();
    	}
    
    	CDialog::OnOK();
    }
    


    At each try, I get an error code at the line "BeginWriting()" of the second SinkWriter (audio SinkWriter), which value is : 0xc00d36da. It should means that some component is already listening to events on this event generator. "MF_E_MULTIPLE_SUBSCRIBERS"...

    I can not include audio data in the file so.

     

    Could you tell me what I should do to solve that problem? Is my idea to combine 2 SinkWriters (one for video and one for audio) and 1 MediaSink is a good one? According my code, are there other mistake(s)?

    I have to succeed, i.e. create a good mpeg4 file, because I need it for my work.

     

    Thank you so much for your future help.

     

    Best regards.

     

     

    Julien

    Friday, July 29, 2011 2:24 PM
  • There should be a 1-1 relation between sink writers and media sinks.  Having multiple sink writers point to the same media sink will only lead to errors.  This particular one is because both sink writers are trying to listen on the media sink's event generator.  To put multiple streams (for example, an audio and video stream) into the same file, use one sink writer and use AddStream to add an additional stream to the sink writer.
    Monday, August 1, 2011 10:37 PM
  • Dear Matt,

     

    Thank you for your explanation. I understand what is the problem and the returned error code now...
    I will now stay concern about your proposal : build a way to generate a mp4 file with only one SinkWriter. I will use it with two different streams : stream 0 for video stream, and stream 1 for audio stream. It should work like this, even if my first attempt to this way was not successful.

    I contact you again if I have some problems with this method.

    It is really important I get this file, for my work, and I really appreciate the help you bring me.

     

    P.S. : If you could give me a code snippet, including an example of parameters which run well for AAC encoding (bits per sample, block alignment, avg bytes per second, number of channels etc.), it would be great!!! I tried a lot of configurations, but without any success yet.

     

    Best regards.


    Tuesday, August 2, 2011 1:26 PM
  • The MFDub sample on the MF blog uses the sink writer to transcode an audio and video stream together.  If you look in mediatranscoder.cpp, the function ConfigureSinkWriter gives an example of initializing audio and video streams from source content parameters.  MakeTargetAudioType demonstrates using MFTranscodeGetAudioOutputAvailableTypes to choose the closest type match supported by the audio encoder.

    The general idea with audio encoding is that you do not choose a fully formed type.  You select parameters that you care about -- perhaps you want a minimum sample rate, and/or to match a certain number of channels -- and choose an output type from the encoder that most closesly matches your desired parameters.

    Tuesday, August 2, 2011 7:15 PM
  • I can creating a video but I can not add a audio. If someone managed to do Could you tell me? please.
    Sorry for my English
    Wednesday, June 25, 2014 3:12 PM