Depth and audio data in the same application (C++) RRS feed

  • Question

  • Hello everybody,

    I’m trying to develop a C++ application using both depth and audio data. After having studied the AudioBasics and SkeletonBasics examples, I’m building a project  trying to merge the functions coming from the examples.  I’m using VS 2010 and the SDK v.1.6.

    The general purpose of this project is to combine the gesture recognition with the beamforming. Right now, I’d like to obtain a dialog window in which I can visualize the skeleton and the direction of the sound source as well. The problem is that I still can’t run them at the same time, despite  I’m keeping  things simple without introducing any change in the original functions.

    Being more specific, I’ve created a single class called CBasics, in which all the members come from the CAudioBasics and CSkeletonBasics classes.  In the main(), I initialize the COM libraries and call the run() function, where I create the main window in the usual way:

    HWND hWndApp = CreateDialogParamW(         
        // Show window
        ShowWindow(hWndApp, nCmdShow);
        const int eventCount = 1;
        HANDLE hEvents[eventCount];
        // Main message loop
        while (WM_QUIT != msg.message) 
    	hEvents[0] = m_hNextSkeletonEvent;
            DWORD dwEvent = MsgWaitForMultipleObjects(eventCount, hEvents, FALSE, INFINITE, QS_ALLINPUT);     		
            if (WAIT_OBJECT_0 == dwEvent)    
            if (PeekMessageW(&msg, NULL, 0, 0, PM_REMOVE))  
                if ((hWndApp != NULL) && IsDialogMessageW(hWndApp, &msg))

    At this point, I'm not sure if I should call the ProcessAudio() and Update() functions in the main message loop, given that I removed the WM_TIMER message in DlgProc() to avoid any synchronization problem.

    So, there us the usual call to MessageRouter(), DlgProc() and CreateFirstConnected(), where I assumed it was correct to use a single m_pNuiSensor object for the initialization:

    HRESULT CBasics::CreateFirstConnected()
    if (NULL != m_pNuiSensor)
            if (SUCCEEDED(hr))
    m_hNextSkeletonEvent = CreateEventW(NULL, TRUE, FALSE, NULL); 
    hr = m_pNuiSensor->NuiSkeletonTrackingEnable(m_hNextSkeletonEvent, 0);
    	 hr = InitializeAudioStream();
    HRESULT CBasics::InitializeAudioStream()
    HRESULT hr = m_pNuiSensor->NuiGetAudioSource(&m_pNuiAudioSource);
    hr = m_pNuiAudioSource->QueryInterface(IID_IMediaObject, (void**)&m_pDMO); 
     hr = m_pNuiAudioSource->QueryInterface(IID_IPropertyStore, (void**)&m_pPropertyStore);
    PROPVARIANT pvSysMode;

    At this point, there should be the call to ProcessSkeleton(), DrawSkeleton(), SkeletonToScreen() and DrawBone() on one side, according to the Event Modelling, while on the other side I should have ProcessAudio(), Update() and all the functions contained in the AudioPanel class. I tried to put these last 2 functions within UpdateSkeletonData(), but it didn't work.

    So, I really don't understand what's missing, I've tried different solutions without any valuable result. I don't know if it's a matter of D2D resources (i.e., defining a single RECT structure, using IDC_VIDEOVIEW and IDC_AUDIOVIEW as handles  for GetWindowRect() ) or if it's something on the general structure. At the end, I can compile the project without having any error, but the .exe doesn't work.

    I found few similar examples in C#, I'm afraid it's not even possible to combine audio and depth data in C++!

    Sorry for the veeeery long explanation, hope at least my point is clear. Any help would be appreciated!


    Wednesday, October 31, 2012 6:56 PM

All replies

  • What is the exact behavior you're seeing?

    You can definitely use both depth and audio in the same application. 

    Audio data should generally be pumped based on a timer message because of the underlying circular buffer. 

    Friday, November 30, 2012 10:12 PM
  • Hi Chris,

    I solved the problem by myself, but I had to change quite a lot of things in my project. I had no previous experience on Windows applications, dialog boxes, etc., so I didn't know how to manage a timer and the relative messages sent to the procedure. It looked pretty confusing to me to manage at the same time the color/depth stream, acquired with an event-based model, and the audio stream, based on a timer-driven model.

    Anyway, these are the changes I made:

    - I copied the entire SkeletalViewer example (instead of the SkeletonBasics) to output all the streams available, without any valuable change except for the initialization-flag value

    - I created a separate thread for the audio data following the Nui_ProcessThread() function, to initialize the resources for audio stream as done in the AudioBasics example and to set the timer properly

    - I wrote a separate AudioClass with all the members I needed for the application

    - I corrected thousands of mistakes

    I'm still working on it, but it seems to run correctly now.

    However, thanks for your interest!

    Saturday, December 1, 2012 2:14 PM