none
Native Kinect Interaction RRS feed

  • Question

  • Hi,

      in our company we use native C++ kinect SDK. I'm trying to get to work hand interaction but I've stuck in some details. I didn't found any example in SDK and in Internet how to use the native API for interaction and there is little to no documentation. There are 2 examples in SDK C# and managed C++ but there is no native C++ example.

     What I've done so far is a I've used GetKinectCoreWindowForCurrentThread to get core window I subscribed successfully to pointer events using SubscribePointerEntered/Moved/Exited. I'm receiving the evtents in massage loop correctly. But what I was unable to do is to correctly register handlers in GestureRecognizer for example RegisterManipulationStartedHandler. Here is how I tried to implement the IKinectGestureRecognizerManipulationHandler and register it:

    class myHandler : public IKinectGestureRecognizerManipulationHandler
    {
        virtual HRESULT STDMETHODCALLTYPE OnManipulationStarted(IKinectManipulationStartedEventArgs *args)
        {
            DebugTrace(L"Manipulation started");
            return S_OK;
        }
    
        virtual HRESULT STDMETHODCALLTYPE OnManipulationUpdated(IKinectManipulationUpdatedEventArgs *args)
        {
            DebugTrace(L"Manipulation updated");
            return S_OK;
        }
    
        virtual HRESULT STDMETHODCALLTYPE OnManipulationInertiaStarting(IKinectManipulationInertiaStartingEventArgs *args)
        {
            DebugTrace(L"Manipulation inertia starting");
            return S_OK;
        }
    
        virtual HRESULT STDMETHODCALLTYPE OnManipulationCompleted(IKinectManipulationCompletedEventArgs *args)
        {
            DebugTrace(L"Manipulation completed");
            return S_OK;
        }
    
        HRESULT STDMETHODCALLTYPE QueryInterface(REFIID riid, LPVOID * ppvObj)
        {
            return E_NOINTERFACE;
        }
    
        ULONG STDMETHODCALLTYPE AddRef()
        {
            return 0;
        }
        ULONG STDMETHODCALLTYPE Release()
        {
            return 0;
        }
    };
    
    myHandler g_manipulatorHandler;
    
    HRESULT CBodyBasics::InitializeDefaultSensor()
    {
        HRESULT hr;
    
        hr = GetDefaultKinectSensor(&m_pKinectSensor);
        if (FAILED(hr))
        {
            return hr;
        }
    
        if (m_pKinectSensor)
        {
            // Initialize the Kinect and get coordinate mapper and the body reader
            IBodyFrameSource* pBodyFrameSource = NULL;
    
            hr = m_pKinectSensor->Open();
    
            ...
    
            if (SUCCEEDED(hr))
            {
                hr = CreateKinectGestureRecognizer(&m_pGestureRecognizer);
            }
    
            if (SUCCEEDED(hr))
            {
                hr = m_pGestureRecognizer->RegisterManipulationStartedHandler(&g_manipulatorHandler);
            }
    
            SafeRelease(pBodyFrameSource);
        }
    
        if (!m_pKinectSensor || FAILED(hr))
        {
            SetStatusMessage(L"No ready Kinect found!", 10000, true);
            return E_FAIL;
        }
    
        return hr;
    }

    For the propose of testing I skipped implementation of QueryInterface, AddRef and Release because they are not called at all which I double checked in debuger. I tried to trigger the manipulation gesture by closing my hand but I still don't get any debug messages from IKinectGestureRecognizerManipulationHandler. The closed hand gesture is likely to be detected correctly because this example build on BodyBasic from SDK detects closed hand correctly in the same time.

    Can please someone provide an example or help me with this specific problem

    Thanks in advance,

    Pawel Dondziak


    Tuesday, February 17, 2015 5:43 PM

All replies

  • Did the call to register succeed? What was the HR value since I don't see that you checked that?

    Typically you are going to attach this interface onto some type of UIElement like a button, so i assume that is what myHandler is going to be. You should be implementing QueryInterface and at least returning 1 from AddRef as you are not following the COM convention since you don't know how the underlying code using this component. It may be doing a QueryInterface for you to return a pointer to the instantiated object but you are returning E_NOINTERFACE and not populating the return pointer. 


    Carmine Sirignano - MSFT

    Wednesday, February 18, 2015 8:15 PM
  • Thanks for you ideas Carmine. I've tried them but still I don't receive any callbacks. I've changed the implementation of IKinectGestureRecognizerManipulationHandler interface to this:

    class ManipulationHandler : public IKinectGestureRecognizerManipulationHandler
    {
    public:
        virtual HRESULT STDMETHODCALLTYPE OnManipulationStarted(IKinectManipulationStartedEventArgs *args)
        {
            DebugTrace(L"OnManipulationStarted");
            return S_OK;
        }
    
        virtual HRESULT STDMETHODCALLTYPE OnManipulationUpdated(IKinectManipulationUpdatedEventArgs *args)
        {
            DebugTrace(L"OnManipulationUpdated");
            return S_OK;
        }
    
        virtual HRESULT STDMETHODCALLTYPE OnManipulationInertiaStarting(IKinectManipulationInertiaStartingEventArgs *args)
        {
            DebugTrace(L"OnManipulationInteriaStarting");
            return S_OK;
        }
    
        virtual HRESULT STDMETHODCALLTYPE OnManipulationCompleted(IKinectManipulationCompletedEventArgs *args)
        {
            DebugTrace(L"OnManipulationCompleted");
            return S_OK;
        }
    
        STDMETHODIMP QueryInterface(REFIID riid, void** ppvObject)
        {
            IUnknown *punk = nullptr;
    
            if (riid == IID_IUnknown)
            {
                punk = static_cast<IUnknown*>(this);
            }
    
            if (riid == IID_IKinectGestureRecognizerManipulationHandler)
            {
                punk = this;
            }
    
            if (!punk)
            {
                return E_NOINTERFACE;
            }
    
            punk->AddRef();
            return S_OK;
        }
    
        STDMETHODIMP_(ULONG) AddRef()
        {
            return ++m_cRef;
        }
    
        STDMETHODIMP_(ULONG) Release()
        {
            ULONG cRef = --m_cRef;
    
            if (cRef == 0)
            {
                delete this;
            }
    
            return cRef;
        }
    
    private:
        ULONG m_cRef;
    };

    but the problem is that this didn't compile because IID_IKinectGestureRecognizerManipulationHandler is only declared but not defined in Kinect.h. I've tried with a generated IID but it didn't help. I've also examine more carefully the HRESULT from RegisterManipulationStartedHandler which is always set to S OK. I've also tried to set the bounding rect in gestureRecognizer still with no result:

    if (hr == S_OK)
            {
                hr = CreateKinectGestureRecognizer(&m_pGestureRecognizer);
            }
    
            if (hr == S_OK)
            {
                m_pGestureRecognizer->RegisterManipulationStartedHandler(m_pManipulationHandler.get());
            }
    
            if (hr == S_OK)
            {
                DebugTrace(L"Kinect initialized successfully\n");
    
                RectF rect = {0.f, 0.f, 1080.f, 1920.f};
                m_pGestureRecognizer->put_BoundingRect(rect);
                m_pGestureRecognizer->get_BoundingRect(&rect);
                DebugTrace(L"Bounding rect: %f, %f, %f, %f", rect.X, rect.Y, rect.Width, rect.Height);
            }

    To summarize for now I've 2 questions:

    1. From where I can get correct IID_IKinectGestureRecognizerManipulationHandler ?

    2. Do I need to set BoundingRect of gesture recognizer and how to do it correctly ?

    Is it somehow possible to receive from you a very simplistic example which uses this interfaces correctly? I would like to solve this problem together with you and maybe present it on Berlin Kinect Meetups.

    Thanks in advance,

    Pawel Dondziak



    Monday, February 23, 2015 1:43 PM
  • Just use __uuidof(), https://msdn.microsoft.com/en-us/library/zaah6a61.aspx

    __uuidof(IKinectGestureRecognizerManipulationHandler)

    As for a sample, I don't have one available. The simplest thing would be to use a UI framework we have already provided like WPF or Windows Store XAML/DX. Since a store application can be written in C++ as well as managed, any UI implementation work you are doing may be faster to complete using that framework.

    Porting a desktop Win32 application is pretty easy since you don't have to re-write most of your c++ code as this can be wrapped and compiled as a library. Using any c++ code can be used as long as you are not accessing native c++ libraries that are not allowed. See For Windows Runtime apps in C++/Win32 and COM:
    https://msdn.microsoft.com/en-us/library/windows/apps/br211369.aspx


    Carmine Sirignano - MSFT



    Tuesday, February 24, 2015 6:51 PM