none
Kinect Interaction does only track left hand? RRS feed

  • Question

  • Hello guys,

    I've been trying to implement the kinect interaction wich came with the new Kinect SDK 1.7 in C++. It's been going well so far: I have successfully created an INuiInteractionStream instance by calling the function NuiCreateInteractionStream() and then successfully processed the depth and skeleton data with the functions InteractionStream->ProcessDepth() and InteractionStream->ProcessSkeleton().

    This all seems to work fine (no error messages and FAILED(hr) always false) until i get to the actual interaction frame with the following lines of code:

    if (!( WAIT_OBJECT_0 == WaitForSingleObject(m_hNextHandFrameEvent, 0) ))    
    {        
       return;    
    }     
    
    NUI_INTERACTION_FRAME interactionFrame;    
    hr = mInteractionStream->GetNextFrame(0, &interactionFrame);    
    if(FAILED(hr))    
    {        
      return;
    }

    Now comes the depressing part:

    I tried a lot of changes in my polymorphed function of INuiInteractionClient but it always comes down to this: After i call the GetNextFrame() function the value of interactionFrame.UserInfos->HandPointerInfos->HandType is always either zero or one, meaning "No hand" or "Left hand". The HandEventType works for the left hand in the way it should, if i close my hand it's set to "GRIP" and if i release it's set to "GRIP_RELEASED".

    My class for the InteractionClient looks like this:

    	class interactionClient : public INuiInteractionClient
    	{
    		public:
    			interactionClient() {}
    			~interactionClient() {}
    			STDMETHODIMP_(ULONG) AddRef()                                    
    			{ return 1;     }
    			STDMETHODIMP_(ULONG) Release()                                   
    			{ return 1;     }
    			STDMETHODIMP QueryInterface(REFIID riid, void **ppv)     
    			{ return S_OK;  }
    			STDMETHODIMP GetInteractionInfoAtLocation(DWORD skeletonTrackingId, NUI_HAND_TYPE handType, FLOAT x, FLOAT y, NUI_INTERACTION_INFO *pInteractionInfo)
    			{
    				if(pInteractionInfo)
    				{
    					if(NUI_HAND_TYPE_RIGHT == handType)
    					{
    						pInteractionInfo->IsGripTarget = true;
    						pInteractionInfo->PressTargetControlId  = 1;
    					}
    					else if(NUI_HAND_TYPE_LEFT == handType)
    					{
    						pInteractionInfo->IsGripTarget = true;
    						pInteractionInfo->PressTargetControlId  = 2;
    					}	
    					else if(NUI_HAND_TYPE_NONE == handType)
    					{
    						pInteractionInfo->PressTargetControlId  = 0;
    					}	
    
    					pInteractionInfo->PressAttractionPointX = 0.f;
    					pInteractionInfo->PressAttractionPointY = 0.f;
    		
    					return S_OK;
    				}
    				return E_POINTER;
    			}
    			
    
    	};

    In addition it does'nt seem to matter what value i write into pInteractionInfo->IsGripTarget. Even if i set it to false every time the function is called the application behaves the same (only left hand is being tracked correctly..)

    With debug print outs i was able to determine that the function of the class interactionClient "GetInteractionInfoAtLocation" is in fact being called by the kinect and the value of "NUI_HAND_TYPE handType" is in fact sometimes RIGHT and sometimes LEFT but never NONE.

    I hope you understand my problem :) Does anyone has an idea on what's causing this?

    best regards

    -Kinecticuz


    • Edited by Kinecticuz Wednesday, April 10, 2013 4:07 PM
    Wednesday, April 10, 2013 4:06 PM

Answers

  • I'm having somewhat the same problem. 

    Currently I get left and right hand events from GetInteractionInfoAtLocation and from interactionFrame.UserInfos->HandPointerInfos->HandType. However, I'm having problem with the grip state always returning NONE 

    void InteractionManager::ReceivedStreamAlert()
    {
    	NUI_INTERACTION_FRAME interactionFrame = { 0 };
    
    	HRESULT results = m_interactionStream->GetNextFrame(0, &interactionFrame );
    
    	if(FAILED(results)) return; 
    	
    	for(int i = 0; i < NUI_SKELETON_COUNT; i++)
    	{
    		NUI_USER_INFO user = interactionFrame.UserInfos[i];
    		
    		if (user.SkeletonTrackingId != 0)
    		{
    			for(int j = 0; j < NUI_USER_HANDPOINTER_COUNT; j++)
    			{
    				NUI_HANDPOINTER_INFO hand = user.HandPointerInfos[j];
    				NUI_HANDPOINTER_STATE state  = (NUI_HANDPOINTER_STATE)hand.State;
    
    				if(state & NUI_HANDPOINTER_STATE_INTERACTIVE)
    					cout << "Interactive: " << hand.X << " " << hand.Y <<endl;
    				if(state & NUI_HANDPOINTER_STATE_PRESSED)
    					cout << "Pressed Button" << endl;
    
    				if(hand.State != NUI_HANDPOINTER_STATE_NOT_TRACKED)
    				{
    					//if(m_pInteractionStreamFunction != NULL)
    					//	m_pInteractionStreamFunction(hand);
    					switch(hand.HandEventType)
    					{
    						case 0:
    							cout << hand.HandType << " NUI_HAND_EVENT_TYPE_NONE" << endl;
    							break;
    						case 1:
    							cout << hand.HandType << " NUI_HAND_EVENT_TYPE_GRIP" << endl; //This never output
    							break;
    						case 2:
    							cout << hand.HandType << " NUI_HAND_EVENT_TYPE_GRIPRELEASE" << endl; //This never outputs
    							break;
    					}
    				}
    			}
    		}
    	}
    }


    Also i wrote my GetInteractionInfoAtLocation function as this.

    HRESULT STDMETHODCALLTYPE GetInteractionInfoAtLocation(DWORD skeletonTrackingId, NUI_HAND_TYPE handType, FLOAT x, FLOAT y, NUI_INTERACTION_INFO *pInteractionInfo)


    • Edited by Goss84 Wednesday, April 10, 2013 6:00 PM
    • Marked as answer by Kinecticuz Thursday, April 11, 2013 9:33 AM
    Wednesday, April 10, 2013 5:51 PM

All replies

  • I'm having somewhat the same problem. 

    Currently I get left and right hand events from GetInteractionInfoAtLocation and from interactionFrame.UserInfos->HandPointerInfos->HandType. However, I'm having problem with the grip state always returning NONE 

    void InteractionManager::ReceivedStreamAlert()
    {
    	NUI_INTERACTION_FRAME interactionFrame = { 0 };
    
    	HRESULT results = m_interactionStream->GetNextFrame(0, &interactionFrame );
    
    	if(FAILED(results)) return; 
    	
    	for(int i = 0; i < NUI_SKELETON_COUNT; i++)
    	{
    		NUI_USER_INFO user = interactionFrame.UserInfos[i];
    		
    		if (user.SkeletonTrackingId != 0)
    		{
    			for(int j = 0; j < NUI_USER_HANDPOINTER_COUNT; j++)
    			{
    				NUI_HANDPOINTER_INFO hand = user.HandPointerInfos[j];
    				NUI_HANDPOINTER_STATE state  = (NUI_HANDPOINTER_STATE)hand.State;
    
    				if(state & NUI_HANDPOINTER_STATE_INTERACTIVE)
    					cout << "Interactive: " << hand.X << " " << hand.Y <<endl;
    				if(state & NUI_HANDPOINTER_STATE_PRESSED)
    					cout << "Pressed Button" << endl;
    
    				if(hand.State != NUI_HANDPOINTER_STATE_NOT_TRACKED)
    				{
    					//if(m_pInteractionStreamFunction != NULL)
    					//	m_pInteractionStreamFunction(hand);
    					switch(hand.HandEventType)
    					{
    						case 0:
    							cout << hand.HandType << " NUI_HAND_EVENT_TYPE_NONE" << endl;
    							break;
    						case 1:
    							cout << hand.HandType << " NUI_HAND_EVENT_TYPE_GRIP" << endl; //This never output
    							break;
    						case 2:
    							cout << hand.HandType << " NUI_HAND_EVENT_TYPE_GRIPRELEASE" << endl; //This never outputs
    							break;
    					}
    				}
    			}
    		}
    	}
    }


    Also i wrote my GetInteractionInfoAtLocation function as this.

    HRESULT STDMETHODCALLTYPE GetInteractionInfoAtLocation(DWORD skeletonTrackingId, NUI_HAND_TYPE handType, FLOAT x, FLOAT y, NUI_INTERACTION_INFO *pInteractionInfo)


    • Edited by Goss84 Wednesday, April 10, 2013 6:00 PM
    • Marked as answer by Kinecticuz Thursday, April 11, 2013 9:33 AM
    Wednesday, April 10, 2013 5:51 PM
  • Implicitly, you just solved my problem: I wasn't aware of that the user.HandPointerInfos is another array with actual two Info's for two hands (embarrassing..).

    Anyways, thanks a lot. The part of my application after getting the next interaction frame is now almost the same as yours and works, wich means your problem must be coming from either how you process the depth and skeleton data or from your InteractionClient class. I'll post my code below (reduced to a minimum), maybe it helps you.

    void getDepthData(){
    
    
    	//WAIT AND CHECK IF IT'S TIME TO PROCESS NEW DEPTH IMAGE DATA
    	if (!( WAIT_OBJECT_0 == WaitForSingleObject(m_hNextDepthFrameEvent, 0) ))
    	{
    		return;
    	}
    
    	HRESULT hr;
    	NUI_IMAGE_FRAME imageFrame;
    
    	// GET DEPTH FRAME
    	hr = NUISensor->NuiImageStreamGetNextFrame(m_pDepthStreamHandle, 0, &imageFrame);
    	if (FAILED(hr))
    	{
    		return;
    	}
    
    	BOOL nearMode;
    	INuiFrameTexture* pTexture;
    
    	hr = NUISensor->NuiImageFrameGetDepthImagePixelFrameTexture(
    		m_pDepthStreamHandle, &imageFrame, &nearMode, &pTexture);
    	if (FAILED(hr))
    	{
    		NUISensor->NuiImageStreamReleaseFrame(m_pDepthStreamHandle, &imageFrame);
    		return;
    	}
    
    	NUI_LOCKED_RECT LockedRect;
    
    	// LOCK THE FRAME SO THE KINECT KNOWS NOT TO MODIFY IT
    	pTexture->LockRect(0, &LockedRect, NULL, 0);
    
    	// TEST IF THE RECEIVED DATA IS VALID
    	if (LockedRect.Pitch != 0)
    	{
    		//HAND INTERACTION
    		mInteractionStream->ProcessDepth(LockedRect.size,LockedRect.pBits,imageFrame.liTimeStamp);
    	}
    
    	pTexture->UnlockRect(0);
    	pTexture->Release();
    	NUISensor->NuiImageStreamReleaseFrame(m_pDepthStreamHandle, &imageFrame);
    
    }
    
    void getSkeletonData(void)
    {
        if (NULL == NUISensor)
        {
            return;
        }
    	//RESET SKELETONFRAME TO ZERO VALUES
    	NUI_SKELETON_FRAME	skeletonFrameLocal = {0};
    	mSkeletonFrame = skeletonFrameLocal;
    
    
    	//GET NEXT SKELETON DATA FRAME FROM SENSOR
        HRESULT hr = NUISensor->NuiSkeletonGetNextFrame(0, &mSkeletonFrame);
        if ( FAILED(hr) )
        {
            return;
        }
        
    	//PROCESS SKELETON DATA FOR HAND GESTURES
    	Vector4 tempVec = {0};
    	NUISensor->NuiAccelerometerGetCurrentReading(&tempVec);
    
    	mInteractionStream->ProcessSkeleton(NUI_SKELETON_COUNT,pskeletonFrame->SkeletonData,&tempVec,pskeletonFrame->liTimeStamp);
    }
    
    class interactionClient : public INuiInteractionClient
    {
        public:
            interactionClient()                                                 {}
            ~interactionClient()                                                {}
            STDMETHODIMP_(ULONG)    AddRef()                                    { return S_OK;     }
            STDMETHODIMP_(ULONG)    Release()                                   { return S_OK;     }
            STDMETHODIMP            QueryInterface(REFIID riid, void **ppv)     { return S_OK;  }
            STDMETHODIMP GetInteractionInfoAtLocation(DWORD skeletonTrackingId, NUI_HAND_TYPE handType, FLOAT x, FLOAT y, NUI_INTERACTION_INFO *pInteractionInfo)
            {
                
                if(pInteractionInfo)
                {
                    pInteractionInfo->IsGripTarget          = true;
                    pInteractionInfo->PressTargetControlId  = 0;
                    pInteractionInfo->PressAttractionPointX = 0.f;
                    pInteractionInfo->PressAttractionPointY = 0.f;
        
                    return S_OK;
                }
                return E_POINTER;
            }
            
    
    };

    If your problem doesn't go away please consider opening a new thread, cause my problem is now solved by you ;-)

    Thanks again.

    best regards

    -Kinecticuz

    Thursday, April 11, 2013 9:32 AM
  • Hi Kinecticuz, could you explain how to use your class (interactionClient)

    I would like to get hand grip in C++ but still don't know how to get

    I tried to add your code

        if (NULL != m_pNuiSensor)
        {
            // Initialize the Kinect and specify that we'll be using skeleton
            //hr = m_pNuiSensor->NuiInitialize(NUI_INITIALIZE_FLAG_USES_SKELETON); 
            hr = m_pNuiSensor->NuiInitialize(NUI_INITIALIZE_FLAG_USES_SKELETON | NUI_INITIALIZE_FLAG_USES_COLOR | NUI_INITIALIZE_FLAG_USES_DEPTH_AND_PLAYER_INDEX); 
            if (SUCCEEDED(hr))
            {
    			
                // Create an event that will be signaled when skeleton data is available
                m_hNextSkeletonEvent = CreateEventW(NULL, TRUE, FALSE, NULL);
    
                // Open a skeleton stream to receive skeleton data
                hr = m_pNuiSensor->NuiSkeletonTrackingEnable(m_hNextSkeletonEvent, 0); 
    
    			m_interactionClient = new interactionClient();
    			hr = NuiCreateInteractionStream(m_pNuiSensor,m_interactionClient, &m_interactionStream);
    			if(FAILED(hr))
    				return hr;
    			m_hNextHandFrameEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
    			hr = m_interactionStream->Enable(m_hNextHandFrameEvent);
    
    			if(FAILED(hr))
    				return hr;
    				
            }
        }
    


    but it got error at  hr = NuiCreateInteractionStream(m_pNuiSensor,m_interactionClient, &m_interactionStream);

    SkeletonBasics-D2D.exe not found or not built by the last incremental link; performing full link
    1>SkeletonBasics.obj : error LNK2019: unresolved external symbol __imp__NuiCreateInteractionStream@12 referenced in function "private: long __thiscall CSkeletonBasics::CreateFirstConnected(void)" (?CreateFirstConnected@CSkeletonBasics@@AAEJXZ)

    PS: I use the SkeletonBasic-D2D sample from SDK 1.7


    • Edited by King Nguyen Friday, April 12, 2013 9:10 AM add more detail
    Friday, April 12, 2013 3:42 AM
  • Hi, i'm using the class interactionClient only at one place in my whole project: At the initialization of the InteractionStream.

    You should initialize your kinect sensor at one point in your code like this (with different flags maybe):


            HRESULT hr = NUISensor->NuiInitialize(NUI_INITIALIZE_FLAG_USES_SKELETON | NUI_INITIALIZE_FLAG_USES_DEPTH_AND_PLAYER_INDEX); 

    If your line differs from this one, try to change your flags to those two..

    After that, i create an instance of my class interactionClient and give that instance to the NuiCreateInteractionStream like this:

    			mInteractionClient = new interactionClient();
    
    			hr = NuiCreateInteractionStream(NUISensor,mInteractionClient, &mInteractionStream);
    
    

    In addition, i give the function a variable of type INuiInteractionStream, you cannot instantiate this variable it jut have to be a pointer with that particular type. The function NuiCreateInteractionStream now gives you a working instance of the INuiInteractionStream type to your pointer.

    Now you just have to create an event an then enable the InteractionStream like this:

    			m_hNextHandFrameEvent = CreateEvent(NULL, TRUE, FALSE, NULL);
    			hr = mInteractionStream->Enable(m_hNextHandFrameEvent);
    
    			if(FAILED(hr))
    				return;
    
            

    I hope this helps you ;)

    best regards

    -Kinecticuz

    Friday, April 12, 2013 9:07 AM
  • Hi Kinecticuz,

    I could initialize the InteractionStream.

    Now I try to process the InteractionStream . But it never jump into ProcessInteraction(),  while ProcessSkeleton() works well

    Could you give me any advice?

    void CSkeletonBasics::Update()
    {
        if (NULL == m_pNuiSensor)
        {
            return;
        }
    
        if ( WAIT_OBJECT_0 == WaitForSingleObject(m_hNextSkeletonEvent, 0) )
        {
            ProcessSkeleton();
        }
    	if ( WAIT_OBJECT_0 == WaitForSingleObject(m_hNextHandFrameEvent, 0) )
        {
            ProcessInteraction();
        }
    
    }

    Tuesday, April 16, 2013 9:46 AM
  • you need  processDepth()

    if (WAIT_OBJECT_0 == WaitForSingleObject(m_hNextDepthFrameEvent, 0))
    		{
    			ProcessDepthFrame();
    		}
    		if (WAIT_OBJECT_0 == WaitForSingleObject(m_hNextSkeletonFrameEvent, 0))
    		{
    			ProcessSkeleton();
    		}
    		if (WAIT_OBJECT_0 == WaitForSingleObject(m_hNextInteractionEvent, 0))
    		{
    			ProcessInteraction();
    		}


    • Edited by pphamtuan Tuesday, December 10, 2013 1:29 AM
    Tuesday, December 10, 2013 1:28 AM