none
Get world coordinate from depth image RRS feed

  • Question

  • Hello all!

    I use the following code to get the coordinates of depth image pixels:

    //...

    unsigned short* pBufferRun = (unsigned short*)LockedRect.pBits;

    for(int row = 0; row < m_depthHeight; ++row)
    {
     for(int col = 0; col < m_depthWidth; ++col)
     {
      int increment = (row * m_depthWidth + col);
      USHORT depth = NuiDepthPixelToDepth(*(pBufferRun+increment));
      Vector4 vp   = NuiTransformDepthImageToSkeleton(col, row, depth);
      //...record vp.x, vp.y, vp.z
     }
    }

    Is the value in vp.x, vp.y and vp.z the world coordinate of pixel (row, col) ?

    The data look strange however, when displayed as 3D point clouds.

    And when I conduct a transform on them, like rotation,  the result is not correct :(

    Could someone point out where the mistake is please?

    Thank you very much!


    • Edited by walkerzz Wednesday, August 29, 2012 3:25 PM
    Wednesday, August 29, 2012 3:23 PM

Answers

  • In your code, replace this:

      USHORT depth = NuiDepthPixelToDepth(*(pBufferRun+increment));
      Vector4 vp   = NuiTransformDepthImageToSkeleton(col, row, depth);

    with this:

      Vector4 vp   = NuiTransformDepthImageToSkeleton(col, row, *(pBufferRun+increment));

    In other words, pass the original packed depth value, without first unpacking it via NuiDepthPixelToDepth.


    John | K4W Dev

    • Marked as answer by walkerzz Thursday, September 20, 2012 8:52 AM
    Tuesday, September 4, 2012 9:13 PM

All replies

  • Looks like you are passing in the depth image pixel value, not the depth.

    http://msdn.microsoft.com/en-us/library/nuiskeleton.nuitransformdepthimagetoskeleton.aspx

    • The depth value in millimeters of the depth image pixel, shifted left by three bits. The left shift enables you to pass the value from the depth image directly into this function.

    Friday, August 31, 2012 12:26 AM
  • Thanks for the reply!

    Then what is the return value of the function NuiDepthPixelToDepth ? In

    http://msdn.microsoft.com/en-us/library/nuiimagecamera.nuidepthpixeltodepth.aspx

    it says the function "Unpacks the depth value from the packed pixel format",

    so I thought it returns the depth value of the corresponding pixel. Is this an misunderstanding ?

    And how should I use the function NuiTransformDepthImageToSkeleton ? I mean, how to set the 3rd parameter?

    Thanks again!


    • Edited by walkerzz Saturday, September 1, 2012 3:51 PM
    Saturday, September 1, 2012 3:50 PM
  • In your code, replace this:

      USHORT depth = NuiDepthPixelToDepth(*(pBufferRun+increment));
      Vector4 vp   = NuiTransformDepthImageToSkeleton(col, row, depth);

    with this:

      Vector4 vp   = NuiTransformDepthImageToSkeleton(col, row, *(pBufferRun+increment));

    In other words, pass the original packed depth value, without first unpacking it via NuiDepthPixelToDepth.


    John | K4W Dev

    • Marked as answer by walkerzz Thursday, September 20, 2012 8:52 AM
    Tuesday, September 4, 2012 9:13 PM
  • Thank you very much!

    Your suggestion gives the correct z-coordinate as depth value in meters!

    However I'm still confused:

    I used NUI_IMAGE_TYPE_DEPTH in NuiImageStreamOpen(), not NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX,

    as discussed in the forum and in the FAQ(http://social.msdn.microsoft.com/Forums/en-US/kinectsdknuiapi/thread/4da8c75e-9aad-4dc3-bd83-d77ab4cd2f82) Question 10, the depth value should be left shift by three bits : depth << 3.

    But why in my code the raw depth should be used?

    And though the z-coordinate is correct, the x and y coordinates are still not right, is there any further processing needed ?

    Thanks!

    Wednesday, September 5, 2012 1:38 PM
  • Hi,I have a problem to interupt you .Sorry to trouble you.I want to show kinect data in opengl ,and get X,Y,Z,and R,G,B through the functions NuiTransformDepthImageToSkeleton() and NuiImageGetColorPixelCoordinatesFromDepthPixel(),but i don't know how to set parameters in functions.My code and showimage are pasted below,my X,Y,Z is  -3.952,2.964,7.056,R,G,B is also wrong ,it doesn't show normally.I don't know the reason.Thank you for helping me.
     void CKinectApplicationView::CollectData(long &d)
    {

            IplImage *colorImage=NULL,*depthImage = NULL;  
            colorImage = cvCreateImage(cvSize(320, 240), 8, 3);  
            depthImage = cvCreateImage(cvSize(320,240),8,1);
            int KINECT_DEPTH_W = depthImage->width,KINECT_DEPTH_H = depthImage->height;
            long& degree = d;
            HANDLE h1 =CreateEvent(NULL,TRUE,FALSE,NULL);
            HANDLE m_pDepthStreamHandle = NULL;
            HANDLE h3 = CreateEvent(NULL,TRUE,FALSE,NULL);
            HANDLE m_pVideoStreamHandle = NULL;
            HRESULT hr= NuiImageStreamOpen(NUI_IMAGE_TYPE_DEPTH,NUI_IMAGE_RESOLUTION_320x240,0,2,h1,&m_pDepthStreamHandle);
            hr= NuiImageStreamOpen(NUI_IMAGE_TYPE_COLOR,NUI_IMAGE_RESOLUTION_640x480,0,2,h3,&m_pVideoStreamHandle);        
            if(labs(degree)%ANGLE==0)
            {
                    cvZero(depthImage);
                    cvZero(colorImage);
                    const NUI_IMAGE_FRAME *pImageFrameD = NULL;
                    const NUI_IMAGE_FRAME *pImageFrameC = NULL;
                    if(WaitForSingleObject(h1,INFINITE)==WAIT_OBJECT_0&&WaitForSingleObject(h3,INFINITE)==WAIT_OBJECT_0);//判断是否得到了新的数据;
                    {
                        hr = NuiImageStreamGetNextFrame(m_pDepthStreamHandle,0,&pImageFrameD);//得到该帧数据;
                            hr = NuiImageStreamGetNextFrame(m_pVideoStreamHandle,0,&pImageFrameC);
                            if(S_OK==hr)
                            {
                                    INuiFrameTexture *pTexture1 = pImageFrameD->pFrameTexture;
                                NUI_LOCKED_RECT lockrect1;
                                pTexture1->LockRect(0,&lockrect1,NULL,0);
                                    byte *pbuffer = (byte*)lockrect1.pBits;
                                    USHORT *pBufferRun = (USHORT*)lockrect1.pBits;
                                    if(lockrect1.Pitch!=0)
                                    {
                                            KinectPoint temp;
                                            long colorx,colory;
                                            for(int i=0;i<KINECT_DEPTH_H;i++)
                                            {
                                                    uchar* ptr1 = (uchar*)(depthImage->imageData+i*depthImage->widthStep);
                                                    uchar* ptr2= (uchar*)(colorImage->imageData+i*colorImage->widthStep);
                                                    byte *pbufferd = (byte*)lockrect1.pBits+i*lockrect1.Pitch;
                                                    USHORT *pBufferRund = (USHORT*)pbufferd;
                                                    for(int j=0;j<KINECT_DEPTH_W;j++)
                                                    {        
                                                            int index = i*KINECT_DEPTH_W+j;
                                                            ptr1[KINECT_DEPTH_W-1-j] = (byte)(256*pBufferRund[j]/0x0fff);                                        
                                                            USHORT realdepth = *pBufferRun<<3;
                                                            if (outfile.is_open())
                                                            {
                                                                    outfile<<realdepth<<endl;
                                                            }                
                                                        Vector4 v = NuiTransformDepthImageToSkeleton((float)j/320.0f,(float)i/240.0f,realdepth);
                                                            temp.x = v.x;
                                                            temp.y = v.y;
                                                            temp.z = v.z;
                                                            NuiImageGetColorPixelCoordinatesFromDepthPixel(NUI_IMAGE_RESOLUTION_640x480,0,(float)j/320.0f,(float)i/240.0f,realdepth,&colorx,&colory);
                                                            int a = (colorx+(colory-1)*KINECT_DEPTH_W)*4;
                                                            temp.R = pbuffer[a+2];
                                                            temp.G = pbuffer[a+1];
                                                            temp.B = pbuffer[a];
                                                            ptr2[3*(KINECT_DEPTH_W-1-j)] = temp.B;
                                                            ptr2[3*(KINECT_DEPTH_W-1-j)+1] = temp.G;  
                                                            ptr2[3*(KINECT_DEPTH_W-1-j)+2] = temp.R;  
                                                    
                                                            pBufferRun++;
                                                            VectorPointClouds.push_back(temp);
                                                    }                
                                            }
                                            cvNamedWindow("colorimage",1);
                                            cvShowImage("colorimage",colorImage);
                                            cvNamedWindow("depthimage",1);
                                            cvShowImage("depthimage",depthImage);
                                            int key =cvWaitKey(30);
                                    }        
                            }  
             }        
                    NuiImageStreamReleaseFrame(m_pDepthStreamHandle,pImageFrameD);
                    NuiImageStreamReleaseFrame( m_pVideoStreamHandle, pImageFrameC);
          }
    }

    alert me

    Monday, December 3, 2012 1:17 PM