none
Depth Color alignment RRS feed

  • General discussion

  • I am attempting to generate a 3D mesh that combines the color and depth streams. I know this question has been asked several times but I don't see any code examples so let me volunteer mine and perhaps the issue can be resolved. I started from the c++ skeleton viewer example code. I have the depth stream at 320x240 and video stream at 640x480 and the mesh I generate has 320x240 vertices and I set color per vertex so I could in theory make the video stream 320x240 but this throws an error in the init method so I keep it at 640x480 and just skip every other color value. For the depth stream I use NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX since NUI_IMAGE_TYPE_DEPTH also throws an error so I must deal with shifting the bits as mentioned in other posts

    In the Nui_GotDepthAlert() method I removed the drawDevice stuff and made it simply set the position of the vertex associated with the pixel. one thing I notice is that the Vector4 returned is in meters not mm 

     

      BYTE * pBuffer = (BYTE*) LockedRect.pBits;
      USHORT * pBufferRun = (USHORT*) pBuffer;
      for( int y = 0 ; y < 240 ; y++ )
      {
       for( int x = 0 ; x < 320 ; x++ )
       {
    		Vector4 v = NuiTransformDepthImageToSkeletonF(x/320.0f,y/240.0f, *pBufferRun >> 3);
    		setPointPos(x,y,v.x,v.y,v.z); //set the vertex position
        pBufferRun++;
       }
    			
      }

     


    in the Nui_GotVideoAlert() method I set the vertex color based on the pixel the NuiImageGetColorPixelCoordinatesFromDepthPixel() method call does effect the offset but it moves too much, in fact the offset is closer if I simply line the colors and depths one to one. 

     

     

      BYTE * pBuffer = (BYTE*) LockedRect.pBits;
    
    	long colorx,colory;
    	int colIndx = 0;
      for( int y = 0 ; y < 240 ; y++ )
      {
       for( int x = 0 ; x < 320 ; x++ )
       {
    			 USHORT depth = (short)(App->getPoint(x,y).z * 1000)<<3; //convert to mm
    			 NuiImageGetColorPixelCoordinatesFromDepthPixel(NUI_IMAGE_RESOLUTION_640x480,0,x,y,depth,&colorx,&colory);
    			 colIndx = (colorx + colory*640)*4; //use the transformed coordinates
    			 // colIndx = (x*2 + y*2*640)*4; //just map them one to one.. 
    			 setPointColor( x ,y,	pBuffer[colIndx+2], pBuffer[colIndx+1], pBuffer[colIndx] );  
       }				
      }

    please let me know if more information is needed

     


    • Changed type Eddy Escardo-Raffo [MSFT] Friday, July 15, 2011 2:27 AM This is a code example, not a question looking for a specific answer
    Friday, July 15, 2011 2:10 AM

All replies

  • Are you requesting skeleton tracking? You don't have to use tracking to use the projection into skeleton space. If so I believe it uses both depth and color and requires the depth and color to be in a specific format. That may be the source of your errors though that isn't your question. The IR and RGB sensors are approximately 2.5CM apart. The nominal focal length and field of view is provided in the MSR_NuiImageCamera.h header. Ideally the camera and depth sensor axes lie in a plane and are parallel. I believe there is calibration done at the time of manufacture to check how true that is.

    You need the depth to find the corresponding image pixel and you can't go the other way. So logically you project a ray from the depth focal point through the screen pixel for the distance of depth. You then find the direction that is from the camera focal point and project that onto the plane of the image. If there is calibration then you don't have access to it. If there is calibration you may be able to deduce it from what they provide versus what you come up with.

    Likely your problem is the skeleton to depth projection provided is wrong. A given depth transforms to a fixed Z. So points on a sphere become points on a plane and points on a plane become points on a sphere. I belie the X and Y are wrong as well, only right when they are on the axes. It's an approximation as well, specifically an approximation of tangent. They are definitely wrong because they are dependant upon the transformed Z and it's wrong. The collection of points a given distance from the focal point is not a line in a plane perpendicular to the z-axis, but points on a circle instead.

    You can verify that by pointing the Kinect at a wall and seeing how flat your point cloud is. It's not surprising 1:1 comes close, the device is designed to get that close at near the minimum depth. How close that is depends upon distance. A plane through the focals points and left edges of the images of both sensor intersect at some distance, i.e. sin(fov.d/2)*z=sin(fov.i/2)*z-2.5 or z~=94cm. The right edges do to, but that's behind the Kinect.

    Your result rather beg the question of whether the depth to color is correct. The error to with depth to skeleton would just make your mesh all wrong, but if the depth to image is correct it would still be colored correctly. So it may be depth to image is wrong as well. You might try alpha blending the depth over the image to see if looks right. You have to do that on a pixel by pixel basis though.

    Friday, July 15, 2011 12:15 PM