none
Kinect Bridge With Open CV Basics D2D and Point Cloud Library RRS feed

  • Question

  • Hello all!

    I already managed to implement PCL (PointCloudLibrary) in my KinectBridgeWithOpenCVBasics-D2D project and visualize the cloud in a seperate window but now i fail at a simple thing like reading the kinect depth data properly.
    Where can i read x, y, z value for every pixel in this example project? Anybody can help me with this??
    it seems that OpenCVFrameHelper::GetDepthData reads just the z value...

    thanks for your help!

    Thursday, July 11, 2013 5:15 PM

Answers

All replies

  • Can you elaborate more on what you mean for the x,y values? The pixel value is acquired from the column(x) and row(y) offset in the DepthImageFrame values that you copied to your byte array.

    // KinectHelper.h file
    // class property contains the following byte array
    BYTE* m_pDepthBuffer;
    // This is populated in the UpdateDepthFrame function
    HRESULT KinectHelper<Image>::UpdateDepthFrame(DWORD waitMillis /* = 0 */)
    {
    ...
        // Check if image is valid
        if (lockedRect.Pitch != 0)
        {
    ...
            if (size != m_depthBufferSize)
            {
                delete [] m_pDepthBuffer;
                m_pDepthBuffer = new BYTE[size];
                m_depthBufferSize = size;
            }
            memcpy_s(m_pDepthBuffer, size, pBuffer, size);
    ...
        }
    ...
    }

    As for getting depth value, you have to keep in mind this is a raw(packed) depth value. You have to remove the player index:
    http://msdn.microsoft.com/en-us/library/nuiimagecamera.nuidepthpixeltodepth.aspx

    // KinectHelper.h file
    template <typename Image>
    HRESULT KinectHelper<Image>::DepthShortToRgb(USHORT depth, UINT8* redPixel, UINT8* greenPixel, UINT8* bluePixel) const
    {
    ...
        SHORT realDepth = NuiDepthPixelToDepth(depth);
        USHORT playerIndex = NuiDepthPixelToPlayerIndex(depth);
    ...
    }
    If you are trying to get x,y values from an OpenCV construct, you will need to post the question to the OpenCV forums/support.

    Carmine Si - MSFT

    Thursday, July 11, 2013 7:38 PM
  • NuiTransformDepthImageToSkeleton will transform a pixel in the depth image to real world coordinates (skeleton point)

    Anson Tsao MSFT

    Friday, July 12, 2013 6:22 AM
  • Hy!
    thanks for your fast reply and sorry if i was unclear. i want to copy the real world coordinates of every pixel into my point_cloud for further segmentation.

    i already tried NuiTransformDepthImageToSkeleton (returns Vector4, the real world coordinates of the pixel) and it works so far, only i get returned valid data just for the bright spots in the depth image (and they all have the same vector4.z), also the area close to the coordinate system and the stripe on the left side seem wrong (see in picture)?! anybody has a clue what i´m doing wrong here? vector4.x and vector4.y look good to me...

    that´s the code block i use:

    //clear clouds
    		cloud->clear();
    		cloudRGB->clear();
    
    		// Convert the point from depth space to real world space and copy it in point_cloud
    		for (UINT y = 0; y < depthHeight; y++)
    		{
    			// Get row pointer for depth Mat
    			USHORT* pDepthRow = m_depthMat->ptr<USHORT>(y);
    
    			for (UINT x = 0; x < depthWidth; x++)
    			{
    				depth = pDepthRow[x];
    				depthX = (LONG)x;
    				depthY = (LONG)y;
    				vector4 = NuiTransformDepthImageToSkeleton(depthX, depthY, depth, depthResolution);
    				
    				// Requires a valid depth value
    				if (vector4.z > FLT_EPSILON)
    				{
    					// copy data into point clouds
    					pcl::PointXYZ basic_point;
    					basic_point.x = vector4.x;
    					basic_point.y = vector4.y;
    					basic_point.z = vector4.z;
    					cloud->points.push_back(basic_point);
    
    					pcl::PointXYZRGBA point_rgba;
    					point_rgba.x = basic_point.x;
    					point_rgba.y = basic_point.y;
    					point_rgba.z = basic_point.z;
    					uint32_t rgb = (static_cast<uint32_t>(r) << 16 |
    							static_cast<uint32_t>(g) << 8 | static_cast<uint32_t>(b));
    					point_rgba.rgb = *reinterpret_cast<float*>(&rgb);
    					cloudRGB->points.push_back (point_rgba);
    				}
    			}
    		}

    here is a picture of the depth image and the transformed point cloud so you can see what i mean

    Tuesday, July 16, 2013 11:56 AM
  • Is pDepthRow:

    depth = pDepthRow[x];

    the raw(packed) depth value? For example in the code above, m_pDepthBuffer, values are used for this function.

    http://msdn.microsoft.com/en-us/library/jj663871.aspx
    usDepthValue

    Type: USHORT
    [in] The depth value (in millimeters) of the depth image pixel, shifted left by three bits. The left shift enables you to pass the value from the depth image directly into this function.

    For the real depth value, you then remove the player index from that:

    SHORT realDepth = NuiDepthPixelToDepth(depth);
    http://social.msdn.microsoft.com/Forums/en-US/56b6298c-7365-44fe-917e-bc3eadd91212/get-world-coordinate-from-depth-image

    Carmine Si - MSFT

    Tuesday, July 16, 2013 5:14 PM
  • Hey guys!

    Thanks a lot for your help, everything works fine now!

    I was using depth-values which were obviously already unpacked.

    So for all the other users:
    -> don´t let you get confused by USHORT depth (packed and unpacked depth values are the same data type)
    -> from OpenCVFrameHelper::GetDepthData use pBufferRun, depthWidth, depthHeight and see code below
         NuiTransformDepthImageToSkeleton needs unpacked depth data!!!!

    // Convert the point from depth space to real world space and copy it in point_cloud
    		for (UINT y = 0; y < depthHeight; y++)
    		{
    			for (UINT x = 0; x < depthWidth; x++)
    			{
    				depthX = (LONG)x;
    				depthY = (LONG)y;
    				vector4 = NuiTransformDepthImageToSkeleton(depthX, depthY, pBufferRun[y * depthWidth + x]);
    				
    				// Requires a valid depth value
    				if (vector4.z > FLT_EPSILON)
    				{
    					// copy data into point clouds
    					pcl::PointXYZ basic_point;
    					basic_point.x = vector4.x;
    					basic_point.y = vector4.y;
    					basic_point.z = vector4.z;
    					cloud->points.push_back(basic_point);
    
    					pcl::PointXYZRGBA point_rgba;
    					point_rgba.x = basic_point.x;
    					point_rgba.y = basic_point.y;
    					point_rgba.z = basic_point.z;
    					uint32_t rgb = (static_cast<uint32_t>(r) << 16 |
    							static_cast<uint32_t>(g) << 8 | static_cast<uint32_t>(b));
    					point_rgba.rgb = *reinterpret_cast<float*>(&rgb);
    					cloudRGB->points.push_back (point_rgba);
    				}
    			}
    		}

    Tuesday, July 16, 2013 11:36 PM
  • Hi

    could you give a working procedure tipp how you integrated the PCL library into the Kinect sample using VStudio 12 or VStudio 13?

    Thanks!

    Monday, August 18, 2014 11:42 PM