none
Modifying Kinect Depth data RRS feed

  • Question

  • I'm want to roughly isolate part of the depth image (the face). I know the location of the face by using OpenCV. What I'm trying to figure out is how to go from having a x, y, width, height rectangle of the face in the 2D image to 'cropping' the depth stream.

    This is the code I'm using to process the depth and fusion.

    void processDepth()
    {
        NUI_IMAGE_FRAME depthFrame = { 0 };
        ERROR_CHECK( kinect->NuiImageStreamGetNextFrame( depthStreamHandle, 0, &depthFrame ) );
    
        BOOL nearMode = FALSE;
        INuiFrameTexture *frameTexture = 0;
        kinect->NuiImageFrameGetDepthImagePixelFrameTexture( depthStreamHandle, &depthFrame, &nearMode, &frameTexture );
    
        NUI_LOCKED_RECT depthData = { 0 };
    
        frameTexture->LockRect( 0, &depthData, 0, 0 );
        if ( depthData.Pitch == 0 ) {
            std::cout << "zero" << std::endl;
        }
    
        processKinectFusion( (NUI_DEPTH_IMAGE_PIXEL*)depthData.pBits, depthData.size);
    
        ERROR_CHECK( kinect->NuiImageStreamReleaseFrame( depthStreamHandle, &depthFrame ) );
    }
    
    void processKinectFusion( const NUI_DEPTH_IMAGE_PIXEL* depthPixel, int depthPixelSize) 
    {
        // DepthImagePixel 
        HRESULT hr = ::NuiFusionDepthToDepthFloatFrame( depthPixel, width, height, m_pDepthFloatImage,
                            NUI_FUSION_DEFAULT_MINIMUM_DEPTH, NUI_FUSION_DEFAULT_MAXIMUM_DEPTH, TRUE );
        if (FAILED(hr)) {
            throw std::runtime_error( "::NuiFusionDepthToDepthFloatFrame failed." );
        }
    
        Matrix4 worldToBGRTransform = { 0.0f };
        worldToBGRTransform.M11 = 256 / 512;
        worldToBGRTransform.M22 = 256 / 384;
        worldToBGRTransform.M33 = 256 / 512;
        worldToBGRTransform.M41 = 0.5f;
        worldToBGRTransform.M42 = 0.5f;
        worldToBGRTransform.M44 = 1.0f;
        Matrix4 worldToCameraTransform;
    
        m_pVolume->GetCurrentWorldToCameraTransform( &worldToCameraTransform );
        hr = m_pVolume->ProcessFrame( m_pDepthFloatImage, NUI_FUSION_DEFAULT_ALIGN_ITERATION_COUNT,
                                        NUI_FUSION_DEFAULT_INTEGRATION_WEIGHT, &worldToCameraTransform );
    
        worldToCameraTransform;
        if (FAILED(hr)) {
            ++trackingErrorCount;
            if ( trackingErrorCount >= 100 ) {
                trackingErrorCount = 0;
                m_pVolume->ResetReconstruction( &IdentityMatrix(), nullptr );
            }
    
            return;
        }
    
        // PointCloud
        hr = m_pVolume->CalculatePointCloud( m_pPointCloud, &worldToCameraTransform );
        if (FAILED(hr)) {
            throw std::runtime_error( "CalculatePointCloud failed." );
        }
    
    // PointCloud
        hr = ::NuiFusionShadePointCloud( m_pPointCloud, &worldToCameraTransform,
                                    &worldToBGRTransform, m_pShadedSurface, nullptr );
    
    
        if (FAILED(hr)) {
            throw std::runtime_error( "::NuiFusionShadePointCloud failed." );
        }
    
        INuiFrameTexture * pShadedImageTexture = m_pShadedSurface->pFrameTexture;
    
        NUI_LOCKED_RECT ShadedLockedRect;
        hr = pShadedImageTexture->LockRect(0, &ShadedLockedRect, nullptr, 0);
    
        if (FAILED(hr)) {
            throw std::runtime_error( "LockRect failed." );
        }
    
        pShadedImageTexture->UnlockRect(0);
        }
    };

    The result looks like this now, goal is to only have the face shown.

    http://i.imgur.com/IhCo5LP.png

    Tuesday, May 28, 2013 10:15 AM

Answers

  • You need the color stream too, to search the face. When you get the area where the face is, you can search the same area in the depth stream. Every depth pixel that are not in this area get a depth value of 1 for example. This depth value will be to close for kinect fusion and only the depth pixels of the face area should be shown.
    Tuesday, May 28, 2013 11:35 AM

All replies

  • You need the color stream too, to search the face. When you get the area where the face is, you can search the same area in the depth stream. Every depth pixel that are not in this area get a depth value of 1 for example. This depth value will be to close for kinect fusion and only the depth pixels of the face area should be shown.
    Tuesday, May 28, 2013 11:35 AM
  • Thanks this is indeed what I need to do, and I guess it's pretty basic too, but I'm new to this so I need a little head start on how to actually filter the depth stream.
    Tuesday, May 28, 2013 12:04 PM
  • You should begin from the basics and read the samples in c++ Depth-D3D and Green Screen-D2D , and try to program the both again but with help from OpenCV (not with the windows c++ api). When you are ready with it, I'm sure ,you will know how to filter the depth stream. The next step would be to program Kinect Fusion Basics-D2D again with OpenCV. When you are ready with it, you  will know what the kinect Fusion workflow is. Then you are able to do everything with that information. (but not texture mapping : (   ) .
    Wednesday, May 29, 2013 11:30 AM