none
CL NUI SDK... RRS feed

  • Question

  • this is CLNUICamera API

    「 IMPORT(bool) GetNUICameraDepthFrameRAW(CLNUICamera cam, PUSHORT pData, int waitTimeout = 2000); 」

    PUSHORT pData;

    is this get Depth Value, right ?

    the maximum ?? the minimum??

    my english is bad, so...

    i hope you know my question...:)

    Sunday, February 5, 2012 12:37 PM

Answers

  • Have you taken a look at the Skeleton Viewer sample in the SDK? This will show you how to get the depth frame.
    • Marked as answer by rEnEkToN Tuesday, February 7, 2012 2:54 AM
    • Unmarked as answer by rEnEkToN Tuesday, February 7, 2012 3:51 AM
    • Marked as answer by rEnEkToN Tuesday, February 7, 2012 3:56 AM
    Monday, February 6, 2012 4:47 PM
  • Your depth value returned by the SDK is a 13 bit number, stored in a 16 bit unsigned short variable.  If you want to convert that value to another color scale, then you just need to decide how you want to map the value to a color.  For example, if you want far away to be black, then you would need to do something like this:

    USHORT color = 4000 - depthValue;

    Then you need to map from the range of (0 to 4000) to something like (0 to 255).  So you rescale the values like this:

    float finalColor = color * 255.0f / 4000.0f;

    You can also do other mappings, such as offsetting the zero point, or zooming into a more restricted depth range to provide a better contrast between neighboring values.

    • Marked as answer by rEnEkToN Tuesday, February 7, 2012 2:53 AM
    • Unmarked as answer by rEnEkToN Tuesday, February 7, 2012 3:51 AM
    • Marked as answer by rEnEkToN Tuesday, February 7, 2012 3:56 AM
    Monday, February 6, 2012 5:27 PM
  • You are right - that was a type in my post.  It should actually be 8000, since that is the maximum distance that can be read out of the depth image.

    EDIT: no that isn't right - the depth values can range from 800 to 4000 mm.  If you are seeing values in the range up to 8000 then there is something not correct...

    • Edited by Jason Zink Tuesday, February 7, 2012 9:23 PM
    • Marked as answer by rEnEkToN Wednesday, February 8, 2012 2:47 AM
    Tuesday, February 7, 2012 6:30 AM
  • you are probably not using the complete depth range in your image, so why not find the min and max values of the depth image and use that to scale the color range?  For example, try this:

    float colorOutput = (rawDepth - minDepth) / (maxDepth - minDepth)

    That should give better contrast.  Notice again that if you want to flip the colors (i.e. lighter is closer) then you take 1.0 - colorOutput.  This value is then mapped to your color space, which you have already indicated is 0-255.

    • Marked as answer by rEnEkToN Wednesday, February 8, 2012 7:53 AM
    Wednesday, February 8, 2012 6:10 AM
  • What is the problem that you are still having?  The normalize function maps your depth values into a range from 0-1, which correlate to your specified min and max depths.  You just have to choose your color based on the values 0-1.  Is it not working, or do you want to change the image somehow, or something else?
    • Marked as answer by rEnEkToN Monday, February 13, 2012 2:53 AM
    Thursday, February 9, 2012 6:22 AM
  • The 13-bit number could theoretically support values up to 8192 - that is true.  However, the SDK explicitly says that the maximum depth value is 4000 mm.  If you use 8192 as your maximum value then you are just going to be missing out on extra precision for your visualization.

    I was looking at that other image you posted, and they are doing lighting calculations on the reconstructed geometry.  It looks like a form of Ambient Occlusion based on the depth buffer.  If you are interested in performing an implementation of this algorithm, you can take a look at my D3D11 engine here.

    • Marked as answer by rEnEkToN Monday, February 13, 2012 2:47 AM
    Thursday, February 9, 2012 9:45 PM
  • Jason is, of course, right on for the data that can be returned from "default" depth mode.  For Near Mode (supported on Kinect for Windows devices, not Xbox devices), the range is 400-3000.  The current range can be retrieved from the managed API in DepthImageStream.MinDepth/MaxDepth, and the values themselves are in NuiImageCamera.h:

    #define NUI_IMAGE_DEPTH_MAXIMUM               ((4000 << NUI_IMAGE_PLAYER_INDEX_SHIFT) | NUI_IMAGE_PLAYER_INDEX_MASK)
    #define NUI_IMAGE_DEPTH_MINIMUM               (800 << NUI_IMAGE_PLAYER_INDEX_SHIFT)
    #define NUI_IMAGE_DEPTH_MAXIMUM_NEAR_MODE     ((3000 << NUI_IMAGE_PLAYER_INDEX_SHIFT) | NUI_IMAGE_PLAYER_INDEX_MASK)
    #define NUI_IMAGE_DEPTH_MINIMUM_NEAR_MODE     (400 << NUI_IMAGE_PLAYER_INDEX_SHIFT)

    The "13th bit" of the depth is used to help express one of the reserved values, i.e. "Unknown depth":

    #define NUI_IMAGE_DEPTH_NO_VALUE              0
    #define NUI_IMAGE_DEPTH_TOO_FAR_VALUE         (0x0fff << NUI_IMAGE_PLAYER_INDEX_SHIFT)
    #define NUI_DEPTH_DEPTH_UNKNOWN_VALUE         (0x1fff << NUI_IMAGE_PLAYER_INDEX_SHIFT)


    -Adam Smith [MSFT]

    • Marked as answer by rEnEkToN Monday, February 13, 2012 2:47 AM
    Saturday, February 11, 2012 4:37 AM
  • The pixel value in the data that is returned from the APIs will contain a value in bits 3-15 which, when shifted right by NUI_IMAGE_PLAYER_INDEX_SHIFT, will be in the range 800-4000 or 0 or 0x0fff (4095) or 0x1fff (or -1 if you treat the value as signed), and which is directly interpreted as a depth value in mm from the sensor.

    -Adam Smith [MSFT]

    • Marked as answer by rEnEkToN Tuesday, February 14, 2012 1:17 PM
    Monday, February 13, 2012 10:46 PM

All replies

  • i ask question it's too easy ??
    i search so long time :(
    just know depth distance 0 ~ 10000 mm
    Monday, February 6, 2012 6:45 AM
  • this is RGB Frame↓

    but i want Gray Frame...
    what can i do

    Monday, February 6, 2012 6:56 AM
  • i have try
    PUSHORT pData;
    cvNormalize  255 ~ 0
    but the result is not correct ↓

    what can i do
    :|...
    Monday, February 6, 2012 7:01 AM
  • i want the result just like openNI's sample↓

    Monday, February 6, 2012 7:03 AM
  • Have you taken a look at the Skeleton Viewer sample in the SDK? This will show you how to get the depth frame.
    • Marked as answer by rEnEkToN Tuesday, February 7, 2012 2:54 AM
    • Unmarked as answer by rEnEkToN Tuesday, February 7, 2012 3:51 AM
    • Marked as answer by rEnEkToN Tuesday, February 7, 2012 3:56 AM
    Monday, February 6, 2012 4:47 PM
  • Your depth value returned by the SDK is a 13 bit number, stored in a 16 bit unsigned short variable.  If you want to convert that value to another color scale, then you just need to decide how you want to map the value to a color.  For example, if you want far away to be black, then you would need to do something like this:

    USHORT color = 4000 - depthValue;

    Then you need to map from the range of (0 to 4000) to something like (0 to 255).  So you rescale the values like this:

    float finalColor = color * 255.0f / 4000.0f;

    You can also do other mappings, such as offsetting the zero point, or zooming into a more restricted depth range to provide a better contrast between neighboring values.

    • Marked as answer by rEnEkToN Tuesday, February 7, 2012 2:53 AM
    • Unmarked as answer by rEnEkToN Tuesday, February 7, 2012 3:51 AM
    • Marked as answer by rEnEkToN Tuesday, February 7, 2012 3:56 AM
    Monday, February 6, 2012 5:27 PM
  • wow
    thank you very much~
    Tuesday, February 7, 2012 2:54 AM
  • wow
    thank you very much~
    Tuesday, February 7, 2012 2:54 AM
  • i have try that

    this is my code:

    void __fastcall TForm1::Paper11Click(TObject *Sender)
    {
            CLNUICamera KinectCamera = CreateNUICamera();
    
            if(!StartNUICamera(KinectCamera))
            {
                    ShowMessage("Not Found!!");
            }
            else
            {
                    IplImage *Img_DepthRGB,
                             *Img_DepthGray;
    
                    int Img_Height = 480,
                        Img_Width = 640;
    
                    PUSHORT pDepthRaw;
    
                    PDWORD datadepth = (PDWORD) malloc(Img_Width*Img_Height*4);
    
    
                    Img_DepthRGB = cvCreateImage(cvSize(Img_Width,Img_Height), 8U, 4);
                    Img_DepthGray = cvCreateImage(cvSize(Img_Width,Img_Height), 8U, 1);
    
    
                    pDepthRaw = new USHORT[Img_Width*Img_Height];
    
                    cvNamedWindow("Depth_RGB",1);
                    cvNamedWindow("Depth_Gray",1);
    
                    while(1)
                    {
                            //====宣告欄===
                            int Img_Length = 0;
    
                            CvMat src,
                                  dst;
    
                            //unsigned short
                            unsigned short *Depth_Value = new unsigned short [Img_Width*Img_Height],
                                  *data;
    
                            CvScalar Pixel_Value_GRAY,
                                     Pixel_Value_RGB;
                            //=============
    
                            GetNUICameraDepthFrameRGB32(KinectCamera, datadepth, 0);
    
                            GetNUICameraDepthFrameRAW(KinectCamera, pDepthRaw, 0);
    
                            //指派用戶數據給數組頭
                            cvSetData(Img_DepthRGB, datadepth, Img_DepthRGB->widthStep);
    
                            for(int i=0;i<Img_Width*Img_Height;i++)
                            {
                                    Depth_Value[i] = pDepthRaw[i];
                            }
    
                            src = cvMat(Img_Height, Img_Width,CV_16UC1,Depth_Value);
    
                            dst = cvMat(Img_Height, Img_Width,CV_16UC1,Depth_Value);
    
                            //cvNormalize(&src,&dst,255,0,CV_MINMAX,NULL);//CV_MINMAX
    
                            data = dst.data.us;
    
                            for(int i=0;i<Img_Height;i++)
                            {
                                    for(int j=0;j<Img_Width;j++)
                                    {                                                     
                                            Pixel_Value_GRAY = cvGet2D(Img_DepthGray,i,j);
    
                                            Pixel_Value_GRAY.val[0] = (255-((4000-data[i*Img_Width+j])*255/4000));
                                            
                                            cvSet2D(Img_DepthGray,i,j,Pixel_Value_GRAY);   
                                    }
                            }
    
                            cvShowImage( "Depth_RGB", Img_DepthRGB);
                            cvShowImage( "Depth_Gray", Img_DepthGray);
    
                            if( cvWaitKey(1) == 'q')
                            {
                                    cvSaveImage("Img_DepthRGB.jpg",Img_DepthRGB);
                                    cvSaveImage("Img_DepthGray.jpg",Img_DepthGray);
                                    cvWaitKey(1);
                                    break;
                            }
    
                            delete []Depth_Value;
    
                            //ShowMessage(AnsiString("END"));
                            //break;
                    }
    
                    StopNUICamera(KinectCamera);
                    cvDestroyWindow("Depth_RGB");
                    cvDestroyWindow("Depth_Gray");
            }
    }

    the result ↓

    Tuesday, February 7, 2012 3:50 AM
  • Thanks for your reply. I still have some questions.

    You said that each depth value is recorded by using a 13-bit number in SDK.  My questions are as follows:

    1. In SDK, white and black mean the farest distance and nearest distance in a depth map, respectively. right? 

    2. If the answer to the above question is "YES" and we want to invert the depth map (i.e, black = shorest, white = farest), the transfer function seems to be

       USHORT color = 8191 - depthValue;

    Could you explain why 4000 is used in your transfer function (i.e., USHORT color = 4000 - depthValue;) ?

    Tuesday, February 7, 2012 5:56 AM
  • You are right - that was a type in my post.  It should actually be 8000, since that is the maximum distance that can be read out of the depth image.

    EDIT: no that isn't right - the depth values can range from 800 to 4000 mm.  If you are seeing values in the range up to 8000 then there is something not correct...

    • Edited by Jason Zink Tuesday, February 7, 2012 9:23 PM
    • Marked as answer by rEnEkToN Wednesday, February 8, 2012 2:47 AM
    Tuesday, February 7, 2012 6:30 AM
  • Thank you my Savior~~ :D
    So how can i correct my code :|
    result just like this
    • Edited by rEnEkToN Wednesday, February 8, 2012 4:11 AM
    Wednesday, February 8, 2012 2:46 AM
  • you are probably not using the complete depth range in your image, so why not find the min and max values of the depth image and use that to scale the color range?  For example, try this:

    float colorOutput = (rawDepth - minDepth) / (maxDepth - minDepth)

    That should give better contrast.  Notice again that if you want to flip the colors (i.e. lighter is closer) then you take 1.0 - colorOutput.  This value is then mapped to your color space, which you have already indicated is 0-255.

    • Marked as answer by rEnEkToN Wednesday, February 8, 2012 7:53 AM
    Wednesday, February 8, 2012 6:10 AM
  • thank you :D
    i try find the maximum and minimum method
    code↓↓↓

    while(1)
                    {
                            //====宣告欄===
                            int Img_Length = 0;
    
                            CvMat src,
                                      dst;
    
                            //unsigned short
                            unsigned short *Depth_Value = new unsigned short [Img_Width*Img_Height],
                                                     *data,
                                                     Maximum_Value=0,
                                                     Minimum_Value=0;
    
                            CvScalar Pixel_Value_GRAY,
                                          Pixel_Value_RGB;
                            //=============
    
                            GetNUICameraDepthFrameRGB32(KinectCamera, datadepth, 0);
    
                            GetNUICameraDepthFrameRAW(KinectCamera, pDepthRaw, 0);
    
                            //指派用戶數據給數組頭
                            cvSetData(Img_DepthRGB, datadepth, Img_DepthRGB->widthStep);
    
                            Maximum_Value = pDepthRaw[0];
                            Minimum_Value = pDepthRaw[0];
    
                            for(int i=0;i<Img_Width*Img_Height;i++)
                            {
                                    Depth_Value[i] = pDepthRaw[i];
                                    if(Maximum_Value < pDepthRaw[i])
                                    {
                                            Maximum_Value = pDepthRaw[i];
                                    }
    
                                    if(Minimum_Value > pDepthRaw[i])
                                    {
                                            Minimum_Value = pDepthRaw[i];
                                    }
                            }
                            //ShowMessage(AnsiString("Maximum_Value")+AnsiString(Maximum_Value));
                            //ShowMessage(AnsiString("Minimum_Value")+AnsiString(Minimum_Value));
    
    
                            src = cvMat(Img_Height, Img_Width,CV_16UC1,Depth_Value);//CV_64FC1
    
                            dst = cvMat(Img_Height, Img_Width,CV_16UC1,Depth_Value);//CV_64FC1
    
                            //cvNormalize(&src,&dst,255,0,CV_MINMAX,NULL);//CV_MINMAX
    
                            data = dst.data.us;
    
                            for(int i=0;i<Img_Height;i++)
                            {
                                    for(int j=0;j<Img_Width;j++)
                                    {
                                            Pixel_Value_GRAY = cvGet2D(Img_DepthGray,i,j);
    
                                            Pixel_Value_GRAY.val[0] = (1-((data[i*Img_Width+j]-Minimum_Value)/
                                                                      (Maximum_Value-Minimum_Value)))*255 ;
    
                                            cvSet2D(Img_DepthGray,i,j,Pixel_Value_GRAY);
                                    }
                            }
    
                            cvShowImage( "Depth_RGB", Img_DepthRGB);
                            cvShowImage( "Depth_Gray", Img_DepthGray);
    
                            if( cvWaitKey(1) == 'q')
                            {
                                    cvSaveImage("Img_DepthRGB.jpg",Img_DepthRGB);
                                    cvSaveImage("Img_DepthGray.jpg",Img_DepthGray);
                                    cvWaitKey(1);
                                    break;
                            }
    
                            delete []Depth_Value;
    
                            //ShowMessage(AnsiString("END"));
                            //break;
                    }

    i try two pic
    1.

    Maximum: 2047
    Minimum: 472

    2.

    Maximum: 2047
    Minimum: 765

    Wednesday, February 8, 2012 7:51 AM
  • the result :(((((( ↓

    Wednesday, February 8, 2012 7:52 AM
  • i correct float
    the result will be better
    Wednesday, February 8, 2012 8:17 AM
  • this formula :
    float colorOutput = (rawDepth - minDepth) / (maxDepth - minDepth)
    is equal cvNormalize ??
    @.@...


    • Edited by rEnEkToN Thursday, February 9, 2012 4:48 AM
    Wednesday, February 8, 2012 8:21 AM
  • what can i do ?? :(((
    Thursday, February 9, 2012 4:49 AM
  • What is the problem that you are still having?  The normalize function maps your depth values into a range from 0-1, which correlate to your specified min and max depths.  You just have to choose your color based on the values 0-1.  Is it not working, or do you want to change the image somehow, or something else?
    • Marked as answer by rEnEkToN Monday, February 13, 2012 2:53 AM
    Thursday, February 9, 2012 6:22 AM
  • You said "the depth values can range from 800 to 4000 mm". you mean that the distance about depth is from 800 to 4000 mm. right?

    Actually, in the depth map, the depth distance between Kinect and an object should be transfered to an intensity value.  For 13-bit format, the range of intensity value about depth is from 0 to 8191.   Could you explain why 4000 is used in your transfer function (i.e., USHORT color = 4000 - depthValue;) ? I think the transfer function should be modified as 

    USHORT color = 8191 - depthValue;

    Thursday, February 9, 2012 8:37 AM
  • Thanks for you kind response.

    I have a question. Why is the result generated by using NUI different to that generated by OpenNI?  It seems that it is easy to see objects with different depths in the result generated by using NUI. Compared with the result generated by using OpenNI, my result seems not correct. Do you know the reasons?

    OpenNI


    NUI

    Thursday, February 9, 2012 8:49 AM
  • The 13-bit number could theoretically support values up to 8192 - that is true.  However, the SDK explicitly says that the maximum depth value is 4000 mm.  If you use 8192 as your maximum value then you are just going to be missing out on extra precision for your visualization.

    I was looking at that other image you posted, and they are doing lighting calculations on the reconstructed geometry.  It looks like a form of Ambient Occlusion based on the depth buffer.  If you are interested in performing an implementation of this algorithm, you can take a look at my D3D11 engine here.

    • Marked as answer by rEnEkToN Monday, February 13, 2012 2:47 AM
    Thursday, February 9, 2012 9:45 PM
  • Jason is, of course, right on for the data that can be returned from "default" depth mode.  For Near Mode (supported on Kinect for Windows devices, not Xbox devices), the range is 400-3000.  The current range can be retrieved from the managed API in DepthImageStream.MinDepth/MaxDepth, and the values themselves are in NuiImageCamera.h:

    #define NUI_IMAGE_DEPTH_MAXIMUM               ((4000 << NUI_IMAGE_PLAYER_INDEX_SHIFT) | NUI_IMAGE_PLAYER_INDEX_MASK)
    #define NUI_IMAGE_DEPTH_MINIMUM               (800 << NUI_IMAGE_PLAYER_INDEX_SHIFT)
    #define NUI_IMAGE_DEPTH_MAXIMUM_NEAR_MODE     ((3000 << NUI_IMAGE_PLAYER_INDEX_SHIFT) | NUI_IMAGE_PLAYER_INDEX_MASK)
    #define NUI_IMAGE_DEPTH_MINIMUM_NEAR_MODE     (400 << NUI_IMAGE_PLAYER_INDEX_SHIFT)

    The "13th bit" of the depth is used to help express one of the reserved values, i.e. "Unknown depth":

    #define NUI_IMAGE_DEPTH_NO_VALUE              0
    #define NUI_IMAGE_DEPTH_TOO_FAR_VALUE         (0x0fff << NUI_IMAGE_PLAYER_INDEX_SHIFT)
    #define NUI_DEPTH_DEPTH_UNKNOWN_VALUE         (0x1fff << NUI_IMAGE_PLAYER_INDEX_SHIFT)


    -Adam Smith [MSFT]

    • Marked as answer by rEnEkToN Monday, February 13, 2012 2:47 AM
    Saturday, February 11, 2012 4:37 AM
  • Do you mean that the pixel value 4000 (i.e. an integer) in the depth map equals 4000 mm? That is, when the pixel value is one, the distance between kinect and an object is 1 mm. 

    I guess that  there is a transfer function between the pixel value (i.e. an integer) and the exact depth. 

     
    Monday, February 13, 2012 9:01 AM
  • The pixel value in the data that is returned from the APIs will contain a value in bits 3-15 which, when shifted right by NUI_IMAGE_PLAYER_INDEX_SHIFT, will be in the range 800-4000 or 0 or 0x0fff (4095) or 0x1fff (or -1 if you treat the value as signed), and which is directly interpreted as a depth value in mm from the sensor.

    -Adam Smith [MSFT]

    • Marked as answer by rEnEkToN Tuesday, February 14, 2012 1:17 PM
    Monday, February 13, 2012 10:46 PM
  • why in my PC can't find this file "NuiImageCamera.h" ??
    Tuesday, February 14, 2012 1:16 PM
  • thank you

    Jason Zink

    Adam Smith
    :))

    Tuesday, February 14, 2012 1:18 PM