none
Trouble with NuiImageGetColorPixelCoordinatesFromDepthPixel RRS feed

  • Question

  • Hi everyone,

    I am trying to get the color pixel coordinates that correspond to the depth pixels using this function but the results don't seem to match up.

    When saving the color and depth data, I noticed the color image is flipped horizontally compared to the depth data. Is this of any concern with this function?

    Also, the documentation states that the first parameter of this function is the resolution of the color image, but when I use that, the program crashes. Instead, if I enter the resolution of the depth image, I actually get data from the function.

    Can anyone explain how to use this function properly?

    Thursday, July 14, 2011 11:20 PM

Answers

  • I'm sorry for your troubles. The only resolution currently supported by this function is NUI_IMAGE_RESOLUTION_640x480 (or ImageResolution.Resolution640x480 in C#). Sorry about that. If your RGB image is in a different resolution than 640x480, you will need to scale values returned by NuiImageGetColorPixelCoordinatesFromDepthPixel in order to select correct pixel within your RGB image frame.

    You're also right that if you're using NUI_IMAGE_TYPE_DEPTH (as opposed to NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX) then the image will be flipped with respect to the NUI_IMAGE_TYPE_COLOR image. To deal with this, the value you pass in as lDepthX in call to NuiImageGetColorPixelCoordinatesFromDepthPixel should be (320-depthX), where depthX represents the x-coordinate of your depth image pixel. You use 320 because NuiImageGetColorPixelCoordinatesFromDepthPixel assumes a 320x240 depth image size. So, if your depth image has a resolution other than 320x240, you will also have to scale values before calling this function.

    It looks like many people are having problems with this, so I will paste some sample code. This code will be in C#, since it is a smaller amount of code than C++ equivalent. Code can be replaced into SkeletalViewer C# sample, assuming that initialization is changed to:

        nui.Initialize(RuntimeOptions.UseDepth | RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);
        nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);

    Also,

        ImageFrame lastColorFrame = null;

    was added to MainWindow class declarations and

        lastColorFrame = e.ImageFrame;

    was added to nui_ColorFrameReady event handler. The following code would replace implementation of convertDepthFrame in MainWindow.xaml.cs in sample:

    byte[] convertDepthFrame(byte[] depthFrame16)
    {
      for (int i16 = 0, i32 = 0; i16 < depthFrame16.Length && i32 < depthFrame32.Length; i16 += 2, i32 += 4)
      {
        int realDepth = (depthFrame16[i16 + 1] << 8) | (depthFrame16[i16]);
        bool showColor = (realDepth > 1000 && realDepth < 2000);
        byte intensity = (byte)(255 - (255 * realDepth / 0x0fff));
    
        depthFrame32[i32 + RED_IDX] = 0;
        depthFrame32[i32 + GREEN_IDX] = 0;
        depthFrame32[i32 + BLUE_IDX] = 0;
    
        // choose different display colors based on player
        if (showColor && lastColorFrame != null)
        {
          int depthX = (i16 / 2) % 320;
          int depthY = (i16 / 2) / 320;
          int colorX, colorY;
    
          nui.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(ImageResolution.Resolution640x480, new ImageViewArea(), 320 - depthX, depthY, (short)(realDepth << 3), out colorX, out colorY);
    
          int colorIdx = 4 * (colorY * 640 + colorX);
          byte[] colorPixels = lastColorFrame.Image.Bits;
    
          depthFrame32[i32 + RED_IDX] = colorPixels[colorIdx + RED_IDX];
          depthFrame32[i32 + GREEN_IDX] = colorPixels[colorIdx + GREEN_IDX];
          depthFrame32[i32 + BLUE_IDX] = colorPixels[colorIdx + BLUE_IDX];
        }
        else
        {
          depthFrame32[i32 + RED_IDX] = (byte)(intensity / 2);
          depthFrame32[i32 + GREEN_IDX] = (byte)(intensity / 2);
          depthFrame32[i32 + BLUE_IDX] = (byte)(intensity / 2);
        }
      }
      return depthFrame32;
    }

    Hope this helps,
    Eddy


    I'm here to help
    Friday, July 15, 2011 1:46 AM