none
Depth to color pixel mapping does not work anymore (used to work with the Beta SDK) RRS feed

  • Question

  • Hello,

    I wrote a simple test application (in C++) to black our all areas of the image that a not a player (according to the depth & player index frame). The result with the new NuiImageGetColorPixelCoordinatesFromDepthPixelAtResolution is an image that is shifted to the right. If you look carefully you see my fist where my head is supposed to be. The dotted points are expected since I am mapping the 320x240 depth frame to a 640x480 color frame. If a pixel does not have a mapping, I leave it black.

    This used to work fine with the Beta 2 SDK. The mapped depth pixel for a "player" were pretty well aligned with the corresponding color pixels of the player. With the 1.0 SDK they are completely off:

    If necessary I can post some code illustrating this on the Skeletal viewer example?

    -Florian

    Tuesday, February 7, 2012 9:55 PM

Answers

  • Initial note: the 6th parameter (the depth value) should not be shifted bia NuiDepthPixelToDepth - it should be passed "as is" to the function.  I appreciate that the documentation could be crisper here.

    While it will make no practical difference, it's probably more clear to have your first loops be from 0 to depthHeight and 0 to depthWidth rather than half the color dimensions.  This confused me at first, though they do end up being the same values for the resolutions you've chosen.

    >1) Does the mapping from depth to color pixel change once the drepth and color stream are open?

    I don't believe so, no.

    >2) What is the depth value (arg 6) passed to convertion function used for?

    It's a critical piece of the math - that is, the distance from the camera affects the amount of divergence between the two cameras.

    I hope that simply removing the NuiDepthPixelToDepth will address your issues.  If not, please post your updated sample and we can take another look.

    I might also recommend that you consider using your mask to affect a 2x2 region of pixels in color space - this will avoid your gaps and should work reasonably well.

    Alternately, you can of course run depth at 640x480, and there are other techniques that can be used as well.


    -Adam Smith [MS]

    • Marked as answer by F1orian Friday, February 10, 2012 1:42 AM
    Wednesday, February 8, 2012 1:55 AM

All replies

  • Yes - if you could post a code snippet, that'd be great.  There may very well be a bug here, but we have also had a number of successful color<->depth mapping samples as well.  Hopefully there's something simple we can tweak in your code.


    -Adam Smith [MS]

    Wednesday, February 8, 2012 12:37 AM
  • Hi,

    Below is the code snip.

    I have a few additional questions I could not clarify by reading the documentation / code samples.

    1) Does the mapping from depth to color pixel change once the drepth and color stream are open?

    2) What is the depth value (arg 6) passed to convertion function used for?

    Thanks,

    -Florian

    // snip ---- for each depth + player index frame I am updating a 'mask' 
    
            memset(m_pMask, 0, m_Width*m_Height*sizeof(unsigned char));
    
            USHORT* packedPixel = (USHORT*)frame;
            for (int y = 0; y < m_Height/2; ++y) {
                for (int x = 0; x < m_Width/2; ++x) {
    
                    LONG colorX = 0;
                    LONG colorY = 0;
                    m_pNuiSensor->NuiImageGetColorPixelCoordinatesFromDepthPixelAtResolution(
                         NUI_IMAGE_RESOLUTION_640x480,
                         NUI_IMAGE_RESOLUTION_320x240,
                         NULL,
                         x,
                         y,
                         NuiDepthPixelToDepth(*packedPixel),
                         &colorX,
                         &colorY
                    );
    
                    if (colorX >= 0 && colorX <= m_Width && colorY >= 0 && colorY <= m_Height) {
                        m_pMask[(m_Width*colorY)+colorX] = *packedPixel & NUI_IMAGE_PLAYER_INDEX_MASK ? 255 : 0;
                    }
    
                    ++packedPixel;
                }
            }
    
    // snip ---- for each color frame I combine the mask (created above) with the color frame. 
    //        If the mask value is 0, the result is back, if the mask value is 255, that color pixel remains unchanged
    // in this exmaple source is a black image, target is the color frame and mask is the mask we created above
    
    
        int* pTarget = target;
        const int* pSrc = source;
        const unsigned char* pMask = mask;
    
        for (int y = 0; y < m_Height; ++y) {
            for (int x = 0; x < m_Width; ++x) {
    
                if (*pMask == 0) {
                    *pTarget = *pSrc;
                }
    
                ++pTarget;
                ++pSrc;
                ++pMask;
            }
        }


    Wednesday, February 8, 2012 1:17 AM
  • Initial note: the 6th parameter (the depth value) should not be shifted bia NuiDepthPixelToDepth - it should be passed "as is" to the function.  I appreciate that the documentation could be crisper here.

    While it will make no practical difference, it's probably more clear to have your first loops be from 0 to depthHeight and 0 to depthWidth rather than half the color dimensions.  This confused me at first, though they do end up being the same values for the resolutions you've chosen.

    >1) Does the mapping from depth to color pixel change once the drepth and color stream are open?

    I don't believe so, no.

    >2) What is the depth value (arg 6) passed to convertion function used for?

    It's a critical piece of the math - that is, the distance from the camera affects the amount of divergence between the two cameras.

    I hope that simply removing the NuiDepthPixelToDepth will address your issues.  If not, please post your updated sample and we can take another look.

    I might also recommend that you consider using your mask to affect a 2x2 region of pixels in color space - this will avoid your gaps and should work reasonably well.

    Alternately, you can of course run depth at 640x480, and there are other techniques that can be used as well.


    -Adam Smith [MS]

    • Marked as answer by F1orian Friday, February 10, 2012 1:42 AM
    Wednesday, February 8, 2012 1:55 AM
  • Works great thanks!
    Friday, February 10, 2012 1:43 AM