none
Formula for space transformations RRS feed

  • Question

  • Hi, would you have any problems with making public the formulas for the transformations between color and depth and depth to skeleton? I would like to use the graphics processor to transform between color and depth because the SDK only seems to convert one point at a time which would be slow if I wanted to transform the entire image, and I just want to do a simple shader for that. Thanks.
    Friday, June 17, 2011 7:25 PM

Answers

  • Eric Rudolph (Dev on Kinect for PC team):

    It's not so simple. At the factory, the two cameras are aligned and the exact mapping between them is computed and burned into the Kinect's f/w. We read that (binary) blob out and do the mapping from one image space to another. On the xbox, this is done on the GPU. On the PC, we haven't decided what to do in the GPU space yet (since we can't demand everybody have the same graphics chip like on the xbox!), so initially, the mapping will probably be done by the CPU. in any case, we don't have a way to make public the details of that binary blob, so you'll have to wait, or be very creative. Hint: I imagine if you map a grid of points from one space to another, you can get a pretty good idea of the mapping function... then use the GPU...


    ~ Eric R., Senior Dev, MS Research You can probably figure out more about me if you try!
    Saturday, June 18, 2011 4:42 AM

All replies

  • Eric Rudolph (Dev on Kinect for PC team):

    It's not so simple. At the factory, the two cameras are aligned and the exact mapping between them is computed and burned into the Kinect's f/w. We read that (binary) blob out and do the mapping from one image space to another. On the xbox, this is done on the GPU. On the PC, we haven't decided what to do in the GPU space yet (since we can't demand everybody have the same graphics chip like on the xbox!), so initially, the mapping will probably be done by the CPU. in any case, we don't have a way to make public the details of that binary blob, so you'll have to wait, or be very creative. Hint: I imagine if you map a grid of points from one space to another, you can get a pretty good idea of the mapping function... then use the GPU...


    ~ Eric R., Senior Dev, MS Research You can probably figure out more about me if you try!
    Saturday, June 18, 2011 4:42 AM
  • That's a very interesting detail about the underlying mechanism.   Thank you for sharing it.

    I assume this means the mapping is fixed for a given device and doesn't need to be recalculated if you adjust the tilt, or during whatever magic the Kinect Tuner might perform?

    The only API exposed is from depth to color.  Given the depth resolution is lower, the reverse mapping requires interpolation, which if I have to do I'd like to cache ahead of time.

    I was rather surprised to see how very different the fields of view of my depth and color cameras are.   My depth information maps into the rectangle (23,33)-(621,484) in color image coordinates.   There's also a pronounced fish-eye sort of distortion.   Are those within the range of typical mappings?

    One more related question: The Camera.GetColorPixelCoordinatesFromDepthPixel() method takes a "depthValue" as well as the expected X & Y.   I get the same mapping passing all 16 bits, just the 13 bits of actual depth or a constant 0.   Could you provide some guidance as to what is expected?

    -Blake

    • Edited by Blake Coverett Saturday, June 18, 2011 12:11 PM One more Question
    Saturday, June 18, 2011 11:26 AM
  • Hi,

    that indeed were intersting details and findings about the mechanisms. I want to mask the RGB image with the depth map and thus need a reliable (and fast) transform for this...

     

    Best Regards,

    Christoph

    Monday, June 20, 2011 10:51 AM
  • By the way, the mapping range sounds realistic (but not being an actual rectangle because of the fisheye effect).

    However, I can't confirm that the "depthValue" parameter for the Camera.GetColorPixelCoordinatesFromDepthPixel() function is insignificant: when passing different values, I get very different mappings.

     

    Christoph

    Monday, June 20, 2011 11:41 AM