none
Color in 3D volume using Kinect Fusion toolkit RRS feed

  • Question

  • Is there a method to incorporate the color from the Color camera into the 3D volume created by INuiFusion functions? For example, how can the matrix pWorldToBGRTransform be used in the NuiFusionShadePointCloud method to produce colored surfaces instead of gray ones in the ShadedSurface image?

    Ultimately, I would like to be able to export a mesh with a color associated to each triangle. Do you have any idea how to produce such an output using the Color camera of the Kinect?

    Thursday, March 28, 2013 7:45 PM

Answers

  • John from  Kinect for Windows development team says :

    "Kinect Fusion has no built-in support for rendering a texture-mapped image.

    However, you can read the data out and then apply your own texture map to it. For example, INuiFusionReconstruction::CalculatePointCloud will give you a point cloud consisting of every visible pixel in the reconstruction model, as viewed from a specified camera pose."

    • Marked as answer by wxy123 Friday, April 19, 2013 3:11 AM
    Monday, April 1, 2013 2:36 AM

All replies

  • John from  Kinect for Windows development team says :

    "Kinect Fusion has no built-in support for rendering a texture-mapped image.

    However, you can read the data out and then apply your own texture map to it. For example, INuiFusionReconstruction::CalculatePointCloud will give you a point cloud consisting of every visible pixel in the reconstruction model, as viewed from a specified camera pose."

    • Marked as answer by wxy123 Friday, April 19, 2013 3:11 AM
    Monday, April 1, 2013 2:36 AM
  • Thank you John. I understand the CalculatePointCloud function now but I wanted to make sure that there was no way to add color to the Fusion volume based on information gathered by the Color camera of the Kinect.
    Monday, April 1, 2013 5:40 PM
  • Hy "wxy123"!

    Could you please post some of your ideas regarding the CalculatePointCloud function? I still don't really now how to use is. Some snippets would be more than helpful for me. What I need is the additional color information in the saved .obj file. Did you managed that?

    Thanks


    • Edited by Patrick_Ba Friday, April 12, 2013 10:28 AM
    Friday, April 12, 2013 9:29 AM
  • Hi Patrick_Ba,

    The CalculatePointCloud function only gives you a cloud of point as seen from a given camera pose (no color). As far as I can tell based on the response from John above, once you have that cloud of points, the Kinect SDK does not provide any way to color this cloud of point except for the function NuiFusionShadePointCloud. The way I understand it, that function will either give you a shaded gray-scaled image of the cloud of points or an image of the cloud of point where each pixel is colored according to the X,Y,Z  values of the Voxel where that pixel comes from. If you want to apply other color information to the cloud of points, you are on your own.

    Saturday, April 13, 2013 2:01 PM
  • Ok, that means I can save this colored image for example to the disk, and also the gray image. that's a first step. I also know the camera parameter. But what I really would like to have is to produce a text file like the .obj file with the additional color information. e.g.: x y z 255 255 255 (r g b)

    hmm.... i am computer graphics beginner. does someone have some hints? I don't understand the gap between the rgb image ( i guess 648x480) and the thousands of points I have in the exported mesh file. is it even possible to produce the information? 

    Sunday, April 14, 2013 2:08 PM
  • the exported mesh file is the full volume in 3D. The images obtained with the CalculatePointCloud and NuiFusionShadePointCloud functions are only 2D images of the volume from a given point of view of the camera. There is no direct link between the images and the volume (and therefore the mesh file). That was my question from the beginning. There is no function in the Kinect SKD to color the points of the volume (and of the mesh file). You can only see what it would look like from a camera point of view in shaded-gray or pseudo colors. To color the points of the volume to produce a color mesh file, you have to acquire your own images and treat them to produce some kind of texture to be applied to the volume. That is something out of my league and there is no support in the SDK. You would have to get help probably from outside this forum. Let me know if you find something.

    Sunday, April 14, 2013 3:44 PM
  • Hy!

    Thank you for answering. Hmmm... I tried to find some information in the web. Some "forums" said that UV mapping could work here. I don't know... I am not really happy with that, because I want to apply the mapping to the cloud and not to meshes. 

    I will continue with some research :-(

    Tuesday, April 16, 2013 9:08 AM
  • Do you really need the Fusion 3D volume reconstruction? Just in case you don't know, if you are just trying to get a cloud of points with colors, you can do it with just the depth and color streams.

    Wednesday, April 17, 2013 5:11 PM
  • I stopped the texturing now... . And yes, now I try to get a cloud from the kinect without the Fusion Application, and "register" rgb to the depth stream. It's the first time I use Kinect, but I tink that shouldnt be a big problem... Hopefully :-)
    Friday, April 19, 2013 2:17 PM