locked
Point Cloud / Depth stream RRS feed

  • Question

  •  
    I am having issues getting the depth stream to look correct.  I am coding using Win forms and coding for fun extensions to display the depth image in a picture box.  The issue is I get repeating lines as the image gets deeper as in attached picture.  I do not think it is in has anything to do with using Coding for fun I also see this when I try to create a point cloud using the below code.  The below code also has a scaling/depth issue.  Using the below code to create a point cloud creates rows of points in the Y direction with an incorrect depth.  I anyone help with the display and/or the point cloud?
     
     
     
     
            Dim res As New List(Of ColorVector3)
            Dim depHeight As Integer = _Depth.Image.Height
            Dim depWidth As Integer = _Depth.Image.Width
            Dim vidHeight As Integer = _Video.Image.Height
            Dim vidWidth As Integer = _Video.Image.Width
    
            If (vidHeight <> depHeight Or vidWidth <> depWidth) Then
                MessageBox.Show("Depth and video images are not the same")
                Exit Sub
            End If
    
            Dim depthData() As Byte = _Depth.Image.Bits
            Dim colorData() As Byte = _Video.Image.Bits
            Dim cnt As Integer
    
            For i As Integer = 0 To depthData.Length - 1 Step 2
                Dim depthPixel As Integer = (depthData(i + 1) << 8) Or depthData(i)
                Dim byteArray As Byte() = {depthPixel, depthPixel}
                Dim value2 As UInt16 = BitConverter.ToUInt16(byteArray, 0)
                depthPixel = BitConverter.ToInt16(BitConverter.GetBytes(value2 << 3), 0)
                Dim x As Integer = (i / 2) Mod depWidth
                Dim y As Integer = (i / 2) / depWidth
    
                Dim v As Vector = _kinectNui.SkeletonEngine.DepthImageToSkeleton(CSng(x / 640.0F), CSng(y / 480.0F), CShort(depthPixel))
    
                If (v.Z > 0) Then
                    Dim cv As ColorVector3 = New ColorVector3()
                    cv.X = v.X
                    cv.Y = v.Z
                    cv.Z = v.Y
                    If (withColor) Then
                        Dim colorX As Integer
                        Dim colorY As Integer
    
                        _kinectNui.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(_Video.Resolution, _Video.ViewArea, 320 - (x / 2), (y / 2), (depthPixel << 3), colorX, colorY)
                        colorX = Math.Max(0, Math.Min(vidWidth - 1, colorX))
                        colorY = Math.Max(0, Math.Min(vidHeight - 1, colorY))
                        Dim colIndex As Integer = 4 * (colorX + (colorY * vidWidth))
                        cv.B = (colorData(colIndex + 0))
                        cv.G = (colorData(colIndex + 1))
                        cv.R = (colorData(colIndex + 2))
                    Else
                        cv.B = 255
                        cv.G = 255
                        cv.R = 255
                    End If
                    res.Add(cv)
                End If
                'Next x
            Next i


    • Edited by JohnKes Saturday, January 28, 2012 3:42 PM
    Saturday, January 28, 2012 3:39 PM

All replies

  • Have you checked the produced values at two different points on this image?  For example, in the black area, is your loading algorithm producing the same value in each of the repeating bands?

    The reason that I ask is that it looks like your drawing is using a subset of the total number of bits that are available.  For example, it could be that you are only using the bottom 8 bits of the depth value somehow.  This could be due to using only an 8 bit render target, and the values are getting truncated when you write to it.

    I don't see anything immediately wrong with your code, so I'm not sure if it has to do with your calculations or the copying to the output image...

    Sunday, January 29, 2012 2:12 PM
  • The depth is not really right it could be because I am not exactly sure what the units are.  The below pictures show the depth image and the generated point cloud.  The picture is on me sitting, you can see in the right view of the point cloud it is not correct.  The point cloud was produced with the above code.  The depth image is being displayed in a picture box with the below code. 

    What are the units returned from DepthImageToSkeleton?

     

    Public Function DepthToBitmap(ByVal myImage As Microsoft.Research.Kinect.Nui.PlanarImage) As Bitmap
    
            For i = 0 To myImage.Bits.Length - 1 Step 2
                Dim this As Integer = (myImage.Bits(i + 1) << 8 Or myImage.Bits(i))
                this <<= 2
                myImage.Bits(i) = this And &HFF
                myImage.Bits(i + 1) = (this >> 8)
    
            Next i
            Dim bmap As New Bitmap(myImage.Width, myImage.Height, PixelFormat.Format16bppRgb555)
            ' bmap.Width = PImage.Width
    
            Dim bmapdata As BitmapData = bmap.LockBits(New Rectangle(0, 0, myImage.Width, myImage.Height), ImageLockMode.WriteOnly, bmap.PixelFormat)
            Dim ptr As IntPtr = bmapdata.Scan0
            Marshal.Copy(myImage.Bits, 0, ptr, myImage.Width * myImage.BytesPerPixel * myImage.Height)
            bmap.UnlockBits(bmapdata)
            Return bmap
        End Function

     


     
    • Edited by JohnKes Sunday, January 29, 2012 5:47 PM
    Sunday, January 29, 2012 5:44 PM
  • I think it it has somthing to do with the data types.  If I look at the sample code it is correct c# but when using vb it is not correct. 

    Does the below mean I do not need to get the lower 12 bits? 
     
    depthValue
    Type: Int16
    The depth value of the depth image pixel, in millimeters, shifted left by 3. The left-shift enables you to pass the value from the depth image directly into this function.

     

     Dim byteArray As Byte() = {depthData(i), depthData(i + 1)}
                Dim value2 As UShort = BitConverter.ToUInt16(byteArray, 0)
                Dim depthPixel As Integer = BitConverter.ToInt16(BitConverter.GetBytes(value2 << 3), 0)
    
    Dim v As Vector = _kinectNui.SkeletonEngine.DepthImageToSkeleton(CSng(x) / 640.0F, CSng(y) / 480.0F, (depthPixel))

     

     
    • Edited by JohnKes Monday, January 30, 2012 4:45 PM
    Sunday, January 29, 2012 9:31 PM
  • I think I see where the problem is.  In your code from the original sample, you have the following:

     Dim depthPixel As Integer = (depthData(i + 1) << 8) Or depthData(i)
     Dim byteArray As Byte() = {depthPixel, depthPixel}
    
    

    This constructs the full precision (13-bit) depth value in depthPixel, but then you copy that value into two array elements of type byte.  This truncates the data in depthPixel to 8 bits in both array elements, which effectively copies the lower 8-bits into both the lower and upper 8-bit section of the depth value.  If you need to use this array method, then I think it should look like this (warning: I am not vb.net literate!) :

    Dim byteArray As Byte() = {depthData(i + 1) << 8, depthData(i)}
    
    

    This way the byte array gets teh proper data in both upper and lower bytes.  Depending on how the byte array is used, you might have to swap those two elements...

    Monday, January 30, 2012 5:54 PM
  • I wish I could say that is it.  I caught that and have since changed the code to what is below.  From everything I have read it should work.  I am working with Windows forms and not WPF not sure if this has anything to do with it. 

    The _Depth.Image.Bits is created in DepthFrameReady event  _Depth = e.ImageFrame.

    Can anyone tell me what I should expect DepthPixel to equal in the below line.  I get 730is this correct?

    Dim depthPixel As Int16 = CInt((2 << 8)) Or CInt

    (220)

     

     

    What about after the shift?  I get 5840
    depthPixel = depthPixel << 3
    

     


     
            Dim depthData() As Byte = _Depth.Image.Bits
            Dim colorData() As Byte = _Video.Image.Bits
            For i As Integer = 0 To depthData.Length - 1 Step 2
                'Dim depthPixel As Int16 = CInt((depthData(i + 1) * 255)) + CInt(depthData(i))
                'depthPixel = depthPixel << 3
                Dim depthPixel As Int16 = CInt((depthData(i + 1) << 8)) Or CInt(depthData(i))
                'depthPixel = depthPixel << 3
                Dim x As Integer = (i / 2) Mod depWidth
                Dim y As Integer = (i / 2) / depWidth
    
                Dim v As Vector = _kinectNui.SkeletonEngine.DepthImageToSkeleton(CSng(x) / 640.0F, CSng(y) / 480.0F, CShort(depthPixel) << 3)
    
                If (v.Z > 0) Then
                    Dim cv As ColorVector3 = New ColorVector3()
                    cv.X = v.X
                    cv.Y = v.Z
                    cv.Z = v.Y
                    If (withColor) Then
                        Dim colorX As Integer
                        Dim colorY As Integer
    
                        _kinectNui.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(_Video.Resolution, _Video.ViewArea, 320 - (x / 2), (y / 2), (depthPixel << 3), colorX, colorY)
                        colorX = Math.Max(0, Math.Min(vidWidth - 1, colorX))
                        colorY = Math.Max(0, Math.Min(vidHeight - 1, colorY))
                        Dim colIndex As Integer = 4 * (colorX + (colorY * vidWidth))
                        cv.B = (colorData(colIndex + 0))
                        cv.G = (colorData(colIndex + 1))
                        cv.R = (colorData(colIndex + 2))
                    Else
                        cv.B = 255
                        cv.G = 255
                        cv.R = 255
                    End If
                    res.Add(cv)
                End If
                'Next x
            Next i
    


     

    Here is the cod that is being used to display the depth image in a picture box.  I not sure why but if I change this code the point cloud also changes.  I think if the below can be corrected the point cloud will also be correct.

     

    Public Function DepthToBitmap(ByVal myImage As Microsoft.Research.Kinect.Nui.PlanarImage) As Bitmap
    
            For i = 0 To myImage.Bits.Length - 1 Step 2
                Dim this As Integer = CInt((myImage.Bits(i + 1) << 8) Or myImage.Bits(i))
                this <<= 2
                myImage.Bits(i) = this And &HFF
                myImage.Bits(i + 1) = this >> 8
            Next i
          
            Dim bmap As New Bitmap(myImage.Width, myImage.Height, PixelFormat.Format16bppRgb555)
            Dim bmapdata As BitmapData = bmap.LockBits(New Rectangle(0, 0, myImage.Width, myImage.Height), ImageLockMode.WriteOnly, bmap.PixelFormat)
            Dim ptr As IntPtr = bmapdata.Scan0
            Marshal.Copy(myImage.Bits, 0, ptr, myImage.Width * myImage.BytesPerPixel * myImage.Height)
            bmap.UnlockBits(bmapdata)
            Return bmap
        End Function







    • Edited by JohnKes Tuesday, January 31, 2012 2:31 AM
    Tuesday, January 31, 2012 1:15 AM
  • No, 2 << 8 should be 10 0000 0000 in binary, which is 512 in decimal.  That is probably where the issue is being introduced (or at least part of it).  Try a different way to convert from bytes of data to a 16 bit integer...

    Is your depth information configured to include the player index, or is it purely the depth data?

    Tuesday, January 31, 2012 6:39 AM
  • I am using Depth only.  I will try your suggestion a new way to convert.  I have tried 2*255 + 220 .  I think it had the same result and will try again.

    I see in my post above the 220 got placed on the line below 2 << 8 or 220 = ???  732  ???


    • Edited by JohnKes Tuesday, January 31, 2012 3:00 PM
    Tuesday, January 31, 2012 12:03 PM
  • I'm still trying to understand what is going on with the VB.net data types - can you post how many bits there are in each of these types:

    CInt, CShort, Int16, Byte, and Integer

    It seems that all of these types are being used, but I don't know if there is truncation happening between the individual conversions.  For example, depthPixel is declared as an Int16 but is then filled with a bitwise or combination of two CInt's.  Then it is later converted to a CShort.  I would suggest to generate it in one data type, and keep it in that data type for all of the calculations and uses.

    Tuesday, January 31, 2012 8:46 PM
  • I think I should have everything as Int16 and have tried this.  The DepthImageToSkeleton function documents say the depth should be Short.  I think you are correct that somthing is happening in the conversions of the data types.  I have tried what should be correct and also tried combinations and things to change based on the data type being used.  What do you suggest to be the correct one?  Is the DepthImageToSkeleton need to be signed or unsigned?

     

    Byte- 8 bit unsigned integer

     Int16 - 16 bit signed integer

    Integer - 32 bit signed integer

    Cint – Convert to Integer

    Short - Holds signed 16-bit (2-byte) integers ranging in value from -32,768 through 32,767

    CShort – Convert to Short

     

    Wednesday, February 1, 2012 1:35 AM
  • Hi, I know it's probably too late and you don't need it now, but in case it helps someone else,

    for the display thing, try changing the endianness of the ints

    + I think that to get the correct depth data, you do a right shift by 3 to get rid of the player bits and not a left one. Sorry if I misunderstood what you're doing, I read the posts rather quickly. Max value that should be returned for depth should be 4095
    • Edited by chrych Tuesday, April 10, 2012 5:19 PM added info
    Tuesday, April 10, 2012 5:09 PM