none
How to compare two depth frame data? RRS feed

  • Question

  • Hi All,

            I am trying to save two depth data in 1 s and then to compare them. If there is any difference, it will trigger some event. I am wondering what is the best way to do it. My first thought is to convert the 'byte' to 'array', and then do the subtraction between them. Any suggestion will be appreciated. Here is my code:

            int i =0;
            void newSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
            {
                
                using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
                {
                    if (depthFrame == null)
                    {
                        return;
                    }
                    byte[] pixels = GenerateColoredBytes(depthFrame);
                    byte senar1;
                    byte senar2=null;

                    if (i == 0)
                    {
                        senar1 = pixels;
                    }
                    else if (i == 30)
                    {
                        senar2 = pixels;
                    }
                    i++;
                   // if senar1 is different with senar2

                   // some action will be triggered;
                    int stride = depthFrame.Width * 4;
                    image1.Source =
                        BitmapSource.Create(depthFrame.Width, depthFrame.Height,
                        96, 96, PixelFormats.Bgr32, null, pixels, stride);

                }
            }

    Thursday, May 10, 2012 11:44 PM

All replies

  • I am afraid if I convert to 'array', there may be information lose and waste the cpu.
    • Edited by ZDsam Thursday, May 10, 2012 11:51 PM
    Thursday, May 10, 2012 11:45 PM
  • I'm not quite sure what your GenerateColoredBytes function is doing; you didn't include its implementation.

    You could compare the raw pixel data from two depth frames:

    if (i == 0)
    {
        if (this.array1 == null) this.array1 = new byte[depthFrame.PixelDataLength];
        depthFrame.CopyPixelDataTo(this.array1);
    }
    else if (i == 30)
    {
        if (this.array2 == null) this.array2 = new byte[depthFrame.PixelDataLength];
        depthFrame.CopyPixelDataTo(this.array2);

        for (int i = 0; i < this.array2.Length; i++)
        {
            if (array1[i] != array2[i])
            {
                // DIFFERENT!
                break;
            }
        }
    }

    However, due to noise inherent in the depth data, at least some pixels will almost certainly be different every time.

    To account for noise, you'll probably want to allow for some variation in individual pixels, and only consider the frames "different" if more than a certain percentage of pixels have changed by a significant amount. Here's a simple example:

       
        int perPixelThreshold = 50 << DepthImageFrame.PlayerIndexBitmaskWidth;  // 50 mm
        int percentPixelLimit = 5;
        int changedPixelCount = 0;
       
        for (int i = 0; i < this.array2.Length; i++)
        {
            if (Math.Abs(array1[i] - array2[i]) > perPixelThreshold)
            {
                ++changedPixelCount;
            }
        }

        if (changedPixelCount * 100 / depthFrame.PixelDataLength >= percentPixelLimit)
        {
            // DIFFERENT!
        }

    This will flag the frames as different if at least 5% of the pixels differ by more than 50 mm.

    Your mileage may vary... you may need to tweak the values of perPixelThreshold and percentPixelLimit, depending on your needs.

    John
    K4W Dev

    Friday, May 11, 2012 6:16 PM
  • Thanks John.
    Friday, May 11, 2012 9:21 PM
  • Hi John,

         I try your code, but I found some problem coming up: 

        1. there is no extension for array1 and array 2, did I missing a reference?

         2. The pixels in my function are the color pixels, so I think compare method should be different, right?

        And my GenerateColoredBytes(depthFrame) function is defined as :

        private byte[] GenerateColoredBytes(DepthImageFrame depthFrame)
            {

                //get the raw data from kinect with the depth for every pixel
                short[] rawDepthData = new short[depthFrame.PixelDataLength];
                depthFrame.CopyPixelDataTo(rawDepthData);

                //use depthFrame to create the image to display on-screen
                //depthFrame contains color information for all pixels in image
                //Height x Width x 4 (Red, Green, Blue, empty byte)
                Byte[] pixels = new byte[depthFrame.Height * depthFrame.Width * 4];

                //Bgr32  - Blue, Green, Red, empty byte
                //Bgra32 - Blue, Green, Red, transparency
                //You must set transparency for Bgra as .NET defaults a byte to 0 = fully transparent

                //hardcoded locations to Blue, Green, Red (BGR) index positions       
                const int BlueIndex = 0;
                const int GreenIndex = 1;
                const int RedIndex = 2;

                
                //loop through all distances
                //pick a RGB color based on distance
                for (int depthIndex = 0, colorIndex = 0;
                    depthIndex < rawDepthData.Length && colorIndex < pixels.Length;
                    depthIndex++, colorIndex += 4)
                {
                    //get the player (requires skeleton tracking enabled for values)
                    int player = rawDepthData[depthIndex] & DepthImageFrame.PlayerIndexBitmask;

                    //gets the depth value
                    int depth = rawDepthData[depthIndex] >> DepthImageFrame.PlayerIndexBitmaskWidth;

                    //.9M or 2.95'
                    if (depth <= 900)
                    {
                        //we are very close
                        pixels[colorIndex + BlueIndex] = 0;
                        pixels[colorIndex + GreenIndex] = 0;
                        pixels[colorIndex + RedIndex] = 0;
              

                    }
                    // .9M - 2M or 2.95' - 6.56'
                    else if (depth > 900 && depth < 2000)
                    {
                        //we are a bit further away
                        pixels[colorIndex + BlueIndex] = 0;
                        pixels[colorIndex + GreenIndex] = 255;
                        pixels[colorIndex + RedIndex] = 0;
                    }
                    // 2M+ or 6.56'+
                    else if (depth > 2000)
                    {
                        //we are the farthest
                        pixels[colorIndex + BlueIndex] = 0;
                        pixels[colorIndex + GreenIndex] = 0;
                        pixels[colorIndex + RedIndex] = 0;
                    }


                    ////equal coloring for monochromatic histogram
                    //byte intensity = CalculateIntensityFromDepth(depth);
                    //pixels[colorIndex + BlueIndex] = intensity;
                    //pixels[colorIndex + GreenIndex] = intensity;
                    //pixels[colorIndex + RedIndex] = intensity;


                    ////Color all players "gold"
                    //if (player > 0)
                    //{
                    //    pixels[colorIndex + BlueIndex] = Colors.Gold.B;
                    //    pixels[colorIndex + GreenIndex] = Colors.Gold.G;
                    //    pixels[colorIndex + RedIndex] = Colors.Gold.R;
                    //}

                }
              

                return pixels;
            }

           I am new for this, thank you for your help.


    • Edited by ZDsam Friday, May 11, 2012 9:38 PM
    Friday, May 11, 2012 9:34 PM
  • I'm not quite sure what your GenerateColoredBytes function is doing; you didn't include its implementation.

    You could compare the raw pixel data from two depth frames:

    if (i == 0)
    {
        if (this.array1 == null) this.array1 = new byte[depthFrame.PixelDataLength];
        depthFrame.CopyPixelDataTo(this.array1);
    }
    else if (i == 30)
    {
        if (this.array2 == null) this.array2 = new byte[depthFrame.PixelDataLength];
        depthFrame.CopyPixelDataTo(this.array2);

        for (int i = 0; i < this.array2.Length; i++)
        {
            if (array1[i] != array2[i])
            {
                // DIFFERENT!
                break;
            }
        }
    }

    However, due to noise inherent in the depth data, at least some pixels will almost certainly be different every time.

    To account for noise, you'll probably want to allow for some variation in individual pixels, and only consider the frames "different" if more than a certain percentage of pixels have changed by a significant amount. Here's a simple example:

       
        int perPixelThreshold = 50 << DepthImageFrame.PlayerIndexBitmaskWidth;  // 50 mm
        int percentPixelLimit = 5;
        int changedPixelCount = 0;
       
        for (int i = 0; i < this.array2.Length; i++)
        {
            if (Math.Abs(array1[i] - array2[i]) > perPixelThreshold)
            {
                ++changedPixelCount;
            }
        }

        if (changedPixelCount * 100 / depthFrame.PixelDataLength >= percentPixelLimit)
        {
            // DIFFERENT!
        }

    This will flag the frames as different if at least 5% of the pixels differ by more than 50 mm.

    Your mileage may vary... you may need to tweak the values of perPixelThreshold and percentPixelLimit, depending on your needs.

    John
    K4W Dev

          I modify the code, right now it can compare the two array. But right now the problem is even the there is nothing change in the enviornment, the message would still show up. Any suggestions? Here is my code:

         short[] array1 = new short[depthFrame.PixelDataLength];
                    short[] array2 = new short[depthFrame.PixelDataLength];
                    if (i == 0)
                    {

                        depthFrame.CopyPixelDataTo(array1);
                    }
                    else if (i == 60)
                    {

                        depthFrame.CopyPixelDataTo(array2);
                        i = 0;
                    }
                    i++;
                    int perPixelThreshold = 1000 << DepthImageFrame.PlayerIndexBitmaskWidth;  // 50 mm
                    int percentPixelLimit = 60;
                    int changedPixelCount = 0;
                    if (array1 != null && array2 != null)
                    {
                        for (int j = 0; j < array2.Length; j++)
                        {
                            if (Math.Abs(array1[j] - array2[j]) > perPixelThreshold)
                            {
                                ++changedPixelCount;
                            }
                        }if (changedPixelCount * 100 / depthFrame.PixelDataLength >= percentPixelLimit)
                    {
                        if (IsGetUpActive)
                        {
                            IsGetUpActive = false;
                            MessageBoxResult dres;
                            dres = MessageBox.Show("Attention! ", "Warning", MessageBoxButton.OK);
                            if (dres == MessageBoxResult.OK)
                            {
                                IsGetUpActive = true;
                            }     
                        }

                    }

                   

    Friday, May 11, 2012 10:22 PM