none
CoordinateMapping-WPF working slow in my Kinect v2. application. RRS feed

  • Question

  • I have taken the reference of CoordinateMapping-WPF example which is given in the SDK Browser of Kinect for Windows SDK 2.0.

    And i created a .cs file for the same so that i can use it my different WPF application but it is working very slow.

    When i am near to the camera(approx 15cm to 20cm from the camera) it works fine and smoothly but when i am going far from camera(approx 4ft to 5ft from the camera) then it freeze one single frame for some seconds and then changes it and then again freeze that for some seconds and then changes it and it goes on and the example is working properly.

    So where i am going wrong, i just won't understand.

    Please help with this, its really urgent.

    Any help will be appreciable. 

    Thank You.


    DK

    Friday, January 2, 2015 12:10 PM

All replies

  • Is your custom code doing anything when it reaches certain depth distances? If so, are you releasing/disposing your frames and objects properly?

    Sr. Enterprise Architect | Trainer | Consultant | MCT | MCSD | MCPD | SharePoint TS | MS Virtual TS |Windows 8 App Store Developer | Linux Gentoo Geek | Raspberry Pi Owner | Micro .Net Developer | Kinect For Windows Device Developer |blog: http://dgoins.wordpress.com

    Friday, January 2, 2015 5:51 PM
  • 15cm to 20cm is too close for the sensor to detect any depth/ir/body information so as Dwight suggested, are you forgetting to release a body frame somewhere. Do the samples have the same issue? Verify your code execution to ensure you know what is happening in the callbacks.

    Carmine Sirignano - MSFT

    Saturday, January 3, 2015 1:05 AM
  • Hi Dwight,

    Thank you so much for your reply.

    I have a KinectController.cs class file as follows:

    private readonly int bytesPerPixel = (PixelFormats.Bgr32.BitsPerPixel + 7) / 8; private KinectSensor kinectSensor = null; private CoordinateMapper coordinateMapper = null; private MultiSourceFrameReader multiFrameSourceReader = null; private WriteableBitmap bitmap = null; private uint bitmapBackBufferSize = 0; private DepthSpacePoint[] colorMappedToDepthPoints = null; private string statusText = null; public KinectControler() { Main(); } public void Main() { this.kinectSensor = KinectSensor.GetDefault(); this.multiFrameSourceReader = this.kinectSensor.OpenMultiSourceFrameReader(FrameSourceTypes.Depth | FrameSourceTypes.Color | FrameSourceTypes.BodyIndex); this.multiFrameSourceReader.MultiSourceFrameArrived += this.Reader_MultiSourceFrameArrived; this.coordinateMapper = this.kinectSensor.CoordinateMapper; FrameDescription depthFrameDescription = this.kinectSensor.DepthFrameSource.FrameDescription; int depthWidth = depthFrameDescription.Width; int depthHeight = depthFrameDescription.Height; FrameDescription colorFrameDescription = this.kinectSensor.ColorFrameSource.FrameDescription; int colorWidth = colorFrameDescription.Width; int colorHeight = colorFrameDescription.Height; this.colorMappedToDepthPoints = new DepthSpacePoint[colorWidth * colorHeight]; this.bitmap = new WriteableBitmap(colorWidth, colorHeight, 96.0, 96.0, PixelFormats.Bgra32, null); // Calculate the WriteableBitmap back buffer size this.bitmapBackBufferSize = (uint)((this.bitmap.BackBufferStride * (this.bitmap.PixelHeight - 1)) + (this.bitmap.PixelWidth * this.bytesPerPixel)); this.kinectSensor.IsAvailableChanged += this.Sensor_IsAvailableChanged; this.kinectSensor.Open(); }

    public ImageSource ImageSource { get { return this.bitmap; } }

    private void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
    {
          int depthWidth = 0;
          int depthHeight = 0;

          DepthFrame depthFrame = null;
          ColorFrame colorFrame = null;
          BodyIndexFrame bodyIndexFrame = null;
          bool isBitmapLocked = false;

          MultiSourceFrame multiSourceFrame = e.FrameReference.AcquireFrame();

          // If the Frame has expired by the time we process this event, return.
          if (multiSourceFrame == null)
          {
              return;
          }

          // We use a try/finally to ensure that we clean up before we exit the function.  
          // This includes calling Dispose on any Frame objects that we may have and unlocking the bitmap back buffer.
          try
          {
          depthFrame = multiSourceFrame.DepthFrameReference.AcquireFrame();
          colorFrame = multiSourceFrame.ColorFrameReference.AcquireFrame();
          bodyIndexFrame = multiSourceFrame.BodyIndexFrameReference.AcquireFrame();

          // If any frame has expired by the time we process this event, return.
          // The "finally" statement will Dispose any that are not null.
          if ((depthFrame == null) || (colorFrame == null) || (bodyIndexFrame == null))
          {
               return;
          }

          // Process Depth
          FrameDescription depthFrameDescription = depthFrame.FrameDescription;

          depthWidth = depthFrameDescription.Width;
          depthHeight = depthFrameDescription.Height;

          // Access the depth frame data directly via LockImageBuffer to avoid making a copy
          using (KinectBuffer depthFrameData = depthFrame.LockImageBuffer())
          {
              this.coordinateMapper.MapColorFrameToDepthSpaceUsingIntPtr(
              depthFrameData.UnderlyingBuffer,
              depthFrameData.Size,
              this.colorMappedToDepthPoints);
          }

          // We're done with the DepthFrame 
          depthFrame.Dispose();
          depthFrame = null;

          // Process Color

          // Lock the bitmap for writing
          this.bitmap.Lock();
          isBitmapLocked = true;

          colorFrame.CopyConvertedFrameDataToIntPtr(this.bitmap.BackBuffer, this.bitmapBackBufferSize, ColorImageFormat.Bgra);

          // We're done with the ColorFrame 
          colorFrame.Dispose();
          colorFrame = null;

          // We'll access the body index data directly to avoid a copy
          using (KinectBuffer bodyIndexData = bodyIndexFrame.LockImageBuffer())
          {
               unsafe
               {
                    byte* bodyIndexDataPointer = (byte*)bodyIndexData.UnderlyingBuffer;

                    int colorMappedToDepthPointCount = this.colorMappedToDepthPoints.Length;

                    fixed (DepthSpacePoint* colorMappedToDepthPointsPointer = this.colorMappedToDepthPoints)
                    {
                        // Treat the color data as 4-byte pixels
                        uint* bitmapPixelsPointer = (uint*)this.bitmap.BackBuffer;

                        // Loop over each row and column of the color image
                        // Zero out any pixels that don't correspond to a body index
                        for (int colorIndex = 0; colorIndex < colorMappedToDepthPointCount; ++colorIndex)
                        {
                             float colorMappedToDepthX = colorMappedToDepthPointsPointer[colorIndex].X;
                             float colorMappedToDepthY = colorMappedToDepthPointsPointer[colorIndex].Y;

                             // The sentinel value is -inf, -inf, meaning that no depth pixel corresponds to this color pixel.
                             if (!float.IsNegativeInfinity(colorMappedToDepthX) &&
                                  !float.IsNegativeInfinity(colorMappedToDepthY))
                             {
                                    // Make sure the depth pixel maps to a valid point in color space
                                    int depthX = (int)(colorMappedToDepthX + 0.5f);
                                    int depthY = (int)(colorMappedToDepthY + 0.5f);

                                    // If the point is not valid, there is no body index there.
                                    if ((depthX >= 0) && (depthX < depthWidth) && (depthY >= 0) && (depthY < depthHeight))
                                    {
                                          int depthIndex = (depthY * depthWidth) + depthX;

                                          // If we are tracking a body for the current pixel, do not zero out the pixel
                                          if (bodyIndexDataPointer[depthIndex] != 0xff)
                                          {
                                              continue;
                                          }
                                     }
                               }

                               bitmapPixelsPointer[colorIndex] = 0;
                       }
                     }

                     this.bitmap.AddDirtyRect(new Int32Rect(0, 0, this.bitmap.PixelWidth, this.bitmap.PixelHeight));
                   }
                  }
                }
                finally
                {
                    if (isBitmapLocked)
                    {
                        this.bitmap.Unlock();
                    }

                    if (depthFrame != null)
                    {
                        depthFrame.Dispose();
                    }

                    if (colorFrame != null)
                    {
                        colorFrame.Dispose();
                    }

                    if (bodyIndexFrame != null)
                    {
                        bodyIndexFrame.Dispose();
                    }
                }
            }


    And i am calling it in my MainWindow.xaml.cs file as follows:

    KinectControler kinectControl;
    
    public MainWindow()
    {
          InitializeComponent();
          kinectControl = new KinectControler();
          camera.Source = kinectControl.ImageSource;
    }

    "camera" is a Image Control.

    So i didn's find any issue in this and i am also disposing color and depth frame.

    And i am also not doing anything when it reaches some distance.

    So what can be the issue??

    Please suggest as soon as possible.

    Thank You. 

     

    DK

    Monday, January 5, 2015 9:16 AM
  • Hi Caemine,

    Thanks for Reply.

    If i am trying to dispose the bodyframe then it gives me the Exception as "Disposed Element cannot be Processed".

    And the Example is works just fine.

    So what can be the issue? I have also posted my code as the reply of Dwight Comment.

    So Whats the issue in this?

    Please suggest as soon as possible.

    Thank You.


    DK

    Monday, January 5, 2015 9:22 AM
  • Because the frame got already disposed. Where is it failing? Review how .net uses the using statement:

    http://msdn.microsoft.com/en-us/library/yh598w02.aspx

    At the end of the using statement, the object is already disposed.


    Carmine Sirignano - MSFT

    Monday, January 5, 2015 8:52 PM
  • Hi again Carmine,

    Thanks for your reply and your valuable time.

    I do understand hows the "using" statement works in .net.

    My main concern is that why the it works good when user is near to the camera and it starts lagging when the user far from the camera.

    If my program has some issue in it then it should not work at all.

    But it works when the user is near and starts lagging when the user is far.

    I just won't understand this.

    Please Reply on this.

    Thank You.


    DK

    Tuesday, January 6, 2015 6:24 AM
  • It behaves this way because only when you are getting all frame data does the full execution of the callback function execute. If there are no bodies detected, the bodyIndexFrame is going to have nothing in it and that would be where I would focus where the issue is occurring. Any delay's returning from the callback has a ripple effect on perf.

    You may be better served to copy the data to a buffer and process it on another thread. This will ensure that you are returning/disposing all the Kinect data and not holding up the runtime and the watchdog is ensured to keep providing your process frame data.

    You can profile your application you should see where in that block you are spending the majority of the time. Alternatively, you can setup breakpoints/logging to determine the code path.

     

    Carmine Sirignano - MSFT

    Tuesday, January 6, 2015 6:01 PM
  • Disposing the body frame, then it runs smoothly. Thanks!

    YU Xinbo

    Thursday, March 14, 2019 2:08 PM