none
Acquire RGB values of each CameraSpacePoint in C# RRS feed

  • Question

  • I have obtained an array of CameraSpacePoint by mapping the entire frame from depth space to camera space. I want to get the RGB values of each point in C#. Please see the code snippet below:

    var colorFrame = multiSourceFrame.ColorFrameReference.AcquireFrame();
    var depthFrame = multiSourceFrame.DepthFrameReference.AcquireFrame();
    
    var depthWidth  = depthFrame.FrameDescription.Width;
    var depthHeight = depthFrame.FrameDescription.Height;
    
    var depthData     = new ushort[depthWidth * depthHeight];
    var camerapoints = new CameraSpacePoint[depthData.Length];
    var colorpoints    = new ColorSpacePoint[depthData.Length];
    
    depthFrame.CopyFrameDataToArray(depthData);
    this.coordinateMapper.MapDepthFrameToCameraSpace(depthData, camerapoints);
    this.coordinateMapper.MapDepthFrameToColorSpace(depthData, colorpoints);
    

    One possible approach can be copying color frame data to a byte array and iterate through each ColorSpacePoint as following:

    var colorWidth  = colorFrame.FrameDescription.Width;
    var colorHeight = colorFrame.FrameDescription.Height;
    
    // Assuming BGRA format here
    var pixels = new byte[colorWidth * colorHeight * 4];
    colorFrame.CopyFrameDataToArray(pixels);
    
    var bgraPoints = new byte[depthWidth * depthHeight * 4];
    
    for (var index = 0; index < depthData.Count; index++)
    {
        var u = colorpoints[index].X;
        var v = colorpoints[index].Y;
        var pixelsBaseIndex = v * depthWidth + u;
        Array.Copy(pixels, 4 * pixelsBaseIndex, bgraPoints, 4 * index, 4);
    }

    In this way, I can use following code to get the RGB information of the first CameraSpacePoint:

    var index = 0;
    var blue   = bgraPoints[4 * index + 0];
    var green = bgraPoints[4 * index + 1];
    var red    = bgraPoints[4 * index + 2];
    var alpha = bgraPoints[4 * index + 3];

    However, I am worried about its performance. Is there exist any better/faster way to acquire RGB values of each CameraSpacePoint in C#?
    • Edited by Ravi 123 Tuesday, November 21, 2017 12:45 PM Better Formatting
    Tuesday, November 21, 2017 12:28 PM

Answers

  • One thing I can see is,

    int pixelsBaseIndex = (int)(v * depthWidth + u);

    u and v are the X and Y components of a ColorSpacePoint. ColorSpacePoints give you coordinates in ColorSpace. ColorSpace is 1920x1080. So when you want to convert a two dimensional coordinate to a one dimentional one, you have to use colorWidth.

    • Marked as answer by Ravi 123 Saturday, February 10, 2018 4:23 AM
    Thursday, November 23, 2017 4:41 PM

All replies

  • No you have to acquire both Depth and Color frames, copy them in buffers and then deal with CoordinateMapper to remap coordinates from one space to another.

    One thing you can improve though is where you copy your frames over to local buffers. Think that color is 1920x1080x4bytes ~=7.9MB and depth is 512*424*2bytes = 424KB. So all in all something like 8MB per frame * 30 frames per second = 240MB/s. A lot of data per sec to be allocating and deallocating frequently.

    You can preallocate the byte arrays for each stream and do memory pinning. You basically declare to your GC that these buffers are not to be messed with,so GC won't accidentally collect them as can be done with your code. The SDK also has some CopyFrameDataToIntPtr which you can call.You basicaly give this call the IntPtr of the GCHandle you'll create for each buffer and it will copy the data to that preallocated memory block each frame, meaning there won't be any allocations per frame for the data. If you look up pinning or GCHandle in the forums here,you'll find examples I wager.

    PS: You know what the color format is by looking at the FrameDescription of the ColorSource. Depends on what the default was for your sensor though. Point is , you don't have to guess. Also do check it. Mine is YUV2 so when copying data from the frame to the buffer, I need to convert the data to the format of my choosing before copying them to my buffer. That's usually a call like CopyConvertedDataTo..().

    Wednesday, November 22, 2017 5:52 PM
  • Hi Nikolaos Patsiouras,

    Thank you very much for your suggestions. I am in the process of incorporating them one by one. Meanwhile, I found something surprising. Please see the code snippet below:

    for (var index = 0; index < depthData.Length; index++)
    {
        var u = colorSpacePoints[index].X;
        var v = colorSpacePoints[index].Y;
        int pixelsBaseIndex = (int)(v * depthWidth + u);
    
        byte blue  = pixels[4 * pixelsBaseIndex + 0];
        byte green = pixels[4 * pixelsBaseIndex + 1];
        byte red   = pixels[4 * pixelsBaseIndex + 2];
        byte alpha = pixels[4 * pixelsBaseIndex + 3];
    }
    

    It is showing following error:

    IndexOutOfRangeException: Index was outside the bounds of the array.

    I monitored minimum and the maximum value of pixelsBaseIndex variable and realize that minimum value is -2147483648. Please see code below, which I used to debug the issue:

    List<int> allIndex = new List<int>();
    for (var index = 0; index < depthData.Length; index++)
    {
        var u = colorpoints[index].X;
        var v = colorpoints[index].Y;
        int pixelsBaseIndex = (int)(v * depthWidth + u);
    
        allIndex.Add(pixelsBaseIndex);
    }
    
    var maxIndex = allIndex.Max();
    var minIndex = allIndex.Min();
    Console.WriteLine(minIndex);//Prints -2147483648
    Console.WriteLine((maxIndex < pixels.Length) && (minIndex >= 0));//Prints False

    It is really surprising to me. What is missing here? Can you please help me to fix it?

    • Edited by Ravi 123 Thursday, November 23, 2017 9:56 AM
    Thursday, November 23, 2017 9:54 AM
  • One thing I can see is,

    int pixelsBaseIndex = (int)(v * depthWidth + u);

    u and v are the X and Y components of a ColorSpacePoint. ColorSpacePoints give you coordinates in ColorSpace. ColorSpace is 1920x1080. So when you want to convert a two dimensional coordinate to a one dimentional one, you have to use colorWidth.

    • Marked as answer by Ravi 123 Saturday, February 10, 2018 4:23 AM
    Thursday, November 23, 2017 4:41 PM