none
Extracting raw RGB data from detected faces. RRS feed

  • Question

  • Hi,

    I am playing around with the FaceBasicsD2D sample for C++, provided in Kinect V2 SDK. After detecting all faces within the sensor's FOV, I want to recognize a specific face. I know I will have to use OpenCV for the recognition part but, before I do that, I want to extract the pixel data for each face detected by the sensor. Is there an API within the Kinect library that can provide me any of the pixel properties (RGB, HSI, CMY)?

    Thanks!!! 

    Thursday, January 22, 2015 8:08 PM

Answers

  • The Kinect face api's will provide a bound box that will assist with isolating where in the image the face is. You will have to pull the pixel values from the respective frames based on that.

    https://msdn.microsoft.com/en-us/library/microsoft.kinect.face.ifaceframeresult.aspx

    • get_FaceBoundingBoxInColorSpace - Gets the face bounding box in color space.
    • get_FaceBoundingBoxInInfraredSpace - Gets the face bounding box in infrared space.

    Carmine Sirignano - MSFT


    Friday, January 23, 2015 6:47 PM
  • Are you asking...What are the RGB pixel values from the Kinect color image bounded by a box?

    If that is your question...

    1) get a pointer to the raw RGB data

    2) index into that data based on the bounding box's top left XY position

    3) use the width and height of the bounding box to index into the raw RGB data starting at the XY position.

    You will need to write code with some pointer math to get just the pixels from the raw RGB data bounded by the box.
    Or maybe there is a simple open source library you can use somewhere. Maybe imagemagick


    --Dale

    Monday, May 25, 2015 4:22 PM

All replies

  • The Kinect face api's will provide a bound box that will assist with isolating where in the image the face is. You will have to pull the pixel values from the respective frames based on that.

    https://msdn.microsoft.com/en-us/library/microsoft.kinect.face.ifaceframeresult.aspx

    • get_FaceBoundingBoxInColorSpace - Gets the face bounding box in color space.
    • get_FaceBoundingBoxInInfraredSpace - Gets the face bounding box in infrared space.

    Carmine Sirignano - MSFT


    Friday, January 23, 2015 6:47 PM
  • Hello, I am also want to get face image from Kinect face and color frame. I have succeed on getting bound box on image. based on your suggestion that we need to pull the pixel from bound box frame, how can I create a frame in color space based on coordinate that I got from bound box ?

    Thank You


    Adityo Setyonugroho



    Friday, May 22, 2015 2:27 PM
  • Are you asking...What are the RGB pixel values from the Kinect color image bounded by a box?

    If that is your question...

    1) get a pointer to the raw RGB data

    2) index into that data based on the bounding box's top left XY position

    3) use the width and height of the bounding box to index into the raw RGB data starting at the XY position.

    You will need to write code with some pointer math to get just the pixels from the raw RGB data bounded by the box.
    Or maybe there is a simple open source library you can use somewhere. Maybe imagemagick


    --Dale

    Monday, May 25, 2015 4:22 PM
  • Thanks for your reply. I still working on it slowly, I tried to look at CoordinateMapping example, and I noticed something maybe useful for me.

    fixed (DepthSpacePoint* colorMappedToDepthPointsPointer = this.colorMappedToDepthPoints)
                            {
                                // Treat the color data as 4-byte pixels
                                uint* bitmapPixelsPointer = (uint*)this.bitmap.BackBuffer;
    
                                // Loop over each row and column of the color image
                                // Zero out any pixels that don't correspond to a body index
                                for (int colorIndex = 0; colorIndex < colorMappedToDepthPointCount; ++colorIndex)
                                {
                                    float colorMappedToDepthX = colorMappedToDepthPointsPointer[colorIndex].X;
                                    float colorMappedToDepthY = colorMappedToDepthPointsPointer[colorIndex].Y;
    
                                    // The sentinel value is -inf, -inf, meaning that no depth pixel corresponds to this color pixel.
                                    if (!float.IsNegativeInfinity(colorMappedToDepthX) &&
                                        !float.IsNegativeInfinity(colorMappedToDepthY))
                                    {
    
    Continue . . . . .

    And I found out about

    ColorSpacePoint*

    Can I use coordinate and size ( Width and Length ) from boundbox for these example ?

    Thanks


    Adityo Setyonugroho

    Saturday, June 6, 2015 7:15 AM
  • No, none of that code will help you. Instead...
    1) get a pointer to the raw RGB data
    Use the APIs, e.g. CopyConvertedFrameDataToArray(), for this. You will need to choose your RGB pixel format with the ColorImageFormat Enumeration

    2) index into that data based on the bounding box's top left XY position
    3) use the width and height of the bounding box to index into the raw RGB data starting at the XY position.
    Index into the color data using your bounding box coordinates with pointer math. See http://www.cs.umd.edu/class/sum2003/cmsc311/Notes/BitOp/pointer.html


    --Dale

    Saturday, June 6, 2015 11:08 AM
  • Thanks for your reply and patience hahaha, I kind a slowly programmer hehe, I'm still learning c# haha.

    Please be patience with me haha, I tried to do slowly one by one. So far, here is What I can do:

    (For Example static x and y, x = 700 , y = 400 with width = 400 and height = 400)

    1. Preparing container for raw data:

    private byte[] pixels = null;

    2. I have initiated pixels:

    this.pixels = new byte[colorFrameDescription.Width * colorFrameDescription.Height * colorFrameDescription.BytesPerPixel];

    So the length of pixels = 8294400

    3. And also do the same thing for pixels for bounding box (400x400):

    this.pixelsForBoundingBox = new Byte[400 * 400 * colorFrameDescription.BytesPerPixel];

    So the length for pixelsForBoundingBox = 640000

    4. Copy raw data into array:

    FrameDescription colorFrameDescription = colorFrame.FrameDescription;
                        if (colorFrame.RawColorImageFormat == ColorImageFormat.Bgra)
                            colorFrame.CopyRawFrameDataToArray(pixels);
                        else
                            colorFrame.CopyConvertedFrameDataToArray(this.pixels, ColorImageFormat.Bgra);

    5. Initiate Coordinate:

    int coorX = 700;
    int coorY = 400;

    I so confused because the size of array is 1920*1080*4 (because of bytes). I got x and y, what index of pixels array that I should access as initial ?

    I haven't learned anything about pointers array in c# yet, but I found https://msdn.microsoft.com/en-us/library/28k1s2k6.aspx . can I use it ?


    Adityo Setyonugroho

    Tuesday, June 9, 2015 8:38 AM
  • Thanks for your ideaaa, I tried so many code and ways. Finally, I can get pixels from raw using coordinate from bounding box. Is it the good way to do it or there is more stable and accurate way ?

    here is my code:

    private void Reader_ColorFrameArrived(object sender, ColorFrameArrivedEventArgs e)
            {
                // ColorFrame is IDisposable
                using (ColorFrame colorFrame = e.FrameReference.AcquireFrame())
                {
                    if (colorFrame != null)
                    {
                        FrameDescription colorFrameDescription = colorFrame.FrameDescription;
                        if (colorFrame.RawColorImageFormat == ColorImageFormat.Bgra)
                            colorFrame.CopyRawFrameDataToArray(pixels);
                        else
                            colorFrame.CopyConvertedFrameDataToArray(this.pixels, ColorImageFormat.Bgra);
    
                        //Initialize Emgu CV image then assign byte array of pixels to it
                        //Image<Bgra, byte> img = new Image<Bgra, byte>(1920, 1080);
                        //img.Bytes = pixels;
    
                        int coorX = 0;
                        int coorY = 0;
    
                        if (initialX <= 50 || initialY <= 50)
                        {
                            coorX = 51;
                            coorY = 51;
                        }
    
                        else
                        {
                            coorX = initialX;
                            coorY = initialY;
                        }
    
                            int initialCoordinateOnPixels = (coorX + ((coorY - 1) * 1920)) * 4;
    
                            int i = 0;
                            int j = 0;
    
                            for (int y = 0; y < 400; y++)
                            {
                                for (int x = 0; x < 1600; x++)
                                {
                                    pixelsForBoundingBox[i] = pixels[initialCoordinateOnPixels + j];
                                    i++;
                                    j++;
                                }
                                j = 0;
                                initialCoordinateOnPixels = initialCoordinateOnPixels + (1920 * 4);
                            }
    
                            //Image<Bgra, byte> img = new Image<Bgra, byte>(1920, 1080);
                            Image<Bgra, byte> img = new Image<Bgra, byte>(400, 400);
                            img.Bytes = pixelsForBoundingBox;
    
                            imgBox.Image = img;
                    }
                }
            }


    Adityo Setyonugroho

    Tuesday, June 9, 2015 1:22 PM