none
How do I capture just a face image from the RBG camera... RRS feed

  • Question

  • Hi guys,

    I want to be able to drop a still image from the RBG camera every 5 seconds or at a certain event being called, i.e. TakeFacePic, I'll want to store these still images local in a file and then use Face.com's API to analyses the faces facial expression/mood, etc. I was wondering how I could do this using the Kinect SDK and also if it was possible just to take a picture of the user's face. Considering I can track the user head position very easily with the depth camera. The reason I want to snap a picture of just the face is I don't want to take pictures of random onlookers faces, I'm just interested in the person looking at and closest to the Kinect or more precisely the display above/below it. Thanks in advance.

    Regards,

    Mark Dunne

    Wednesday, July 13, 2011 1:58 PM

Answers

  • Mark,

    Sorry for the long delay replying. I'm just now looking back at questions I missed in the past. Keep in mind that I'm not an expert in WPF, so you should redirect any follow-up questions about UI or WPF specifics to the WPF forums (http://social.msdn.microsoft.com/Forums/en/wpf/threads). Also, the following code is just meant as a sample and does not do much checking for errors or boundary conditions, but hopefully it points you in the right direction.

    Starting from C# SkeletalViewer sample (installed to C:\Users\Public\Documents\Microsoft Research KinectSDK Samples\NUI\SkeletalViewer\CS), add following declarations to MainWindow class in MainWindow.xaml.cs file:

    Point lastHeadPoint = new Point(-1,-1);
    Point lastShoulderLeftPoint = new Point(-1,-1);
    Point lastShoulderRightPoint = new Point(-1, -1);
    

    Then, in getDisplayPosition function in same file, add the following after call to nui.NuiCamera.GetColorPixelCoordinatesFromDepthPixel:

    if (joint.ID == JointID.Head)
    {
      lastHeadPoint = new Point(colorX,colorY);
    }
    else if (joint.ID == JointID.ShoulderRight)
    {
      lastShoulderRightPoint = new Point(colorX, colorY);
    }
    else if (joint.ID == JointID.ShoulderLeft)
    {
      lastShoulderLeftPoint = new Point(colorX, colorY);
    }
    
    

    Finally, replace code in nui_ColorFrameReady method with:

    // 32-bit per pixel, RGBA image
    PlanarImage Image = e.ImageFrame.Image;
    BitmapSource fullBitmap = BitmapSource.Create(
    Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, Image.Bits, Image.Width * Image.BytesPerPixel);
          
    if (lastHeadPoint.X >= 0)
    {
      double minX = Math.Min(Math.Min(lastShoulderLeftPoint.X,lastShoulderRightPoint.X), lastHeadPoint.X);
      double maxX = Math.Max(Math.Max(lastShoulderLeftPoint.X, lastShoulderRightPoint.X), lastHeadPoint.X);
      double maxY = Math.Max(lastShoulderLeftPoint.Y, lastShoulderRightPoint.Y);
      double minY = lastHeadPoint.Y - (maxY - lastHeadPoint.Y);
    
      Int32Rect cropRect = new Int32Rect((int)Math.Max(0,minX), (int)Math.Max(0,minY), (int)(maxX-minX), (int)(maxY-minY));
      video.Source = new CroppedBitmap(fullBitmap, cropRect);
    }
    else 
    {
      video.Source = fullBitmap;
    }
    

    Again, this is not production-level code, but hopefully the calculation of minX, maxX, minY and maxY will be helpful to you.
    Eddy


    I'm here to help
    Tuesday, August 16, 2011 2:45 AM

All replies

  •  

    There's a sample in OpenCV called FaceDetect which does exactly that

    Tim

     


    TM
    Thursday, July 14, 2011 2:58 PM
  • There's no face recognition API in the SDK currently. You'll have to figure out from color, skeleton or depth data where the face is, then map that to the color image to extract the part of the color image with the face.

    You could try using the color image and recognizing faces by their geometry and features. The same approach might work with depth data -- lighting and skin tone aren't a factor in the depth data, which might make it easier to identify a face-shaped thing.

    Or you could try using the skeleton's head joint as the face center, and then scaling your face image width and height based on the visible width of the shoulders. (Person farther from camera -> shoulder joints closer together in image space -> smaller rectangle to enclose the face.)

    If it makes sense in your application, you could prompt the user to stand in a certain place to simplify face finding. Maybe even simplify it to the point of just using the entire frame, if they stand close enough.

    TimMacP's suggestion is probably doing some variant on one of the above, so looking at it might be useful, even if you don't use OpenCV in your project.

    Thursday, July 14, 2011 5:22 PM
  • Cheers Robert,

     

    That's a big help. I hope to use Face.com's API to analyse the images I drop into a folder. Can you suggest some C# code to capture an image from the RGB video stream available in the Kinect SDK.

     

    Thanks in advance.

    Mark.

    Thursday, July 14, 2011 6:39 PM
  • Mark,

    Sorry for the long delay replying. I'm just now looking back at questions I missed in the past. Keep in mind that I'm not an expert in WPF, so you should redirect any follow-up questions about UI or WPF specifics to the WPF forums (http://social.msdn.microsoft.com/Forums/en/wpf/threads). Also, the following code is just meant as a sample and does not do much checking for errors or boundary conditions, but hopefully it points you in the right direction.

    Starting from C# SkeletalViewer sample (installed to C:\Users\Public\Documents\Microsoft Research KinectSDK Samples\NUI\SkeletalViewer\CS), add following declarations to MainWindow class in MainWindow.xaml.cs file:

    Point lastHeadPoint = new Point(-1,-1);
    Point lastShoulderLeftPoint = new Point(-1,-1);
    Point lastShoulderRightPoint = new Point(-1, -1);
    

    Then, in getDisplayPosition function in same file, add the following after call to nui.NuiCamera.GetColorPixelCoordinatesFromDepthPixel:

    if (joint.ID == JointID.Head)
    {
      lastHeadPoint = new Point(colorX,colorY);
    }
    else if (joint.ID == JointID.ShoulderRight)
    {
      lastShoulderRightPoint = new Point(colorX, colorY);
    }
    else if (joint.ID == JointID.ShoulderLeft)
    {
      lastShoulderLeftPoint = new Point(colorX, colorY);
    }
    
    

    Finally, replace code in nui_ColorFrameReady method with:

    // 32-bit per pixel, RGBA image
    PlanarImage Image = e.ImageFrame.Image;
    BitmapSource fullBitmap = BitmapSource.Create(
    Image.Width, Image.Height, 96, 96, PixelFormats.Bgr32, null, Image.Bits, Image.Width * Image.BytesPerPixel);
          
    if (lastHeadPoint.X >= 0)
    {
      double minX = Math.Min(Math.Min(lastShoulderLeftPoint.X,lastShoulderRightPoint.X), lastHeadPoint.X);
      double maxX = Math.Max(Math.Max(lastShoulderLeftPoint.X, lastShoulderRightPoint.X), lastHeadPoint.X);
      double maxY = Math.Max(lastShoulderLeftPoint.Y, lastShoulderRightPoint.Y);
      double minY = lastHeadPoint.Y - (maxY - lastHeadPoint.Y);
    
      Int32Rect cropRect = new Int32Rect((int)Math.Max(0,minX), (int)Math.Max(0,minY), (int)(maxX-minX), (int)(maxY-minY));
      video.Source = new CroppedBitmap(fullBitmap, cropRect);
    }
    else 
    {
      video.Source = fullBitmap;
    }
    

    Again, this is not production-level code, but hopefully the calculation of minX, maxX, minY and maxY will be helpful to you.
    Eddy


    I'm here to help
    Tuesday, August 16, 2011 2:45 AM
  • I assume this can work equally as well in VB?

     

    PS. This is GREAT stuff. Can't wait to try it!

     

     


    Thomas M. Riverside Robotics www.robots-and-androids.com
    Tuesday, August 23, 2011 5:53 AM