none
How to track eyes using Kinect? RRS feed

  • Question

  • The requirement is to locate a rectangle around each eye on a face in 3D space. There should be a way to track eyes using the Microsoft Kinect SDK. According to this 

    The Face Tracking SDK uses the Kinect coordinate system to output its 3D tracking results. The origin is located at the camera’s optical center (sensor), Z axis is pointing towards a user, Y axis is pointing up. The measurement units are meters for translation and degrees for rotation angles.

    Adding

    ...
    Debug3DShape("OuterCornerOfRightEye", faceTrackFrame.Get3DShape()
        [FeaturePoint.OuterCornerOfRightEye]);
    Debug3DShape("InnerCornerRightEye", faceTrackFrame.Get3DShape()
        [FeaturePoint.InnerCornerRightEye]);
    Debug3DShape("InnerCornerLeftEye", faceTrackFrame.Get3DShape()
        [FeaturePoint.InnerCornerLeftEye]);
    Debug3DShape("OuterCornerOfLeftEye", faceTrackFrame.Get3DShape()
        [FeaturePoint.OuterCornerOfLeftEye]);
    ...
    private void Debug3DShape(string s, Vector3DF v)
    {
        Debug.WriteLine(s + " X " + v.X + " Y " + v.Y + " Z " + v.Z);
    }

    to CreateResult() in Microsoft.Kinect.Toolkit.FaceTracking prints

    OuterCornerOfRightEye X -0.05728126 Y 0.04850625 Z -0.1144406
    InnerCornerRightEye X -0.01584376 Y 0.04850625 Z -0.1279687
    InnerCornerLeftEye X 0.01584374 Y 0.04850625 Z -0.1279687
    OuterCornerOfLeftEye X 0.05728124 Y 0.04850625 Z -0.1144406

    when the SDK starts tracking a face. I should be able to use these coordinates to paint a box around each eye but the Z coordinate should probably be closer to 1.0, not -0.1.. or -0.2.. (based on my setup) so I don't trust the numbers. Is the XYZ supposed to be the location of the FeaturePoint in 3D space relative to the sensor? Do I misunderstand the Kinect coordinate system? Am I using the Kinect SDK incorrectly? Does it matter that I am using an Xbox 360 Kinect Sensor? (Microsoft does not guarantee full compatibility for Kinect for Windows applications and the Xbox 360 Kinect Sensor)

    Adding

    if (trackSucceeded) {
        ...
        if (headPointsObj != null) 
            for (int i = 0; i < 2; i++) 
                DebugHeadPoint(i, headPointsObj.Points);
    }
    private void DebugHeadPoint(int i, Vector3DF[] points) {
        if (points == null) throw new ArgumentNullException("points");
        Debug.WriteLine("HeadPoint[" + i + "] X " + points[i].X + 
            " Y " + points[i].Y + 
            " Z " + points[i].Z);
    }

    to FaceTracker.cs::Track() prints this

    HeadPoint[0] X 0.01227657 Y -0.2290326 Z 1.319978
    HeadPoint[1] X 0.00613283 Y -0.02143053 Z 1.280334
    HeadPoint[0] X 0.003939687 Y -0.2297621 Z 1.319813
    HeadPoint[1] X -0.003732742 Y -0.02388078 Z 1.277723
    ...

    These numbers seem correct based on the setup. The FeaturePoints print only once but the HeadPoints print continuously while trackSucceeded. Are FeaturePoint values relative to the HeadPoint?


    Wednesday, June 19, 2013 2:06 PM

Answers

  • Since you are calling Get3DShape the values are in the local coordinate system created for the face mesh. If you are trying to determine the mapped location in relation to the color frame, then you need to call GetProjected3DShape. Since color is a 2D plane, the feature points are X and Y components. If you want the projected mesh vertices, then you use GetTriangles() and map out the vertices for that area.

    The coordinate system is shown here:

    http://msdn.microsoft.com/en-us/library/jj130970.aspx#ID4EWF

    Have a look at the FraceTrackingBasics-WPF sample. You can iterate the feature points and if you want to find the correct enumerated type you can dump this out as follows:

                    for (int i = 0; i < this.facePoints.Count; i++)
                    {
                        Debug.Print("X:{0}, Y:{1} - Type:{2} ", this.facePoints[i].X, facePoints[i].Y, ((FeaturePoint)i).ToString() );
                        ...

    }

    Thursday, June 20, 2013 10:10 PM

All replies

  • Since you are calling Get3DShape the values are in the local coordinate system created for the face mesh. If you are trying to determine the mapped location in relation to the color frame, then you need to call GetProjected3DShape. Since color is a 2D plane, the feature points are X and Y components. If you want the projected mesh vertices, then you use GetTriangles() and map out the vertices for that area.

    The coordinate system is shown here:

    http://msdn.microsoft.com/en-us/library/jj130970.aspx#ID4EWF

    Have a look at the FraceTrackingBasics-WPF sample. You can iterate the feature points and if you want to find the correct enumerated type you can dump this out as follows:

                    for (int i = 0; i < this.facePoints.Count; i++)
                    {
                        Debug.Print("X:{0}, Y:{1} - Type:{2} ", this.facePoints[i].X, facePoints[i].Y, ((FeaturePoint)i).ToString() );
                        ...

    }

    Thursday, June 20, 2013 10:10 PM
  • @Carmine: Thank you very much for the answer. I asked this question earlier here on stackoverflow, then crossposted it here on msdn after getting no correct answer for several days. I'll credit you with the answer and award you the bounty if you post a link to this answer on msdn over there. Or I can post the link to this answer myself if you prefer.
    Sunday, June 23, 2013 2:41 PM