none
WPF SDK 1.5 - Face Tracking withour skeleton

    Question

  • I recently discovered, thanks to this forum (This Question), that c# and c++ examples provided with the SDK have different capabilities (the firts one works only with skeleton information, the latter with skeleton and color-depth(optional) information).

    Now I wonder why I cannot start the tracking with the only color information in c#, the function call always return HeadSearchFailed. Do I do something wrong in the call? The tracker only works when I provide the skeleton information. I tried to cheat and manually provide the head joint data, but no luck with that.

    if (this.faceTracker != null)
                        {
                            FaceTrackFrame faceTrackFrame = this.faceTracker.Track(
                                this.colorImageFormat,
                                this.colorImage,
                                this.depthImageFormat,
                                this.depthImage);
    
                            if (faceTrackFrame.TrackSuccessful)
                            {
                                //Work on Data
                            }
                        }

    Wednesday, May 30, 2012 8:49 AM

Answers

  • Re-posting here from another question:

    Unfortunately, the WPF sample assumes that the face tracking is used only to track face of people with tracked skeletons. So your modification is not enough. You would need to modify plenty of code in this sample to make it working like you want. So I would recomend creating your own C# project and call into the interop for FT SDK directly if you want to use only color + depth.

    For example, in the function: OnAllFramesReady() in FaceTrackingViewer.xaml.cs we see the following loop:

      // Update the list of trackers and the trackers with the current frame information
                    foreach (Skeleton skeleton in this.skeletonData)
                    {
                        if (skeleton.TrackingState == SkeletonTrackingState.Tracked
                            || skeleton.TrackingState == SkeletonTrackingState.PositionOnly)
                        {
                            // We want keep a record of any skeleton, tracked or untracked.
                            if (!this.trackedSkeletons.ContainsKey(skeleton.TrackingId))
                            {
                                this.trackedSkeletons.Add(skeleton.TrackingId, new SkeletonFaceTracker());
                            }

                            // Give each tracker the upated frame.
                            SkeletonFaceTracker skeletonFaceTracker;
                            if (this.trackedSkeletons.TryGetValue(skeleton.TrackingId, out skeletonFaceTracker))
                            {
                                skeletonFaceTracker.OnFrameReady(this.Kinect, colorImageFormat, colorImage, depthImageFormat, depthImage, skeleton);
                                skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
                            }
                        }
                    }

    The method OnFrameReady() will then call the face tracker. But it will happen only when there is an active skeleton to track.

    You would need to modify this code to call the face tracker even when there is not active skeleton. It was not the original goal of the sample and so it was not done that way :-)

    Also, to call the face tracker with only color and depth you need to call this method from FTInterop.cs:

    int StartTracking(ref FaceTrackingSensorData sensorData, ref Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    and pass the region of interest (roi) where you think the face is located. If you don't know, you can pass the Rect for the full frame, i.e. (0,0,639,479).

    You can also modify this interop, so it matches native C++ COM API better, meaning the Rect parameter can actually be passed as NULL to the native API, in which case the FT API will search the whole RGB frame for a face to start tracking. To do this you would need to modify the interop this way:

    [PreserveSig]
    int StartTracking(ref FaceTrackingSensorData sensorData, Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    note that "ref Rect roi" is now defined as "Rect roi" so you can pass null.

    you also need to redefine Rect structure as a class this way (so you can pass null as roi parameter):

    [StructLayout(LayoutKind.Sequential)]
    public class Rect
    {
        public Rect() { }
        ...
    }

    Now, you can call the face tracker this way:

    hr = this.faceTrackerInteropPtr.StartTracking(ref faceTrackSensorData, null, null, this.frame.ResultPtr); // starts

    hr = this.faceTrackerInteropPtr.ContinueTracking(ref faceTrackSensorData, null, this.frame.ResultPtr);  // continues

    where skeleton is null and region of interest is null too.

    Sorry, it does not look simple...

    • Marked as answer by MaDoss Thursday, May 31, 2012 3:36 PM
    Wednesday, May 30, 2012 9:41 PM

All replies

  • As the post above you link to states right now its limited to only skeleton,color, and depth in c#. Also,  face tracking requires a person to stand up initially to detect the skeleton then your face can be tracked once a skeleton is available.


    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/


    • Edited by The Thinker Wednesday, May 30, 2012 9:14 PM
    Wednesday, May 30, 2012 9:13 PM
  • Re-posting here from another question:

    Unfortunately, the WPF sample assumes that the face tracking is used only to track face of people with tracked skeletons. So your modification is not enough. You would need to modify plenty of code in this sample to make it working like you want. So I would recomend creating your own C# project and call into the interop for FT SDK directly if you want to use only color + depth.

    For example, in the function: OnAllFramesReady() in FaceTrackingViewer.xaml.cs we see the following loop:

      // Update the list of trackers and the trackers with the current frame information
                    foreach (Skeleton skeleton in this.skeletonData)
                    {
                        if (skeleton.TrackingState == SkeletonTrackingState.Tracked
                            || skeleton.TrackingState == SkeletonTrackingState.PositionOnly)
                        {
                            // We want keep a record of any skeleton, tracked or untracked.
                            if (!this.trackedSkeletons.ContainsKey(skeleton.TrackingId))
                            {
                                this.trackedSkeletons.Add(skeleton.TrackingId, new SkeletonFaceTracker());
                            }

                            // Give each tracker the upated frame.
                            SkeletonFaceTracker skeletonFaceTracker;
                            if (this.trackedSkeletons.TryGetValue(skeleton.TrackingId, out skeletonFaceTracker))
                            {
                                skeletonFaceTracker.OnFrameReady(this.Kinect, colorImageFormat, colorImage, depthImageFormat, depthImage, skeleton);
                                skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
                            }
                        }
                    }

    The method OnFrameReady() will then call the face tracker. But it will happen only when there is an active skeleton to track.

    You would need to modify this code to call the face tracker even when there is not active skeleton. It was not the original goal of the sample and so it was not done that way :-)

    Also, to call the face tracker with only color and depth you need to call this method from FTInterop.cs:

    int StartTracking(ref FaceTrackingSensorData sensorData, ref Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    and pass the region of interest (roi) where you think the face is located. If you don't know, you can pass the Rect for the full frame, i.e. (0,0,639,479).

    You can also modify this interop, so it matches native C++ COM API better, meaning the Rect parameter can actually be passed as NULL to the native API, in which case the FT API will search the whole RGB frame for a face to start tracking. To do this you would need to modify the interop this way:

    [PreserveSig]
    int StartTracking(ref FaceTrackingSensorData sensorData, Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    note that "ref Rect roi" is now defined as "Rect roi" so you can pass null.

    you also need to redefine Rect structure as a class this way (so you can pass null as roi parameter):

    [StructLayout(LayoutKind.Sequential)]
    public class Rect
    {
        public Rect() { }
        ...
    }

    Now, you can call the face tracker this way:

    hr = this.faceTrackerInteropPtr.StartTracking(ref faceTrackSensorData, null, null, this.frame.ResultPtr); // starts

    hr = this.faceTrackerInteropPtr.ContinueTracking(ref faceTrackSensorData, null, this.frame.ResultPtr);  // continues

    where skeleton is null and region of interest is null too.

    Sorry, it does not look simple...

    • Marked as answer by MaDoss Thursday, May 31, 2012 3:36 PM
    Wednesday, May 30, 2012 9:41 PM
  • Also, to call the face tracker with only color and depth you need to call this method from FTInterop.cs:

    int StartTracking(ref FaceTrackingSensorData sensorData, ref Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    and pass the region of interest (roi) where you think the face is located. If you don't know, you can pass the Rect for the full frame, i.e. (0,0,639,479).

    You can also modify this interop, so it matches native C++ COM API better, meaning the Rect parameter can actually be passed as NULL to the native API, in which case the FT API will search the whole RGB frame for a face to start tracking. To do this you would need to modify the interop this way:

    [PreserveSig]
    int StartTracking(ref FaceTrackingSensorData sensorData, Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    note that "ref Rect roi" is now defined as "Rect roi" so you can pass null.

    you also need to redefine Rect structure as a class this way (so you can pass null as roi parameter):

    [StructLayout(LayoutKind.Sequential)]
    public class Rect
    {
        public Rect() { }
        ...
    }

    Now, you can call the face tracker this way:

    hr = this.faceTrackerInteropPtr.StartTracking(ref faceTrackSensorData, null, null, this.frame.ResultPtr); // starts

    hr = this.faceTrackerInteropPtr.ContinueTracking(ref faceTrackSensorData, null, this.frame.ResultPtr);  // continues

    where skeleton is null and region of interest is null too.

    Sorry, it does not look simple...

    Thank for your answer: so it's true, the API as they are provided (including the toolkit) do not allow to operate without skeleton, you have to rewrite some of the code inside Toolkit.Facetracking. I understood what you wrote (more or less), I'll try to apply it in my code and see what results.

    Thursday, May 31, 2012 8:39 AM
  • if trying to start a new thread to clean out the discussion if a new thread you could write part 2 in title.


    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/

    Thursday, May 31, 2012 11:58 AM
  • I managed to make it work: it's still really slow compared to the other example, but it's a starting point. I modified the two projects provided with the SDK (Microsot.Kinect.Toolkit and ~.FaceTracking) as Nikolai described.

    Be careful though, every time you install a new example the SDK tool overwrites the projects, so rename their folder.

    Thursday, May 31, 2012 2:14 PM
  • if trying to start a new thread to clean out the discussion if a new thread you could write part 2 in title.


    I thought to change topic because the question where different ("Are the examples different?", "How can I obtain the same results?"), no intention of double-posting.
    Thursday, May 31, 2012 3:39 PM
  • Also, please remember that the C# API that you refer to is really just a C# sample with internal sample API. The Face Tracking SDK has only C++ public supported API. The C# sample was intended as a sample only. You can modify it as you wish :-) The interop layer was intended to demo the lower level calls to the native API and higher level FaceTracker was about demoing the more higher level approach to the APIs. Hopefully, there will be a fully supported public C# API as well in the future.

    There will still be differences in speed between native and managed code since interop layer slows things down a bit.

    Thanks!

    Saturday, June 02, 2012 6:01 AM
  • I managed to make it work: it's still really slow compared to the other example, but it's a starting point. I modified the two projects provided with the SDK (Microsot.Kinect.Toolkit and ~.FaceTracking) as Nikolai described.

    Be careful though, every time you install a new example the SDK tool overwrites the projects, so rename their folder.

    Hi MaDoss,

    Can you please provide the C# working code to do face tracking without the skeleton data? I am trying using the sensor polling method but never succeed.

    Appreciate your help.

    Thanks

    Monday, November 12, 2012 4:58 AM