none
Kinect SDK 1.5 - Face Tracking : WPF tracking problems RRS feed

  • Question

  • I'm working with the new SDK (1.5) and I noticed deep differences between c++ and WPF examples provided with it: the c++ appears to be much faster than the WPF one in detecting and tracking the face.

    The first one is almost on the fly, while the second one takes seconds (tens of seconds, often) to track the face. Did anyone found out why? I noticed that the skeletonframe provided shows the property "Trackingmode = default", even though I set the kinect skeleton stream on seated.

    Wednesday, May 23, 2012 12:07 PM

Answers

  • The difference between C# sample and C++ sample is the following: FT SDK can track faces in 2 models: 1)based on color + depth + skeleton OR 2) color + depth only (no skeleton). The C++ sample uses both modes, so the active skeleton is not required to start tracking - it starts in color + depth mode only and then when a skeleton is available from Kinect (when you stand up) it switches to using color + depth + skeleton. The C# sample is limited to only color + depth + skeleton mode. See this code:

    FaceTrackFrameframe = this.faceTracker.Track(colorImageFormat, colorImage, depthImageFormat, depthImage, skeletonOfInterest);

    So when the skeleton is not passed it will fail for a given frame (even though the C# interop API provides a second mode with color + depth only, it is just not used in WPF app). So the delay is caused by the fact that Kinect is initialized with the full skeleton tracking (not seated) and when you start the sample app the skeletal tracking is not yet tracking your skeleton until you stand up. Performance wise - C# sample is a bit slower, but still can do real time tracking since its API is just a wrapper of the same face tracking engine that does most of the work. The interop and marshalling are not that expensive in this case.


    Thursday, May 24, 2012 7:09 PM
  • Unfortunately, the whole WPF sample assumes that the face tracking is used only for actively tracked skeletons. So you modification is not enough. You would need to modify plenty of code in this sample to make it working like you want. So I would recomend creating your own C# project and call into the interop for FT SDK directly if you want to use only color + depth.

    For example, in the function: OnAllFramesReady() in FaceTrackingViewer.xaml.cs we see the following loop:

      // Update the list of trackers and the trackers with the current frame information
                    foreach (Skeleton skeleton in this.skeletonData)
                    {
                        if (skeleton.TrackingState == SkeletonTrackingState.Tracked
                            || skeleton.TrackingState == SkeletonTrackingState.PositionOnly)
                        {
                            // We want keep a record of any skeleton, tracked or untracked.
                            if (!this.trackedSkeletons.ContainsKey(skeleton.TrackingId))
                            {
                                this.trackedSkeletons.Add(skeleton.TrackingId, new SkeletonFaceTracker());
                            }

                            // Give each tracker the upated frame.
                            SkeletonFaceTracker skeletonFaceTracker;
                            if (this.trackedSkeletons.TryGetValue(skeleton.TrackingId, out skeletonFaceTracker))
                            {
                                skeletonFaceTracker.OnFrameReady(this.Kinect, colorImageFormat, colorImage, depthImageFormat, depthImage, skeleton);
                                skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
                            }
                        }
                    }

    The method OnFrameReady() will then call the face tracker. But it will happen only when there is an active skeleton to track.

    You would need to modify this code to call the face tracker even when there is not active skeleton. It was not the original goal of the sample and so it was not done that way :-)

    Also, to call the face tracker with only color and depth you need to call this method from FTInterop.cs:

    int StartTracking(ref FaceTrackingSensorData sensorData, ref Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    and pass the region of interest (roi) where you think the face is located. If you don't know, you can pass the Rect for the full frame, i.e. (0,0,639,479).

    You can also modify this interop, so it matches native C++ COM API better, meaning the Rect parameter can actually be passed as NULL to the native API, in which case the FT API will search the whole RGB frame for a face to start tracking. To do this you would need to modify the interop this way:

    [PreserveSig]
    int StartTracking(ref FaceTrackingSensorData sensorData, Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    note that "ref Rect roi" is now defined as "Rect roi" so you can pass null.

    you also need to redefine Rect structure as a class this way (so you can pass null as roi parameter):

    [StructLayout(LayoutKind.Sequential)]
    public class Rect
    {
        public Rect() { }
        ...
    }

    Now, you can call the face tracker this way:

    hr = this.faceTrackerInteropPtr.StartTracking(ref faceTrackSensorData, null, null, this.frame.ResultPtr); // starts

    hr = this.faceTrackerInteropPtr.ContinueTracking(ref faceTrackSensorData, null, this.frame.ResultPtr);  // continues

    where skeleton is null and region of interest is null too.

    Sorry, it does not look simple...

    • Marked as answer by MaDoss Thursday, May 31, 2012 3:34 PM
    Wednesday, May 30, 2012 8:02 PM

All replies

  • I did some others test: apparently, with the provided c# instructions, the tracker still search for the full skeleton, not the new partial one. So if you want to enable it you have to be standing.

    Do I forget something? Do I set something wrong?

    Wednesday, May 23, 2012 3:28 PM
  • The affect you are seeing is usually caused by delayed data which is compounded by the layers that are in use.

    As with any c++ to c# interop has some delay. Data has to be marshaled between c++ and managed and there is no guarntee of how fast this will happen. When using Kinect or any media api, you are dealing with very large datasets that must be delivered very quickly. Delays will result in poor quality or dropped data/frames.

    Additionally, with a managed runtime, you have to content with the Garbage Collector. If the GC runs this is going to cause more delays. There is no way to know when the GC will run and when it does, it freezes all managed threads. Your unmanaged code has no idea this happens, the result is non-deterministic behavior.

    A faster machine and managing your managed allocations can mitigate some of this, but if you need to deal in sub-millisecond timings it is best to be native.

    Wednesday, May 23, 2012 7:08 PM
  • Thanks for the reply Carmine, I can understand the delay int delivering information and data transmission, but the point is that the module appears to be searching for the full skeleton (confirmed by the "Default" marker on the skeletondata frame), not for the seated one. The proof is that when I stand up, the tracker starts right away, while in the c++ example (and routines) the tracker starts as soon I place my face in the FOV of the camera.

    • Proposed as answer by The Thinker Thursday, May 24, 2012 1:00 PM
    • Unproposed as answer by The Thinker Thursday, May 24, 2012 1:00 PM
    Thursday, May 24, 2012 8:56 AM
  • Their is a speed difference because of carmines factors above mentioned and if garabage collection happens while your application is running it gets tricky to get maximum frames. I would if you are wanting those extra frames to code in c++. 

    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/

    Thursday, May 24, 2012 1:02 PM
  • I probably expressed myself badly, I wanted to point out the fact that in the WPF example the facetracker only fires when almost the full body is available, while the c++ one starts right away.

    Am I the only one experiencing this kind of problem?

    Thursday, May 24, 2012 2:53 PM
  • I think that might be a difference between the unmanaged code (c++) and managed code (c#, vb.net). Managed code like carmine said above has be marshaled through unmanaged libraries and code to go to managed code so it has more steps which equals more time to wait for response using kinect.

    So c++ = little to no time for some tasks and managed code = a little bit to a fair amount of time to process commands.

    Their is no way around this it will have some lag in managed code.


    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/

    Thursday, May 24, 2012 3:21 PM
  • I think that might be a difference between the unmanaged code (c++) and managed code (c#, vb.net). Managed code like carmine said above has be marshaled through unmanaged libraries and code to go to managed code so it has more steps which equals more time to wait for response using kinect.

    It would be right if staring enough time at the kinect, framing only my head like the other example, would result in a detection, but this isn't happening to me. I have to stand and let myself letting almost my entire body entering the FOV of kinect: only at that point the facetracker starts.

    Again, I'm sorry if the question was misleading, but I'm wondering if the example in WPF (the only piece of code in c# showing how to operate with facetracking SDK) have some differences with the one c++.

    Thursday, May 24, 2012 3:55 PM
  • I think that might be a difference between the unmanaged code (c++) and managed code (c#, vb.net). Managed code like carmine said above has be marshaled through unmanaged libraries and code to go to managed code so it has more steps which equals more time to wait for response using kinect.

    It would be right if staring enough time at the kinect, framing only my head like the other example, would result in a detection, but this isn't happening to me. I have to stand and let myself letting almost my entire body entering the FOV of kinect: only at that point the facetracker starts.

    Again, I'm sorry if the question was misleading, but I'm wondering if the example in WPF (the only piece of code in c# showing how to operate with facetracking SDK) have some differences with the one c++.

    Your fine sorry if we blab we tend to need more information sometimes in order to help. :) 

    Yes i find that stepping out of view of kinect wheter to left or right side or very far back helps to reset data being used by the kinect and can cause the kinect to correctly start tracking you during debug.

    I think I will start playing around soon with kinect code for head tracking to make a helper app for 3d modeling artists doing dialog for their games so if thats your goal email me at: jefferycarlson@gmail.com because I will start a codeplex project soon and if you have a username over their send it to me in email and i will add you to the project as soon as I can.  Also, if using for security app also email your code and I will take a look at it but i would recommend c++ for login or autenication apps.

    The c++ sample will more then likely have some speed and other very minor differences.


    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/



    Thursday, May 24, 2012 4:36 PM
  • BTW, heres a nice article by kinect devs on how to use face tracking better:

    http://nsmoly.wordpress.com/2012/05/21/face-tracking-sdk-in-kinect-for-windows-1-5/


    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/

    Thursday, May 24, 2012 6:36 PM
  • The difference between C# sample and C++ sample is the following: FT SDK can track faces in 2 models: 1)based on color + depth + skeleton OR 2) color + depth only (no skeleton). The C++ sample uses both modes, so the active skeleton is not required to start tracking - it starts in color + depth mode only and then when a skeleton is available from Kinect (when you stand up) it switches to using color + depth + skeleton. The C# sample is limited to only color + depth + skeleton mode. See this code:

    FaceTrackFrameframe = this.faceTracker.Track(colorImageFormat, colorImage, depthImageFormat, depthImage, skeletonOfInterest);

    So when the skeleton is not passed it will fail for a given frame (even though the C# interop API provides a second mode with color + depth only, it is just not used in WPF app). So the delay is caused by the fact that Kinect is initialized with the full skeleton tracking (not seated) and when you start the sample app the skeletal tracking is not yet tracking your skeleton until you stand up. Performance wise - C# sample is a bit slower, but still can do real time tracking since its API is just a wrapper of the same face tracking engine that does most of the work. The interop and marshalling are not that expensive in this case.


    Thursday, May 24, 2012 7:09 PM
  • The difference between C# sample and C++ sample is the following: FT SDK can track faces in 2 models: 1)based on color + depth + skeleton OR 2) color + depth only (no skeleton). The C++ sample uses both modes, so the active skeleton is not required to start tracking - it starts in color + depth mode only and then when a skeleton is available from Kinect (when you stand up) it switches to using color + depth + skeleton. The C# sample is limited to only color + depth + skeleton mode. See this code:

    FaceTrackFrameframe = this.faceTracker.Track(colorImageFormat, colorImage, depthImageFormat, depthImage, skeletonOfInterest);

    So when the skeleton is not passed it will fail for a given frame (even though the C# interop API provides a second mode with color + depth only, it is just not used in WPF app). So the delay is caused by the fact that Kinect is initialized with the full skeleton tracking (not seated) and when you start the sample app the skeletal tracking is not yet tracking your skeleton until you stand up. Performance wise - C# sample is a bit slower, but still can do real time tracking since its API is just a wrapper of the same face tracking engine that does most of the work. The interop and marshalling are not that expensive in this case.


    Ah, so an active skeleton is required for c# and manged development. Can he initialize the sample with seated skeleton tracking or is it not possible?




    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/



    Thursday, May 24, 2012 7:35 PM
  • Yes, you can use the seated pipeline as well. The Face Tracking SDK only requires 2 head points that are the end of the head bone. If you don't have a skeleton you can use color + depth. It works in both C++ and in C# (if you call into interop). The provided C# sample though uses active standing skeleton, therefore you need to stand up for it to start tracking. You can modify this sample to use the seated skeleton or to use only color + depth...

    Friday, May 25, 2012 6:51 AM
  • I think for OP's sake can you demostrate in a few lines of code how to modify the sample op's using, only showing the part to change to seated skeleton tracking. Maybe yours would be more efficent then his method.


    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/



    Friday, May 25, 2012 2:30 PM
  • Thank you all for the replies, and it's right: only the tracking based on the skeleton is used in the WPF example, that's the reason of the difference.

    Two more question, though:

    1) I tried to modify the example, to use the tracking without the skeleton information, but the tracking is always failing, with or without the rectangle to focus. Here some fo the code:

    internal void OnFrameReady(KinectSensor kinectSensor, ColorImageFormat colorImageFormat, byte[] colorImage, DepthImageFormat depthImageFormat, short[] depthImage)
                {
                    if (this.faceTracker == null)
                    {
                        try
                        {
                            this.faceTracker = new Microsoft.Kinect.Toolkit.FaceTracking.FaceTracker(kinectSensor);
                        }
                        catch (InvalidOperationException)
                        {
                            // During some shutdown scenarios the FaceTracker
                            // is unable to be instantiated.  Catch that exception
                            // and don't track a face.
                            //Debug.WriteLine("AllFramesReady - creating a new FaceTracker threw an InvalidOperationException");
                            this.faceTracker = null;
                        }
                    }
    
                    if (this.faceTracker != null)
                    {
                        FaceTrackFrame frame = this.faceTracker.Track(
                            colorImageFormat,
                            colorImage,
                            depthImageFormat,
                            depthImage,
                            Microsoft.Kinect.Toolkit.FaceTracking.Rect.Empty);
                            //new Microsoft.Kinect.Toolkit.FaceTracking.Rect(100,100,500,400)); 
    
                        this.lastFaceTrackSucceeded = frame.TrackSuccessful;
                        if (this.lastFaceTrackSucceeded)
                        {
                            if (faceTriangles == null)
                            {
                                // only need to get this once.  It doesn't change.
                                faceTriangles = frame.GetTriangles();
                            }
    
                            this.facePointsProjected = frame.GetProjected3DShape();
    
                            this.rotationVector = frame.Rotation;
                            this.translationVector = frame.Translation;
                            this.faceRect = frame.FaceRect;
                            this.facepoints3D = frame.Get3DShape();
                        }
                    }
                }
    Frame is never succesful.

    2) Why is the skeletonframe listing the tracking mode as 'DEFAULT' even if the kinect sensor is set as "Seated"? Is it normal?


    • Edited by MaDoss Friday, May 25, 2012 2:32 PM
    Friday, May 25, 2012 2:31 PM
  • You would'nt be using xbox kinect during debug would you? I dont remember if seated mode worked with a xbox kinect during debug so it might be switching it back to normal because it doesnt support seated for xbox kinect.  Also, try making the person active initially to see if the face tracking mode requires the skeleton active.  make sure your standing up when you first start face tracking then sit down after it recognizes your skeleton as active (in v1 it was a light blue color in the depth stream during debug).  

    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/





    Friday, May 25, 2012 2:39 PM
  • You would'nt be using xbox kinect during debug would you? I dont remember if seated mode worked with a xbox kinect during debug so it might be switching it back to normal because it doesnt support seated for xbox kinect.  Also, try making the person active initially to see if the face tracking mode requires the skeleton active.  make sure your standing up when you first start face tracking then sit down after it recognizes your skeleton as active (in v1 it was a light blue color in the depth stream during debug). 

    No, I'm using the kinect for windows. The point is letting the WPF example work as the c++ one is working, with the tracking starting right away: my target is having an application for a seated user , not standing and seating: I already know the skeleton-based way to track is working well, I'd like to use the other way.

    Friday, May 25, 2012 2:48 PM
  • Well I always have had a second or two delay in vb.net or c# compared to c++ in v1 with skeleton tracking (have v1.5 on my home computer and still playing around with it). Make sure to show us the initialization sub when you use skeleton tracking enabled for the face because we can better help you instead of guessing what your doing.

    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/


    Friday, May 25, 2012 3:28 PM
  • Well I always have had a second or two delay in vb.net or c# compared to c++ in v1 with skeleton tracking (have v1.5 on my home computer and still playing around with it). Make sure to show us the initialization sub when you use skeleton tracking enabled for the face because we can better help you instead of guessing what your doing.

    Yep, sorry for that, I constantly forget to post code.

    This is the initialization of the sensor:

    private void SensorChooserOnKinectChanged(object sender, KinectChangedEventArgs kinectChangedEventArgs)
            {
                KinectSensor oldSensor = kinectChangedEventArgs.OldSensor;
                KinectSensor newSensor = kinectChangedEventArgs.NewSensor;
    
                if (oldSensor != null)
                {
                    oldSensor.AllFramesReady -= KinectSensorOnAllFramesReady;
                    oldSensor.ColorStream.Disable();
                    oldSensor.DepthStream.Disable();
                    oldSensor.DepthStream.Range = DepthRange.Near;
                    oldSensor.SkeletonStream.Disable();
                    oldSensor.SkeletonStream.EnableTrackingInNearRange = true;
                    oldSensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
                }
    
                if (newSensor != null)
                {
                    try
                    {
                        newSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
                        newSensor.DepthStream.Enable(DepthImageFormat.Resolution320x240Fps30);
                        try
                        {
                            // This will throw on non Kinect For Windows devices.
                            newSensor.DepthStream.Range = DepthRange.Near;
                            newSensor.SkeletonStream.EnableTrackingInNearRange = true;
                        }
                        catch (InvalidOperationException)
                        {
                            newSensor.DepthStream.Range = DepthRange.Default;
                            newSensor.SkeletonStream.EnableTrackingInNearRange = false;
                        }
    
                        newSensor.SkeletonStream.AppChoosesSkeletons = true;
                        newSensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
                        newSensor.SkeletonStream.Enable();
                        newSensor.AllFramesReady += KinectSensorOnAllFramesReady;
                    }
                    catch (InvalidOperationException)
                    {
                        // This exception can be thrown when we are trying to
                        // enable streams on a device that has gone away.  This
                        // can occur, say, in app shutdown scenarios when the sensor
                        // goes away between the time it changed status and the
                        // time we get the sensor changed notification.
                        //
                        // Behavior here is to just eat the exception and assume
                        // another notification will come along if a sensor
                        // comes back.
                    }
                }
            }

    Here's the processing code, just like the example, but using the function mentioned above.

     private void OnAllFramesReady(object sender, AllFramesReadyEventArgs allFramesReadyEventArgs)
            {
                ColorImageFrame colorImageFrame = null;
                DepthImageFrame depthImageFrame = null;
                SkeletonFrame skeletonFrame = null;
    
                try
                {
                    colorImageFrame = allFramesReadyEventArgs.OpenColorImageFrame();
                    depthImageFrame = allFramesReadyEventArgs.OpenDepthImageFrame();
                    skeletonFrame = allFramesReadyEventArgs.OpenSkeletonFrame();
    
                    if (colorImageFrame == null || depthImageFrame == null || skeletonFrame == null)
                    {
                        return;
                    }
    
                    // Check for image format changes.  The FaceTracker doesn't
                    // deal with that so we need to reset.
                    if (this.depthImageFormat != depthImageFrame.Format)
                    {
                        this.ResetFaceTracking();
                        this.depthImage = null;
                        this.depthImageFormat = depthImageFrame.Format;
                    }
    
                    if (this.colorImageFormat != colorImageFrame.Format)
                    {
                        this.ResetFaceTracking();
                        this.colorImage = null;
                        this.colorImageFormat = colorImageFrame.Format;
                    }
    
                    // Create any buffers to store copies of the data we work with
                    if (this.depthImage == null)
                    {
                        this.depthImage = new short[depthImageFrame.PixelDataLength];
                    }
    
                    if (this.colorImage == null)
                    {
                        this.colorImage = new byte[colorImageFrame.PixelDataLength];
                    }
                    
                    // Get the skeleton information
                    if (this.skeletonData == null || this.skeletonData.Length != skeletonFrame.SkeletonArrayLength)
                    {
                        this.skeletonData = new Skeleton[skeletonFrame.SkeletonArrayLength];
                    }
    
                    colorImageFrame.CopyPixelDataTo(this.colorImage);
                    depthImageFrame.CopyPixelDataTo(this.depthImage);
                    skeletonFrame.CopySkeletonDataTo(this.skeletonData);
    
                    // Update the list of trackers and the trackers with the current frame information
                    bool skeletonTracked = false;
                    foreach (Skeleton skeleton in this.skeletonData)
                    {
                        if (skeleton.TrackingState == SkeletonTrackingState.Tracked
                            || skeleton.TrackingState == SkeletonTrackingState.PositionOnly)
                        {
                            // We want keep a record of any skeleton, tracked or untracked.
                            if (!this.trackedSkeletons.ContainsKey(skeleton.TrackingId))
                            {
                                this.trackedSkeletons.Add(skeleton.TrackingId, new SkeletonFaceTracker());
                            }
    
                            // Give each tracker the upated frame.
                            SkeletonFaceTracker skeletonFaceTracker;
                            if (this.trackedSkeletons.TryGetValue(skeleton.TrackingId, out skeletonFaceTracker))
                            {
                                skeletonFaceTracker.OnFrameReady(this.Kinect, 
                                    colorImageFormat, 
                                    colorImage, 
                                    depthImageFormat, 
                                    depthImage, 
                                    skeleton);
                                skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
                                skeletonTracked = true;
                            }
                        }
                    }
    
                    if (skeletonTracked == false)
                    {
                        int id = 997;
                        if (!this.trackedSkeletons.ContainsKey(id))
                        {
                            this.trackedSkeletons.Add(id, new SkeletonFaceTracker());
                        }
    
                        // Give each tracker the upated frame.
                        SkeletonFaceTracker skeletonFaceTracker;
                        if (this.trackedSkeletons.TryGetValue(id, out skeletonFaceTracker))
                        {
                            skeletonFaceTracker.OnFrameReady(this.Kinect,
                                colorImageFormat,
                                colorImage,
                                depthImageFormat,
                                depthImage);
                            skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
                            skeletonTracked = true;
                        }
                    }
    
                    this.RemoveOldTrackers(skeletonFrame.FrameNumber);
    
                    this.InvalidateVisual();
                }
                finally
                {
                    if (colorImageFrame != null)
                    {
                        colorImageFrame.Dispose();
                    }
    
                    if (depthImageFrame != null)
                    {
                        depthImageFrame.Dispose();
                    }
    
                    if (skeletonFrame != null)
                    {
                        skeletonFrame.Dispose();
                    }
                }
            }

    Friday, May 25, 2012 3:37 PM
  • Well I always have had a second or two delay in vb.net or c# compared to c++ in v1 with skeleton tracking (have v1.5 on my home computer and still playing around with it). Make sure to show us the initialization sub when you use skeleton tracking enabled for the face because we can better help you instead of guessing what your doing.

    Yep, sorry for that, I constantly forget to post code.

    This is the initialization of the sensor:

    private void SensorChooserOnKinectChanged(object sender, KinectChangedEventArgs kinectChangedEventArgs)
            {
                KinectSensor oldSensor = kinectChangedEventArgs.OldSensor;
                KinectSensor newSensor = kinectChangedEventArgs.NewSensor;
    
                if (oldSensor != null)
                {
                    oldSensor.AllFramesReady -= KinectSensorOnAllFramesReady;
                    oldSensor.ColorStream.Disable();
                    oldSensor.DepthStream.Disable();
                    oldSensor.DepthStream.Range = DepthRange.Near;
                    oldSensor.SkeletonStream.Disable();
                    oldSensor.SkeletonStream.EnableTrackingInNearRange = true;
                    oldSensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
                }
    
                if (newSensor != null)
                {
                    try
                    {
                        newSensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
                        newSensor.DepthStream.Enable(DepthImageFormat.Resolution320x240Fps30);
                        try
                        {
                            // This will throw on non Kinect For Windows devices.
                            newSensor.DepthStream.Range = DepthRange.Near;
                            newSensor.SkeletonStream.EnableTrackingInNearRange = true;
                        }
                        catch (InvalidOperationException)
                        {
                            newSensor.DepthStream.Range = DepthRange.Default;
                            newSensor.SkeletonStream.EnableTrackingInNearRange = false;
                        }
    
                        newSensor.SkeletonStream.AppChoosesSkeletons = true;
                        newSensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
                        newSensor.SkeletonStream.Enable();
                        newSensor.AllFramesReady += KinectSensorOnAllFramesReady;
                    }
                    catch (InvalidOperationException)
                    {
                        // This exception can be thrown when we are trying to
                        // enable streams on a device that has gone away.  This
                        // can occur, say, in app shutdown scenarios when the sensor
                        // goes away between the time it changed status and the
                        // time we get the sensor changed notification.
                        //
                        // Behavior here is to just eat the exception and assume
                        // another notification will come along if a sensor
                        // comes back.
                    }
                }
            }

    Here's the processing code, just like the example, but using the function mentioned above.

     private void OnAllFramesReady(object sender, AllFramesReadyEventArgs allFramesReadyEventArgs)
            {
                ColorImageFrame colorImageFrame = null;
                DepthImageFrame depthImageFrame = null;
                SkeletonFrame skeletonFrame = null;
    
                try
                {
                    colorImageFrame = allFramesReadyEventArgs.OpenColorImageFrame();
                    depthImageFrame = allFramesReadyEventArgs.OpenDepthImageFrame();
                    skeletonFrame = allFramesReadyEventArgs.OpenSkeletonFrame();
    
                    if (colorImageFrame == null || depthImageFrame == null || skeletonFrame == null)
                    {
                        return;
                    }
    
                    // Check for image format changes.  The FaceTracker doesn't
                    // deal with that so we need to reset.
                    if (this.depthImageFormat != depthImageFrame.Format)
                    {
                        this.ResetFaceTracking();
                        this.depthImage = null;
                        this.depthImageFormat = depthImageFrame.Format;
                    }
    
                    if (this.colorImageFormat != colorImageFrame.Format)
                    {
                        this.ResetFaceTracking();
                        this.colorImage = null;
                        this.colorImageFormat = colorImageFrame.Format;
                    }
    
                    // Create any buffers to store copies of the data we work with
                    if (this.depthImage == null)
                    {
                        this.depthImage = new short[depthImageFrame.PixelDataLength];
                    }
    
                    if (this.colorImage == null)
                    {
                        this.colorImage = new byte[colorImageFrame.PixelDataLength];
                    }
                    
                    // Get the skeleton information
                    if (this.skeletonData == null || this.skeletonData.Length != skeletonFrame.SkeletonArrayLength)
                    {
                        this.skeletonData = new Skeleton[skeletonFrame.SkeletonArrayLength];
                    }
    
                    colorImageFrame.CopyPixelDataTo(this.colorImage);
                    depthImageFrame.CopyPixelDataTo(this.depthImage);
                    skeletonFrame.CopySkeletonDataTo(this.skeletonData);
    
                    // Update the list of trackers and the trackers with the current frame information
                    bool skeletonTracked = false;
                    foreach (Skeleton skeleton in this.skeletonData)
                    {
                        if (skeleton.TrackingState == SkeletonTrackingState.Tracked
                            || skeleton.TrackingState == SkeletonTrackingState.PositionOnly)
                        {
                            // We want keep a record of any skeleton, tracked or untracked.
                            if (!this.trackedSkeletons.ContainsKey(skeleton.TrackingId))
                            {
                                this.trackedSkeletons.Add(skeleton.TrackingId, new SkeletonFaceTracker());
                            }
    
                            // Give each tracker the upated frame.
                            SkeletonFaceTracker skeletonFaceTracker;
                            if (this.trackedSkeletons.TryGetValue(skeleton.TrackingId, out skeletonFaceTracker))
                            {
                                skeletonFaceTracker.OnFrameReady(this.Kinect, 
                                    colorImageFormat, 
                                    colorImage, 
                                    depthImageFormat, 
                                    depthImage, 
                                    skeleton);
                                skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
                                skeletonTracked = true;
                            }
                        }
                    }
    
                    if (skeletonTracked == false)
                    {
                        int id = 997;
                        if (!this.trackedSkeletons.ContainsKey(id))
                        {
                            this.trackedSkeletons.Add(id, new SkeletonFaceTracker());
                        }
    
                        // Give each tracker the upated frame.
                        SkeletonFaceTracker skeletonFaceTracker;
                        if (this.trackedSkeletons.TryGetValue(id, out skeletonFaceTracker))
                        {
                            skeletonFaceTracker.OnFrameReady(this.Kinect,
                                colorImageFormat,
                                colorImage,
                                depthImageFormat,
                                depthImage);
                            skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
                            skeletonTracked = true;
                        }
                    }
    
                    this.RemoveOldTrackers(skeletonFrame.FrameNumber);
    
                    this.InvalidateVisual();
                }
                finally
                {
                    if (colorImageFrame != null)
                    {
                        colorImageFrame.Dispose();
                    }
    
                    if (depthImageFrame != null)
                    {
                        depthImageFrame.Dispose();
                    }
    
                    if (skeletonFrame != null)
                    {
                        skeletonFrame.Dispose();
                    }
                }
            }

    This the the part that might cause the problem:

    newSensor.DepthStream.Range = DepthRange.Default;
                            newSensor.SkeletonStream.EnableTrackingInNearRange = false;


    Try actually logging the exception error to file or use msgbox instead and then exit the sub so it will continue running the program.

    In addition, their should be a debug or XBOX_KINECT pre-processor directive you can use for this code so you use the above code only for kinect for xbox device above instead of on errors:

    #if DEBUG newSensor.DepthStream.Range = DepthRange.Near; newSensor.SkeletonStream.EnableTrackingInNearRange = false; #end if

    Can someone confirm the xbox kinect pre-processor directive? I think their would be one with how windows phone has them sometimes for debugging purposes and using the emulator.

    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/




    Friday, May 25, 2012 3:44 PM
  • This the the part that might cause the problem:

    newSensor.DepthStream.Range = DepthRange.Default;
                            newSensor.SkeletonStream.EnableTrackingInNearRange = false;

    Try actually logging the exception error to file or use msgbox instead and then exit the sub so it will continue running the program.

    In addition, their should be a debug processor directive you can use for this code so you use the above code only for kinect for xbox device above instead of on errors:

    #if DEBUG
    newSensor.DepthStream.Range = DepthRange.Near;
                            newSensor.SkeletonStream.EnableTrackingInNearRange = false;
    #end if


    The code is never entering that section of code: as I said, I'm working on a Kinect for Windows device, so it's set properly, as you can see from the runtime information:

    SkeletonStream    {Microsoft.Kinect.SkeletonStream}    Microsoft.Kinect.SkeletonStream
    AppChoosesSkeletons    true    bool
    EnableTrackingInNearRange    true    bool
    FrameSkeletonArrayLength    6    int
    IsEnabled    true    bool
    IsSmoothingEnabled    false    bool
    SmoothParameters    Smoothing:0.0 Correction:0.0 Prediction:0.0 JitterRadius:0.0 MaxDeviationRadius:0.0    Microsoft.Kinect.TransformSmoothParameters
    TrackingMode    Seated    Microsoft.Kinect.SkeletonTrackingMode

    DepthStream    {Microsoft.Kinect.DepthImageStream}    Microsoft.Kinect.DepthImageStream
    base    {Microsoft.Kinect.DepthImageStream}    Microsoft.Kinect.ImageStream {Microsoft.Kinect.DepthImageStream}
    Format    Resolution320x240Fps30    Microsoft.Kinect.DepthImageFormat
    MaxDepth    3000    int
    MinDepth    400    int
    NominalDiagonalFieldOfView    70.0    float
    NominalFocalLengthInPixels    285.63    float
    NominalHorizontalFieldOfView    58.5    float
     NominalInverseFocalLengthInPixels    0.00350103271    float
    NominalVerticalFieldOfView    45.6    float
    Range    Near    Microsoft.Kinect.DepthRange
    TooFarDepth    4095    int
    TooNearDepth    0    int
    UnknownDepth    -1    int




    Friday, May 25, 2012 4:01 PM
  • I feel the same way :| sometimes about this kinect stuff its so frustrating because you find these weird bugs as your playing around and then the bug makes a simple task take forever because you have to take a long workaround.

    Sorry if i got the preprocessor directive wrong but I think for purposes of debugging and testing they would be in their even in managed code environements so if someone tests with a xbox kinect they dont have failure with the code. But i think the debug one is built into visual studio or express version of c# and visual basic but i think just visual studio. 

    Also, try changing the old sensor to default tracking mode and false on enabletrackinginnearrange once you stop using it:

     if (oldSensor != null)
                {
                    oldSensor.AllFramesReady -= KinectSensorOnAllFramesReady;
                    oldSensor.ColorStream.Disable();
                    oldSensor.DepthStream.Disable();
                    oldSensor.DepthStream.Range = DepthRange.Near;
                    oldSensor.SkeletonStream.Disable();
                    oldSensor.SkeletonStream.EnableTrackingInNearRange = true;
                    oldSensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
                }
    I dont have anything else i can try but those two sections because the other code looks fine to me. Just email me the whole solution if you need personalized help when i finished my kinectmultipoint project I will start looking at your code. I need to at least upgrade the kinectmultipoint project to v1.5 but iam trying to properly dispose of the emulated devices when I dont need them.

    Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. - "Sherlock holmes" "speak softly and carry a big stick" - theodore roosevelt. Fear leads to anger, anger leads to hate, hate leads to suffering - Yoda. Blog - http://jefferycarlsonblog.blogspot.com/







    Friday, May 25, 2012 4:44 PM
  • I tried modifying some code, but no luck: I cannot start a no-skeleton face tracker in managed code:

    if (this.faceTracker != null)
                    {
                        FaceTrackFrame frame = this.faceTracker.Track(
                            colorImageFormat,
                            colorImage,
                            depthImageFormat,
                            depthImage,
                            Microsoft.Kinect.Toolkit.FaceTracking.Rect.Empty);
                            //new Microsoft.Kinect.Toolkit.FaceTracking.Rect(100,100,500,400)); 
    
                        this.lastFaceTrackSucceeded = frame.TrackSuccessful;
    TrackSuccessful is always false. Any clues?

    Monday, May 28, 2012 1:30 PM
  • Unfortunately, the whole WPF sample assumes that the face tracking is used only for actively tracked skeletons. So you modification is not enough. You would need to modify plenty of code in this sample to make it working like you want. So I would recomend creating your own C# project and call into the interop for FT SDK directly if you want to use only color + depth.

    For example, in the function: OnAllFramesReady() in FaceTrackingViewer.xaml.cs we see the following loop:

      // Update the list of trackers and the trackers with the current frame information
                    foreach (Skeleton skeleton in this.skeletonData)
                    {
                        if (skeleton.TrackingState == SkeletonTrackingState.Tracked
                            || skeleton.TrackingState == SkeletonTrackingState.PositionOnly)
                        {
                            // We want keep a record of any skeleton, tracked or untracked.
                            if (!this.trackedSkeletons.ContainsKey(skeleton.TrackingId))
                            {
                                this.trackedSkeletons.Add(skeleton.TrackingId, new SkeletonFaceTracker());
                            }

                            // Give each tracker the upated frame.
                            SkeletonFaceTracker skeletonFaceTracker;
                            if (this.trackedSkeletons.TryGetValue(skeleton.TrackingId, out skeletonFaceTracker))
                            {
                                skeletonFaceTracker.OnFrameReady(this.Kinect, colorImageFormat, colorImage, depthImageFormat, depthImage, skeleton);
                                skeletonFaceTracker.LastTrackedFrame = skeletonFrame.FrameNumber;
                            }
                        }
                    }

    The method OnFrameReady() will then call the face tracker. But it will happen only when there is an active skeleton to track.

    You would need to modify this code to call the face tracker even when there is not active skeleton. It was not the original goal of the sample and so it was not done that way :-)

    Also, to call the face tracker with only color and depth you need to call this method from FTInterop.cs:

    int StartTracking(ref FaceTrackingSensorData sensorData, ref Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    and pass the region of interest (roi) where you think the face is located. If you don't know, you can pass the Rect for the full frame, i.e. (0,0,639,479).

    You can also modify this interop, so it matches native C++ COM API better, meaning the Rect parameter can actually be passed as NULL to the native API, in which case the FT API will search the whole RGB frame for a face to start tracking. To do this you would need to modify the interop this way:

    [PreserveSig]
    int StartTracking(ref FaceTrackingSensorData sensorData, Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    note that "ref Rect roi" is now defined as "Rect roi" so you can pass null.

    you also need to redefine Rect structure as a class this way (so you can pass null as roi parameter):

    [StructLayout(LayoutKind.Sequential)]
    public class Rect
    {
        public Rect() { }
        ...
    }

    Now, you can call the face tracker this way:

    hr = this.faceTrackerInteropPtr.StartTracking(ref faceTrackSensorData, null, null, this.frame.ResultPtr); // starts

    hr = this.faceTrackerInteropPtr.ContinueTracking(ref faceTrackSensorData, null, this.frame.ResultPtr);  // continues

    where skeleton is null and region of interest is null too.

    Sorry, it does not look simple...

    • Marked as answer by MaDoss Thursday, May 31, 2012 3:34 PM
    Wednesday, May 30, 2012 8:02 PM
  • Thanks Nicolai

    but i think it is not a big diffrent initial face detecting time...

    it takes 30 sec more for me

    but c++ sample detect within 3 sec always....



    Thursday, May 31, 2012 1:37 AM
  • Thanks Nicolai

    but i think it is not a big diffrent initial face detecting time...

    it takes 30 sec more for me

    but c++ sample detect within 3 sec always....

     With the changes suggested by Nikolai, it works more or less the same way to me, it's a little bit slower, but the reason of the increased processing time has been already discussed in previous post in the same thread. I notices it takes longer on slow processor though (I have it on two different PCs).
    Thursday, May 31, 2012 3:36 PM
  • 2) Why is the skeletonframe listing the tracking mode as 'DEFAULT' even if the kinect sensor is set as "Seated"? Is it normal?


    This part is a known bug in the Kinect for Windows runtime. The SkeletonFrame.TrackingMode property always has a value of Default, regardless of the state of SkeletonStream.TrackingMode. You should not rely on SkeletonFrame.TrackingMode to give you a meaningful value. An alternative way to detect whether the current frame was captured in seated mode is to examine the tracking state of any lower-body joint (e.g., spine, hip center, etc.) in one of the tracked skeletons. If its state is NotTracked, then you know that the frame was captured in seated mode.

    John
    K4W Dev

    Thursday, July 12, 2012 4:56 PM
  • I am having a similar problem with this one too can any one tell me why Error.status is always give me a failed response please.

    Status is always "HeadSearchFailed" hence 

     if (this.faceTracker != null)
                    {
                        FaceTrackFrame frame = this.faceTracker.Track(
                            colorImageFormat, colorImage, depthImageFormat, depthImage, skeletonOfInterest);
    // no frame is never successful why ?
     this.lastFaceTrackSucceeded =    frame.TrackSuccessful;
                        System.Diagnostics.Debug.Write(lastFaceTrackSucceeded);
    
    // here lastfacetracking always false ? donno why 
                        if (this.lastFaceTrackSucceeded)
                        {
                            if (faceTriangles == null)
                            {
      // only need to get this once.  It doesn't change.
         faceTriangles = frame.GetTriangles();
                            }
    
         this.facePoints = frame.GetProjected3DShape();
                        }
                    }
    
    // please any one help with this thank you.



    Tuesday, July 31, 2012 3:54 PM
  • Hello everyone!

    I successfully got the face tracking WPF to work in C# completely without the skeleton! It runs much faster and even seems to track much better than with the skeleton. Here is how to make it work:

    First, follow the instructions that Nikolai Smolyanskiy set forth earlier in this thread.

    Once that is done, do the following:

    1. In the FaceTrackingViewer class, add the following properties:

            private Int32 trackingID = 0;
            private int frameNumber = 0;

    2.  Change your OnAllFramesReady(object sender, ...) method to the following:

    private void OnAllFramesReady(object sender, AllFramesReadyEventArgs allFramesReadyEventArgs)
            {
                ColorImageFrame colorImageFrame = null;
                DepthImageFrame depthImageFrame = null;
    
                try
                {
                    depthImageFrame = allFramesReadyEventArgs.OpenDepthImageFrame();
                    colorImageFrame = allFramesReadyEventArgs.OpenColorImageFrame();
    
                    if (depthImageFrame == null)
                    {
                        return;
                    }
    
                    if (colorImageFrame == null)
                    {
                        return;
                    }
    
                    // Check for image format changes.  The FaceTracker doesn't
                    // deal with that so we need to reset.
                    if (this.depthImageFormat != depthImageFrame.Format)
                    {
                        this.ResetFaceTracking();
                        this.depthImage = null;
                        this.depthImageFormat = depthImageFrame.Format;
                    }
    
                    if (this.colorImageFormat != colorImageFrame.Format)
                    {
                        this.ResetFaceTracking();
                        this.colorImage = null;
                        this.colorImageFormat = colorImageFrame.Format;
                    }
    
                    // Create any buffers to store copies of the data we work with
                    if (this.depthImage == null)
                    {
                        this.depthImage = new short[depthImageFrame.PixelDataLength];
                    }
    
                    if (this.colorImage == null)
                    {
                        this.colorImage = new byte[colorImageFrame.PixelDataLength];
                    }
    
                    depthImageFrame.CopyPixelDataTo(this.depthImage);
                    colorImageFrame.CopyPixelDataTo(this.colorImage);
    
                    if (!this.trackedSkeletons.ContainsKey(this.trackingID))
                    {
                        this.trackedSkeletons.Add(this.trackingID, new SkeletonFaceTracker());
                    }
                    
                    // Give each tracker the upated frame.
                    SkeletonFaceTracker skeletonFaceTracker;
                    if (this.trackedSkeletons.TryGetValue(this.trackingID, out skeletonFaceTracker))
                    {
                        skeletonFaceTracker.OnFrameReady(this.Kinect, colorImageFormat, colorImage, depthImageFormat, depthImage);
                        skeletonFaceTracker.LastTrackedFrame = this.frameNumber;
                    }
    
                    this.RemoveOldTrackers(this.frameNumber);
    
                    this.InvalidateVisual();
                }
    
                finally
                {
                    if (depthImageFrame != null)
                    {
                        depthImageFrame.Dispose();
                    }
    
                    if (colorImageFrame != null)
                    {
                        colorImageFrame.Dispose();
                    }
                }
            }


    3. Change the OnFrameReady Method in the SkeletonFaceTracker class to the following:

                internal void OnFrameReady(KinectSensor kinectSensor, ColorImageFormat colorImageFormat, byte[] colorImage, DepthImageFormat depthImageFormat, short[] depthImage)
                {
                    if (this.faceTracker == null)
                    {
                        try
                        {
                            this.faceTracker = new FaceTracker(kinectSensor);
                        }
                        catch (InvalidOperationException)
                        {
                            // During some shutdown scenarios the FaceTracker
                            // is unable to be instantiated.  Catch that exception
                            // and don't track a face.
                            Debug.WriteLine("AllFramesReady - creating a new FaceTracker threw an InvalidOperationException");
                            this.faceTracker = null;
                        }
                    }
    
                    if (this.faceTracker != null)
                    {
                        FaceTrackFrame frame = this.faceTracker.Track(
                            colorImageFormat, colorImage, depthImageFormat, depthImage);
    
                        this.lastFaceTrackSucceeded = frame.TrackSuccessful;
                        if (this.lastFaceTrackSucceeded)
                        {
                            if (faceTriangles == null)
                            {
                                // only need to get this once.  It doesn't change.
                                faceTriangles = frame.GetTriangles();
                            }
    
                            this.facePoints = frame.GetProjected3DShape();
                        }
                    }
                }


    4. Change the following code in the DrawFaceModel method in the SkeletonFaceTracker:

    if (!this.lastFaceTrackSucceeded || this.skeletonTrackingState != SkeletonTrackingState.Tracked)
                    {
                        return;
                    }

    to the following:

    if (!this.lastFaceTrackSucceeded)
                    {
                        return;
                    }

    5. Now go to MainWindow.xaml.cs and you can remove the following 2 lines from SensorChooserOnKinectChanged method:

    newSensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
    newSensor.SkeletonStream.Enable();

    That's it! The face tracker should now be tracking without depending on the skeleton. You should see a huge improvement in terms of speed.

    Again, I must emphasize this:

    Make sure you followed Nikolai Smolyanskiy's steps thoroughly before making these changes. My additions assume that you followed his steps first.

    Let me know if you have any questions. Hope this helped!




    • Edited by Shu Das Tuesday, August 14, 2012 9:52 PM
    • Proposed as answer by Gary_Lewis Tuesday, August 21, 2012 9:38 PM
    Tuesday, August 14, 2012 6:40 PM
  • Shu, would you pretty, pretty please post your whole modified and working project over on CodeProject or similar so that we can all benefit from your hard work instead of each needing to execute a series of complex steps on an already unfamiliar project's source. 
    Sunday, August 19, 2012 1:22 AM
  • Unfortunately, I do not have the code with me. I worked in a lab and the code is on their computer. I'm in college now and I don't have access to the code. But I guarantee that the modifications I made, work.

    Thursday, September 27, 2012 1:28 AM
  • Dear Shu Das, I followed your tutorial step by step, but at the end I couldn't track a face without skeleton. FaceTracker.Track() just doesn't recognize a face and keeps returning FALSE in faceFrame.TrackSuccessful. Here is how I call the Track() method:

    void kinectSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
    {            
        using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
        {
            if (colorImageFrame == null)
                return;
            colorImageFrame.CopyPixelDataTo(colorPixelData);
        }
    
        using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
        {
            if (depthImageFrame == null)
                return;
            depthImageFrame.CopyPixelDataTo(depthPixelData);
        }              
    
        faceFrame = faceTracker.Track(kinectSensor.ColorStream.Format, colorPixelData,
                                    kinectSensor.DepthStream.Format, depthPixelData);
    
        if (faceFrame.TrackSuccessful)
        {
            systemStatus = Status.TrackingFace;
            GetAnimationAmplutide();
        }
        else
        {
            systemStatus = Status.MissingFace;
        }
    }

    Am I missing something?

    Tuesday, October 2, 2012 7:06 PM
  • EDIT: I got it!

    Open a fresh install of the FaceTrackingBasics-WPF solution.
    From Nikolai Smolyanskiy's post:

    Open FaceTracker.cs, go to the Track function (very bottom) and change the following:

                if (this.trackSucceeded)
                {
                    hr = this.faceTrackerInteropPtr.ContinueTracking(ref faceTrackSensorData, headPointsObj, this.frame.ResultPtr);
                }
                else
                {
                    hr = this.faceTrackerInteropPtr.StartTracking(
                        ref faceTrackSensorData, ref regionOfInterest, headPointsObj, this.frame.ResultPtr);
                }

    changes to:

                if (this.trackSucceeded)
                {
                    hr = this.faceTrackerInteropPtr.ContinueTracking(ref faceTrackSensorData, null, this.frame.ResultPtr);
                }
                else
                {
                    hr = this.faceTrackerInteropPtr.StartTracking(
                        ref faceTrackSensorData, null, null, this.frame.ResultPtr);
                }

    Open FtInterop.cs:

    	[PreserveSig]
            int StartTracking(ref FaceTrackingSensorData sensorData, ref Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    changes to:

            [PreserveSig]
            int StartTracking(ref FaceTrackingSensorData sensorData, Rect roi, HeadPoints headPoints, IFTResult faceTrackResult);

    Open Utils.cs:

        [DebuggerDisplay("(l={Left},t={Top},r={Right},b={Bottom})")]
        [StructLayout(LayoutKind.Sequential)]
        public struct Rect
        {
            public Rect(int left, int top, int right, int bottom) : this()
            {
                Left = left;
                Top = top;
                Right = right;
                Bottom = bottom;
            }

    changes to:

        [DebuggerDisplay("(l={Left},t={Top},r={Right},b={Bottom})")]
        [StructLayout(LayoutKind.Sequential)]
        public class Rect
        {
            public Rect() { }
    
            public Rect(int left, int top, int right, int bottom) : this()
            {
                Left = left;
                Top = top;
                Right = right;
                Bottom = bottom;
            }

    Now, follow the instructions given by Shu Das and it should work!

    Note: I found the StartTracking function to be very computationally intensive (my laptop can't play the video stream while running StartTracking), but as soon as it finds a face and starts using ContinueTracking it seems to be fine.


    • Edited by xxzziioonn Tuesday, December 18, 2012 6:14 AM
    • Proposed as answer by xxzziioonn Tuesday, December 18, 2012 6:30 AM
    Tuesday, December 18, 2012 5:37 AM
  • I follow all the above instructions to track without skeleton in c# and everything works fine.

    However when I try to get the FaceRect in the OnFrameReady Method in the SkeletonFaceTracker class:

                    

    ...............

    ................

    if (this.faceTracker != null) { FaceTrackFrame frame = this.faceTracker.Track( colorImageFormat, colorImage, depthImageFormat, depthImage); this.lastFaceTrackSucceeded = frame.TrackSuccessful; if (this.lastFaceTrackSucceeded) { if (faceTriangles == null) { // only need to get this once. It doesn't change. faceTriangles = frame.GetTriangles(); } this.facePoints = frame.GetProjected3DShape(); Microsoft.Kinect.Toolkit.FaceTracking.Rect faceRect = frame.FaceRect; } }

    ...............

    ...............

    I get the following exception:

    An unhandled exception of type 'System.NullReferenceException' occurred in Microsoft.Kinect.Toolkit.FaceTracking.dll

    Additional information: Object reference not set to an instance of an object.

    The problem is in the FaceTrackFrame class:

            public Rect FaceRect
            {
                get
                {
                    this.CheckPtrAndThrow();
                    Rect faceRect;
                    this.faceTrackingResultPtr.GetFaceRect(out faceRect);
                    return faceRect;
                }
            }

    I guess it has something to do with the modification in the Rect class but I don't know how to fix it.

    Can anyone please tell me how to solve this?

    Thanks so much!

    Friday, March 29, 2013 5:38 PM
  • OK,get it. 
    Friday, April 12, 2013 1:14 AM
  • Ok, so I got it to work. Thanks to the advice of everyone above. I have the code on my github / lakehanne. One thing I notice however is that the Kinect doesn't track the object as fast as I want when it is directly above the manikin head I am using. I tried reducing the intensity of the lights in the room to see what happens and it does give a bit of results in this scenario. Is this normal, Nikolai, anyone?
    Friday, December 19, 2014 6:02 PM
  • Okay, I got this fixed. The intensity of lights in the room affects the tracking of a non-skeletal object. I've had to turn off 3 of the 6 lights in the room to get consistency in tracking. Also, I've found the tilt angle of the camera does matter to get optimized pose for this code. Currently, I have it set to about -21 degrees above my object of interest in order to get accuracy in tracking.

    One question I have though is how do I spill out the max of the 121 coordinates on the face tracking window (during execution) so I can see what's going on. I've done this in debug mode and writing to text but since this is an application I'm envisioning for laymen, I would like a technician somewhere to be able to see the max of the coordinates on the same window.

    Thanks in advance!

    Friday, December 19, 2014 6:45 PM