Problems with the region of interest Kinect SDK 1.7 RRS feed

  • Question

  • Hello,

    I'm trying to track a face in kinect SDK 1.7 without tracking a Skeleton, I read a post in which the autor changes some code in Microsoft.Kinect.Toolkit.FaceTracker sample in order to allow us to call the StartTracking method with the RegionOfInterest and Skeleton as null and continuesTracking with HeadPoints as null. It works for me and I tracked a face without a skeleton but it works too slow. The video skips many frames and I can't do any other operations at the same time as the Kinect is working. I need to use a region of interest (Rect roi) in where the kinect must search the face to track but if I call the method track() with a rect it doesn't works, I'm showing the rectangle to know where have I to "put" my face. The post which says how to track a face only with color and depth image is this. The second reply. I hope that someone can help me. Thanks to everyone.

    PS: Excuse my bad english.

    Wednesday, July 17, 2013 9:17 AM

All replies

  • Since RegionOfInterest is only used once in the StartTracking function, the main issue are the headPoints.

    I tracked a face without a skeleton but it works too slow.

    This is the main side effect of not using the headPoints parameter. Without these values, you can minimize the total area a bit, but you are still processing a larger area than you would if you know where the head positions are.

    Are you using the C# Toolkit api's or C++ api's? Does the RegionOfInterest you calculated encompass enough area for the head and neck for all subjects?


    Carmine Sirignano - MSFT

    Wednesday, July 17, 2013 8:05 PM
  • I'm using the C# API and the fist time I tested it, the region of interest was the whole frame and dind't work. Now I'm trying it with a smaller rectangle which encompass  enough area for head and neck but still doesn't work. Any ideas? I can post the code if it helps. Thank you.
    Thursday, July 18, 2013 8:24 AM
  • How close are you to the sensor? Are you enabling near mode if you are closer than .8 meters? What is your depth resolution? Does it work if you are around 1.5 meters away? 

    It will take time for it to lock on, but you should be in the optimal range for it to work better.

    Carmine Sirignano - MSFT

    Friday, July 19, 2013 5:27 PM
  • I tried it 1.5 meters away and without near mode and it didn't work, then I enabled the near mode and didn't work, neither. The depth resolution is 80x60 30 Fps and color resolution 640x480 30 Fps. Thanks again. 
    Monday, July 22, 2013 8:56 AM
  • Why are you using such a low depth resolution? At that resolution and distance, you will be hard pressed to have enough definition to detect facial features. Below is an example. There are approximately 14 x 20 pixels for the face. You may be able to find my nose, it will need to analyze a lot of frames before it can confidently provide all the facial features.

    You should be using Resolution640x480Fps30 (Kinect for Windows device) for optimal results.

    Carmine Sirignano - MSFT

    Tuesday, July 23, 2013 9:07 PM
  • Aja, ok, I'll try with 640x480 and let's see if it works, thanks :)
    Tuesday, July 23, 2013 9:19 PM
  • Nothing... With a resolution of 640x480 it can't track my face. I don't know which can be the problem.
    Wednesday, July 24, 2013 8:31 AM
  • Unfortunately, I am not sure what the issue might be other than lighting or IR interferences. With Skeletal Tracking is it able to track without issues? Do the samples work?

    Carmine Sirignano - MSFT

    Wednesday, July 24, 2013 7:32 PM
  • Light it's not the problem because the place in where I'm working with kinect is very lighting and IR interferences... I don't think so, with skeletal tracking it works perfect, with a good performance and it detects the face relatively fast.
    Wednesday, July 24, 2013 10:27 PM
  • As I already mentioned, without the FT_VECTOR3D headPoints[2] locations, FaceTracking will be slower. Region Of Interest covers the entire frame for where people are and cannot be restricted further. I have not experienced the same issues by modified the samples and setting the headPoints to null. 

    If you are able to generate calculate your own FT_VECTOR3D for the Neck-Head locations, you still don't need ST. Since ST is effectively free, why are you not using it if you know it works with that.

    Carmine Sirignano - MSFT

    Thursday, July 25, 2013 5:25 PM
  • I can't use Sekeletal Tracking because it is a game for disabled childrens, who can be in a wheelchair and there may not be depth between the kid and the chair, so kinect possibly won't be able to track the skeleton.
    Thursday, July 25, 2013 5:58 PM
  • In that case, have you tried with seated tracking mode? Depending on how they are in the chair, there may be enough definition to get the neck/head joints from those skeletons.

    You can try some simple tests with Kinect Explorer and examine how the skeletons are being drawn. If the Head/Neck joints are confidentially tracked(green dots) most of the time, then that should be enough.

    Carmine Sirignano - MSFT

    Thursday, July 25, 2013 7:46 PM
  • Ok, I tried with the seated mode and it detects the face without problems. Now I've got another problem, and it's that it works slow with skeleton. I don't know if the reason is my computer or the code, should I start a new thread for this "new" question? Thanks to everybody.
    Wednesday, July 31, 2013 9:48 AM
  • Ideally, but what type of frame rates are getting with Kinect Explorer in general. Can you sustain 30fps with that sample running? FaceTracking(FT) is a layer on top of the SDK which is CPU dependent. If you are barely hovering in the 29-30fps range, that will drop when adding FT. You might want to try with other systems to see if you get better results.

    Carmine Sirignano - MSFT

    Wednesday, July 31, 2013 9:18 PM
  • I have run the program in a Windows 8 with core i5 2.8 GHz, 12 Gb RAM, nVidia 460 GTX 768 Mb and the FPS until it Tracks the face is 58 - 59, but when it tracks the face the FPS range is 9-10, sometimes it raises 30 FPS but only for 1 second and it comes back again to 9 - 10, I don't know why, I can post the code if it helps.
    Friday, August 2, 2013 10:56 AM
  • Which sample are you using to calculate frame rate with Face Tracking? How are you getting more that 30fps and how are you calculating frame rate? Our samples are frame-locked to frames of the sensor, which is at most 30fps.

    You may need to implement threading into the application to boost the performance. Processing the data from the sensor can be independent of FT. The thread pulling data from the sensor should be kept to a minimum to ensure it is responsive and the runtime is not dropping frames. If you block this thread for longer than 33ms, then you will get poor performance/results.

    Carmine Sirignano - MSFT

    Friday, August 2, 2013 5:50 PM