Kinect v2 Face Detection/Mapping - Obscured Features RRS feed

  • Question

  • I'm currently developing an application that tries to detect if a person is wearing clothing accessories on the head. I plan to use the HD Face API to find the region the item will be worn and then apply some fancy classifier (preferably considering both colour and depth) to work out if the item is being worn. Given the level of detail the HD Face API can provide, I have had success in constructing these regions and am pleased that the heavy lifting around orientation, detection of the head region itself etc. has been left to the experts at Microsoft.

    So far, I have identified the following regions that can be “clothed”: "hat" "left/right ear", "left/right eye", "nose", "lower face (chin/jaw/cheeks)". To give an example of a use-case, imagine that I want to determine if a person is wearing a blue hat. I could use the face points provided by the HD Face API to extract the "hat" region (a combination of these HD points) then run my "blue hat" classifier on this area to yield the result.

    Here comes the problem.

    Some of my accessories cover a substantial area of the face which can prevent the face being detected at all. To give an extreme example, imagine that I want to detect the presence of a gas mask. A person wearing such an item would look quite different to what Microsoft have probably taught the API to recognise as a human face. I am not sure to what extent the algorithm would suffer (or not work at all) in this case.

    I have thrown together a test application to try and glean a bit more information. I found that the algorithm performed remarkably well when I wore hats/glasses etc. but had real trouble when the nose was obscured. It seems that certain features are more important for detection and mapping than others.

    I have a few questions that I hope someone can help me with:

    1) How does the HD Face algorithm detect a face and then go onto map the facial points? What does the algorithm need to be able to see in order to do this? Does it suppose the position of a face based on the presence of a body? Could I make the algorithm more tolerant to unusual looking faces?

    2) Would I be better of using the standard Face API (rather than HD) and then trying to identify the regions (hat, ears, eyes etc.) myself? This API seems to require less detail than the HD version (you aren’t even advised to stand very close) so may be more tolerant to obscured features. If I am forced to identify the regions myself, can you recommend a technique/literature/resource for trying to do this – I have seen plenty of examples for detecting faces in scenes but few that can identify particular features. Whatever I do will need to be robust up to obscured features – the algorithm should detect a face in a gas mask as a face!

    3) I have noticed that the HD Face API does not map the whole head – it stops at the forehead. (I guess this is why they call it the HD Face API right? J) Is there a way to extend the detection points to include the hair etc.?



    Sunday, October 25, 2015 3:03 PM