none
Problems with body detection for small kids (< 40")? RRS feed

  • Question

  • Does Kinect v2 have known problems detecting bodies and/or tracking skeletons for small kids?

    I've heard from a museum whose exhibit uses Kinect (v2) that they have trouble with body tracking for kids under 40".  I haven't been able to get more information on their technical setup, but since I'm working on a museum project that small kids will use, I'm eager to hear if the community has any experiences to relate.

    This question, no doubt, involves the question of sensor height and angle:  it seems Kinect is engineered around placement orthogonal to the ground; the post below claims that skeleton pose can vary with sensor angle, supporting the potential importance of keeping the sensor level for optimal body tracking:

    https://social.msdn.microsoft.com/Forums/en-US/594cf9ed-3fa6-4700-872c-68054cac5bf0/angle-of-kinect-device-and-effect-on-xyz-positional-data?forum=kinectv2sdk

    The Kinect's internal accelerometer readings are presumably used, along with plane-fitting of recovered floor points, by the BodyFrame class FloorClipPlane member:

    https://msdn.microsoft.com/en-us/library/microsoft.kinect.bodyframe.aspx?f=255&MSPPError=-2147217396

    So, again, is there an optimal height and angle to place the Kinect when detecting small kids?

    In experimental contexts I've seen that it's common to place the Kinect at 1m above the floor and level, but I haven't been able to find any documentation that suggests that this is indeed the best setup for tracking.

    The problems the museum staff reported, then, sound like a question of skeleton detection:  if a small child is in a big winter parka, for instance, it's hard for the Kinect to figure out the geometry of the skeleton, since the limbs are less distinct.  The tough scenario for Kinect would be someone in a full burka, where the head is also covered:  that gives the system few skeletal landmarks to orient to, a 3D tracking equivalent to the 2D tracking problems Logitech had in enabling black-skinned users to be recognized by their webcams' facial recognition code.

    Wednesday, January 4, 2017 7:20 PM

Answers

  • That's correct , there is a minimum height for skeleton tracking. If I find the link I will provide it but basically anyone shorter than 40" will not be recognized/tracked. Doesn't make a difference even if the sensor is looking downward. If anything, the higher it is, the bigger the downward angle and stuff like the lower extremities look more aligned to the sensor and there are signs of degradation to the depth data.

    Bear in mind that you are dealing with IR here.IR is generally a bit shoddy when it comes to black color. I usually get small holes in the mask due to large beard or no legs tracked at all if I wear a black jean. It's a question of absorption mostly. So someone wearing a full burka will never even get picked for skeleton tracking, let alone be tracked(even to a lesser extent).

    You will also get problems if the fabric/material or the dye of the clothing is reflective. So if someone wears a shiny jacket or even worse, something that has glitter or glass or whatever of the sort, will not get recognized at all.

    UPDATE: https://support.xbox.com/en-US/xbox-360/accessories/body-tracking-troubleshoot

    • Proposed as answer by Nikolaos Patsiouras Wednesday, January 4, 2017 10:01 PM
    • Marked as answer by KevinCain Thursday, January 5, 2017 2:19 AM
    Wednesday, January 4, 2017 10:01 PM

All replies

  • That's correct , there is a minimum height for skeleton tracking. If I find the link I will provide it but basically anyone shorter than 40" will not be recognized/tracked. Doesn't make a difference even if the sensor is looking downward. If anything, the higher it is, the bigger the downward angle and stuff like the lower extremities look more aligned to the sensor and there are signs of degradation to the depth data.

    Bear in mind that you are dealing with IR here.IR is generally a bit shoddy when it comes to black color. I usually get small holes in the mask due to large beard or no legs tracked at all if I wear a black jean. It's a question of absorption mostly. So someone wearing a full burka will never even get picked for skeleton tracking, let alone be tracked(even to a lesser extent).

    You will also get problems if the fabric/material or the dye of the clothing is reflective. So if someone wears a shiny jacket or even worse, something that has glitter or glass or whatever of the sort, will not get recognized at all.

    UPDATE: https://support.xbox.com/en-US/xbox-360/accessories/body-tracking-troubleshoot

    • Proposed as answer by Nikolaos Patsiouras Wednesday, January 4, 2017 10:01 PM
    • Marked as answer by KevinCain Thursday, January 5, 2017 2:19 AM
    Wednesday, January 4, 2017 10:01 PM
  • Thanks @NPatch for the answer and link!

    This thread suggests that the 40" height is a recommendation, not an absolute:

    "Josh Hutto, on the Xbox Kinect team, (he) explained that there is a recommended height minimum of 40 inches, corresponding roughly to kids 4 and a half years old and up."

    Can anyone say what their experiences have been with kids under 40"?

    Is there an API method to omit this culling check if it is explicit?

    From what I've seen, kids reach 40" between ages 3-6 -- this would seem to be a pretty significant limitation for museum use.  Since lots of museums serving kids have Kinect-based interactives (e.g., the Tech Museum -- https://www.thetech.org/), any idea how they are getting around this limitation with their users?

    • Edited by KevinCain Thursday, January 5, 2017 2:41 AM
    Thursday, January 5, 2017 2:24 AM
  • First off, that thread is talking about Xbox 360,so effectively the whole article and the links are talking about Kinect v1. So it's not relevant. Kinect v2 is the sensor Xbox One(and One S) uses.

    I have a project that was placed in a public space, at 1.1m which is a good enough height for the sensor's plane to be orthoparallel to the user's plane and see the whole user at a normal distance. Most kids that were shorter than the recommended height were never recognized.Also kids that are short tend to go very close to the sensor which is even worse because they block the field of view.

    You might have some luck mounting the Kinect a bit higher and tilting it down. Unless you mount it higher than ~1.8m you probably won't see any degradation to depth data near the feet.

    The article from the forum link you provided uses a Tuner app which is available on Xbox. Honestly I haven't worked with Kinect v1 so I don't know if its also available for PC but I do know there's nothing like that on Kinect v2. Kinect v2 is supposed to have calibration data , from the factory, stored in the hardware and taken into consideration when using CoordinateMapper class. Otherwise you'd have to venture into Computer Vision field and do manual calibration and effectively use your own custom CoordinateMapper.

    There's no way to omit culling.

    I guess this was either an implementation problem or maybe a way to differentiate between animals and very short or kneeling bipeds? I don't know for sure but I think there's nothing you can really do about it.

    At least so long as you're trying to use the official SDK. Open source SDKs tend to have their own custom solutions for Skeleton Tracking so they might support that height. Don't know if there's any SDK though that's good enough for skeleton tracking.

    As far as the Tech Museum goes, is there some particular project you're referring to? Because it's mostly a question of what hardware to use for what kind of project and depending on the audience you want to attract. So if its a project for kids, you look at other sensors perhaps, or if you can't do it with any existing, you either do something custom or you shift the type of interaction to something you can support.




    Thursday, January 5, 2017 9:31 AM
  • Good points, @NPatch, about the Kinect v2 differences from Kinect v1.

    I agree that there must be some compelling reason to limit the skeleton detection.  As you suggest, it may just be to avoid the nuisance of detecting pets, or it may be a response to poor tracking with small humans.    In the former case, being able to omit body culling < 40" height would work fine for controlled environments like museums; in the latter, there's not much we can do.

    To answer your question, the Tech Museum has several Kinect-based exhibits that are visited by a lot of small kids; they are clearly using the standard (Microsoft) API and it seems they would have experience with the ~40" height threshold for body tracking.  I was just appealing to folks with similar environments to share their experiences -- in practice, perhaps this isn't as big a problem as it sounds.

    Thursday, January 5, 2017 9:23 PM
  • Do they use full body tracking for kids then? Because I doubt it.

    You could use a simplified custom tracking by having kids wear stuff like reflective markers and track those in the raw IR data. Kinda like mocap.

    Like I said, depends on the type of interaction you're going for and what you can support.

    Thursday, January 5, 2017 11:36 PM
  • Yes, I'd thought of solutions similar to the ones you suggest:  have the kids hold a marker in each hand and track those in 2D via openCV, or perhaps 'trick' the Kinect with a step for the small kids to stand on, et cetera.

    Clearly we need to do some testing.

    Also, the Tech does use full body tracking for kids in several exhibits, and it works pretty well from my visit -- here's their page:

    "Body Moves: Microsoft’s Kinect technology powers large-scale projectors that lead visitors through three activities — Pose, Balance and Bounce — during which body position, activity and range of motion are measured."

    However, theirs is just one example and I don't mean to over-emphasize it.  Lots of museums use Kinect as part of their exhibits, of course, and many count small kids as their main visitors.  So the museum community would presumably have had to grapple with this; maybe the lack of discussion means it's not much of a problem in practical use.

    Friday, January 6, 2017 12:26 AM
  • Yes but I don't buy it.

    There's a difference to seeing kids above the height constraint interacting with the exhibit and kids below. Did you happen to see kids below 40" using the Balance or Jump parts when you visited? So far I've seen small kids from two open-crowd events in a research center get ignored by Kinect v2. Their parents had to lift them up to satisfy them.

    Also "Lots of museums use Kinect as part of their exhibits, of course, and many count small kids as their main visitors" can be misleading.

    Saying that many museums count kids as main visitors doesn't mean all exhibits have to be accessible to any age or height when it comes to kids(CyberDetectives is recommended for 12+ yo),nor does it mean they don't/won't provide exhibits for adults. You can still count very small kids as visitors, as long as there are some exhibits for them. And your particular museum example doesn't just use Kinect in all its exhibits. Therefore you can't say that it definitely has an exhibit for *small* kids using Kinect, nor can you say it's main audience is kids.

    My point is, perhaps you're overgeneralizing things. Because the problem here is not supporting body tracking for kids, it's for kids under a certain height.

    Fact is that there's no way to alter the behavior of Body Tracking through the SDK, but only filter upon the data it feeds you. Like further filter the 6 bodies to only select one etc. So if the SDK recognizes the kid, all's well, otherwise , oh well.

    The only other way to do something like MS' full Body Tracking is to create your own custom solution for image-based skeleton tracking , kinda like reverse engineering Kinect's algorithm. But that's easier said than done. So either they used an open source framework for that or they created their own which means its *probably* not open source(seriously lot of work and complicated).



    Friday, January 6, 2017 3:12 AM
  • Saying that many museums count kids as main visitors doesn't mean all exhibits have to be accessible to any age or height when it comes to kids...

    @NPatch's point is certainly valid for some museums, but certainly not the one I'm working with in this instance.  I'd like to briefly quote a museum staff member who objected in no uncertain terms:

    "to restrict or exclude visitors due to technological limitations is something I do not believe we should start accepting institutionally. This is a clear case of technology driving the experience, rather than having a great experience being made possible by technology."

    For our purposes, I think we could smoothly degrade from full skeletal tracking to 2D point detection via a web camera, using OpenCV or similar, as I suggested previously. I agree that even if we had access to the closed source Kinect code, which we don't, it's pointless to try to 'fix' Microsoft's work without the minimum information to try (full documentation).  And, even then, we'd be without the years of experience the Kinect team have with their own product.

    Back to your healthy questioning about my Tech Museum example, actually I did see full skeletons recognized for small kids at the 'Jump' and 'Balance' stations, including a couple children whose height is below 40", so I was told by their parents!  I don't doubt your central point that Microsoft is intentionally not passing bodies under 40" to their body manager methods.  I just think that it's not a firm limit since Kinect v2 returns varying distances compared to ground truth, as a function of sensor distance, sensor angle and height, as noted above.




    • Edited by KevinCain Friday, January 6, 2017 3:42 AM
    Friday, January 6, 2017 3:39 AM
  • You also make a valid point. On the other hand even if you test it out and find out, you might also find the threshold is not accurate or that the whole thing is not as reliable as expected. After all there's also environmental noise that can also degrade the data and in situations like this even a small inaccuracy might be enough for unreliable tracking or none at all.

    If you do test this, I'm sure many in this forum,including me, would be very interested to know ,so please report your findings if possible.

    Friday, January 6, 2017 9:58 AM
  • We'll be testing in a separate museum for kids next week, so I'll be sure to write back with an update!
    Sunday, January 8, 2017 5:27 AM
  •  We completed three days of testing 'in the wild' last week, so as promised, I wanted to return with what we observed.

    To start with some good news, we found that even very small kids could interact successfully. Despite the published 40" limitations for the Kinect sensor noted in our thread, we found that smaller visitors were able to do just fine. The smallest child we measured was 30"; another was 33", and several others were < 40". The youngest child we saw succeed was two years old. With much help from the parents, we tried to get smaller kids to flap their arms but they weren't really aware of what they were doing. So we seem to be in good shape to support Andrea's 4-7 age range for the interactive.

    The first two days of tests, we found that hovering parents with outstretched arms would be detected by the Kinect rather than their kids. For the final day we reduced the distance at which we ignore input, which let us detect even small kids with a parent crouched behind them.

    This worked well enough that even when a parent articulated the child's arms like a puppet, we correctly read the child's skeleton and not the parent directly behind.  That's a good thing, because we found that parents would often hover in a crouched position to coach the kids. We did see occasional competition between the figures, but not enough to frustrate the kids or the parents.

    As expected, we saw the smaller kids creep up on the sensor until they touched it without knowing. Some physical barrier may be helpful to prevent this, but it didn't strike us as a necessity. We saw some effects from infrared light bounce off the floor-to-ceiling windows aligned on the sensor axis in the testing space, but we won't have this to contend with in the actual exhibit space. The exhibit will be bathed in indirect sunlight from windows facing the Bay, so I did some mock-up testing for these sunlight bounce conditions last week outside the testing space. The sensor performed well; I didn't notice any noise. Still, we should explicitly test for this (and for cross-talk between the adjacent sensors) in the actual exhibit space as soon as practical.

    Unexpectedly, we saw that older kids would recognize the X-Box logo on the Kinect sensor and assumed this was a commercial title!


    Tuesday, January 17, 2017 8:55 PM
  • Thanks for reporting back!

    Did you by , any chance, test with more than one sensor? Was it consistent?

    Also what was the setup for the sensor's placement? Height and tilting?

    Wednesday, January 18, 2017 9:12 AM
  • We just had a single sensor with us, so consistency is something we'll have to address in the next round of tests.

    The sensor was level, at a height of 1m -- I should have mentioned that!

    Thursday, January 19, 2017 2:43 AM
  • Thanks! I'm interested to see if the findings are consistent between sensors.
    Thursday, January 19, 2017 8:23 AM