grab hand detection using the depth frame? RRS feed

  • Question

  • I am trying to figure out a good way to recognize when a hand is closed using the sdk. I'm very familiar with the skeleton frame, but it only has one joint on the hand and i cant use that to track if the hand is closed. I was wondering if anyone knows of anyway i could use the depth stream to detect a change in the hand to recognize that a fist was made or that the hand was closed?



    Wednesday, August 3, 2011 5:58 PM


All replies

  • There are a couple of posts on this topic. For example:

    You could try asking for more details to the people that posted about specific solutions, but unfortunately we (Microsoft) are not providing support for this scenario at this point.

    Thanks for your interest,

    I'm here to help
    • Marked as answer by Dejanr23 Friday, August 5, 2011 6:03 PM
    Wednesday, August 3, 2011 11:17 PM
  • I would think the starting point is segment the hand, i.e. pick it out. Use the direction from wrist to hand as a normal to a plane passing through the wrist. Start at the point of the hand and use a flood fill algorithm to find all points infront of that plane within a reasonable distance of the hand. Recursion is an easy way to do flood fills. You check the current point then check left, right, top, bottom with a recursive call passing the direction. If the current point does meet the criteria then you don't the check the current direction. What you're doing with recursion is just using a stack to push and pop states. So you could just use the collections stack implementation in .Net with no recursion.

    How elaborate you have to get depends on what you want to do. As an example you could use a horizontal line rather than an oriented plane. It will limit when you recognize an open hand versus fist, but if you just want forearm veritical, hand open facing the camera it can do that. If you want to handle arm tilted 45 degrees to the right then a horizontal line likely isn't going to work too well, a line at 45 degrees would work better. If you want to handle rotation about multiple axes then you start needing a plane. The main reason for bring that up is you can start simple. If you're not strong with 3D then you can start with 2D. You can start with axis aligned and then more to non-axis aligned. All you really need to start is some method of saying a point is part of the hand or not. Render it on the screen like the player indexes in the skeletal viewer, i.e. color the hand, and you get easy feedback on well it's doing it.

    Wednesday, August 3, 2011 11:18 PM