Visual Gesture Builder - Detection of hand/wrist action gesture

    General discussion

  • Hi

    This is probably more like feedback rather than a question, but maybe there's a solution to that problem using VGB.

    So I'd like to be able to detect a hand/wrist-based gesture resulting from a particular movement, specifically a hand waving/flicking gesture. I'm not interested in the state of the body skeleton and how it maps to numeric values, I'd just want to know when the action has been performed. I assume that's a "discrete"gesture, but I'm not sure how I can create one with VGB. One problem is that I can't tell the trainer to only consider hand and wrist joints. I know VGB is currently only a preview, but really, users should be able to select the individual joints they want to include in a gesture. The current options in the training settings (Ignore Lower Body, Left/Right Arm) are way too coarse and therefore insufficient.

    I watched the 2 tutorial videos and they don't really explain how one would go about to create an action-based discrete gesture. Part 1 shows how to train and recognise a "Seated" gesture, which only depends on a particular skeleton state or pose, not an action sequence (hence I would call it a pose or a posture, rather than a gesture). Part 2 shows how skeleton states and transitions in between them can be mapped to values, but again, there's no "event"-type of trigger where one can indicate that a specific movement sequence should be detected. Of course, in the code, one could implement a function that looks for transitions between values around 0 to values around 1 within a particular period of time, but I'd have thought there'd be something more straightforward available in VGB.

    Tuesday, February 3, 2015 2:52 PM

All replies

  • A seated gesture is a discrete gesture; the event triggers when you reach the final pose. There is a beginning and end to the gesture. Whether it is an entire body tracking or specific joints(arm, wrist, hand only) it is the same tagging process.

    As for your feedback, because the state of the hand can fluctuate greatly(thumb can flip and hand tips can become occluded) it is not a good to base your gesture. Fine/granular type movements of this scale will be hard to detect at the optimal tracking ranges for the body tracking system. If you consider tracking a user at 2 meters away the amount of movement in camera space by the hand/wrist will be so small that may not be enough to cancel the noise floor/signal to noise ratio.

    If you need hand/finger based type gesturing, you will want to use a different system altogether and do a heuristic based detection or come up with your own machine learning on that library.

    Carmine Sirignano - MSFT

    Tuesday, February 3, 2015 6:36 PM
  • Thanks for the reply.

    I disagree that the "Seated" gesture is a gesture. The event triggers because the skeleton has reached a particular position that the detector recognises, regardless of what poses preceded the action. Hence, it's a posture. A gesture is a succession of states that the user has to go through (of course, with a certain tolerance) until an end state is reached that then triggers the event. If the user reaches the end state without performing the required in-between poses, the gesture event doesn't trigger.

    As for my particular hand/flick gesture, it's more like a kind of an open-hand swipe, I guess, similar to what the Kinect Toolbox supported with its SwipeGestureDetector (unfortunately it hasn't been updated for v2), so it's not as fine-grained as that. I actually created the gesture in VGB and obtain relatively satisfactory results in the analysis. I'm now trying to integrate the database in my code to see if I can reliably detect the gesture using the method I described above, but I get a "System.InvalidOperationException" when trying to initialise the VisualGestureBuilderDatabase with my gdb file. I see there's a thread dealing with that, so will probably post in there, because I have no idea why that's happening.

    Tuesday, February 3, 2015 7:08 PM