Visual Gesture Builder for recognition of multimodal gestures? RRS feed

  • Question

  • Hello,

    Could you please let me know if Visual Gesture Builder can be used for recognition of multimodal gestures?

    Thank You

    Wednesday, June 10, 2015 7:38 PM


All replies

  • When you refer to multimodal, is it in reference to detecting multiple forms of the same gesture, or that a single gesture will change its form over time (has several phases that it will progress through)? Both are possible with VGB, but would require different training/detection techniques. If you are using a different definition for 'multimodal', please explain, and provide an example of such a gesture that you wish to train.

    Multiple forms: Nobody performs the same gesture identically, which is why VGB uses a machine learning approach instead of heuristics for training gesture detectors. It allows for variance in the same gesture to be learned. However, if the variance is large, it will be difficult to train a detector that has a high level of accuracy. In such cases, you will find that it is easier to train multiple forms of the same gesture and then have your code respond identically when any of the gesture forms are detected. Using Discrete detectors for this type of gesture should be sufficient.

    Multiple phases: When a gesture has multiple phases that it needs to progress through, it is best to use a combination of Continuous and Discrete detectors together. First, you'll train each phase separately, as a Discrete gesture. And then you train a single, continuous gesture that combines all of the phases together. For the continuous gesture, you would want each phase to have a unique value range (phase 1: 0.0-0.3, phase 2: 0.31-0.7, phase 3: 0.71-1.0). Continuous detectors will always fire, even when nothing is happening, so you need to use the corresponding Discrete detectors tell you when that signal is valid. Also, if you expect the gesture to occur in a certain pattern (phase 1 must always occur before phase 2), you can choose to ignore any detection results that do not fit that pattern.

    I hope this explanation helps,


    Wednesday, June 10, 2015 8:43 PM
  • Hi Angela,

    Thanks a lot for your reply. 

    Multimodal gesture I mean is the correlation between audio(rhythm) and gestures. For ex: beat gestures. 

    Friday, June 12, 2015 6:15 AM
  • Then, no, at least in the current version. I think Angela says it in one of the VGB videos.
    Friday, June 12, 2015 8:26 AM