none
Kinect Avateering problems RRS feed

  • Question

  • I am having some problem understanding and using Joint positions and orientations. 

    Using the kinect V2 I want to make a mesh (Ogre3D sinbad model) follow my body in front of the kinect, but I am having some serious problems.

    Currently I am having the kinect to detect my body as well as the Joint positions and orientations. 

    But I cannot figure out what I have to do now. I know that the orientation is a quaternion, but when I apply the quaternion without any further processing to the corresponding bone of the mesh-skeleton, it looks wrong. 

    Can someone explain to me what I have to do know? How to preprocess the quaternions and position for applying them to the mesh. 

    The -z Axis of my current Ogre3D Scene goes INTO the monitor, y goes up. If this matters

    Here is a YT Video of how it currently looks: https://www.youtube.com/watch?v=QC8SnFBb5sg
    • Edited by Glethien Thursday, January 8, 2015 11:47 AM
    Thursday, January 8, 2015 11:41 AM

Answers

  • For everyone who has a similar problem:
    I've found a solution.

    My misstake was to use only the kinect joint data and transform the elbow into the kinect shoulder. As I want to apply the stuff to an Ogre3D model I had to use the parent bone of my current bone.

    ParentBone.Orientation.Inverse * CorrespondingKinectJointOrientation.

    For the humerus bone in Ogre3D it looks like this:

      Ogre::Bone* bone = sinbad->getSkeleton()->getBone("Humerus.L");
       bone->setManuallyControlled(true);
       bone->resetOrientation();   
       Ogre::Quaternion parentOrientation = bone->getParent()->_getFullTransform().extractQuaternion();
    
       Ogre::Quaternion shoulder = KinectHelper::kinectOrientationToOgreQuaternion(&skeleton->getKinectJointOrientation(JointType_ElbowRight)->Orientation);
    
       KinectHelper::convertCoordinateSystem(&shoulder);
    
       Ogre::Quaternion orientation = parentOrientation.Inverse() * shoulder;   
       bone->setOrientation(orientation);

    Tuesday, January 13, 2015 2:11 PM

All replies

  • Kinect is a right hand coordinate system. If you model doesn't align to the orientation of Kinects coordinate system, you will get some odd twisting. Also you may need to apply the orientation for the left joints to the right joints of your character model.

    Carmine Sirignano - MSFT

    Thursday, January 8, 2015 6:52 PM
  • That means that I can apply the orientations directly to the mesh?

    For example:
    In Ogre3D I have a mesh with the humerus bone. The bone will be influenced by which joint? Shoulder? A combination from shoulder and elbow? For me this construct with the joints and bones is not very clear...

    Monday, January 12, 2015 12:11 PM
  • In a non-graphical system it is going be a bit difficult as each bone will affect the mesh differently if things are not as it is expected. It is a lot of trial and error. You may need to play with the bone quaternion conjugate to get the identity, apply the Kinect rotation and then reapply the rotation and see if that helps.  

    Carmine Sirignano - MSFT

    Monday, January 12, 2015 8:32 PM
  • For everyone who has a similar problem:
    I've found a solution.

    My misstake was to use only the kinect joint data and transform the elbow into the kinect shoulder. As I want to apply the stuff to an Ogre3D model I had to use the parent bone of my current bone.

    ParentBone.Orientation.Inverse * CorrespondingKinectJointOrientation.

    For the humerus bone in Ogre3D it looks like this:

      Ogre::Bone* bone = sinbad->getSkeleton()->getBone("Humerus.L");
       bone->setManuallyControlled(true);
       bone->resetOrientation();   
       Ogre::Quaternion parentOrientation = bone->getParent()->_getFullTransform().extractQuaternion();
    
       Ogre::Quaternion shoulder = KinectHelper::kinectOrientationToOgreQuaternion(&skeleton->getKinectJointOrientation(JointType_ElbowRight)->Orientation);
    
       KinectHelper::convertCoordinateSystem(&shoulder);
    
       Ogre::Quaternion orientation = parentOrientation.Inverse() * shoulder;   
       bone->setOrientation(orientation);

    Tuesday, January 13, 2015 2:11 PM
  • for completeness, your convertCoordinateSystem function is that Kinect space to Ogre world/view space or model space?

    Carmine Sirignano - MSFT

    Tuesday, January 13, 2015 6:54 PM
  • I've been trying to do the same in Unity3d but it didn't work. First of all I am using 2 different models, one exported from daz3d and one downloaded from the internet.  However, my results were very bizarre.

    I tried:
    rightShoulder.transform.localRotation = shoulderRightStart * Quaternion.Inverse(rightShoulderOrientation) * rightElbowOrientation;

    And this is the result of me standing in normal pose. You can see that the cylinder aligns perfectly and move(rotates) perfectly with my upper arm. However, the 2 models on the right behaves completely different. Maybe this is because the mecanim bone direction done by automap is different?

    Wednesday, January 14, 2015 7:01 AM
  • for completeness, your convertCoordinateSystem function is that Kinect space to Ogre world/view space or model space?

    Carmine Sirignano - MSFT

    It just flips the w and z value of the quaternion, thus I think it is from the left handed kinect space to the right handed orge space. The values are untouched, except for the sign.

    Here is the code:

    		static void convertCoordinateSystem(Ogre::Quaternion* quat)
    		{
    			quat->w =  -quat->w;
    			quat->x =  quat->x;
    			quat->y =  quat->y;
    			quat->z =  -quat->z;
    		}

    Wednesday, January 14, 2015 8:32 AM
  • I've been trying to do the same in Unity3d but it didn't work. First of all I am using 2 different models, one exported from daz3d and one downloaded from the internet.  However, my results were very bizarre.

    I tried:
    rightShoulder.transform.localRotation = shoulderRightStart * Quaternion.Inverse(rightShoulderOrientation) * rightElbowOrientation;

    And this is the result of me standing in normal pose. You can see that the cylinder aligns perfectly and move(rotates) perfectly with my upper arm. However, the 2 models on the right behaves completely different. Maybe this is because the mecanim bone direction done by automap is different?

    For doing this with Unity, I highly recommend "Rumen Filkov"'s blog entry:

    http://rfilkov.com/2014/08/01/kinect-v2-with-ms-sdk/

    If you write him an eMail explaining what and why you want to do this, he may sends you his working Unity3D Avateering sample. I got it from him and it works like charm. 


    • Edited by Glethien Wednesday, January 14, 2015 8:36 AM
    Wednesday, January 14, 2015 8:35 AM
  • I have his code with me too. I tried to study how is it different from what I'm doing and I discovered that he does not rely on JointOrientations at all but rather compute everything from the ground up with the joints' positions. This makes me wonder why jointOrientations themselves are not preferable because they should be very easy (but they actually aren't).

    Anyway, thanks Glethien.
    Wednesday, January 14, 2015 9:38 AM