none
Unity JointOrientations to Bone Mapping Not Quite Right

    Question

  • I'm trying to map the KinectV2 bone rotations (JointOrientations) to unity bones, but it's not quite right.  If I isolate it to the spine only it seems off by the amount of the initial value of the bind pose.  However, I've tried saving this bind rotation quaternion in the start() code and then multiplying it by Kinect's rotation quaternion, but that doesn't work either.  I tried taking the inverse of that quaternion too, but still no luck.

    I'm sure this is newbie stuff, but does anyone know what I may be doing wrong here?

    Thanks

    Ray

    Thursday, July 24, 2014 5:08 AM

All replies

  • Ray are you using our Unity plug-in?  If so, what version?

    FYI, we have fixed some bugs in the plug-in recently so a newer build may help.  Let me know what you are using and then I can better direct.


    Ben Lower [MSFT]

    Saturday, July 26, 2014 2:17 AM
    Owner
  • Hi Ben,

    Thanks for the reply.  The DLL says version 2.0.1407.14000.  It's basically the version from the KinectForWindows_1407_Unity (Pro Only).zip from the website.

    As far as I can see this is the latest available on the website.  Have you guys done a more recent build?

    Thanks

    Ray

    Saturday, July 26, 2014 3:58 AM
  • Just FYI, this is my code so far.  In start(), I have...

    		GameObject hips = GameObject.Find("hips");
    		hipStart = hips.transform.localRotation;
    		GameObject spine = GameObject.Find("spine");
    		spineStart = spine.transform.localRotation;
    		GameObject neck = GameObject.Find("neck");
    		neckStart = neck.transform.localRotation;
    		GameObject head = GameObject.Find("head");
    		headStart = head.transform.localRotation;
    		GameObject chest = GameObject.Find("chest");
    		chestStart = chest.transform.localRotation;
    
    		GameObject leftClavicle = GameObject.Find("clavicle.L");
    		leftClavicleStart = leftClavicle.transform.localRotation;
    		GameObject leftArm = GameObject.Find("upper_arm.L");
    		leftArmStart = leftArm.transform.localRotation;
    		GameObject leftForearm = GameObject.Find("forearm.L");
    		leftForearmStart = leftForearm.transform.localRotation;
    		GameObject leftHand = GameObject.Find("hand.L");
    		leftHandStart = leftHand.transform.localRotation;
    
    		GameObject rightClavicle = GameObject.Find("clavicle.R");
    		rightClavicleStart = rightClavicle.transform.localRotation;
    		GameObject rightArm = GameObject.Find("upper_arm.R");
    		rightArmStart = rightArm.transform.localRotation;
    		GameObject rightForearm = GameObject.Find("forearm.R");
    		rightForearmStart = rightForearm.transform.localRotation;
    		GameObject rightHand = GameObject.Find("hand.R");
    		rightHandStart = rightHand.transform.localRotation;
    

    Then in update() I have...

    		// get positions from kinect
    		Vector3 neckPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.Neck].Position);
    		Vector3 headPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.Head].Position);
    		Vector3 leftHipPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HipLeft].Position);
    		Vector3 spineBasePosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineBase].Position);
    		Vector3 spineMidPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineMid].Position);
    		Vector3 spineShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineShoulder].Position);
    		Vector3 leftShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ShoulderLeft].Position);
    		Vector3 leftElbowPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ElbowLeft].Position);
    		Vector3 leftHandPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HandLeft].Position);
    		Vector3 leftWristPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.WristLeft].Position);
    		Vector3 leftHandTipPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HandTipLeft].Position);
    		Vector3 leftThumbPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ThumbLeft].Position);
    
    		// get rotations from kinect
    		Quaternion neckOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Neck].Orientation);
    		Quaternion headOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Head].Orientation);
    		Quaternion leftHipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HipLeft].Orientation);
    		Quaternion spineBaseOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineBase].Orientation);
    		Quaternion spineMidOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineMid].Orientation);
    		Quaternion spineShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineShoulder].Orientation);
    		Quaternion leftShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderLeft].Orientation);
    		Quaternion leftElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowLeft].Orientation);
    		Quaternion leftHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandLeft].Orientation);
    		Quaternion leftWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristLeft].Orientation);
    		Quaternion leftHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipLeft].Orientation);
    		Quaternion leftThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbLeft].Orientation);
    
    		// use this local position to update our real model
    		GameObject suitMan = GameObject.Find("Suit Man");
    		MeshFilter meshFilter = suitMan.GetComponentInChildren<MeshFilter>();
    		SkinnedMeshRenderer renderer = suitMan.GetComponentInChildren<SkinnedMeshRenderer>();
    		Mesh myMesh = renderer.sharedMesh;
    
    		// get game objects
    		GameObject neck = GameObject.Find("neck");
    		GameObject head = GameObject.Find("head");
    		GameObject hips = GameObject.Find("hips");
    		GameObject spine = GameObject.Find("spine");
    		GameObject chest = GameObject.Find("chest");
    		GameObject leftClavicle = GameObject.Find("clavicle.L");
    		GameObject leftArm = GameObject.Find("upper_arm.L");
    		GameObject leftForearm = GameObject.Find("forearm.L");
    		GameObject leftHand = GameObject.Find("hand.L");
    
    		// calculate bones from positions
    		Vector3 spineBone = spineMidPosition - spineBasePosition;
    		Vector3 neckBone = neckPosition - spineShoulderPosition;
    		Vector3 headBone = headPosition - neckPosition;
    		Vector3 chestBone = spineShoulderPosition - spineMidPosition;
    		Vector3 leftShoulderBone = leftShoulderPosition - spineShoulderPosition;
    		Vector3 leftArmBone = leftElbowPosition - leftShoulderPosition;
    
    		// spine
    		Quaternion myTest = Quaternion.FromToRotation (new Vector3(0.0f, 1.0f, 0.0f), spineBone);
    //		myTest = QuaternionFromVector4(spineBaseOrientation);
    		Quaternion myTest5 = Quaternion.FromToRotation (spineBone, chestBone);
    		spine.transform.localRotation = Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest;
    		chest.transform.localRotation = Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest5;
    
    		// neck
    		Quaternion myTest2 = Quaternion.FromToRotation (chestBone, neckBone);
    //		myTest2 = Quaternion.Inverse(spineShoulderOrientation) * QuaternionFromVector4(neckOrientation);
    		Debug.Log("neck rotation="+myTest2.ToString());
    		neck.transform.localRotation = Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest2;
    
    		// head
    		Quaternion myTest3 = Quaternion.FromToRotation (neckBone, headBone);
    //		myTest3 = Quaternion.Inverse(neckOrientation) * QuaternionFromVector4(headOrientation);
    		Debug.Log("head rotation="+myTest3.ToString());
    		head.transform.localRotation = Quaternion.Inverse(headStart) * Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest3;
    
    		// left shoulder
    		Quaternion myTest4 = Quaternion.FromToRotation (chestBone, leftShoulderBone);
    //		myTest4 = Quaternion.Inverse(spineShoulderOrientation) * leftShoulderOrientation;
    		Debug.Log("shoulder rotation="+myTest4.ToString());
    		leftClavicle.transform.localRotation = Quaternion.Inverse(leftClavicleStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse (hipStart) * myTest4;
    

    The body and head seem to work fine by the shoulder is going a little haywire.  Also, when I try and use the rotations (commented out code) it just goes nuts for everything.

    Regards

    Ray


    Saturday, July 26, 2014 4:07 AM
  • I also tried this using the rotations instead, but then the neck and the head don't rotate at all and the shoulder still goes haywaire...


    // hips // Quaternion myTest = Quaternion.FromToRotation (new Vector3(0.0f, 1.0f, 0.0f), spineBone); Quaternion myTest = spineBaseOrientation; Quaternion myTest0 = Quaternion.AngleAxis(myTest.eulerAngles.y, new Vector3(0.0f, 1.0f, 0.0f)); hips.transform.localRotation = Quaternion.Inverse(hipStart) * myTest0; // spine Quaternion myTest6 = Quaternion.AngleAxis(myTest.eulerAngles.x, new Vector3(1.0f, 0.0f, 0.0f)) * Quaternion.AngleAxis(myTest.eulerAngles.z, new Vector3(0.0f, 0.0f, 1.0f)); spine.transform.localRotation = Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest6; // chest Quaternion myTest5 = Quaternion.Inverse(spineBaseOrientation) * spineMidOrientation; chest.transform.localRotation = Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest5; // neck // Quaternion myTest2 = Quaternion.FromToRotation (chestBone, neckBone); Quaternion myTest2 = Quaternion.Inverse(spineShoulderOrientation) * neckOrientation; Debug.Log("neck rotation="+myTest2.ToString()); neck.transform.localRotation = Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest2; // head // Quaternion myTest3 = Quaternion.FromToRotation (neckBone, headBone); Quaternion myTest3 = Quaternion.Inverse(neckOrientation) * headOrientation; Debug.Log("head rotation="+myTest3.ToString()); head.transform.localRotation = Quaternion.Inverse(headStart) * Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest3; // left shoulder // Quaternion myTest4 = Quaternion.FromToRotation (chestBone, leftShoulderBone); Quaternion myTest4 = Quaternion.Inverse(spineShoulderOrientation) * leftShoulderOrientation; Debug.Log("shoulder rotation="+myTest4.ToString()); leftClavicle.transform.localRotation = Quaternion.Inverse(leftClavicleStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse (hipStart) * myTest4;


    Monday, July 28, 2014 6:45 PM
  • Replied to your other thread, but the JointOrientations are absolute. To relative you'll need to do 

    var relative = parentQuaternion.Inverse() * quaternion;

    as well as possibly rotate the quaternion depending upon the axis orientation of your world. For Unity, I've found I need to flip the Z-axis due to the right-handed vs left-handed in Unity. I know quaternions technically aren't handed, but this was necessary anyway. To flip an axis, you need to negate the axis and the angle, like so:

    static Quaternion ConvertToLeftHanded(Quaternion q)
    {
        return new Quaternion(q.x, q.y, -q.z, -q.w);
    }



    Joshua Blake
    Kinect for Windows MVP
    OpenKinect Community Founder
    Technical Director, InfoStrat Advanced Technology Group
    Blog: http://nui.joshland.org
    Twitter: http://twitter.com/joshblake

    Wednesday, July 30, 2014 12:28 AM
    Moderator
  • That's pretty much what I had in my second example, except without the handedness change of quaternions (which I also read were not handed).  But, still no go.  The spine rotation seems to work well, but the head and neck don't seem to be moving at all and the shoulder is just way out still.  Here's the code as it stands...

    		// get positions from kinect
    		Vector3 neckPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.Neck].Position);
    		Vector3 headPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.Head].Position);
    		Vector3 leftHipPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HipLeft].Position);
    		Vector3 spineBasePosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineBase].Position);
    		Vector3 spineMidPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineMid].Position);
    		Vector3 spineShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineShoulder].Position);
    		Vector3 leftShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ShoulderLeft].Position);
    		Vector3 leftElbowPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ElbowLeft].Position);
    		Vector3 leftHandPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HandLeft].Position);
    		Vector3 leftWristPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.WristLeft].Position);
    		Vector3 leftHandTipPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HandTipLeft].Position);
    		Vector3 leftThumbPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ThumbLeft].Position);
    		Vector3 rightShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ShoulderRight].Position);
    		Vector3 rightElbowPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ElbowRight].Position);
    
    		// get rotations from kinect
    		Quaternion neckOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Neck].Orientation);
    		Quaternion headOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Head].Orientation);
    		Quaternion leftHipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HipLeft].Orientation);
    		Quaternion spineBaseOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineBase].Orientation);
    		Quaternion spineMidOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineMid].Orientation);
    		Quaternion spineShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineShoulder].Orientation);
    		Quaternion leftShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderLeft].Orientation);
    		Quaternion leftElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowLeft].Orientation);
    		Quaternion leftHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandLeft].Orientation);
    		Quaternion leftWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristLeft].Orientation);
    		Quaternion leftHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipLeft].Orientation);
    		Quaternion leftThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbLeft].Orientation);
    		Quaternion rightShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderRight].Orientation);
    		Quaternion rightElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowRight].Orientation);
    		Quaternion rightHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandRight].Orientation);
    		Quaternion rightWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristRight].Orientation);
    		Quaternion rightHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipRight].Orientation);
    		Quaternion rightThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbRight].Orientation);
    
    		// use this local position to update our real model
    		GameObject suitMan = GameObject.Find("Suit Man");
    		MeshFilter meshFilter = suitMan.GetComponentInChildren<MeshFilter>();
    		SkinnedMeshRenderer renderer = suitMan.GetComponentInChildren<SkinnedMeshRenderer>();
    		Mesh myMesh = renderer.sharedMesh;
    
    		// get game objects
    		GameObject neck = GameObject.Find("neck");
    		GameObject head = GameObject.Find("head");
    		GameObject hips = GameObject.Find("hips");
    		GameObject spine = GameObject.Find("spine");
    		GameObject chest = GameObject.Find("chest");
    		GameObject leftClavicle = GameObject.Find("clavicle.L");
    		GameObject leftArm = GameObject.Find("upper_arm.L");
    		GameObject leftForearm = GameObject.Find("forearm.L");
    		GameObject leftHand = GameObject.Find("hand.L");
    		GameObject rightClavicle = GameObject.Find("clavicle.R");
    		GameObject rightArm = GameObject.Find("upper_arm.R");
    		GameObject rightForearm = GameObject.Find("forearm.R");
    		GameObject rightHand = GameObject.Find("hand.R");
    
    		// calculate bones from positions
    		Vector3 spineBone = spineMidPosition - spineBasePosition;
    		Vector3 neckBone = neckPosition - spineShoulderPosition;
    		Vector3 headBone = headPosition - neckPosition;
    		Vector3 chestBone = spineShoulderPosition - spineMidPosition;
    		Vector3 leftShoulderBone = leftShoulderPosition - spineShoulderPosition;
    		Vector3 leftArmBone = leftElbowPosition - leftShoulderPosition;
    		Vector3 rightShoulderBone = rightShoulderPosition - spineShoulderPosition;
    		Vector3 rightArmBone = rightElbowPosition - rightShoulderPosition;
    
    		// hips
    		Quaternion myTest = spineBaseOrientation;
    		Quaternion myTest0 = Quaternion.AngleAxis(myTest.eulerAngles.y, new Vector3(0.0f, 1.0f, 0.0f));
    		hips.transform.localRotation = Quaternion.Inverse(hipStart) * myTest0;
    
    		// spine
    		Quaternion myTest6 = Quaternion.AngleAxis(myTest.eulerAngles.x, new Vector3(-1.0f, 0.0f, 0.0f)) * Quaternion.AngleAxis(myTest.eulerAngles.z, new Vector3(0.0f, 0.0f, 1.0f));
    		spine.transform.localRotation = Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest6;
    
    		// chest
    		Quaternion myTest5 = Quaternion.Inverse(spineBaseOrientation) * spineMidOrientation;
    		chest.transform.localRotation = Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest5;
    
    		// neck
    		Quaternion myTest2 = Quaternion.Inverse(spineShoulderOrientation) * neckOrientation;
    		Debug.Log("neck rotation="+myTest2.ToString());
    		neck.transform.localRotation = Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest2;
    
    		// head
    		Quaternion myTest3 = Quaternion.Inverse(neckOrientation) * headOrientation;
    		Debug.Log("head rotation="+myTest3.ToString());
    		head.transform.localRotation = Quaternion.Inverse(headStart) * Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest3;
    
    		// left shoulder
    		Quaternion myTest4 = Quaternion.Inverse(spineShoulderOrientation) * leftShoulderOrientation;
    		Debug.Log("shoulder rotation="+myTest4.ToString());
    		leftClavicle.transform.localRotation = Quaternion.Inverse(leftClavicleStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse (hipStart) * myTest4;


    And as per your advice I changed my conversion to Quaternions to read...

    	private Quaternion QuaternionFromVector4(Windows.Kinect.Vector4 inVector)
    	{
    		return new Quaternion(inVector.X, inVector.Y, -inVector.Z, -inVector.W);
    	}

    Here's how it looks if I stand in front of the Kinect in a t-pose (Btw, only spine, neck, head and left shoulder are plugged in at this point)...




    • Edited by _Ray Price_ Wednesday, July 30, 2014 4:51 PM clarity
    Wednesday, July 30, 2014 4:43 AM
  • Interestingly, I tried this with a different model and got a different rotation using the same code.  How can that be?  Is there some assumption here that I have to transform my model into some kind of base pose (t-pose?) before I apply the Kinect rotations?


    • Edited by _Ray Price_ Wednesday, July 30, 2014 10:25 PM clarity
    Wednesday, July 30, 2014 9:44 PM
  • So yeah. I'm getting the same thing!

    How are JointOrientations are absolute?

    Is the value of each joint absolute to the neck being parent to all at 0,0,0,0?

    What is the value of the points?

    Here's what I'm finding from recording skeleton standing in front and spitting out the following:
    JointOrientations bone

    bone.Orientation.X  +

    " "+ bone.Orientation.Y  + " "+ bone.Orientation.Z  + " "+ bone.Orientation.W

    SpineBase = -0.003993207 0.9973675 -0.05752403 -0.04396751 
    SpineMid = -0.003899194 0.9972942 -0.05753048 -0.04559736 
    Neck = -0.008536474 0.9930239 -0.1077237 -0.04718429 
    Head = 0 0 0 0 
    ShoulderLeft = 0.7685696 -0.6336123 0.02236257 0.08565006 
    ElbowLeft = 0.8011876 -0.1689823 -0.5193725 -0.2445313 
    WristLeft = 0.6290028 0.3751864 -0.5778305 -0.3601424 
    HandLeft = 0.6086057 0.4009017 -0.5622655 -0.3908123 
    
    ShoulderRight = 0.7867208 0.6096304 -0.09320773 0.02708374 
    ElbowRight = 0.7174323 0.1195038 0.6718429 -0.1401314 
    WristRight = 0.7895443 -0.09435756 0.1834147 -0.5779925 
    HandRight = 0.7634736 -0.1194136 0.1618041 -0.6137329 
    
    HipLeft = -0.6884276 0.7018301 -0.1602328 0.08847217 KneeLeft = -0.6741655 0.09152499 0.731129 -0.05073958 AnkleLeft = -0.5685248 0.08804499 0.8059186 -0.1397243 FootLeft = 0 0 0 0 
    HipRight = 0.6641833 0.7015544 -0.2135341 -0.1452072 KneeRight = 0.7370378 0.1156219 0.6658675 -0.005235625 AnkleRight = 0.5236376 0.1570109 0.8365145 0.03734707 FootRight = 0 0 0 0 
    
    

    Ken

    Wednesday, July 30, 2014 10:23 PM
  • Here's the first few of mine as euler angles after being converted to local space (relative to parent bone) using quaternion.inverse(parent) * child.

    spine rotation=(0.0, 0.4, 0.0)
    neck rotation=(357.8, 0.0, 359.8)
    head rotation=(0.0, 0.0, 0.0)
    left shoulder rotation=(359.4, 357.4, 258.7)
    left arm rotation=(353.9, 108.3, 4.9)

    Interestingly, even if I tilt my head all the way to the side, I still only get...

    neck rotation=(357.9, 0.0, 359.9)

    • Edited by _Ray Price_ Wednesday, July 30, 2014 10:36 PM
    Wednesday, July 30, 2014 10:34 PM
  • I had similar initial results with arms facing the wrong way when I was testing some models. I think it is necessary to multiply the relative rotation by the bone/joint's bind pose rotation:

    Quaternion newRotation = bindRotation * parentQ.Inverse() * childQ;

    (Possibly bindRotation should be multiplied after childQ but not sure.)

    Haven't had a chance to test this but this might help.


    Joshua Blake
    Kinect for Windows MVP
    OpenKinect Community Founder
    Technical Director, InfoStrat Advanced Technology Group
    Blog: http://nui.joshland.org
    Twitter: http://twitter.com/joshblake

    Wednesday, July 30, 2014 10:41 PM
    Moderator
  • Hi Josh,

    Yeah, I've been trying that too, hence the...

    head.transform.localRotation = Quaternion.Inverse(headStart) * Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest3;
    

    But I may have got my sources wrong or overestimated the need to walk up the chain.  Where are you picking your bind pose rotations from?  I have the following code in my start()...

    		GameObject hips = GameObject.Find("hips");
    		hipStart = hips.transform.localRotation;
    		GameObject spine = GameObject.Find("spine");
    		spineStart = spine.transform.localRotation;
    		GameObject neck = GameObject.Find("neck");
    		neckStart = neck.transform.localRotation;
    		GameObject head = GameObject.Find("head");
    		headStart = head.transform.localRotation;
    		GameObject chest = GameObject.Find("chest");
    		chestStart = chest.transform.localRotation;
    
    		GameObject leftClavicle = GameObject.Find("clavicle.L");
    		leftClavicleStart = leftClavicle.transform.localRotation;
    		GameObject leftArm = GameObject.Find("upper_arm.L");
    		leftArmStart = leftArm.transform.localRotation;
    		GameObject leftForearm = GameObject.Find("forearm.L");
    		leftForearmStart = leftForearm.transform.localRotation;
    		GameObject leftHand = GameObject.Find("hand.L");
    		leftHandStart = leftHand.transform.localRotation;

    But I think this may not be the right place, as it seems each model can be slightly different in it's first frame.

    But even so, that aside the neck rotation is not changing AT ALL in the data coming back from the Kinect, and also when I have my arm out straight from my shoulder, the local (inverse(parent)*child) as euler angles is coming back as...

    left arm rotation=(353.9, 108.3, 4.9)

    Which looks VERY strange considering the shoulder and upper arm should be practically in alignment at that point.

    Wednesday, July 30, 2014 10:47 PM
  • Ray,

    If your model/rigging is set up hierarchically (such that the leftForearm transform is a child of the leftArm transform) then I don't think you need to walk the hierarchy. Just multiply the appropriate joint's local start rotation, e.g.

    forearmL.transform.localRotation = leftForearmStart * inverse(leftShoulderKinect) * leftElbowKinect;

    (I think...)

    Which specific joints were you using to get the left arm rotation = (353.9, 108.3, 4.9)? The bone -> joint mapping parent/child is not intuitive IMHO. For example, if you want the rotation of the elbow joint, then IIRC you need to do inverse(elbowJointLeft.Rotation) * wristJointLeft.Rotation. 

    For shoulder, you would need to do inverse(shoulderJointLeft.Rotation) * elbowJointLeft.Rotation (i.e. elbowJointLeft is the "current" joint. I know.)

    The euler angle you suggested would be consistent with the inverse(shoulderSpineJoint) * shoulderJointLeft (or right), which in the current API is really the angle between the SpineMid -> SpineShoulder bone and the SpineShoulder->ShoulderLeft bone, which would be nearly a 90 degree angle even with your arm straight out. (The absolute SpineShoulder->ShoulderLeft bone rotation, stored in the ShoulderLeft Joint, would also be a similar angle unless you lean your body.)


    Joshua Blake
    Kinect for Windows MVP
    OpenKinect Community Founder
    Technical Director, InfoStrat Advanced Technology Group
    Blog: http://nui.joshland.org
    Twitter: http://twitter.com/joshblake

    Wednesday, July 30, 2014 10:58 PM
    Moderator
  • Hi Josh,

    Thanks for your reply again.  For left arm I was doing...

    Quaternion myTest5 = Quaternion.Inverse(leftShoulderOrientation) * leftElbowOrientation;
    		

    Yet you're suggesting this is coming back with data that looks like I was using Inverse(midSpine) * leftShoulder?

    As for not needing to walk up the hierarchy, I have tried that too, but in your code...

    forearmL.transform.localRotation = leftForearmStart * inverse(leftShoulderKinect) * leftElbowKinect;

    Where are you picking up the leftForearmStart data from?  Because it seems each model is potentially posed different in it's setting in the start() code (and not necessarily T).  Surely before we apply the local rotation we need to apply some kind of rotation to put the model into it's t-pose since it seems that setting all the rotations to zero in the editor does NOT produce a t-pose for all models either.  How does Unity animation controller do this?

    Also, why do the head/neck angles not change even when my head is fully tilted?  Something odd is going on here.


    • Edited by _Ray Price_ Wednesday, July 30, 2014 11:41 PM
    Wednesday, July 30, 2014 11:39 PM
  • Is there any way we can test the rotations coming out of the Kinect SDK and compare them against the Unity plug-in.  I tried Kinect Studio, but it looks like I can see the raw rotation data.
    Thursday, July 31, 2014 6:21 PM
  • Ok, so I dug into the SDK and modified one of the samples to get the RAW quaternions to compare with what Unity is getting while I stand in a t-pose.  In C# in the raw SDK, I get...

    spine base = 0.005743555,0.9976618,0.06572407,0.0178384
    spine mid = 0.005864908,0.9976272,0.06571335,0.0196806
    spine shoulder = 0.004756359,0.9984642,0.05081941,0.0215392
    left shoulder = 0.7682579,-0.6372728,-0.009667866,-0.05974606
    left elbow = -0.2287592,0.2777363,0.7082614,0.6073694
    head = 0,0,0,0
    neck = 0.004115697,0.9992705,0.03131068,0.02147636

    In Unity I get...

    spine base = -0.0042993,0.9973056,0.0537723,-0.04971632
    spine mid = -0.003943218,0.9976128,0.05379959,-0.04311278
    spine shoulder = -0.005316815,0.9985826,0.03853904,-0.03632149
    left shoulder = 0.7760947,-0.6279857,-0.05753399,0.000930806
    left elbow = -0.3599467,0.4590581,0.6543876,0.4811246
    head = 0,0,0,0
    neck = -0.007668501,0.9991321,0.01916226,-0.03618148

    Give or take a little expected deviation they seem to be about the same, which is good.  So I tried again with the raw SDK with my head turned 90 degrees to the left and got...

    head = 0,0,0,0
    neck = -0.001263987,0.9981489,0.01254379,-0.05949736

    So for sure even the base SDK is not picking up head rotation.

    Are there issues in the rotation code for the SDK?  Is this why we haven't seen an avateering demo yet?

    Thursday, July 31, 2014 7:09 PM
  • Next step is to take the raw quaternion data and convert it into a local rotation using Inverse(parent) * child and print the euler angles to see if they make sense.  The code I used is...

    		// get rotations from kinect
    		Quaternion neckOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Neck].Orientation);
    		Quaternion headOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Head].Orientation);
    		Quaternion leftHipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HipLeft].Orientation);
    		Quaternion spineBaseOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineBase].Orientation);
    		Quaternion spineMidOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineMid].Orientation);
    		Quaternion spineShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineShoulder].Orientation);
    		Quaternion leftShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderLeft].Orientation);
    		Quaternion leftElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowLeft].Orientation);
    		Quaternion leftHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandLeft].Orientation);
    		Quaternion leftWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristLeft].Orientation);
    		Quaternion leftHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipLeft].Orientation);
    		Quaternion leftThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbLeft].Orientation);
    		Quaternion rightShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderRight].Orientation);
    		Quaternion rightElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowRight].Orientation);
    		Quaternion rightHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandRight].Orientation);
    		Quaternion rightWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristRight].Orientation);
    		Quaternion rightHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipRight].Orientation);
    		Quaternion rightThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbRight].Orientation);
    
    		// print local rotations using inverse parent * child
    		Quaternion spineMidLocal = Quaternion.Inverse(spineBaseOrientation) * spineMidOrientation;
    		Quaternion spineShoulderLocal = Quaternion.Inverse(spineMidOrientation) * spineShoulderOrientation;
    		Quaternion leftShoulderLocal = Quaternion.Inverse(spineShoulderOrientation) * leftShoulderOrientation;
    		Quaternion leftElbowLocal = Quaternion.Inverse(leftShoulderOrientation) * leftElbowOrientation;
    		Quaternion neckLocal = Quaternion.Inverse(spineShoulderOrientation) * neckOrientation;
    		Quaternion headLocal = Quaternion.Inverse(neckOrientation) * headOrientation;
    		Debug.Log("kinect local rotation spine mid = "+spineMidLocal.eulerAngles.ToString());
    		Debug.Log("kinect local rotation spine shoulder = "+spineShoulderLocal.eulerAngles.ToString());
    		Debug.Log("kinect local rotation left shoulder = "+leftShoulderLocal.eulerAngles.ToString());
    		Debug.Log("kinect local rotation left arm = "+leftElbowLocal.eulerAngles.ToString());
    		Debug.Log("kinect local rotation neck = "+neckLocal.eulerAngles.ToString());
    		Debug.Log("kinect local rotation head = "+headLocal.eulerAngles.ToString());
    

    The results I get from this are...

    kinect local rotation spine mid = (0.0, 359.7, 0.0)
    kinect local rotation spine shoulder = (1.8, 359.7, 359.8)
    kinect local rotation left shoulder = (0.4, 1.8, 257.9)
    kinect local rotation left arm = (4.4, 261.6, 6.3)
    kinect local rotation neck = (2.2, 0.0, 359.8)
    kinect local rotation head = (0.0, 0.0, 0.0)

    These numbers strike me as a little odd.  In a t-pose the arm and the shoulder are almost in alignment, so how can the rotation be 261.6 in the Y axis?  Also, how is the spine-shoulder (point up to behind the neck) not pretty much in alignment with the spine mid (point from spine base to middle of spine)?

    Thursday, July 31, 2014 7:27 PM
  • head rotations are fixed. You will get the rotation of the head from the Face API's.

    Carmine Sirignano - MSFT

    Thursday, July 31, 2014 7:58 PM
    Owner
  • Good to know, thank you. Now I just need to figure out what's going on with the other rotations.
    Thursday, July 31, 2014 8:16 PM
  • Ray:

    Do you have avatar following the Kinect skeleton bone Joint Orientation movement yet?

    I'm using XNA (right handed coordinate system) so I don't have to translate like you do.

    My forearm seems to draw ok with out the wrist and hand is fairly close but it's still not cool, no rotation of the hand, just positional relative. But atleast it not cut off or blown up like giant hulk arm, or sometimes it ends up looking like fantastic 4 rubber guy arm.  Although very intensely cool, not the motif I'm looking for.

     Quaternion childQ = QuaternionFromVector4(bone.Orientation);
    
                Quaternion parentQ = QuaternionFromVector4(skeleton.JointOrientations[parentJoint.JointType].Orientation);
                Quaternion parentQInverse = new Quaternion();
                Quaternion.Inverse(ref parentQ, out parentQInverse);
    
      combined = Matrix.Transform(avatarBoneMatrix, parentQInverse * childQ)

    KenCrazy avatar with deformed left forearm, wrist and hand.


    Ken

    Thursday, July 31, 2014 8:55 PM
  • No, I still don't have it working yet.  I'm starting to wonder what the hell these orientations represent.  The fact that the stick figure you show here, AND Kinect Studio shows the rotations as perpendicular vectors to the bone is making me wonder if these things are even world rotation representations of the joint at all.  What you're showing here shows some kind of rotation vector on the stick-model.  Do you know how that's calculated?
    Thursday, July 31, 2014 9:01 PM

  • It's a little different code for that skeleton than the avatar ...

                    // Get the constraint direction in world space from the parent orientation
                    Matrix matConstraintLocalToWorld = KinectHelper.JointOrientationToXNAMatrix(skeleton.JointOrientations[jc.Joint].Orientation);
    
                    Vector3 constraintDirWs = jc.Dir;
                    constraintDirWs.Normalize();
                    Vector3 rotatedConstraintDirWs = Vector3.Transform(constraintDirWs, matConstraintLocalToWorld);
                    rotatedConstraintDirWs.Normalize();
    
                    // Draw the constraint direction line itself
                    rotatedConstraintDirWs *= 0.5f;
                    this.AddRotatedLine(rotatedConstraintDirWs, skeleton, i, Color.DeepPink);

    This is how jc.DIR got set at setup:

            public void AddDefaultConstraints()
            {
                // Constraints indexed on end joint (i.e. constrains the bone between start and end), but constraint applies at the start joint (i.e. the parent)
                // The constraint is defined in the local coordinate system of the parent bone, relative to the parent bone direction 
    
                // Acts at Hip Center (constrains hip center to spine bone)
                this.AddBoneOrientationConstraint(JointType.SpineBase, new Vector3(0.0f, 1.0f, 0.3f), 90.0f);
    
                // Acts at Spine (constrains spine to shoulder center bone)
                this.AddBoneOrientationConstraint(JointType.SpineShoulder, new Vector3(0.0f, 1.0f, 0.0f), 50.0f);
    
                // Acts at Shoulder Center (constrains shoulder center to head bone)
                this.AddBoneOrientationConstraint(JointType.Head, new Vector3(0.0f, 1.0f, 0.3f), 45.0f);
    
                // Acts at Shoulder Joint (constraints shoulder-elbow bone)
                this.AddBoneOrientationConstraint(JointType.ElbowLeft, new Vector3(0.1f, 0.7f, 0.7f), 80.0f);   // along the bone, (i.e +Y), and forward +Z, enable 80 degrees rotation away from this
                this.AddBoneOrientationConstraint(JointType.ElbowRight, new Vector3(-0.1f, 0.7f, 0.7f), 80.0f);
    
                // Acts at Elbow Joint (constraints elbow-wrist bone)
                this.AddBoneOrientationConstraint(JointType.WristLeft, new Vector3(0.0f, 0.0f, 1.0f), 90.0f);   // +Z (i.e. so rotates up or down with twist, when arm bent, stop bending backwards)
                this.AddBoneOrientationConstraint(JointType.WristRight, new Vector3(0.0f, 0.0f, 1.0f), 90.0f);
    
                // Acts at Wrist Joint (constrains wrist-hand bone)
                this.AddBoneOrientationConstraint(JointType.HandLeft, new Vector3(0.0f, 1.0f, 0.0f), 45.0f);    // +Y is along the bone
                this.AddBoneOrientationConstraint(JointType.HandRight, new Vector3(0.0f, 1.0f, 0.0f), 45.0f);
    
                // Acts at Hip Joint (constrains hip-knee bone)
                this.AddBoneOrientationConstraint(JointType.KneeLeft, new Vector3(0.5f, 0.7f, 0.1f), 65.0f);
                this.AddBoneOrientationConstraint(JointType.KneeRight, new Vector3(-0.5f, 0.7f, 0.1f), 65.0f);
    
                // Acts at Knee Joint (constrains knee-ankle bone)
                this.AddBoneOrientationConstraint(JointType.AnkleRight, new Vector3(0.0f, 0.7f, -1.0f), 65.0f); // enable bending backwards with -Z
                this.AddBoneOrientationConstraint(JointType.AnkleLeft, new Vector3(0.0f, 0.7f, -1.0f), 65.0f);
    
                // Acts at Ankle Joint (constrains ankle-foot bone)
                this.AddBoneOrientationConstraint(JointType.FootRight, new Vector3(0.0f, 0.3f, 0.5f), 40.0f);
                this.AddBoneOrientationConstraint(JointType.FootLeft, new Vector3(0.0f, 0.3f, 0.5f), 40.0f);
            }


    Ken

    Thursday, July 31, 2014 9:22 PM
  • Where does this object's Dir variable get initialized?...

    Vector3 constraintDirWs = jc.Dir;


    • Edited by _Ray Price_ Thursday, July 31, 2014 10:06 PM
    Thursday, July 31, 2014 10:03 PM
  • it's called in the constructor method of the main game.

                // Setup the bone orientation constraint system
                this.boneOrientationConstraints.AddDefaultConstraints()

    Ken

    Thursday, July 31, 2014 11:01 PM
  • Is this sample code in the SDK?  I couldn't see it in the sample browser.  Which directory is it in?  Thanks.
    Friday, August 01, 2014 1:57 PM
  • Is this sample code in the SDK?  I couldn't see it in the sample browser.  Which directory is it in?
    Friday, August 01, 2014 4:07 PM
  • I started with with 1.8 SDK sample avateering.  Let me see about setting up a github or source control for the changes I made, there were a great deal of them to get v1 migrated to v2.
    http://msdn.microsoft.com/en-us/library/jj131041.aspx

    Here's what I'm thinking this morning, as last night I hacked so many different variations that Edison would be proud.   Now I'm back to noodling mode. 

    The vector4 which is returned from JointOrientation.Orientation.  Of course, the primitive auto generated code doc is completely worthless ( method Foo  gets/sets a Foo).    Here's what I think I know of the puzzle.

    1> We believe the Vector4 returned is of Unit Type of Quarternion (Yaw,Pitch,Roll ) and which order?
    2> Below suggests it might be referenced as view down each bone (per V1 spec for Matrix).
    3> Head is (0,0,0) it seems "absolute" rotations related to the head. FK to find the hand position.

    The old stuff had a discussion here:
    http://social.msdn.microsoft.com/Forums/en-US/4e90b413-b302-4c29-a8da-cb08ebc53f9c/kinect-jointorientation-and-bonerotation-matrix-questions?forum=kinectsdk 

    I'm going to push my work to codeplex.


    Ken

    Friday, August 01, 2014 4:08 PM
  • That would be great.  If I can see how you're handling the rotations you've got setup there for the stick figure which seem to be working, maybe it'll shine some light on what's going on.

    It's supposed to be a quaternion, so yes it's an angle around a particular vector.  And it seems to work fine for the root (spine), but for anything other than that even transferring it into local space doesn't seem to get the required results.  Of course I'm sure this is down to my own ineptitude, as it seems to be returning the same results as the base SDK.  I'll continue to hack too and let you know if I make any progress.

    Friday, August 01, 2014 4:12 PM
  • This is it for Visual Studio and GIT. It's C# code.

    https://kinectavateeringv2migration.codeplex.com/


    Ken

    Friday, August 01, 2014 4:42 PM
  • I put your sample code in AvateeringXNA.cs into a method called

            private Matrix processWholeBody(JointOrientation bone, Body body, Matrix bindRoot, ref Matrix[] boneTransforms)

    called it from the beginning of method:

    SetJointTransformation

    Basically for the extremities the code did the following sequences:

    1. convert bone.orientation to XNAMatrix
    2. use XNAMatrix to decompose the XNA Quaternion rot (basically just converted k.Orientation to X.Q
    3. map the K.vector4 quaternions to XNA.Quaternions (Y to X, Z to Y, X to Z, W=W) ** this is the devil.
    4. create a new XNA Matrix from avatar mapped Quaternion
    5. pass this to ReplaceBoneMatrix method (next step is inside other method)
    6. boneTransforms[arraySize of Avatar effects/joints) change it's "Translation" to the next value calc'd

    So basically, the "SetJoinTransformation" is trying to find the new / mapped "Translation" of the avatar's effect joint's position. 

    The Kinect gives the solution via the JointOrientation.Orientation vector4 (you say it's a Quaternion). Why can't the SDK docs say that?  

    I think these values are Normalized to what?   2pi = 1 or pi=1 Given they go negative, then pi = 1.

    Further given the suggestions, they might be relative to the previous joint, although it's stated they are absolute. If they are absolute then to what? The head being zero. So then we should translate the inverse related to the head and not the previous parent joint?   IDK. thinking out loud here.

    The code cloned to codeplex, is hacked away to the point that I don't know what I've tried and I'm sick of trying random transforms combinations. Your help is very much appreciated.

    I've recorded a V2 stream of me moving my arms around (you can see my ghost dog walking in wonder if it's play time.) I'll put this into the git repo as content too.

    I'm new to 3d animation, so it's my first rodeo. So the body form of Dude, is fixed, i'm attempting to just have the arms flail around starting at the shoulder joint. The current version as of right now, x,yz,w mappings figgering is just trying to do the left elbow, wrist, & hand.  Everytime I attempt to add the LeftShoulder the thing goes super crazy. 

    Just some thoughts on annotating what I'm doing there.


    Ken

    Friday, August 01, 2014 5:47 PM
  • The documentation says...

    Joint normals

    There is a normal for each skeleton joint that describes rotation. Rotation is expressed as the vector (in world space) perpendicular to the attached bones in the joint hierarchy. For example, to determine the roll of the right elbow, the immediate parent joint in the hierarchy, the right shoulder, is used to determine the plane of the bone.

    Which doesn't make sense because a) there is no 'normal' property on the joint, and b) if this orientation were a normal, why would it have a 'w' component?

    Friday, August 01, 2014 6:05 PM
  • the Kinect gesture builder allow you to see the raw rotation data:

    Right click in the right point cloud view > body frame> body with joint orientation. Then you will see the raw orientation

    Friday, August 01, 2014 6:55 PM
  • Friday, August 01, 2014 7:24 PM
    Owner
  • Here is an explanation of the values:

    http://social.msdn.microsoft.com/Forums/en-US/31c9aff6-7dab-433d-9af9-59942dfd3d69/kinect-v20-preview-sdk-jointorientation-vs-boneorientation?forum=kinectv2sdk


    Carmine Sirignano - MSFT

    Hi Carmine,

    Thank you so much for this link.  If this value was truely an expression of the bone rotation as a quaternion in world space then...

    Quaternion.Inverse(leftShoulderOrientation) * leftElbowOrientation

    should give the local orientation relative to the parent, right?  I should then be able to assign this in Unity to the bone.transform.localRotation property, but alas this seems to result in pretty much chaos.

    Regards

    Ray


    • Edited by _Ray Price_ Saturday, August 02, 2014 4:41 AM
    Saturday, August 02, 2014 3:46 AM
  • Latest zero progress update.

    I'm using XNA not unity But the problem/solution domain is very similar:

    #1> Convert Kinect Vector4 Quarternion to XNA/Someother.Quarternion

    #2> Matrix.Transform this result into a new worldmap matrix for each of the bones.

    #1> code question of how to map the Yaw, Pitch, Roll into a XNAQuarternion.

     Quaternion QuaternionFromVector4(Microsoft.Kinect.Vector4 inVector)
       {
         //Quaternion.CreateFromYawPitchRoll (Single, Single, Single)
         Quaternion converted = Quaternion.CreateFromYawPitchRoll(inVector.X, inVector.Z, inVector.Y);
    
                return converted;
            }

    or is it    yaw= Y,  pitch = Z and roll=X?  

    What is the mapping to covert a Kinect Quarternion to an XNA or anybody elses?

    Basically, once we have a valid parent and object bone quarternion, then how to transform it?

            private Matrix processBone2(JointOrientation bone, Body skeleton, Matrix bindRoot, ref Matrix[] boneTransforms)
            {
                JointType parentIdx = KinectHelper.ParentBoneJoint(bone.JointType);
                // Calculate the absolute/world equivalents of the hierarchical rotation
                Quaternion parentRotation = KinectHelper.QuaternionFromVector4(skeleton.JointOrientations[parentIdx].Orientation);
                Quaternion relativeRotation = KinectHelper.QuaternionFromVector4(bone.Orientation);
    
                // Create a new world rotation Quaternion.Inverse(leftShoulderOrientation) * leftElbowOrientation
                Quaternion worldRotation = (parentRotation * relativeRotation);
                    //Quaternion.Multiply(parentRotation, relativeRotation);
    
                Matrix temp = new Matrix();
                Matrix.Transform(ref boneTransforms[meshJointId], ref relativeRotation, out temp);
                return temp;
            }

    There are just too many indeterminate to sit and guess combinations.

    If the XNA quarternion mapping /creation is wrong. Then everything down stream of that is bogus.

    When it's said that Y is the Roll then I assume that Z parameter for a standard Quarterion "roll" yes?

    latest failure update


    Ken

    Thursday, August 07, 2014 1:33 AM
  • Hi Ken,

    Thanks for the update post.  Where did you get the QuaternionFromVector4 code from?  Carmine said...

    Each quaternion is the absolute orientation of the parent bone.

    Which seems to suggest it is already a quaternion (thus needs no conversion), so I'm curious if you pulled this from a KinectV2 sample somewhere.  Unfortunately despite Carmine's remark I've done some tests with Quaternion.Inverse(Parent)*Child which should give the local rotation, but with the same bend and rotation in the upper arm and forearm the result is different rotation values as the shoulder is rotated, which suggests that the returned value is NOT local to the parent bone after the usual quaternion world to local conversion.

    Would be nice if Microsoft would throw us a bone here.  (Get it!? :P).

    • Edited by _Ray Price_ Thursday, August 07, 2014 2:24 AM
    Thursday, August 07, 2014 2:14 AM
  • @_Ray Price_, I was working with Unity and Kinect. and  I noticed you have your own custom method GetVector3FromCameraSpacePoint. I was wondering if you could elaborate on how you converted to a Unity Vector3?
    Wednesday, January 28, 2015 1:52 AM
  • Can u share ur project code source ??
    Monday, March 02, 2015 11:47 AM