Asked by:
Unity JointOrientations to Bone Mapping Not Quite Right

I'm trying to map the KinectV2 bone rotations (JointOrientations) to unity bones, but it's not quite right. If I isolate it to the spine only it seems off by the amount of the initial value of the bind pose. However, I've tried saving this bind rotation quaternion in the start() code and then multiplying it by Kinect's rotation quaternion, but that doesn't work either. I tried taking the inverse of that quaternion too, but still no luck.
I'm sure this is newbie stuff, but does anyone know what I may be doing wrong here?
Thanks
Ray
Question
All replies


Hi Ben,
Thanks for the reply. The DLL says version 2.0.1407.14000. It's basically the version from the KinectForWindows_1407_Unity (Pro Only).zip from the website.
As far as I can see this is the latest available on the website. Have you guys done a more recent build?
Thanks
Ray

Just FYI, this is my code so far. In start(), I have...
GameObject hips = GameObject.Find("hips"); hipStart = hips.transform.localRotation; GameObject spine = GameObject.Find("spine"); spineStart = spine.transform.localRotation; GameObject neck = GameObject.Find("neck"); neckStart = neck.transform.localRotation; GameObject head = GameObject.Find("head"); headStart = head.transform.localRotation; GameObject chest = GameObject.Find("chest"); chestStart = chest.transform.localRotation; GameObject leftClavicle = GameObject.Find("clavicle.L"); leftClavicleStart = leftClavicle.transform.localRotation; GameObject leftArm = GameObject.Find("upper_arm.L"); leftArmStart = leftArm.transform.localRotation; GameObject leftForearm = GameObject.Find("forearm.L"); leftForearmStart = leftForearm.transform.localRotation; GameObject leftHand = GameObject.Find("hand.L"); leftHandStart = leftHand.transform.localRotation; GameObject rightClavicle = GameObject.Find("clavicle.R"); rightClavicleStart = rightClavicle.transform.localRotation; GameObject rightArm = GameObject.Find("upper_arm.R"); rightArmStart = rightArm.transform.localRotation; GameObject rightForearm = GameObject.Find("forearm.R"); rightForearmStart = rightForearm.transform.localRotation; GameObject rightHand = GameObject.Find("hand.R"); rightHandStart = rightHand.transform.localRotation;
Then in update() I have...
// get positions from kinect Vector3 neckPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.Neck].Position); Vector3 headPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.Head].Position); Vector3 leftHipPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HipLeft].Position); Vector3 spineBasePosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineBase].Position); Vector3 spineMidPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineMid].Position); Vector3 spineShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineShoulder].Position); Vector3 leftShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ShoulderLeft].Position); Vector3 leftElbowPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ElbowLeft].Position); Vector3 leftHandPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HandLeft].Position); Vector3 leftWristPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.WristLeft].Position); Vector3 leftHandTipPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HandTipLeft].Position); Vector3 leftThumbPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ThumbLeft].Position); // get rotations from kinect Quaternion neckOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Neck].Orientation); Quaternion headOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Head].Orientation); Quaternion leftHipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HipLeft].Orientation); Quaternion spineBaseOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineBase].Orientation); Quaternion spineMidOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineMid].Orientation); Quaternion spineShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineShoulder].Orientation); Quaternion leftShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderLeft].Orientation); Quaternion leftElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowLeft].Orientation); Quaternion leftHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandLeft].Orientation); Quaternion leftWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristLeft].Orientation); Quaternion leftHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipLeft].Orientation); Quaternion leftThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbLeft].Orientation); // use this local position to update our real model GameObject suitMan = GameObject.Find("Suit Man"); MeshFilter meshFilter = suitMan.GetComponentInChildren<MeshFilter>(); SkinnedMeshRenderer renderer = suitMan.GetComponentInChildren<SkinnedMeshRenderer>(); Mesh myMesh = renderer.sharedMesh; // get game objects GameObject neck = GameObject.Find("neck"); GameObject head = GameObject.Find("head"); GameObject hips = GameObject.Find("hips"); GameObject spine = GameObject.Find("spine"); GameObject chest = GameObject.Find("chest"); GameObject leftClavicle = GameObject.Find("clavicle.L"); GameObject leftArm = GameObject.Find("upper_arm.L"); GameObject leftForearm = GameObject.Find("forearm.L"); GameObject leftHand = GameObject.Find("hand.L"); // calculate bones from positions Vector3 spineBone = spineMidPosition  spineBasePosition; Vector3 neckBone = neckPosition  spineShoulderPosition; Vector3 headBone = headPosition  neckPosition; Vector3 chestBone = spineShoulderPosition  spineMidPosition; Vector3 leftShoulderBone = leftShoulderPosition  spineShoulderPosition; Vector3 leftArmBone = leftElbowPosition  leftShoulderPosition; // spine Quaternion myTest = Quaternion.FromToRotation (new Vector3(0.0f, 1.0f, 0.0f), spineBone); // myTest = QuaternionFromVector4(spineBaseOrientation); Quaternion myTest5 = Quaternion.FromToRotation (spineBone, chestBone); spine.transform.localRotation = Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest; chest.transform.localRotation = Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest5; // neck Quaternion myTest2 = Quaternion.FromToRotation (chestBone, neckBone); // myTest2 = Quaternion.Inverse(spineShoulderOrientation) * QuaternionFromVector4(neckOrientation); Debug.Log("neck rotation="+myTest2.ToString()); neck.transform.localRotation = Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest2; // head Quaternion myTest3 = Quaternion.FromToRotation (neckBone, headBone); // myTest3 = Quaternion.Inverse(neckOrientation) * QuaternionFromVector4(headOrientation); Debug.Log("head rotation="+myTest3.ToString()); head.transform.localRotation = Quaternion.Inverse(headStart) * Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest3; // left shoulder Quaternion myTest4 = Quaternion.FromToRotation (chestBone, leftShoulderBone); // myTest4 = Quaternion.Inverse(spineShoulderOrientation) * leftShoulderOrientation; Debug.Log("shoulder rotation="+myTest4.ToString()); leftClavicle.transform.localRotation = Quaternion.Inverse(leftClavicleStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse (hipStart) * myTest4;
The body and head seem to work fine by the shoulder is going a little haywire. Also, when I try and use the rotations (commented out code) it just goes nuts for everything.
Regards
Ray

I also tried this using the rotations instead, but then the neck and the head don't rotate at all and the shoulder still goes haywaire...
// hips // Quaternion myTest = Quaternion.FromToRotation (new Vector3(0.0f, 1.0f, 0.0f), spineBone); Quaternion myTest = spineBaseOrientation; Quaternion myTest0 = Quaternion.AngleAxis(myTest.eulerAngles.y, new Vector3(0.0f, 1.0f, 0.0f)); hips.transform.localRotation = Quaternion.Inverse(hipStart) * myTest0; // spine Quaternion myTest6 = Quaternion.AngleAxis(myTest.eulerAngles.x, new Vector3(1.0f, 0.0f, 0.0f)) * Quaternion.AngleAxis(myTest.eulerAngles.z, new Vector3(0.0f, 0.0f, 1.0f)); spine.transform.localRotation = Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest6; // chest Quaternion myTest5 = Quaternion.Inverse(spineBaseOrientation) * spineMidOrientation; chest.transform.localRotation = Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest5; // neck // Quaternion myTest2 = Quaternion.FromToRotation (chestBone, neckBone); Quaternion myTest2 = Quaternion.Inverse(spineShoulderOrientation) * neckOrientation; Debug.Log("neck rotation="+myTest2.ToString()); neck.transform.localRotation = Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest2; // head // Quaternion myTest3 = Quaternion.FromToRotation (neckBone, headBone); Quaternion myTest3 = Quaternion.Inverse(neckOrientation) * headOrientation; Debug.Log("head rotation="+myTest3.ToString()); head.transform.localRotation = Quaternion.Inverse(headStart) * Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest3; // left shoulder // Quaternion myTest4 = Quaternion.FromToRotation (chestBone, leftShoulderBone); Quaternion myTest4 = Quaternion.Inverse(spineShoulderOrientation) * leftShoulderOrientation; Debug.Log("shoulder rotation="+myTest4.ToString()); leftClavicle.transform.localRotation = Quaternion.Inverse(leftClavicleStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse (hipStart) * myTest4;

Replied to your other thread, but the JointOrientations are absolute. To relative you'll need to do
var relative = parentQuaternion.Inverse() * quaternion;
as well as possibly rotate the quaternion depending upon the axis orientation of your world. For Unity, I've found I need to flip the Zaxis due to the righthanded vs lefthanded in Unity. I know quaternions technically aren't handed, but this was necessary anyway. To flip an axis, you need to negate the axis and the angle, like so:
static Quaternion ConvertToLeftHanded(Quaternion q)
{
return new Quaternion(q.x, q.y, q.z, q.w);
}
Joshua Blake
Kinect for Windows MVP
OpenKinect Community Founder
Technical Director, InfoStrat Advanced Technology Group
Blog: http://nui.joshland.org
Twitter: http://twitter.com/joshblake 
That's pretty much what I had in my second example, except without the handedness change of quaternions (which I also read were not handed). But, still no go. The spine rotation seems to work well, but the head and neck don't seem to be moving at all and the shoulder is just way out still. Here's the code as it stands...
// get positions from kinect Vector3 neckPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.Neck].Position); Vector3 headPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.Head].Position); Vector3 leftHipPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HipLeft].Position); Vector3 spineBasePosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineBase].Position); Vector3 spineMidPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineMid].Position); Vector3 spineShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.SpineShoulder].Position); Vector3 leftShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ShoulderLeft].Position); Vector3 leftElbowPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ElbowLeft].Position); Vector3 leftHandPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HandLeft].Position); Vector3 leftWristPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.WristLeft].Position); Vector3 leftHandTipPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.HandTipLeft].Position); Vector3 leftThumbPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ThumbLeft].Position); Vector3 rightShoulderPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ShoulderRight].Position); Vector3 rightElbowPosition = GetVector3FromCameraSpacePoint(body.Joints[Kinect.JointType.ElbowRight].Position); // get rotations from kinect Quaternion neckOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Neck].Orientation); Quaternion headOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Head].Orientation); Quaternion leftHipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HipLeft].Orientation); Quaternion spineBaseOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineBase].Orientation); Quaternion spineMidOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineMid].Orientation); Quaternion spineShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineShoulder].Orientation); Quaternion leftShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderLeft].Orientation); Quaternion leftElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowLeft].Orientation); Quaternion leftHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandLeft].Orientation); Quaternion leftWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristLeft].Orientation); Quaternion leftHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipLeft].Orientation); Quaternion leftThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbLeft].Orientation); Quaternion rightShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderRight].Orientation); Quaternion rightElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowRight].Orientation); Quaternion rightHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandRight].Orientation); Quaternion rightWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristRight].Orientation); Quaternion rightHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipRight].Orientation); Quaternion rightThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbRight].Orientation); // use this local position to update our real model GameObject suitMan = GameObject.Find("Suit Man"); MeshFilter meshFilter = suitMan.GetComponentInChildren<MeshFilter>(); SkinnedMeshRenderer renderer = suitMan.GetComponentInChildren<SkinnedMeshRenderer>(); Mesh myMesh = renderer.sharedMesh; // get game objects GameObject neck = GameObject.Find("neck"); GameObject head = GameObject.Find("head"); GameObject hips = GameObject.Find("hips"); GameObject spine = GameObject.Find("spine"); GameObject chest = GameObject.Find("chest"); GameObject leftClavicle = GameObject.Find("clavicle.L"); GameObject leftArm = GameObject.Find("upper_arm.L"); GameObject leftForearm = GameObject.Find("forearm.L"); GameObject leftHand = GameObject.Find("hand.L"); GameObject rightClavicle = GameObject.Find("clavicle.R"); GameObject rightArm = GameObject.Find("upper_arm.R"); GameObject rightForearm = GameObject.Find("forearm.R"); GameObject rightHand = GameObject.Find("hand.R"); // calculate bones from positions Vector3 spineBone = spineMidPosition  spineBasePosition; Vector3 neckBone = neckPosition  spineShoulderPosition; Vector3 headBone = headPosition  neckPosition; Vector3 chestBone = spineShoulderPosition  spineMidPosition; Vector3 leftShoulderBone = leftShoulderPosition  spineShoulderPosition; Vector3 leftArmBone = leftElbowPosition  leftShoulderPosition; Vector3 rightShoulderBone = rightShoulderPosition  spineShoulderPosition; Vector3 rightArmBone = rightElbowPosition  rightShoulderPosition; // hips Quaternion myTest = spineBaseOrientation; Quaternion myTest0 = Quaternion.AngleAxis(myTest.eulerAngles.y, new Vector3(0.0f, 1.0f, 0.0f)); hips.transform.localRotation = Quaternion.Inverse(hipStart) * myTest0; // spine Quaternion myTest6 = Quaternion.AngleAxis(myTest.eulerAngles.x, new Vector3(1.0f, 0.0f, 0.0f)) * Quaternion.AngleAxis(myTest.eulerAngles.z, new Vector3(0.0f, 0.0f, 1.0f)); spine.transform.localRotation = Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest6; // chest Quaternion myTest5 = Quaternion.Inverse(spineBaseOrientation) * spineMidOrientation; chest.transform.localRotation = Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest5; // neck Quaternion myTest2 = Quaternion.Inverse(spineShoulderOrientation) * neckOrientation; Debug.Log("neck rotation="+myTest2.ToString()); neck.transform.localRotation = Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest2; // head Quaternion myTest3 = Quaternion.Inverse(neckOrientation) * headOrientation; Debug.Log("head rotation="+myTest3.ToString()); head.transform.localRotation = Quaternion.Inverse(headStart) * Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest3; // left shoulder Quaternion myTest4 = Quaternion.Inverse(spineShoulderOrientation) * leftShoulderOrientation; Debug.Log("shoulder rotation="+myTest4.ToString()); leftClavicle.transform.localRotation = Quaternion.Inverse(leftClavicleStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse (hipStart) * myTest4;
And as per your advice I changed my conversion to Quaternions to read...
private Quaternion QuaternionFromVector4(Windows.Kinect.Vector4 inVector) { return new Quaternion(inVector.X, inVector.Y, inVector.Z, inVector.W); }
Here's how it looks if I stand in front of the Kinect in a tpose (Btw, only spine, neck, head and left shoulder are plugged in at this point)...
 Edited by _Ray Price_ Wednesday, July 30, 2014 4:51 PM clarity

Interestingly, I tried this with a different model and got a different rotation using the same code. How can that be? Is there some assumption here that I have to transform my model into some kind of base pose (tpose?) before I apply the Kinect rotations?
 Edited by _Ray Price_ Wednesday, July 30, 2014 10:25 PM clarity

So yeah. I'm getting the same thing!
How are JointOrientations are absolute?
Is the value of each joint absolute to the neck being parent to all at 0,0,0,0?
What is the value of the points?
Here's what I'm finding from recording skeleton standing in front and spitting out the following:
JointOrientations bonebone.Orientation.X +
" "+ bone.Orientation.Y + " "+ bone.Orientation.Z + " "+ bone.Orientation.W
SpineBase = 0.003993207 0.9973675 0.05752403 0.04396751 SpineMid = 0.003899194 0.9972942 0.05753048 0.04559736 Neck = 0.008536474 0.9930239 0.1077237 0.04718429 Head = 0 0 0 0 ShoulderLeft = 0.7685696 0.6336123 0.02236257 0.08565006 ElbowLeft = 0.8011876 0.1689823 0.5193725 0.2445313 WristLeft = 0.6290028 0.3751864 0.5778305 0.3601424 HandLeft = 0.6086057 0.4009017 0.5622655 0.3908123 ShoulderRight = 0.7867208 0.6096304 0.09320773 0.02708374 ElbowRight = 0.7174323 0.1195038 0.6718429 0.1401314 WristRight = 0.7895443 0.09435756 0.1834147 0.5779925 HandRight = 0.7634736 0.1194136 0.1618041 0.6137329 HipLeft = 0.6884276 0.7018301 0.1602328 0.08847217 KneeLeft = 0.6741655 0.09152499 0.731129 0.05073958 AnkleLeft = 0.5685248 0.08804499 0.8059186 0.1397243 FootLeft = 0 0 0 0 HipRight = 0.6641833 0.7015544 0.2135341 0.1452072 KneeRight = 0.7370378 0.1156219 0.6658675 0.005235625 AnkleRight = 0.5236376 0.1570109 0.8365145 0.03734707 FootRight = 0 0 0 0
Ken

Here's the first few of mine as euler angles after being converted to local space (relative to parent bone) using quaternion.inverse(parent) * child.
spine rotation=(0.0, 0.4, 0.0)
neck rotation=(357.8, 0.0, 359.8)
head rotation=(0.0, 0.0, 0.0)
left shoulder rotation=(359.4, 357.4, 258.7)
left arm rotation=(353.9, 108.3, 4.9)
Interestingly, even if I tilt my head all the way to the side, I still only get...
neck rotation=(357.9, 0.0, 359.9)
 Edited by _Ray Price_ Wednesday, July 30, 2014 10:36 PM

I had similar initial results with arms facing the wrong way when I was testing some models. I think it is necessary to multiply the relative rotation by the bone/joint's bind pose rotation:
Quaternion newRotation = bindRotation * parentQ.Inverse() * childQ;
(Possibly bindRotation should be multiplied after childQ but not sure.)
Haven't had a chance to test this but this might help.
Joshua Blake
Kinect for Windows MVP
OpenKinect Community Founder
Technical Director, InfoStrat Advanced Technology Group
Blog: http://nui.joshland.org
Twitter: http://twitter.com/joshblake 
Hi Josh,
Yeah, I've been trying that too, hence the...
head.transform.localRotation = Quaternion.Inverse(headStart) * Quaternion.Inverse(neckStart) * Quaternion.Inverse(chestStart) * Quaternion.Inverse(spineStart) * Quaternion.Inverse(hipStart) * myTest3;
But I may have got my sources wrong or overestimated the need to walk up the chain. Where are you picking your bind pose rotations from? I have the following code in my start()...
GameObject hips = GameObject.Find("hips"); hipStart = hips.transform.localRotation; GameObject spine = GameObject.Find("spine"); spineStart = spine.transform.localRotation; GameObject neck = GameObject.Find("neck"); neckStart = neck.transform.localRotation; GameObject head = GameObject.Find("head"); headStart = head.transform.localRotation; GameObject chest = GameObject.Find("chest"); chestStart = chest.transform.localRotation; GameObject leftClavicle = GameObject.Find("clavicle.L"); leftClavicleStart = leftClavicle.transform.localRotation; GameObject leftArm = GameObject.Find("upper_arm.L"); leftArmStart = leftArm.transform.localRotation; GameObject leftForearm = GameObject.Find("forearm.L"); leftForearmStart = leftForearm.transform.localRotation; GameObject leftHand = GameObject.Find("hand.L"); leftHandStart = leftHand.transform.localRotation;
But I think this may not be the right place, as it seems each model can be slightly different in it's first frame.
But even so, that aside the neck rotation is not changing AT ALL in the data coming back from the Kinect, and also when I have my arm out straight from my shoulder, the local (inverse(parent)*child) as euler angles is coming back as...
left arm rotation=(353.9, 108.3, 4.9)
Which looks VERY strange considering the shoulder and upper arm should be practically in alignment at that point.

Ray,
If your model/rigging is set up hierarchically (such that the leftForearm transform is a child of the leftArm transform) then I don't think you need to walk the hierarchy. Just multiply the appropriate joint's local start rotation, e.g.
forearmL.transform.localRotation = leftForearmStart * inverse(leftShoulderKinect) * leftElbowKinect;
(I think...)
Which specific joints were you using to get the left arm rotation = (353.9, 108.3, 4.9)? The bone > joint mapping parent/child is not intuitive IMHO. For example, if you want the rotation of the elbow joint, then IIRC you need to do inverse(elbowJointLeft.Rotation) * wristJointLeft.Rotation.
For shoulder, you would need to do inverse(shoulderJointLeft.Rotation) * elbowJointLeft.Rotation (i.e. elbowJointLeft is the "current" joint. I know.)
The euler angle you suggested would be consistent with the inverse(shoulderSpineJoint) * shoulderJointLeft (or right), which in the current API is really the angle between the SpineMid > SpineShoulder bone and the SpineShoulder>ShoulderLeft bone, which would be nearly a 90 degree angle even with your arm straight out. (The absolute SpineShoulder>ShoulderLeft bone rotation, stored in the ShoulderLeft Joint, would also be a similar angle unless you lean your body.)
Joshua Blake
Kinect for Windows MVP
OpenKinect Community Founder
Technical Director, InfoStrat Advanced Technology Group
Blog: http://nui.joshland.org
Twitter: http://twitter.com/joshblake 
Hi Josh,
Thanks for your reply again. For left arm I was doing...
Quaternion myTest5 = Quaternion.Inverse(leftShoulderOrientation) * leftElbowOrientation;
Yet you're suggesting this is coming back with data that looks like I was using Inverse(midSpine) * leftShoulder?
As for not needing to walk up the hierarchy, I have tried that too, but in your code...
forearmL.transform.localRotation = leftForearmStart * inverse(leftShoulderKinect) * leftElbowKinect;
Where are you picking up the leftForearmStart data from? Because it seems each model is potentially posed different in it's setting in the start() code (and not necessarily T). Surely before we apply the local rotation we need to apply some kind of rotation to put the model into it's tpose since it seems that setting all the rotations to zero in the editor does NOT produce a tpose for all models either. How does Unity animation controller do this?
Also, why do the head/neck angles not change even when my head is fully tilted? Something odd is going on here.
 Edited by _Ray Price_ Wednesday, July 30, 2014 11:41 PM


Ok, so I dug into the SDK and modified one of the samples to get the RAW quaternions to compare with what Unity is getting while I stand in a tpose. In C# in the raw SDK, I get...
spine base = 0.005743555,0.9976618,0.06572407,0.0178384
spine mid = 0.005864908,0.9976272,0.06571335,0.0196806
spine shoulder = 0.004756359,0.9984642,0.05081941,0.0215392
left shoulder = 0.7682579,0.6372728,0.009667866,0.05974606
left elbow = 0.2287592,0.2777363,0.7082614,0.6073694
head = 0,0,0,0
neck = 0.004115697,0.9992705,0.03131068,0.02147636In Unity I get...
spine base = 0.0042993,0.9973056,0.0537723,0.04971632
spine mid = 0.003943218,0.9976128,0.05379959,0.04311278
spine shoulder = 0.005316815,0.9985826,0.03853904,0.03632149
left shoulder = 0.7760947,0.6279857,0.05753399,0.000930806
left elbow = 0.3599467,0.4590581,0.6543876,0.4811246
head = 0,0,0,0
neck = 0.007668501,0.9991321,0.01916226,0.03618148
Give or take a little expected deviation they seem to be about the same, which is good. So I tried again with the raw SDK with my head turned 90 degrees to the left and got...
head = 0,0,0,0
neck = 0.001263987,0.9981489,0.01254379,0.05949736
So for sure even the base SDK is not picking up head rotation.
Are there issues in the rotation code for the SDK? Is this why we haven't seen an avateering demo yet?

Next step is to take the raw quaternion data and convert it into a local rotation using Inverse(parent) * child and print the euler angles to see if they make sense. The code I used is...
// get rotations from kinect Quaternion neckOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Neck].Orientation); Quaternion headOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.Head].Orientation); Quaternion leftHipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HipLeft].Orientation); Quaternion spineBaseOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineBase].Orientation); Quaternion spineMidOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineMid].Orientation); Quaternion spineShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.SpineShoulder].Orientation); Quaternion leftShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderLeft].Orientation); Quaternion leftElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowLeft].Orientation); Quaternion leftHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandLeft].Orientation); Quaternion leftWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristLeft].Orientation); Quaternion leftHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipLeft].Orientation); Quaternion leftThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbLeft].Orientation); Quaternion rightShoulderOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ShoulderRight].Orientation); Quaternion rightElbowOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ElbowRight].Orientation); Quaternion rightHandOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandRight].Orientation); Quaternion rightWristOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.WristRight].Orientation); Quaternion rightHandTipOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.HandTipRight].Orientation); Quaternion rightThumbOrientation = QuaternionFromVector4(body.JointOrientations[Kinect.JointType.ThumbRight].Orientation); // print local rotations using inverse parent * child Quaternion spineMidLocal = Quaternion.Inverse(spineBaseOrientation) * spineMidOrientation; Quaternion spineShoulderLocal = Quaternion.Inverse(spineMidOrientation) * spineShoulderOrientation; Quaternion leftShoulderLocal = Quaternion.Inverse(spineShoulderOrientation) * leftShoulderOrientation; Quaternion leftElbowLocal = Quaternion.Inverse(leftShoulderOrientation) * leftElbowOrientation; Quaternion neckLocal = Quaternion.Inverse(spineShoulderOrientation) * neckOrientation; Quaternion headLocal = Quaternion.Inverse(neckOrientation) * headOrientation; Debug.Log("kinect local rotation spine mid = "+spineMidLocal.eulerAngles.ToString()); Debug.Log("kinect local rotation spine shoulder = "+spineShoulderLocal.eulerAngles.ToString()); Debug.Log("kinect local rotation left shoulder = "+leftShoulderLocal.eulerAngles.ToString()); Debug.Log("kinect local rotation left arm = "+leftElbowLocal.eulerAngles.ToString()); Debug.Log("kinect local rotation neck = "+neckLocal.eulerAngles.ToString()); Debug.Log("kinect local rotation head = "+headLocal.eulerAngles.ToString());
The results I get from this are...
kinect local rotation spine mid = (0.0, 359.7, 0.0)
kinect local rotation spine shoulder = (1.8, 359.7, 359.8)
kinect local rotation left shoulder = (0.4, 1.8, 257.9)
kinect local rotation left arm = (4.4, 261.6, 6.3)
kinect local rotation neck = (2.2, 0.0, 359.8)
kinect local rotation head = (0.0, 0.0, 0.0)
These numbers strike me as a little odd. In a tpose the arm and the shoulder are almost in alignment, so how can the rotation be 261.6 in the Y axis? Also, how is the spineshoulder (point up to behind the neck) not pretty much in alignment with the spine mid (point from spine base to middle of spine)?



Ray:
Do you have avatar following the Kinect skeleton bone Joint Orientation movement yet?
I'm using XNA (right handed coordinate system) so I don't have to translate like you do.
My forearm seems to draw ok with out the wrist and hand is fairly close but it's still not cool, no rotation of the hand, just positional relative. But atleast it not cut off or blown up like giant hulk arm, or sometimes it ends up looking like fantastic 4 rubber guy arm. Although very intensely cool, not the motif I'm looking for.
Quaternion childQ = QuaternionFromVector4(bone.Orientation); Quaternion parentQ = QuaternionFromVector4(skeleton.JointOrientations[parentJoint.JointType].Orientation); Quaternion parentQInverse = new Quaternion(); Quaternion.Inverse(ref parentQ, out parentQInverse); combined = Matrix.Transform(avatarBoneMatrix, parentQInverse * childQ)
Ken
Ken

No, I still don't have it working yet. I'm starting to wonder what the hell these orientations represent. The fact that the stick figure you show here, AND Kinect Studio shows the rotations as perpendicular vectors to the bone is making me wonder if these things are even world rotation representations of the joint at all. What you're showing here shows some kind of rotation vector on the stickmodel. Do you know how that's calculated?

It's a little different code for that skeleton than the avatar ...
// Get the constraint direction in world space from the parent orientation Matrix matConstraintLocalToWorld = KinectHelper.JointOrientationToXNAMatrix(skeleton.JointOrientations[jc.Joint].Orientation); Vector3 constraintDirWs = jc.Dir; constraintDirWs.Normalize(); Vector3 rotatedConstraintDirWs = Vector3.Transform(constraintDirWs, matConstraintLocalToWorld); rotatedConstraintDirWs.Normalize(); // Draw the constraint direction line itself rotatedConstraintDirWs *= 0.5f; this.AddRotatedLine(rotatedConstraintDirWs, skeleton, i, Color.DeepPink);
This is how jc.DIR got set at setup:
public void AddDefaultConstraints() { // Constraints indexed on end joint (i.e. constrains the bone between start and end), but constraint applies at the start joint (i.e. the parent) // The constraint is defined in the local coordinate system of the parent bone, relative to the parent bone direction // Acts at Hip Center (constrains hip center to spine bone) this.AddBoneOrientationConstraint(JointType.SpineBase, new Vector3(0.0f, 1.0f, 0.3f), 90.0f); // Acts at Spine (constrains spine to shoulder center bone) this.AddBoneOrientationConstraint(JointType.SpineShoulder, new Vector3(0.0f, 1.0f, 0.0f), 50.0f); // Acts at Shoulder Center (constrains shoulder center to head bone) this.AddBoneOrientationConstraint(JointType.Head, new Vector3(0.0f, 1.0f, 0.3f), 45.0f); // Acts at Shoulder Joint (constraints shoulderelbow bone) this.AddBoneOrientationConstraint(JointType.ElbowLeft, new Vector3(0.1f, 0.7f, 0.7f), 80.0f); // along the bone, (i.e +Y), and forward +Z, enable 80 degrees rotation away from this this.AddBoneOrientationConstraint(JointType.ElbowRight, new Vector3(0.1f, 0.7f, 0.7f), 80.0f); // Acts at Elbow Joint (constraints elbowwrist bone) this.AddBoneOrientationConstraint(JointType.WristLeft, new Vector3(0.0f, 0.0f, 1.0f), 90.0f); // +Z (i.e. so rotates up or down with twist, when arm bent, stop bending backwards) this.AddBoneOrientationConstraint(JointType.WristRight, new Vector3(0.0f, 0.0f, 1.0f), 90.0f); // Acts at Wrist Joint (constrains wristhand bone) this.AddBoneOrientationConstraint(JointType.HandLeft, new Vector3(0.0f, 1.0f, 0.0f), 45.0f); // +Y is along the bone this.AddBoneOrientationConstraint(JointType.HandRight, new Vector3(0.0f, 1.0f, 0.0f), 45.0f); // Acts at Hip Joint (constrains hipknee bone) this.AddBoneOrientationConstraint(JointType.KneeLeft, new Vector3(0.5f, 0.7f, 0.1f), 65.0f); this.AddBoneOrientationConstraint(JointType.KneeRight, new Vector3(0.5f, 0.7f, 0.1f), 65.0f); // Acts at Knee Joint (constrains kneeankle bone) this.AddBoneOrientationConstraint(JointType.AnkleRight, new Vector3(0.0f, 0.7f, 1.0f), 65.0f); // enable bending backwards with Z this.AddBoneOrientationConstraint(JointType.AnkleLeft, new Vector3(0.0f, 0.7f, 1.0f), 65.0f); // Acts at Ankle Joint (constrains anklefoot bone) this.AddBoneOrientationConstraint(JointType.FootRight, new Vector3(0.0f, 0.3f, 0.5f), 40.0f); this.AddBoneOrientationConstraint(JointType.FootLeft, new Vector3(0.0f, 0.3f, 0.5f), 40.0f); }
Ken

Where does this object's Dir variable get initialized?...
Vector3 constraintDirWs = jc.Dir;
 Edited by _Ray Price_ Thursday, July 31, 2014 10:06 PM




I started with with 1.8 SDK sample avateering. Let me see about setting up a github or source control for the changes I made, there were a great deal of them to get v1 migrated to v2.
http://msdn.microsoft.com/enus/library/jj131041.aspxHere's what I'm thinking this morning, as last night I hacked so many different variations that Edison would be proud. Now I'm back to noodling mode.
The vector4 which is returned from JointOrientation.Orientation. Of course, the primitive auto generated code doc is completely worthless ( method Foo gets/sets a Foo). Here's what I think I know of the puzzle.
1> We believe the Vector4 returned is of Unit Type of Quarternion (Yaw,Pitch,Roll ) and which order?
2> Below suggests it might be referenced as view down each bone (per V1 spec for Matrix).
3> Head is (0,0,0) it seems "absolute" rotations related to the head. FK to find the hand position.
The old stuff had a discussion here:
http://social.msdn.microsoft.com/Forums/enUS/4e90b413b3024c29a8dacb08ebc53f9c/kinectjointorientationandbonerotationmatrixquestions?forum=kinectsdkI'm going to push my work to codeplex.
Ken

That would be great. If I can see how you're handling the rotations you've got setup there for the stick figure which seem to be working, maybe it'll shine some light on what's going on.
It's supposed to be a quaternion, so yes it's an angle around a particular vector. And it seems to work fine for the root (spine), but for anything other than that even transferring it into local space doesn't seem to get the required results. Of course I'm sure this is down to my own ineptitude, as it seems to be returning the same results as the base SDK. I'll continue to hack too and let you know if I make any progress.

This is it for Visual Studio and GIT. It's C# code.
https://kinectavateeringv2migration.codeplex.com/
Ken

I put your sample code in AvateeringXNA.cs into a method called
private Matrix processWholeBody(JointOrientation bone, Body body, Matrix bindRoot, ref Matrix[] boneTransforms)
called it from the beginning of method:
SetJointTransformation
Basically for the extremities the code did the following sequences:
 convert bone.orientation to XNAMatrix
 use XNAMatrix to decompose the XNA Quaternion rot (basically just converted k.Orientation to X.Q
 map the K.vector4 quaternions to XNA.Quaternions (Y to X, Z to Y, X to Z, W=W) ** this is the devil.
 create a new XNA Matrix from avatar mapped Quaternion
 pass this to ReplaceBoneMatrix method (next step is inside other method)
 boneTransforms[arraySize of Avatar effects/joints) change it's "Translation" to the next value calc'd
So basically, the "SetJoinTransformation" is trying to find the new / mapped "Translation" of the avatar's effect joint's position.
The Kinect gives the solution via the JointOrientation.Orientation vector4 (you say it's a Quaternion). Why can't the SDK docs say that?
I think these values are Normalized to what? 2pi = 1 or pi=1 Given they go negative, then pi = 1.
Further given the suggestions, they might be relative to the previous joint, although it's stated they are absolute. If they are absolute then to what? The head being zero. So then we should translate the inverse related to the head and not the previous parent joint? IDK. thinking out loud here.
The code cloned to codeplex, is hacked away to the point that I don't know what I've tried and I'm sick of trying random transforms combinations. Your help is very much appreciated.
I've recorded a V2 stream of me moving my arms around (you can see my ghost dog walking in wonder if it's play time.) I'll put this into the git repo as content too.
I'm new to 3d animation, so it's my first rodeo. So the body form of Dude, is fixed, i'm attempting to just have the arms flail around starting at the shoulder joint. The current version as of right now, x,yz,w mappings figgering is just trying to do the left elbow, wrist, & hand. Everytime I attempt to add the LeftShoulder the thing goes super crazy.
Just some thoughts on annotating what I'm doing there.
Ken

The documentation says...
Joint normals
There is a normal for each skeleton joint that describes rotation. Rotation is expressed as the vector (in world space) perpendicular to the attached bones in the joint hierarchy. For example, to determine the roll of the right elbow, the immediate parent joint in the hierarchy, the right shoulder, is used to determine the plane of the bone.
Which doesn't make sense because a) there is no 'normal' property on the joint, and b) if this orientation were a normal, why would it have a 'w' component?


Here is an explanation of the values:
Carmine Sirignano  MSFT
 Proposed as answer by Carmine Si  MSFTMicrosoft employee, Owner Friday, August 01, 2014 7:24 PM

Here is an explanation of the values:
Carmine Sirignano  MSFT
Hi Carmine,
Thank you so much for this link. If this value was truely an expression of the bone rotation as a quaternion in world space then...
Quaternion.Inverse(leftShoulderOrientation) * leftElbowOrientation
should give the local orientation relative to the parent, right? I should then be able to assign this in Unity to the bone.transform.localRotation property, but alas this seems to result in pretty much chaos.
Regards
Ray
 Edited by _Ray Price_ Saturday, August 02, 2014 4:41 AM

Latest zero progress update.
I'm using XNA not unity But the problem/solution domain is very similar:
#1> Convert Kinect Vector4 Quarternion to XNA/Someother.Quarternion
#2> Matrix.Transform this result into a new worldmap matrix for each of the bones.
#1> code question of how to map the Yaw, Pitch, Roll into a XNAQuarternion.
Quaternion QuaternionFromVector4(Microsoft.Kinect.Vector4 inVector) { //Quaternion.CreateFromYawPitchRoll (Single, Single, Single) Quaternion converted = Quaternion.CreateFromYawPitchRoll(inVector.X, inVector.Z, inVector.Y); return converted; }
or is it yaw= Y, pitch = Z and roll=X?
What is the mapping to covert a Kinect Quarternion to an XNA or anybody elses?
Basically, once we have a valid parent and object bone quarternion, then how to transform it?
private Matrix processBone2(JointOrientation bone, Body skeleton, Matrix bindRoot, ref Matrix[] boneTransforms) { JointType parentIdx = KinectHelper.ParentBoneJoint(bone.JointType); // Calculate the absolute/world equivalents of the hierarchical rotation Quaternion parentRotation = KinectHelper.QuaternionFromVector4(skeleton.JointOrientations[parentIdx].Orientation); Quaternion relativeRotation = KinectHelper.QuaternionFromVector4(bone.Orientation); // Create a new world rotation Quaternion.Inverse(leftShoulderOrientation) * leftElbowOrientation Quaternion worldRotation = (parentRotation * relativeRotation); //Quaternion.Multiply(parentRotation, relativeRotation); Matrix temp = new Matrix(); Matrix.Transform(ref boneTransforms[meshJointId], ref relativeRotation, out temp); return temp; }
There are just too many indeterminate to sit and guess combinations.
If the XNA quarternion mapping /creation is wrong. Then everything down stream of that is bogus.
When it's said that Y is the Roll then I assume that Z parameter for a standard Quarterion "roll" yes?
Ken

Hi Ken,
Thanks for the update post. Where did you get the QuaternionFromVector4 code from? Carmine said...
Each quaternion is the absolute orientation of the parent bone.
Which seems to suggest it is already a quaternion (thus needs no conversion), so I'm curious if you pulled this from a KinectV2 sample somewhere. Unfortunately despite Carmine's remark I've done some tests with Quaternion.Inverse(Parent)*Child which should give the local rotation, but with the same bend and rotation in the upper arm and forearm the result is different rotation values as the shoulder is rotated, which suggests that the returned value is NOT local to the parent bone after the usual quaternion world to local conversion.
Would be nice if Microsoft would throw us a bone here. (Get it!? :P).
 Edited by _Ray Price_ Thursday, August 07, 2014 2:24 AM

