Answered by:
How to programatically modify a joint in the Skeleton
Question

Hi,
I am looking to achieve manipulation of the Kinect skeleton data on the fly. I have built an app that renders the Skeleton captured in 3D using the Helix toolkit. I use the XYZ position data for each joint to render the 3D model (a series of sphere's and cylinders to represent the joints and bones).
I have built a "Pause" button that when clicked, stops capturing in real time and keeps the last rendered skeleton information as well as the rendering on the screen.
What I am looking to do is rotate a selected joint (we will discuss the right elbow joint for the purpose of this) by a certain number of degrees around either an X, Y or Z axis, depending on what I choose.
I have built classes to carry out the necessary calculations to do this transformation (Matrix4x4, Matrix3x3, Vector3D, Point3D classes). I now need to apply the calculations to a selected joint to rotate as I require. Here's what I do:
for (int i = 0; i < lastSkeletons.Count(); i++) { var item = lastSkeletons[i]; // Generate an X axis rotation matrix. Matrix4x4 rotateMat = Matrix4x4.RotationAroundX(45); // Rotate 45 degrees for example // Current matrix for bone. I can implicitly convert the Matrix4 from the Kinect
// to my own Matrix4x4 class. Matrix4x4 sourceMat = item.BoneOrientations[JointType.ElbowRight].HierarchicalRotation.Matrix; // Multiply rotation matrix by bone source matrix. Matrix4x4 rotated = rotateMat * sourceMat; // Set back into the Kinect matrix after modifying its value. item.BoneOrientations[JointType.ElbowRight].HierarchicalRotation.Matrix = (Matrix4)rotated; // Update the skeleton with the modified item. lastSkeletons[i] = item; } // Call render method. Render();
So I generate a rotation matrix, get the hierarchy matrix from the bonerotation within the Kinect and multiple these together to get the transformed matrix. I then set the Kinect hierarchy matrix to the value of the modified matrix. Firstly, is this the correct approach for manipulating the data?
Secondly, updating the Matrix obviously is not refreshing the position property of the joint. How do I either...
 The problem I have is that the Render() method using the Position class to render the skeleton in 3D (item.Joints[JointType.ElbowRight].Position). Pull the actual XYZ position from the modified Matrix? OR
 Force the position info in the Kinect skeleton to refresh using the updated matrix info?
Thanks in advance! Also, if anyone has any pointers on how I can modify the rotation of the joint in a different way Id appreciate any pointers!
Thanks!
Rob
Answers

Hi Rob,
initially the skeleton data structure only provided positional info of the joints. With the update of the SDK to version 1.5 joint orientation information was added to the joint data structure. These orientation matrices are derived from the positional data internally and are recalculated after you change the positional info and access them again. Doing it the other way around is not going to work unless you do it yourselves outside the skeleton class.
I think you have two options:
1. Calculate joint lengths based on a source skeleton and calculate new positions by walking through the skeleton hierachically and applying your hierarchical orientation matrices.
2. Render a skeleton based on rotational information in the skeleton.
kind regards,
Roland Smeenk
 Proposed as answer by Carmine Si  MSFTMicrosoft employee Tuesday, January 7, 2014 11:10 PM
 Marked as answer by Carmine Si  MSFTMicrosoft employee Tuesday, January 14, 2014 11:45 PM
All replies

Hi Rob,
initially the skeleton data structure only provided positional info of the joints. With the update of the SDK to version 1.5 joint orientation information was added to the joint data structure. These orientation matrices are derived from the positional data internally and are recalculated after you change the positional info and access them again. Doing it the other way around is not going to work unless you do it yourselves outside the skeleton class.
I think you have two options:
1. Calculate joint lengths based on a source skeleton and calculate new positions by walking through the skeleton hierachically and applying your hierarchical orientation matrices.
2. Render a skeleton based on rotational information in the skeleton.
kind regards,
Roland Smeenk
 Proposed as answer by Carmine Si  MSFTMicrosoft employee Tuesday, January 7, 2014 11:10 PM
 Marked as answer by Carmine Si  MSFTMicrosoft employee Tuesday, January 14, 2014 11:45 PM
