Answered by:
Using the translation in face tracking to move a 3D object in D3D
Question

I would like to use the translation vector obtained from face tracking to move an object in 3D space. How do I do this? In particular, how do I specify my world, view and projection matrices to do this?
For instance, I draw a cube in D3D with a cubeWorld matrix, and a view and projection matrix. All vertices of my 3D geometry are transformed via the view and projection matrix to screen space. If I apply the translation vector obtained from face tracking directly to the cubeWorld matrix (i.e. using a matrix multiply of cubeWorld and translation vector), my results do not work. So, how do I specify my transformation matrices, i.e., the view and projection matrix (and the world matrix of the cube 3D object) in order to be able to translate the cube in 3D with the head movements?Any suggestions will be very helpful
Thanks
Answers

I worked out the solution by multiplying the point returned from kinect with the inverse of the view matrix. This brings the point into the world space. As the head moves, the translations applied to the object allows the object to follow the head.
Don't forget to flip the x or z coordinate system (by multiplying the x or z coordinate with 1), depending on how the view matrix is defined and if the object motion in the image is what is desired.
All replies

This is a common problem in 3D given the different coordinate spaces. A lot of this will depend on what you are trying to determine. It may be a simple task such as joystick movements into 3D space. Normalize the vector first in the Face Tracking coordinates and then apply this to some Translation matrix you are building.
An object has its own 3D coordinate space where translations are applied to the model called the World matrix. The View Matrix is the translation, rotation, and then scale to move that object into the camera coordinate system. This matrix is generated based on the information you know between where the camera location and the model. When model space is based on the origin, the math is easier. Are you trying to take face movements to move the model or the camera? If you apply the vector translation to the View Matrix, this will affect the camera.

Thanks for the reply Carmine Si.
I am trying to use the face movements to move a 3D model, not the camera. The camera view matrix is fixed using a left hand coordinate system with eye at origin, up as the positive yvector, and looking down the z axis. Perspective matrix is set to the field of view of about 45, with near and far 1.0, 2000.0.
An example of what I am trying to do is essentially, if the face moves, the object follows the face movements, i.e. move your head left, object moves left, turn your head one way, the object turns in a similar manner.
While working with the raw data from Get3DPose, I understand the translations returned each time instance are translations from camera origin. However, my application of the translations to the world matrix of an object (after I have multiplied them by inverse of the view matrix) does not track the face movements. Hence, my question as to how to apply the points in camera space to 3D objects.

I worked out the solution by multiplying the point returned from kinect with the inverse of the view matrix. This brings the point into the world space. As the head moves, the translations applied to the object allows the object to follow the head.
Don't forget to flip the x or z coordinate system (by multiplying the x or z coordinate with 1), depending on how the view matrix is defined and if the object motion in the image is what is desired.