none
Missing AUs: FaceModel->GetAUCount() wants 8, FTResult->GetAUCoefficients() provides only 6 RRS feed

  • Question

  • FTResult->GetAUCoefficients() provides only 6 AUs

    When I query FaceModel->GetAUCount() to get the number used/needed in the model, the result is 8

    So...what do I do regarding the difference of 2?


    --Dale

    Monday, August 19, 2013 4:23 PM

Answers

  • Thank you. This does help. With this, reading eggavatar.cpp again, and testing, I think I better understand. Would you please review the following and correct as needed?

    Facetracking in SDK v1.7...

    • can detect and return to the dev 11 SUs. Some of these are Candide-3 while 2 of them are not. Differences are at http://msdn.microsoft.com/en-us/library/jj130970.aspx
    • can detect and return to the dev 6 AU coefficients. There are also documented at http://msdn.microsoft.com/en-us/library/jj130970.aspx
    • IFTModel Get3DShape() and GetProjectedShape() accept these 11 SUs and 6 AUCoeffs. They can both additionally accept 2 optional AUCoeffs (for a total of 8).
    • AUCoeff7 is not a Candid3 standard. It represents the blink of the left eye lid
    • AUCoeff8 is not a Candid3 standard. It represents the blink of the right eye lid
    • The SDK currently does not detect or provide these additional 2 AUCoeffs. They are optional and it is left to the dev to use or derive them. A hint of such usage is in eggavatar.cpp in the Multiface sample. CAUTION: the code could confuse a reader and suggest that 7 and 8 represent upper and lower eye lids. Actual testing w/ the SDK v1.7 instead demonstrates that Coeff7 and 8 represent blink for left and eye eyes when interpreted by Get3DShape()

    --Dale



    • Edited by Dale Phurrough Thursday, August 29, 2013 9:28 PM corrected upper/lower -> left/right
    • Marked as answer by Dale Phurrough Wednesday, September 4, 2013 8:48 PM
    Thursday, August 29, 2013 7:43 PM

All replies

  • GetAUCoefficients maps to the illustrated AU's on the Face Tracking page. The other 2 AU's are not part of these but are used for the eyelids which you may or may not want to use. Since the values may not be significant, you might want to implement a method much like the FaceTracking sample(s):

    BOOL EggAvatar::SetCandideAU(const float* AU, const int numberAU)
    {
    ...
            // Draw the eyelids based on the other AU
            if (numberAU < 8 || (AU[6] == 0 && AU[7] == 0))
            {
    ...
            }
            else // If we capture more data on the eyes, just use it.
            {
                m_LowerEyeLid = AU[7];
                m_UpperEyeLid = AU[6];
            }
            // Check that the eyelids are not excessive!
    ...
    }


    Carmine Sirignano - MSFT

    Tuesday, August 20, 2013 4:49 PM
  • I appreciate our reply. Let me restate my question so it is clearer.

    SDK 1.7 Face tracking API tracks and outputs only 6 AUs

    SDK 1.7 Face tracking API Get3DShape() and GetTriangles() desire to have 8 AUs.

    Are you suggesting that I look at the SetCandideAU() in teh sample code and do the reverse of that whole function? For exammple, I wantt AU7 to feed into Get3DShape, not the m_LowerEyeLid to feed into a custom drawing routing.


    --Dale

    Wednesday, August 21, 2013 9:51 PM
  • Because Get3DShape does not have a dependency on AU's passed in. You can define how many you want to use you are not limited to 6 or 8, but some might not give a result you are expecting.
    IFTModel::Get3DShape Method
    http://msdn.microsoft.com/en-us/library/microsoft.kinect.facetracking.iftmodel.get3dshape.aspx
    ...
    const
    FLOAT *pAUCoefs,
             UINT auCount,
    ...

    The AU's retrieved with GetAUCoefficients will be 6, but are by no means are all of the available AU's. We only document six AUs and 11 SUs, which are a subset of what is defined in the Candide3 model (http://www.icg.isy.liu.se/candide/). The AU's are morph targets and you can mix and match whichever ones you want to modify the output mesh, independent of what the FaceTracking data gives you. For example, you might want to make a smile more extreme by tweaking AU values. This provides the flexibility for you to apply these values to other meshes that are not part of the SDK.


    Carmine Sirignano - MSFT

    Friday, August 23, 2013 6:31 PM
  • That does makes sense and I have looked at the Candide3 origin site.

    In this inquiry, I am only inquiring into Get3DShape and the private code that implements and returns the vertices of the default Models Microsoft is providing. Does this Microsoft code respond to 6 AUs or 8 AUs sent to it?

    If it is 6, then I will just send 6. If code that implements/derives this model responds to 8, then what are the 7th and 8th AUs? Are they the official Candide #7 and #8? Its unclear to me in light of the other data output which doesn't always align w/ Candide.



    --Dale


    Saturday, August 24, 2013 1:37 AM
  • There are AU's and AU Coefficients. I will try to detail the differences. AU's are morph targets, groups of vertices that make the particular shape. The SDK already has the AU's "baked in" representing the documented 6 shapes. The other 2, eye lids, while not directly an AU, helps with visualizing the openness of the eye, ie. blinking. These are not Canadide3 AU's and are specific to FaceTracking. The docs outline the differences.

    AU Coefficients are values between -1 to 1 that will effect the intensity of that morph. Just like bone deformation for skinned meshes, the coefficient values passed in are factors on the amount the vertices are manipulated to the target shape. Since there are shared vertices 2 values may cancel out on a particular area. When using Facetracking values may not be an issue, but if you had you own animation rig, you could get into some odd results.

    The Get3DShape api takes AU Coefficients not AU's. The API will allow you to pass 8 coefficients into the function, if you pass more, you will get an INVALID_ARGS result. 

    We don't comment on the internals of API's publically. If the functionality of the API, doesn't work to your requirements, please provide detail on what you are trying to do and what you would like to see from the API to enable that functionality. From that, I will take your feedback to the team for their consideration. 


    Carmine Sirignano - MSFT

    Monday, August 26, 2013 7:49 PM
  • Thank you. This does help. With this, reading eggavatar.cpp again, and testing, I think I better understand. Would you please review the following and correct as needed?

    Facetracking in SDK v1.7...

    • can detect and return to the dev 11 SUs. Some of these are Candide-3 while 2 of them are not. Differences are at http://msdn.microsoft.com/en-us/library/jj130970.aspx
    • can detect and return to the dev 6 AU coefficients. There are also documented at http://msdn.microsoft.com/en-us/library/jj130970.aspx
    • IFTModel Get3DShape() and GetProjectedShape() accept these 11 SUs and 6 AUCoeffs. They can both additionally accept 2 optional AUCoeffs (for a total of 8).
    • AUCoeff7 is not a Candid3 standard. It represents the blink of the left eye lid
    • AUCoeff8 is not a Candid3 standard. It represents the blink of the right eye lid
    • The SDK currently does not detect or provide these additional 2 AUCoeffs. They are optional and it is left to the dev to use or derive them. A hint of such usage is in eggavatar.cpp in the Multiface sample. CAUTION: the code could confuse a reader and suggest that 7 and 8 represent upper and lower eye lids. Actual testing w/ the SDK v1.7 instead demonstrates that Coeff7 and 8 represent blink for left and eye eyes when interpreted by Get3DShape()

    --Dale



    • Edited by Dale Phurrough Thursday, August 29, 2013 9:28 PM corrected upper/lower -> left/right
    • Marked as answer by Dale Phurrough Wednesday, September 4, 2013 8:48 PM
    Thursday, August 29, 2013 7:43 PM
  • That does sum it up.

    Carmine Sirignano - MSFT

    Wednesday, September 4, 2013 8:36 PM