none
face recognition aligning/mapping with Kinect calibration values or custom function RRS feed

  • Question

  • Following up for more details on the alignment of Color to Depth data for facetracking. I originally posed on the following 2012 thread but it seems dead for replies. 

    http://social.msdn.microsoft.com/Forums/en-US/db34082d-0157-4749-bbad-d57b830ac224/hd-image-for-face-recognition#eddc7e37-9644-48fc-93ec-fd9b88baac4f

    1. The custom mapping function that can be set for the FaceTracking featureset appears to be called pixel by pixel rather than providing back a whole frame of lookup values. Is this true? If so, a later improvement in the SDK to allow for whole frame lookup results would be appreciated and more performant.
    2. I do not see any direct association of the FaceTracking codebase and a specific INuiSensor. Therefore, I am uncertain that the current FaceTracking codebase uses the calibration values present in each Kinect. How would the Facetracking code know from which of the two Kinects attached to my Windows PC to retrieve the calibration values?
    3. If the answer to #2 is...we don't align depth<->color using the calibration values...then...could I improve the recognition speed/accuracy of FaceTracking by providing frames of color and depth to FT_SENSOR_DATA which have already been aligned using MapDepthFrameToColorFrame() or NuiImageGetColorPixelCoordinateFrameFromDepthPixelFrameAtResolution()?

    --Dale

    Thursday, August 29, 2013 7:17 PM

Answers

  • Noted and will let the team know.

    Carmine Sirignano - MSFT

    Monday, September 9, 2013 6:24 PM
  • as per our discussion offline, define your own function for the mapping as such, and provide the function name to the call:

    // define a function for your mapping method
    HRESULT FTAPI MyDepthToColorFunction( 
        UINT depthFrameWidth, UINT depthFrameHeight, 
        UINT colorFrameWidth, UINT colorFrameHeight,
        FLOAT zoomFactor, POINT viewOffset, 
        LONG depthX, LONG depthY, USHORT depthZ, 
        LONG* pColorX, LONG* pColorY)
    {
        //do something
        *pColorX = 50;
        *pColorY = 50;
    
        return S_OK;
    }
    
    // add the function name as the third parameter
    hr = m_pFaceTracker->Initialize(&videoConfig, pDepthConfig, MyDepthToColorFunction, NULL); 
    

    Carmine Sirignano - MSFT

    Wednesday, March 5, 2014 11:31 PM

All replies

  • While not getting into the internals of SDK, accuracy will improve by providing higher resolution color images. Because this was a consideration, there may not be a 1:1 relationship. Narrowing this down to base functionality, at a pixel level, provides the SDK the flexibility it needs.

    FTRegisterDepthToColor(UINT depthFrameWidth, UINT depthFrameHeight, UINT colorFrameWidth, UINT colorFrameHeight,FLOAT zoomFactor, POINT viewOffset, LONG depthX, LONG depthY, USHORT depthZ, LONG* pColorX, LONG* pColorY);

    Typically FT was designed to operate with first/default sensor. In the instance you are using multiple sensors, you will want to override the callback and provide the mapping functions for the sensor you are using. The runtime allows you to enumerate all sensors and from ask for its INuiSensor::NuiGetCoordinateMapper(taking note of the HRESULT returned). You can test this yourself by running the following code with another application using the sensor:

    INuiSensor* pNuiSensor = nullptr;
    int iSensorCount = 0;
    
    HRESULT hr = NuiGetSensorCount(&iSensorCount);
    if( FAILED(hr) )
        return hr;
    
    // Look at each Kinect sensor
    for( int i = 0; i < iSensorCount; ++i )
    {
        // Create the sensor so we can check status, if we can't create it, move on to the next
        hr = NuiCreateSensorByIndex( i, &pNuiSensor );
    	if( SUCCEEDED(hr) )
            break;
    
        pNuiSensor->Release();
    }
    
    INuiCoordinateMapper* pMapper;
    ULONG length;
    byte* data;
    hr = pNuiSensor->NuiGetCoordinateMapper(&pMapper);
    if( SUCCEEDED(hr) )
    {
    	hr = pMapper->GetColorToDepthRelationalParameters(&length, (void**)&data);
    }
    

    Carmine Sirignano - MSFT

    Wednesday, September 4, 2013 8:33 PM
  • I  have experience with CoordinateMapper. My app already usesit for registering depth/color when a user wants to aligh them.

    What I do not yet see possible is using the FTRegisterDepthToColor callback function that is part of FT::Initialize(). Here is why...

    1. The Initialize() method wants a function which matches the FTRegisterDepthToColor signature.
    2. If this is declared/called within a class, the FTRegisterDepthToColor function must be static; therefore, I loose context to my class's member variables.
    3. That signature has various parameters; none of which are a pointer
    4. Without a pointer, I am not able to pass a pointer to a INuiSensor, CoordinateMapper, or to a class/data structure which could hold a pointer to INuiSensor/CoordinateMapper.

    So far, I have been able to see a way to get access to a INuiSensor or CoordinateMapper inside the FTRegisterDepthToColor callback. Without them, I cannot align depth->color for a Kinect Sensor. Am I missing a method/mechanism to do this?

    I did see that the FTRegisterDepthToColor callback receives a FLOAT zoomFactor, and POINT viewOffset. The former is at least 32bits (big enough for a 32-bit pointer) and the latter at least 64bites (large enough for a 64-bit pointer). And...I see structure members on FT_SENSOR_DATA which are matching and is passed into start/continuetracking. Could I hack in a pointer into one of them? Or will the Facetracking API code use those values and therefore behave poorly given the value of a pointer will be in them?


    --Dale

    Thursday, September 5, 2013 5:51 PM
  • As I stated, you can "create" the same sensor in the callback to get the pointer to the same INuiSensor. Calling NuiCreate... doesn't new up a net instance. 

    Carmine Sirignano - MSFT


    Thursday, September 5, 2013 6:39 PM
  • The same problem exists. You can't NuiCreate.

    When you have multiple kinects attached to the same computer, you don't know which USB id or Index in which to NuiCreate an INuiSensor. The callback has no context from which to identify who did the facetracker->Initialize()

    To explore a workaround, I passed a pointer (the hex value of the pointer) by saving it in the zoom param...and also tried in the offset.x param. In both experiments, the custom mapping function was never called. Passing a crazy value (like that of a pointer) in these params caused the Nui code to somehow not call the custom mapping function. I verified this by putting a console/dbgview debug output in the mapping function.


    --Dale


    • Edited by Dale Phurrough Thursday, September 5, 2013 9:02 PM more detail
    Thursday, September 5, 2013 7:46 PM
  • Since you are passing the FT the depth and color image data, you know which sensor that came from. You will have to track that.

    As for determining the device in your callback, the NuiDeviceConnectionId() method will provide the connection for the sensor. You can either store this globally, or create a singleton instance of a "NuiSensorChooser" that will provide you that.

    // enumerate all sensors
    for (int i = 0; i < iCount; ++i)
    {
        INuiSensor* pNui = nullptr;
        if (SUCCEEDED(NuiCreateSensorByIndex(i, &pNui)))
        {
    	const WCHAR* deviceID = pNui->NuiDeviceConnectionId();  // contains the connections string for the device
    																// plugged into a specific USB port. 
    																// removing the sensor and using a different port will have
    																// a different ID.
            HRESULT hrStatus = pNui->NuiStatus();
    
            CreateNuiDeviceByID(deviceID, hrStatus);
        }
        SafeRelease(pNui);
    }
    
    void CreateNuiDeviceByID(const WCHAR* wcConnectionId, HRESULT sensorStatus)
    {
        INuiSensor* pNuiSensor = nullptr;
        if (SUCCEEDED(NuiCreateSensorById(wcConnectionId, &pNuiSensor)))
        {
            // get coordinate mapper for this device.
        }
    }


    Carmine Sirignano - MSFT

    Friday, September 6, 2013 5:33 PM
  • I have a request for an future adjustment to the API/SDK. Allow a lpParam to be sent along with the callback function pointer in the Initialize() for facetrack.

    I ask this because none of the solutions you provided or I can discover/imagine will work when you have multiple Kinect sensors. Your most recent two (have a global or use a C#/WPF control) both will not resolve this. Why? Because like always, in the callback you do not know what sensor caused the callback.

    Two Kinect Sensors
    Both are pointing different directions.
    Both are doing their own independent facetracking.

    With this, a global doesn't much help. Nor does a C# control. Sure...I can store the "list of currently running sensors" in a global. However, when I'm in the callback, I don't know *which* of the two sensors stored in the global is the one for which I'm about to process the data.

    From what I can tell, the custom mapping functionality wasn't designed with the thought of having multiple Kinect Sensors. Is seems a architectural design/spec oversight. No one's perfect. ;-)

    I did think of one total complete horrible hack. I could have 4 (arbitrary #) custom mapping functions with each of them having exactly the same code. When a facetrack initialize() happens, I maintain a mapping of c++ class pointer (or INuiSensor) to the 1 of 4 function pointers that I pass to the Initialize. Then when I'm in the callback, I can use the address of myself (the custom function) as a lookup into the global mapping setup just before the initialize(). Then I can get the pointer I stored and do work.

    Total hack, but it should work. In the future, it would be great to just have an lpParam. :-)


    --Dale


    • Edited by Dale Phurrough Friday, September 6, 2013 11:34 PM mroe detail
    Friday, September 6, 2013 10:37 PM
  • Noted and will let the team know.

    Carmine Sirignano - MSFT

    Monday, September 9, 2013 6:24 PM
  • I realize this is an old thread, but the only one I could find on the subject.  I've created a custom mapping method but it is never called.  I've tried creating a member which points to the static method:

    mDepthToColorFuncPtr = FTRegisterDepthToColor( &FaceTracker::mapDepthToColor );
    hr = mFaceTracker->Initialize( &mConfigColor, &mConfigDepth, mDepthToColorFuncPtr, 0 );
    And just calling the above in place.
    hr = mFaceTracker->Initialize( &mConfigColor, &mConfigDepth, FTRegisterDepthToColor( &FaceTracker::mapDepthToColor ), 0 );

    Here is my callback.

    HRESULT FaceTracker::mapDepthToColor( UINT depthFrameWidth, UINT depthFrameHeight, UINT colorFrameWidth, UINT colorFrameHeight, FLOAT zoomFactor, POINT viewOffset, LONG depthX, LONG depthY, USHORT depthZ, LONG* pColorX, LONG* pColorY )
    {
    	*pColorX = depthX;
    	*pColorY = depthY;
    	return S_OK;
    }
    

    My input images are identical.  One is the 16-bit infrared channel, the other is an 8-bit RGB (greyscale) version.  I originally resized my input images to fit the FT API's format and it worked great, but the resize process is slow. My theory is that this will go much faster if I can just use the original images and bypass the mapping functions.  That is, if this function ever gets called.

    Wednesday, March 5, 2014 12:00 AM
  • I would actually recommend against passing the sensor, instead opting for an arbitrary user created argument.  The FT API could potentially be decoupled from the Kinect sensor in the future without such ties to the SDK.
    Wednesday, March 5, 2014 12:49 AM
  • as per our discussion offline, define your own function for the mapping as such, and provide the function name to the call:

    // define a function for your mapping method
    HRESULT FTAPI MyDepthToColorFunction( 
        UINT depthFrameWidth, UINT depthFrameHeight, 
        UINT colorFrameWidth, UINT colorFrameHeight,
        FLOAT zoomFactor, POINT viewOffset, 
        LONG depthX, LONG depthY, USHORT depthZ, 
        LONG* pColorX, LONG* pColorY)
    {
        //do something
        *pColorX = 50;
        *pColorY = 50;
    
        return S_OK;
    }
    
    // add the function name as the third parameter
    hr = m_pFaceTracker->Initialize(&videoConfig, pDepthConfig, MyDepthToColorFunction, NULL); 
    

    Carmine Sirignano - MSFT

    Wednesday, March 5, 2014 11:31 PM