none
Facetrack only support depth + player index FTIMAGEFORMAT_UINT16_D13P3? RRS feed

  • Question

  • Hi. By trial and error, my results suggest that the Facetracking features of SDK v1.7 only work if you feed it a depthmap + playerindex via FTIMAGEFORMAT_UINT16_D13P3. Is this true?

    I tried sending it a 640x480 and 320x240 depthmap which were formatted FTIMAGEFORMAT_UINT16_D13P3 but the lower 3 bits are always zeros. When I did this, no faces were found.

    Does SDK v1.7 facetracking require the depthmap pixels to have the player index in the lower 3? Or can we send it another format?


    --Dale

    Saturday, August 24, 2013 1:43 AM

Answers

  • Good catch on the docs, I will get that reported since it is incorrect. We have several samples that use 640x480 with player index. An example of that is typically when we fill the depth frame color to mask a player(GreenScreen or Depth-D3D). However, if you are using and an Xbox 360 Kinect sensor, for dev work, then depth is only 320x240.

    // Open a depth image stream to receive depth frames
    hr = m_pNuiSensor->NuiImageStreamOpen(
        NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX,
        NUI_IMAGE_RESOLUTION_640x480,
        0,
        2,
        m_hNextDepthFrameEvent,
        &m_pDepthStreamHandle);
    if (FAILED(hr) ) { return hr; }

    As for your other points, you are correct, the frame format must be FTIMAGEFORMAT_UINT16_D13P3, but the data itself does not need a player index. Performance is mainly a factor when increasing the resolution. The behavior that may be affected most is tracking if you do not have a player index, but it does work.

    Below are the tests I have done with the FaceTrackingVisualization sample

    SingleFace() 
        : m_hInst(NULL)
        , m_hWnd(NULL)
        , m_hAccelTable(NULL)
        , m_pImageBuffer(NULL)
        , m_pVideoBuffer(NULL)
        , m_depthType(NUI_IMAGE_TYPE_DEPTH) // NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX
        , m_colorType(NUI_IMAGE_TYPE_COLOR)
        , m_depthRes(NUI_IMAGE_RESOLUTION_640x480) // NUI_IMAGE_RESOLUTION_320x240
        , m_colorRes(NUI_IMAGE_RESOLUTION_640x480)
        , m_bNearMode(TRUE)
        , m_bSeatedSkeletonMode(FALSE)
    {}
    MultiFace::MultiFace(): m_hInst(NULL), 
        m_hWnd(NULL), 
        m_nbUsers(2),
        m_hAccelTable(NULL), 
        m_pVideoBuffer(NULL),
        m_pImageBuffer(NULL),
        m_eggavatar(NULL),
        m_depthType(NUI_IMAGE_TYPE_DEPTH), // NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX
        m_colorType(NUI_IMAGE_TYPE_COLOR),
        m_depthRes(NUI_IMAGE_RESOLUTION_640x480), // NUI_IMAGE_RESOLUTION_320x240
        m_colorRes(NUI_IMAGE_RESOLUTION_640x480), 
        m_bNearMode(FALSE),
        m_bSeatedSkeletonMode(FALSE)
    {
    }

    and with and without skeletal hints

    if (m_LastTrackSucceeded)
    {
        //hrFT = m_pFaceTracker->ContinueTracking(&sensorData, hint, m_pFTResult);
        hrFT = m_pFaceTracker->ContinueTracking(&sensorData, NULL, m_pFTResult); // without skeletal hints
    }
    else
    {
        // hrFT = m_pFaceTracker->StartTracking(&sensorData, NULL, hint, m_pFTResult);
        hrFT = m_pFaceTracker->StartTracking(&sensorData, NULL, NULL, m_pFTResult); // without skeletal hints
    }

    Carmine Sirignano - MSFT


    Wednesday, August 28, 2013 5:34 PM

All replies

  • Are you creating a depthmask or is that something you are getting from the SDK without the player index? Are you also using tracking with the Skeleton? Skeletal id and player index will have to match.


    Carmine Sirignano - MSFT

    Monday, August 26, 2013 7:57 PM
  • My intention/goal is to provide a high-resolution (640x480) color and depthmap for facetracking and for gestures. I have the CPU power available and threads dedicated for each.

    I am able to give any combination of skeleton hints, depthmap, and colormaps that the SDK v1.7 allows. For example I could give it a 640x480 depth, 640x480 color, and skeleton hints. However, with that combination the depthmap will not have the player index in the lower 3 bits; the SDK disallows player indexes at 640x480 resolution.

    My inquiry is to better understand what formats of data the Facetracking SDK will accept (or must have). I can find no SDK reference/programming guide documentation for this. I see nowhere written, "you must pass the facetracking API a depthmap with the lower 3 bits containing the player index". Instead, I can only find in the C++ samples that a 320x240 depthmap+player index is give to the FaceTracking API.


    --Dale

    Monday, August 26, 2013 10:58 PM
  • FT will only work with a FTIMAGEFORMAT_UINT16_D13P3 depth image format regardless of resolution. I am still unsure how you are not getting depth frames without a player index value if you are detecting skeletons.  

    If you are experimenting with the API's, always check your HRESULT values. Passing an image format of FTIMAGEFORMAT_UINT16_D16 will produce a failed HRESULT value when trying to track. If the format is _D13P3, where depth is in the upper 13bit and player index bits are 0, it will difficult for FT to track. You will know what works best for your application when you have tested this.

    http://msdn.microsoft.com/en-us/library/microsoft.kinect.facetracking.iftfacetracker.starttracking.aspx

    http://msdn.microsoft.com/en-us/library/microsoft.kinect.facetracking.iftresult.getstatus.aspx

    A depth image frame value player index value will not be 0 when the runtime can approximate the shape of a person in the depth frame. The minimum distance is 400mm(near mode)/800mm for this to happen and whether you are using seated skeletal tracking. You can use 640x480 depth frame with FT but it may have a bit of a performance hit. Using 320x240 may be sufficient in most cases. I have done my own tests this works without issues. Player index values are populated when I was in the tracking range.

    What results do you get if you use the Toolkit Face Tracking Visualization sample?

    Carmine Sirignano - MSFT


    Tuesday, August 27, 2013 8:34 PM
  • The Nui documentation at http://msdn.microsoft.com/en-us/library/jj663864.aspx states that for depth+player index, the resolution of the depthmap is limited to no more than 320x240. I verified this limitation by testing behavior in earlier SDKs. I trust the documentation that this limit is still in place. Is this correct?

    Given then, it is easily possible for me to request of Nui a depthmap=640x480 with no player index *and* ask for the skeleton stream. This is a valid combination. The skeleton stream does not require that I startup Nui wiith NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX

    Facetracking works in some combinations of input. You are welcome to try out my component if you would like at hidale.com/dp-kinect/. This component exposes the features of the SDK in a higher-level manner, therefore I am exercising lots of the features and finding/testing their boundary cases.

    From this thread, I am understanding the following. Is the following true?

    • Face Tracking will function *only* by sending it FTIMAGEFORMAT_UINT16_D13P3 data
    • Face Tracking does not require player indexes in the lower 3 bits. A dev does not have to declare NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX on NuiInitialize
    • Face Tracking performs better if the lower 3 bits have a player index. A dev would need to declare NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX on NuiInitialize. Declaring this forces the depthmap to be no larger than 320x240 in SDKs v1.7 or older.


    --Dale

    Wednesday, August 28, 2013 1:11 PM
  • Good catch on the docs, I will get that reported since it is incorrect. We have several samples that use 640x480 with player index. An example of that is typically when we fill the depth frame color to mask a player(GreenScreen or Depth-D3D). However, if you are using and an Xbox 360 Kinect sensor, for dev work, then depth is only 320x240.

    // Open a depth image stream to receive depth frames
    hr = m_pNuiSensor->NuiImageStreamOpen(
        NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX,
        NUI_IMAGE_RESOLUTION_640x480,
        0,
        2,
        m_hNextDepthFrameEvent,
        &m_pDepthStreamHandle);
    if (FAILED(hr) ) { return hr; }

    As for your other points, you are correct, the frame format must be FTIMAGEFORMAT_UINT16_D13P3, but the data itself does not need a player index. Performance is mainly a factor when increasing the resolution. The behavior that may be affected most is tracking if you do not have a player index, but it does work.

    Below are the tests I have done with the FaceTrackingVisualization sample

    SingleFace() 
        : m_hInst(NULL)
        , m_hWnd(NULL)
        , m_hAccelTable(NULL)
        , m_pImageBuffer(NULL)
        , m_pVideoBuffer(NULL)
        , m_depthType(NUI_IMAGE_TYPE_DEPTH) // NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX
        , m_colorType(NUI_IMAGE_TYPE_COLOR)
        , m_depthRes(NUI_IMAGE_RESOLUTION_640x480) // NUI_IMAGE_RESOLUTION_320x240
        , m_colorRes(NUI_IMAGE_RESOLUTION_640x480)
        , m_bNearMode(TRUE)
        , m_bSeatedSkeletonMode(FALSE)
    {}
    MultiFace::MultiFace(): m_hInst(NULL), 
        m_hWnd(NULL), 
        m_nbUsers(2),
        m_hAccelTable(NULL), 
        m_pVideoBuffer(NULL),
        m_pImageBuffer(NULL),
        m_eggavatar(NULL),
        m_depthType(NUI_IMAGE_TYPE_DEPTH), // NUI_IMAGE_TYPE_DEPTH_AND_PLAYER_INDEX
        m_colorType(NUI_IMAGE_TYPE_COLOR),
        m_depthRes(NUI_IMAGE_RESOLUTION_640x480), // NUI_IMAGE_RESOLUTION_320x240
        m_colorRes(NUI_IMAGE_RESOLUTION_640x480), 
        m_bNearMode(FALSE),
        m_bSeatedSkeletonMode(FALSE)
    {
    }

    and with and without skeletal hints

    if (m_LastTrackSucceeded)
    {
        //hrFT = m_pFaceTracker->ContinueTracking(&sensorData, hint, m_pFTResult);
        hrFT = m_pFaceTracker->ContinueTracking(&sensorData, NULL, m_pFTResult); // without skeletal hints
    }
    else
    {
        // hrFT = m_pFaceTracker->StartTracking(&sensorData, NULL, hint, m_pFTResult);
        hrFT = m_pFaceTracker->StartTracking(&sensorData, NULL, NULL, m_pFTResult); // without skeletal hints
    }

    Carmine Sirignano - MSFT


    Wednesday, August 28, 2013 5:34 PM
  • thank you. You clarified it all for me. :-)

    --Dale

    Thursday, August 29, 2013 12:01 AM