none
Assumptions about frames RRS feed

  • Question

  • totally awsome to finaly see an official kinect SDK!

    I have some questions about how the frame events are triggered

    Will the frame events allways fire in the same order (and on the same thread)  given a perticular combination of depth,video and skeletal, or, can they come in any order (for a single framenumber) if so, what order is that? rgb, depth, skeleton? depth,rgb, skeleton?

    Can one assume that a depthframe with a perticular frame number will always come in before a skeleton frame with that same frame number? (given the next question is true....)

    ... are will all the diffrent type of frames use a common frame number? in other words, can i match up the frame number of a skeletal frame, a depth frame and a rgb frame and know that all three will refer to exactly the same point in time?

    What i really want to do is find the depth data around the hand from the skeletal data, for that i need to know what depth frame to look at given a perticular skeleton frame, is that possible?


    -edit-

    oh, and another one, if skeleton tracking is enabled, but the tracking cant detect any skeleton because the user is not in view, does the skeleton frame event still fire but with an empty skeleton collection?


    -edit2-

    another one:P can i assume that, if a user is in view, a new depth frame is not triggered before the next skeleton frame? i.e can i be sure the followin chain of event never happen:

    depth frame 1 triggered
    depth frame 2 triggered
    skeleton fram for depth frame 1 triggered

    Thursday, June 16, 2011 9:48 PM

Answers

  • Allan and Luke,

    Depth and skeleton frames with the same frame number will always be guaranteed to correspond exactly to each other. The RGB color frames are not guaranteed to match, so for matching against those the best thing to use is the SkeletonFrame.Timestamp field to get frames that were generated as close in time as possible to each other. The reason is that video frames are not guaranteed to be delivered based on the same clock as the depth frames, so even if they might match up at first you would most likely see drift over time.

    Also, there is no guarantee about order of delivery between skeleton, depth and color frames. Given the behavior of some specific SDK release now or in the future, you might see consistency in delivery, but that perceived consistency is not an API guarantee, so you should not rely on it.

    We understand that these API characteristics can make some scenarios harder to implement, so we will try to provide a sample soon. I'll link to sample from this thread whenever we write or find an appropriate one.

    In the meantime, if frame synchronization is really essential for you, you could look into using SkeletonEngine.GetNextFrame and ImageStream.GetNextFrame methods in a polling loop (maybe on a timer) from a thread you control, such as your main UI thread. Note if you decide to take this approach, you shouldn NOT register for DepthFrameReady, SkeletonFrameReady or VideoFrameReady since calling GetNextFrame from main thread and dispatching frame ready events simultaneously is not a supported scenario for current SDK.

    Hope this helps!
    Eddy


    I'm here to help
    Friday, June 17, 2011 9:03 PM

All replies

  • I have the same questions.
    Friday, June 17, 2011 7:58 PM
  • Allan and Luke,

    Depth and skeleton frames with the same frame number will always be guaranteed to correspond exactly to each other. The RGB color frames are not guaranteed to match, so for matching against those the best thing to use is the SkeletonFrame.Timestamp field to get frames that were generated as close in time as possible to each other. The reason is that video frames are not guaranteed to be delivered based on the same clock as the depth frames, so even if they might match up at first you would most likely see drift over time.

    Also, there is no guarantee about order of delivery between skeleton, depth and color frames. Given the behavior of some specific SDK release now or in the future, you might see consistency in delivery, but that perceived consistency is not an API guarantee, so you should not rely on it.

    We understand that these API characteristics can make some scenarios harder to implement, so we will try to provide a sample soon. I'll link to sample from this thread whenever we write or find an appropriate one.

    In the meantime, if frame synchronization is really essential for you, you could look into using SkeletonEngine.GetNextFrame and ImageStream.GetNextFrame methods in a polling loop (maybe on a timer) from a thread you control, such as your main UI thread. Note if you decide to take this approach, you shouldn NOT register for DepthFrameReady, SkeletonFrameReady or VideoFrameReady since calling GetNextFrame from main thread and dispatching frame ready events simultaneously is not a supported scenario for current SDK.

    Hope this helps!
    Eddy


    I'm here to help
    Friday, June 17, 2011 9:03 PM
  • great info, thanks, its not a huge issue if you use Rx or tdl, as long as we're aware of how the frames come in :)

    so you're saying there never would be an instance where two depth frames arrive before the one skelleton frame arrives? i also didnt know that you could use the polling api from .net, thats very useful indeed for xna :) what about missing frames when using the polling api? i guess there is always going to be a potential race condition there with the kinect hardware, do you have any comments on that?

    it would be really great if there was a way to have skelleton frames also include the depth data they where generated from, or having an option to enable this :)

    Saturday, June 18, 2011 11:29 AM
  • Eddy has good detail about this Beta’s behavior.
    That said, we will be improving many things in releases to come. This is one area that we are thinking hard about.
    The guarantees about order, etc... that Eddy mentions today, may change in releases after this beta.
     
    Thanks,
    Rob Relyea (@rrelyea)
    Kinect for Windows Team (@KinectSDKTeam)
    Monday, June 20, 2011 4:27 PM
  • Allan,

    When using polling API there is indeed a chance you will miss some frames. As Rob said, we're looking at ways to improve API into the future, but for right now you will have to handle this scenario.

    Also, API does not currently guarantee that two depth frames will never arrive before the matching skeleton frame arrives, just that depth and skeleton frames with same frame number will indeed match. That being said, I haven't seen this happen in practice yet. I'm just warning you that behavior is not an API guarantee. Again, as Rob said, we hope to be able to provide more such guarantees in the future.

    Eddy


    I'm here to help
    Wednesday, June 22, 2011 7:17 PM
  • great info, thanks. For me personally im interested in the depth/skeleton sync, so if the skeleton frame had an option to include the depth data as well, i'd be set :)

    Sorry i didnt mark as awnser, i meant to  but you beat me to it ;)

    Saturday, June 25, 2011 6:49 PM