Synchronizing ColorFrames to AudioFrames, 400 milisecond delay in RelativeTime? RRS feed

  • Question

  • I'm trying to record a video with audio using the Kinect 2. But I seem unable to match the audio to the video. No matter what I do, the audio seems to be 400ms ahead of the video. I've tried recording the audio using WASAPI (as shown in the SDK sample) and using the AudioBeamFrameReader (also as shown in the SDK sample). Since each AudioBeamFrame and each ColorFrame have a RelativeTime property I tried matching the start of the audio with the start of the video but it seems this +- 400ms mismatch is also present in the RelativeTime property. I've even tried on multiple computers but the effect is the same.

    I'm looking for a sample where the audio and video is record, and stored, and is in sync. But I have not found any projects that have this functionality.

    My current process is as follows.

    I read the color data using a ColorFrameReader, I write this data directly to a compressed AVIStream using Video for Windows and the x264 codec. Of each color frame written I store the RelativeTime.

    I read the audio data using the AudioBeamFrame, I copy the data of each AudioBeamSubFrame to a List<Tuple<TimeSpan, byte[]>>. When I stop recording I try to sync the audio to the video by inserting a silence in the beginning or by throwing some audio data away. I write the audio to the avi file in a IEEE-FLOAT 32 bit 1ch stream.

    To verify that my code is correct (and actually tries to match the audio to the video) I added constant 400ms to the RelativeTime of each audio frame, in this case the audio syncs up nicely. 

    I've been stuck at this for more than a week. Is this the completely wrong way to go? I'm pretty sure its not a simple programming error since two of my colleagues look at this code for a day as well.

    • Edited by roy-t Thursday, December 10, 2015 2:59 PM Obsolete url in footer
    Thursday, December 10, 2015 2:31 PM