Recording Kinect v2 footage and playback

Question

Hi,

Been trying to work out how to record some clips of footage using the Kinect v2 and then play back the footage.

I am using the body basics samples and have found the following functions to record and playback, the record function works and records footage but makes the live feed freeze whilst recording and the playback function freezes the live footage and then seems to show a couple of frames of data before the live body basics feed resumes. The functions I am using are below. I have simply added a couple of buttons to the body basics sample, record and playback and then call the functions in the event handlers, saving to a .xef file.

public static void RecordClip(string filePath, TimeSpan duration)
        {
            using (KStudioClient client = KStudio.CreateClient())
            {
                client.ConnectToService();

                KStudioEventStreamSelectorCollection streamCollection = new KStudioEventStreamSelectorCollection();
                streamCollection.Add(KStudioEventStreamDataTypeIds.Ir);
                streamCollection.Add(KStudioEventStreamDataTypeIds.Depth);
                streamCollection.Add(KStudioEventStreamDataTypeIds.Body);
                streamCollection.Add(KStudioEventStreamDataTypeIds.BodyIndex);

                using (KStudioRecording recording = client.CreateRecording(filePath, streamCollection))
                {
                    recording.StartTimed(duration);
                    while (recording.State == KStudioRecordingState.Recording)
                    {
                        System.Threading.Thread.Sleep(500);
                    }

                    if (recording.State == KStudioRecordingState.Error)
                    {
                        throw new InvalidOperationException("Error:Recording failed!");
                    }
                }

                client.DisconnectFromService();
            }
        }

        public static void PlaybackClip(string filePath, uint loopCount)
        {
            using (KStudioClient client = KStudio.CreateClient())
            {
                client.ConnectToService();

                using (KStudioPlayback playback = client.CreatePlayback(filePath))
                {
                    playback.LoopCount = loopCount;
                    playback.Start();

                    while (playback.State == KStudioPlaybackState.Playing)
                    {
                        System.Threading.Thread.Sleep(500);
                    }

                    if (playback.State == KStudioPlaybackState.Error)
                    {
                        throw new InvalidOperationException("Error: Playback failed!");
                    }
                }

                client.DisconnectFromService();
            }
        }

I can only find minimal documentation on the KStudioClient class, so help would be massively appreciated.

Answer 1

The api's work the same as if you are using the KinectStudio(KStudio) app. Take the scenario when you are playing back a clip from KStudio. This will interrupt the live feed to playback then recorded data, since it is taking over from the driver. If you playback the clip in KStudio you should get the same behavior.

That said Body data is stateless. Even though you may have recorded 500seconds, the amount of body data is generated based on the Depth/IR information in the clip. You need to ensure the clip recording has the frames to acquire a lock-on if you expect to have some consistency during each playback. Even then, you will have some variance given the state of the system you are using will not be the same during each playback.

---

Carmine Sirignano - MSFT

Answer 2

Hello,

The above code will work great when running from a separate command line tool or application, but if you were to add this code directly to one of the WPF samples, then you will fail to see anything visualized during record/playback because the main thread will be frozen. Instead, you will want to call the record and playback functions asynchronously, by adding them as background tasks to the Dispatcher. Here is an example of how you can do this in WPF:

 /// <summary> Delegate for placing a job with no arguments onto the Dispatcher </summary>
 private delegate void NoArgDelegate();

 /// <summary> Delegate for placing a job with a single string argument onto the Dispatcher </summary>
 /// <param name="arg">string argument</param>
 private delegate void OneArgDelegate(string arg);

 ///
 /// Code to initialize MainWindow, KStudioClient, etc. should go here
 ///       

 /// <summary>
 /// Handles the user clicking on the Record button
 /// </summary>
 private void RecordButton_Click(object sender, RoutedEventArgs e)
 {
     this.isRecording = true;
     this.UpdateButtonState();

     // Start running the recording asynchronously
     OneArgDelegate recording = new OneArgDelegate(this.RecordClip);
     recording.BeginInvoke(@"C:\temp\kinectRecording0.xef", null, null);
 }

 /// <summary>
 /// Handles the user clicking on the Play button
 /// </summary>
 private void PlayButton_Click(object sender, RoutedEventArgs e)
 {
     this.isPlaying = true;
     this.UpdateButtonState();
     // Start running the playback asynchronously
     OneArgDelegate playback = new OneArgDelegate(this.PlaybackClip);
     playback.BeginInvoke(@"C:\temp\kinectRecording0.xef", null, null);
 }

 /// <summary>
 /// Records a new .xef file
 /// </summary>
 private void RecordClip(string filePath)
 {
     // Specify which streams should be recorded
     KStudioEventStreamSelectorCollection streamCollection = new KStudioEventStreamSelectorCollection();
                 streamCollection.Add(KStudioEventStreamDataTypeIds.Ir);
                 streamCollection.Add(KStudioEventStreamDataTypeIds.Depth);
                 streamCollection.Add(KStudioEventStreamDataTypeIds.Body);
                 streamCollection.Add(KStudioEventStreamDataTypeIds.BodyIndex);

     // Create the recording object
     using (KStudioRecording recording = this.client.CreateRecording(filePath, streamCollection))
     {
         recording.StartTimed(this.duration);
         while (recording.State == KStudioRecordingState.Recording)
         {
             Thread.Sleep(500);
         }
     }

     // Update UI after the background recording task has completed
     this.isRecording = false;
     this.Dispatcher.BeginInvoke(new NoArgDelegate(UpdateButtonState));
 }

 /// <summary>
 /// Plays back a .xef file to the Kinect sensor
 /// </summary>
 private void PlaybackClip(string filePath)
 {
    // Create the playback object
    using (KStudioPlayback playback = client.CreatePlayback(filePath))
    {
        playback.LoopCount = this.loopCount;
        playback.Start();

        while (playback.State == KStudioPlaybackState.Playing)
        {
            Thread.Sleep(500);
        }
    }

    // Update the UI after the background playback task has completed
    this.isPlaying = false;
    this.Dispatcher.BeginInvoke(new NoArgDelegate(UpdateButtonState));
 }

Answer 3

hi, Kev3

I am  so confused cause I have been learning Kinect for a limited period. Do you know how to  call the functions in the event handlers now?

 

  private void record_Click(object sender, RoutedEventArgs e)
        {
               //What should I do here?
        }

  private void replay_Click(object sender, RoutedEventArgs e)
        {
          //What should I do here?

        }

Answer 4

hi, angela

couly you tell me what the content of the function UpdateButtonState()?

How to declare the this.loopCount\this.duration?

Thank you a lot 

Answer 5

I have posted a basic record/playback sample here, which shows how to use the Kinect tooling APIs: https://github.com/angelaHillier/RecordAndPlaybackBasics-WPF.git