locked
Using KinectAudio for audio playback RRS feed

  • Question

  • Hi Kinect / Unity guys :) !

    I would like to use the AudioSource Reader of the Kinect SDK in Unity in order to have an audio playback. What I did is when I get a valid AudioBeamFrame (not null), I copy the data into a byte array and convert it to a float array in order to be readable by a Unity AudioClip. I have something that is working but the sound I get is intermittent.

    This is my code :

    byte[] aBytes = new byte[_Sensor.AudioSource.SubFrameLengthInBytes];
    subframe.CopyFrameDataToArray(aBytes);
                
    Buffer.BlockCopy(aBytes, 0, audioFloats, 0, aBytes.Length);
    
    
    AudioClip myClip = AudioClip.Create("MyAudioSample", 16000, 1, 16000, false, false);
    myClip.SetData(audioFloats, 0);
    
    audio.clip = myClip;
    audio.Play();

    Have you ever experienced this guys ?

    Any input welcome :)


    Tuesday, January 20, 2015 10:21 PM

Answers

  • You will have to double buffer the audio and typically you will want a separate thread that is always poll the audio in 50ms blocks. Depending on how long you take to process the buffer and capture the next block, you may be missing audio that is going to result in clicks/pops

    Carmine Sirignano - MSFT

    Wednesday, January 21, 2015 7:32 PM
  • Ok guys, I finally found what I was looking for.

    In Unity, the kinect can be used as a Microphone, so you can do this to have an audio playback :

    using UnityEngine;
    using System.Collections;
    public class MicStreamer : MonoBehaviour
    {
    
        void Start(){
            StartCoroutine(StartRecord());
        }
    
        private IEnumerator StartRecord()
        {
            
            audio.clip = Microphone.Start(Microphone.devices[0], true, 5, 16000);
            /*
         the longer the mic recording time, the less often there are "hiccups" in game performance
         but also due to being pitched down, the playback gradually falls farther behind the recording
         */
    
            StartCoroutine(StartPlay(audio.clip));
            yield return new WaitForSeconds(5);
            StartCoroutine(StartRecord()); //swaps audio buffers, begins recording and playback of new buffer
            /* it is necessary to swap buffers, otherwise the audioclip quickly becomes too large and begins to slow down the system */
            
        }
        
        private IEnumerator StartPlay(AudioClip buffer)
        {
            audio.clip = buffer;
            yield return new WaitForSeconds(.01f);
            audio.Play();
        }
    }

    Friday, January 30, 2015 4:57 PM

All replies

  • You will have to double buffer the audio and typically you will want a separate thread that is always poll the audio in 50ms blocks. Depending on how long you take to process the buffer and capture the next block, you may be missing audio that is going to result in clicks/pops

    Carmine Sirignano - MSFT

    Wednesday, January 21, 2015 7:32 PM
  • Yes I was thinking about that but I am not really sure on how to have real multithreading in Unity.
    Wednesday, January 21, 2015 7:56 PM
  • I think that the Duration property of AudioBeamSubFrame is giving me a wrong value. The current value is 160 seconds and given the audio specs : 16 000Hz sample rate, 32-bit (4bytes) precision , 1024 bytes in a frame, so a subframe duration should be : 1024 / 4 = 256 samples in one subframe meaning 256 / 16000 seconds, 16 ms. Am I right ?




    Wednesday, January 21, 2015 9:42 PM
  • Ok guys, I finally found what I was looking for.

    In Unity, the kinect can be used as a Microphone, so you can do this to have an audio playback :

    using UnityEngine;
    using System.Collections;
    public class MicStreamer : MonoBehaviour
    {
    
        void Start(){
            StartCoroutine(StartRecord());
        }
    
        private IEnumerator StartRecord()
        {
            
            audio.clip = Microphone.Start(Microphone.devices[0], true, 5, 16000);
            /*
         the longer the mic recording time, the less often there are "hiccups" in game performance
         but also due to being pitched down, the playback gradually falls farther behind the recording
         */
    
            StartCoroutine(StartPlay(audio.clip));
            yield return new WaitForSeconds(5);
            StartCoroutine(StartRecord()); //swaps audio buffers, begins recording and playback of new buffer
            /* it is necessary to swap buffers, otherwise the audioclip quickly becomes too large and begins to slow down the system */
            
        }
        
        private IEnumerator StartPlay(AudioClip buffer)
        {
            audio.clip = buffer;
            yield return new WaitForSeconds(.01f);
            audio.Play();
        }
    }

    Friday, January 30, 2015 4:57 PM