none
Unity Kinect v2 audio input RRS feed

  • Question

  • I am trying to get audio input to work for with unity. Body tracking works fine. I followed the audio basics tutorial and adapted it for unity in a similar way to the Body tutorial was done.

    Here is a snippet of my current code. Even though  _AudioReader.AcquireLatestBeamFrames (); returns a non null list, it always only has one element in the list, which is null. Ideas?

    void Start () 
    	{
    		_Sensor = KinectSensor.GetDefault();
    
    		if (_Sensor != null)
    		{
    			_Reader = _Sensor.BodyFrameSource.OpenReader();
    
    			
    			if (!_Sensor.IsOpen)
    			{
    				_Sensor.Open();
    			}
    
    			//initialize audio reader
    			var audioSource = _Sensor.AudioSource;
    			this.audioBuffer = new byte[audioSource.SubFrameLengthInBytes];
    			_AudioReader = _Sensor.AudioSource.OpenReader ();
    
    		}   
    	}
    	
    	void Update () 
    	{
    				if (_Reader != null) {
    						var frame = _Reader.AcquireLatestFrame ();
    		
    						if (frame != null) {
    								if (_Data == null) {
    										_Data = new Body[_Sensor.BodyFrameSource.BodyCount];
    								}
    				
    								frame.GetAndRefreshBodyData (_Data);
    				
    								frame.Dispose ();
    								frame = null;
    						}
    				}    
    				if (_AudioReader != null) {
    						var audioFrames = _AudioReader.AcquireLatestBeamFrames ();
    		
    						if (audioFrames != null) {
    						
    								// it gets here just fine
    								if (audioFrames [0] != null) {
    									// it never gives me any audio frames!
    										var subFrameList = audioFrames [0].SubFrames;
    										foreach (AudioBeamSubFrame subFrame in subFrameList) {
    												// Check if beam angle and/or confidence have changed
    												bool updateBeam = false;
    					
    												if (subFrame.BeamAngle != this.beamAngle) {
    														this.beamAngle = subFrame.BeamAngle;
    														updateBeam = true;
    												}
    					
    												if (subFrame.BeamAngleConfidence != this.beamAngleConfidence) {
    														this.beamAngleConfidence = subFrame.BeamAngleConfidence;
    														updateBeam = true;
    												}
    					
    												if (updateBeam) {
    														// Refresh display of audio beam
    														this.AudioBeamChanged ();
    												}
    					
    												// Process audio buffer


    Friday, August 1, 2014 6:00 PM

All replies

  • Does it work at least once or nothing happens at all? How are you disposing the audioFrames object or subFrameList?

    var audioFrames = _AudioReader.AcquireLatestBeamFrames ();

    var subFrameList = audioFrames [0].SubFrames;

    As with all the frame objects they have to be disposed which might be the issue here. subFrameList should be of type AudioBeamFrame which has a .Dispose(), as well as, audioFrames [0]. The returned collection from AcquireLatestBeamFrames may be the issue since that does not have a Dispose, but you may be able to cast it(not tested). I will look into this and report it to the team.

    Carmine Sirignano - MSFT

    Tuesday, August 5, 2014 4:38 PM
  • yes im using Dispose but it never gets there. I feel like i got it to go in there one time before I remembered to put the dispose in, but now it never works. Does the sdk keep some memory of that per process? When I run the audio basics sample, it works fine for me.

    See the rest of my code below.

    // Process audio buffer
    												subFrame.CopyFrameDataToArray (this.audioBuffer);
    					
    												for (int i = 0; i < this.audioBuffer.Length; i += BytesPerSample) {
    														// Extract the 32-bit IEEE float sample from the byte array
    														float audioSample = BitConverter.ToSingle (this.audioBuffer, i);
    						
    														this.accumulatedSquareSum += audioSample * audioSample;
    														++this.accumulatedSampleCount;
    						
    														if (this.accumulatedSampleCount < SamplesPerColumn) {
    																continue;
    														}
    						
    														float meanSquare = this.accumulatedSquareSum / SamplesPerColumn;
    						
    														if (meanSquare > 1.0f) {   
    																// A loud audio source right next to the sensor may result in mean square values
    																// greater than 1.0. Cap it at 1.0f for display purposes.
    																meanSquare = 1.0f;
    														} 
    						
    														// Calculate energy in dB, in the range [MinEnergy, 0], where MinEnergy < 0
    														float energy = MinEnergy;
    						
    														if (meanSquare > 0) {
    																energy = (float)(10.0 * Math.Log10 (meanSquare));
    														}
    						
    														//	lock (this.energyLock)
    														//	{
    														// Normalize values to the range [0, 1] for display
    														this.energy [this.energyIndex] = (MinEnergy - energy) / MinEnergy;
    														this.energyIndex = (this.energyIndex + 1) % this.energy.Length;
    														++this.newEnergyAvailable;
    														currEnergy = this.energy [this.energyIndex];
    														//	}
    						
    														this.accumulatedSquareSum = 0;
    														this.accumulatedSampleCount = 0;
    												}
    										}
    					audioFrames[0].Dispose();
    					audioFrames[0] = null;
    				}

    Tuesday, August 5, 2014 4:55 PM
  • Yes, the runtime is aware of outstanding frame objects and if your process does not release the frames in a timely manner you will not longer get any more updates. If you compare the AudioSample, you have to dispose the AcquireLatestFrame  object.

    I have submitted a bug on this.


    Carmine Sirignano - MSFT

    Tuesday, August 5, 2014 5:27 PM
  • oddly, without me doing anything but restarting my computer, it seems to work now. I suspect what happened is that i originally ran it without disposing the frame. Of course it didnt work and then I added that in but somehow it must have clogged up a port or something that was associated with my unity process
    Tuesday, August 5, 2014 7:05 PM
  • anyone solved it till now??

    It's shame on microsoft team!

    Thursday, June 9, 2016 4:28 PM