none
Read and save ColorFrames in memory at 30 FPS RRS feed

  • General discussion

  • Hello everybody,

    I´m trying to read and save to disk the ColorFrames that are received from the KinectService. I used the ColorBasics-WPF sample as base code to receive ColorFrames and convert them to WriteableBitmaps.

    Unfortunately, converting and saving to disk on the fly takes time and if I do so, I only write at about 6 FPS. To my understanding the missing frames get discarded because my Event execution is taking too long. Without doing this process the event gets in fact called at a 30 FPS rate.

    For this reason, my intention is to keep the frames in memory and write them to disk afterwards when the sequence has finished. However, here is my problem, only a single instance of colorFrame (ColorFrame colorFrame = e.FrameReference.AcquireFrame()) can be acquired and used at a time. If colorFrame is not disposed, e.FrameReference.AcquireFrame() returns null in the next call of the event, therefore a copy/conversion has to be made. If I create this copy (converting to WriteableBitmap for instance) in the event code, from the external thread, the main thread is not able to read this data later. The thrown exception is "System.InvalidException: The calling thread cannot access this object because a different thread owns it". On the other hand, if I reserve this memory previously from the main thread, this event thread is not able to write there either.

    I´ve tried several workarounds such as doing the work asynchronously (won´t work because the colorFrame has to be disposed on time) or sending the job to the SynchronizationContext of the main thread. In any case, the memory is mutually exclusive which means that the external event thread is not able to access the memory of the main thread and viceversa.

    This is the code of this last option I´ve tested (part of a .NET 4.5 Windows Console Application):

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Threading;
    using System.Threading.Tasks;
    using System.IO;
    using System.Windows.Media.Imaging;
    using System.Windows.Media;
    using System.Windows;
    using Microsoft.Kinect;
    
    public class BitmapReader
    {
    	private const int BUFFER_SIZE = 600;
    	private KinectSensor kinect;
    	private ColorFrameReader colorFrameReader = null;
    	private WriteableBitmap[] frameBuffer = null;
    	private int frameCount = 0;
    	private static SynchronizationContext mainContext;
    
    	public BitmapReader(KinectSensor sensor)
    	{
    		mainContext = SynchronizationContext.Current;
    
    		this.kinect = sensor;
    
    		// Connect the colour frame handler and enable frame tracking
    		this.colorFrameReader = kinect.ColorFrameSource.OpenReader();
    		this.colorFrameReader.FrameArrived += FrameReady;
    	   
    		// create the colorFrameDescription from the ColorFrameSource using Bgra format
    		FrameDescription colorFrameDescription = kinect.ColorFrameSource.CreateFrameDescription(ColorImageFormat.Bgra);
    
    		Console.WriteLine("Reserving memory for buffer now.");
    
    		// reserve the bitmap buffer
    		frameBuffer = new WriteableBitmap[BUFFER_SIZE];
    
    		for (int i = 0; i < BUFFER_SIZE; ++i)
    			frameBuffer[i] = new WriteableBitmap(colorFrameDescription.Width, colorFrameDescription.Height, 96.0, 96.0, PixelFormats.Bgr32, null);
    	}
    
    	private void FrameReady(object sender, ColorFrameArrivedEventArgs e)
    	{
    		using (ColorFrame colorFrame = e.FrameReference.AcquireFrame())
    		{
    			if (colorFrame != null)
    			{
    				if (frameCount < BUFFER_SIZE)
    				{
    					mainContext.Send(new SendOrPostCallback(new Action<object>(o =>
    					{
    						CopyColorFrame(colorFrame);
    					})), null);
    				}
    				else
    				{
    					throw new Exception("Color frame buffer is full.");
    				}
    			}
    		}
    	}
    
    	private void CopyColorFrame(ColorFrame colorFrame)
    	{
    		FrameDescription colorFrameDescription = colorFrame.FrameDescription;
    		var colorBitmap = frameBuffer[frameCount];
    
    		// Copy frame to local memory
    		using (KinectBuffer colorBuffer = colorFrame.LockRawImageBuffer())
    		{
    			colorBitmap.Lock();
    
    			// write the new color frame data to the display bitmap
    			colorFrame.CopyConvertedFrameDataToIntPtr(
    				colorBitmap.BackBuffer,
    				(uint)(colorFrameDescription.Width * colorFrameDescription.Height * 4),
    				ColorImageFormat.Bgra);
    
    			// Invalidate
    			// colorBitmap.AddDirtyRect(new Int32Rect(0, 0, colorBitmap.PixelWidth, colorBitmap.PixelHeight));
    
    			colorBitmap.Unlock();
    		}
    
                    ++frameCount;
    	}
    }

    Where the mentioned exception is thrown by the 

    colorBitmap.Lock();
    instruction.

    I suppose there must be a much simpler way to do this.

    Any help is appreciated. Thanks a lot in advance.



    Saturday, November 29, 2014 4:53 PM

All replies

  • I would not mix WPF/.Net specific constructs when dealing with raw byte data. You are best to create a circular buffer (size to be determined by you and the performance of the application/memory) which you will just copy the raw frame data to this buffer, which in-turn you can create a worker queue that pulls the frames from the buffer and then saves to disk. At the same time, you can then use that raw byte buffer for a particular frame as the source frame for your writeable bitmap.

    Performance is going to heavily depend on the system and memory constraints of the system. For high-performance systems, you might want to create your own "recording" library that you write in c++ where you provide that library the raw frames and it will buffer and save independently of your application.


    Carmine Sirignano - MSFT

    Monday, December 1, 2014 7:57 PM