Video codec for saving depth stream

Question

Hello Everyone,

I am using C# for saving the depth video stream. After recording the video, when extracted the frames in MATLAB then each depth image is 3 channel, which is not correct(as far as I know). But, when I tried just to save the frame then it is 1 channel depth image which is correct. So, I found that it is a problem in saving video codec, currently I am using Intel IYUV codec. Can you anyone please help which codec would save exactly same channel data as it is saving a depth image. 

Answer 1

How are you saving off the depth video stream info in C#?  One approach would be to get the depth info and convert it to color info and save that off in a writeable bitmap. 

From there, you can manipulate the color stream in MATLAB and save it off in a video format.

Check out the DepthBasics sample in the SDK browser.  This function converts the depth data to color data.

private unsafe void ProcessDepthFrameData(IntPtr depthFrameData, uint depthFrameDataSize, ushort minDepth, ushort maxDepth)
        {
            // depth frame data is a 16 bit value
            ushort* frameData = (ushort*)depthFrameData;

            // convert depth to a visual representation
            for (int i = 0; i < (int)(depthFrameDataSize / this.depthFrameDescription.BytesPerPixel); ++i)
            {
                // Get the depth for this pixel
                ushort depth = frameData[i];

                // To convert to a byte, we're mapping the depth value to the byte range.
                // Values outside the reliable depth range are mapped to 0 (black).
                this.depthPixels[i] = (byte)(depth >= minDepth && depth <= maxDepth ? (depth / MapDepthToByte) : 0);
            }
        }

Answer 2

Hello Joan, here is code for the depth video stream recording.

IplImage depth = new IplImage(512, 424, BitDepth.U16, 1);
CvVideoWriter DepthWriter;
Width = sensor.DepthFrameSource.FrameDescription.Width;
DHeight = sensor.DepthFrameSource.FrameDescription.Height;
WbDepth = new WriteableBitmap(DWidth, DHeight, 96, 96, PixelFormats.Gray16, null);
int depthshft = (int)SliderDepth.Value;
using (DepthFrame depthframe = frame.DepthFrameReference.AcquireFrame())
ushort* depthdata = (ushort*)depth.ImageData;
if (depthframe != null)
   {
    Depthdata = new ushort[DWidth * DHeight];
    ushort[] Depthloc = new ushort[DWidth * DHeight];
    depthframe.CopyFrameDataToArray(Depthdata);
    for (int i = 0; i < DWidth * DHeight; i++)
        {
          Depthloc[i] = 0x1000;
        }
        colorspacePoint = new ColorSpacePoint[DWidth * DHeight];
        depthspacePoint = new DepthSpacePoint[CWidth * CHeight];
        sensor.CoordinateMapper.MapDepthFrameToColorSpace(Depthloc, colorspacePoint);
        for (int y = 0; y < DHeight; y++)
            {
            for (int x = 0; x < DWidth; x++)
                {
                if (depthshft != 0)
                   {
                    Depthdata[y * DWidth + x] = (ushort)((Depthdata[y * DWidth + x]) << depthshft);
                   }

                 }
             }
         depth.CopyPixelData(Depthdata);
 }
WbDepth.WritePixels(new Int32Rect(0, 0, DWidth, DHeight), Depthdata, strideDep, 0);
ImageDepth.Source = WbDepth;
if (depth != null && DepthWriter.FileName != null) Cv.WriteFrame(DepthWriter, depth);
Cv.ReleaseVideoWriter(DepthWriter);
if (CheckBox_saveD.IsChecked == true)
DepthWriter = new CvVideoWriter(string.Format("{1}\\Scene{0}_DepthRecord.avi", scene, TextBlock_saveloca.Text.ToString()), FourCC.Default, 30.0f, new CvSize(512, 424));
CheckBox_saveD.IsEnabled = false;
if (CheckBox_saveD.IsChecked == true) Cv.ReleaseVideoWriter(DepthWriter);