none
DepthImageFrame not getting passed into function RRS feed

  • Question

  • I'm having an issue with not being able to pass a DepthImageFrame into functions. By the time it's been passed twice, it's somehow become null. The issue occurs within the getConvertedPoint method that is called from determineDominantHand in handTracking method that is called from the kinectSensor_AllFramesReady method. Below is the code path:

    kinectSensor_AllFramesReady:

    private void kinectSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
            {
    
                // Verifies that nothing has happened to the Kinect and that the SkeletonStream + DepthStream are running
                if (kinectSensor == null || !((KinectSensor)sender).SkeletonStream.IsEnabled || !((KinectSensor)sender).DepthStream.IsEnabled)
                {
                    return;
                }
    
                bool blnHaveSkeletonData = false;
                bool blnHaveDepthData = false;
    
                // Verifies that the skeletonFrame contains data and if so:
                //      Instantiates the 'skeletons' Skeleton array and populates it with the data from skeletonFrame
                //      And sets blnHaveSkeletonData to true
                using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
                {
    
                    if (skeletonFrame != null)
                    {
                        if ((skeletons == null) || (skeletons.Length != skeletonFrame.SkeletonArrayLength))
                        {
                            skeletons = new Skeleton[skeletonFrame.SkeletonArrayLength];
                        }
    
                        skeletonFrame.CopySkeletonDataTo(skeletons);
    
                        blnHaveSkeletonData = true;
    
                    }
    
                }
    
                using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
                {
    
                    if (depthImageFrame != null)
                    {
                        if (pixelData == null) //allocate the first time
                        {
                            pixelData = new short[depthImageFrame.PixelDataLength];
                        }
                        blnHaveDepthData = true;
    
                    }
                    else
                    {
                        Debug.WriteLine("No depth data found; application possibly took too much time");
                    }
    
                }
    
                if (blnHaveSkeletonData)
                {
                    DepthImageFrame currentDepthFrame = e.OpenDepthImageFrame();
    
                    if (currentDepthFrame != null)
                    {
    
                        //int trackedSkeletons = 0;
    
                        Boolean blnHello;
    
                        if (chosenSkeleton == null)
                        {
                            Debug.WriteLine("chosenSkeleton is null!");
                            Skeleton userSkeleton = (from s in skeletons where s.TrackingState == SkeletonTrackingState.Tracked select s).FirstOrDefault();
    
                            if (userSkeleton != null)
                            {
    
                                blnHello = isRightHello(userSkeleton);
    
                                if (blnHello == true)
                                {
                                    kinectSensor.SkeletonStream.AppChoosesSkeletons = true;
                                    kinectSensor.SkeletonStream.ChooseSkeletons(userSkeleton.TrackingId);
                                    chosenSkeleton = userSkeleton;
                                    uiInstance.setCursorColor(Brushes.Green);
                                }
                                else
                                {
                                    kinectSensor.SkeletonStream.AppChoosesSkeletons = false;
                                    uiInstance.setCursorColor(Brushes.Red);
                                }
                            }
    
                        } else if (chosenSkeleton.TrackingState == SkeletonTrackingState.Tracked)
                        {
                            uiInstance.setCursorColor(Brushes.White);
                            Debug.WriteLine("chosenSkeleton is tracked!");
                            //Debug.WriteLine(depthImageFrame);
                            handTracking(chosenSkeleton, MainWindow.strCurrHand, currentDepthFrame);
                            //Debug.WriteLine("chosenSkeleton is null!");
                        }
                        else
                        {
                            Debug.WriteLine("chosenSkeleton is not null, but it's not tracked!");
                            chosenSkeleton = null;
                            kinectSensor.SkeletonStream.AppChoosesSkeletons = false;
                        }
                }//End blnHaveSkeletonData if statement
    
                if (blnHaveDepthData == true)
                {
                    using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
                    {
                        if (depthImageFrame != null)
                        {
                            byte[] pixelData32 = ConvertDepthFrame(pixelData, ((KinectSensor)sender).DepthStream);
                            depthImageFrame.CopyPixelDataTo(pixelData);
                            depth.Source = BitmapSource.Create(depthImageFrame.Width, depthImageFrame.Height, 96, 96, PixelFormats.Bgr32, null, pixelData32, depthImageFrame.Width * Bgr32BytesPerPixel);
    
                            totalFrames++;
    
                            DateTime cur = DateTime.Now;
                            if (cur.Subtract(lastTime) > TimeSpan.FromSeconds(1))
                            {
                                int frameDiff = totalFrames - lastFrames;
                                lastFrames = totalFrames;
                                lastTime = cur;
                                frameRate.Text = frameDiff.ToString() + " fps";
                            }
    
                            //this.kinectDepthImage.Source = this.outputBitmap;
    
                            // DISPLAY OR PROCESS IMAGE DATA IN pixelData HERE
                            depthImageFrame.Dispose();
                        }
                    }
                    
                }// blnHaveDepthData end
    
            } //kinectSensor_allFramesReady end


    handTracking:

    void handTracking(Skeleton currSkeleton, String strCurrHand, DepthImageFrame inputDepthFrame){
                //Debug.WriteLine(strCurrHand);
    
                Double dblUIHandX = 0;
                Double dblUIHandY = 0;
                Boolean blnValidHand = false;
    
                //strCurrHand = "right";
    
                if (inputDepthFrame == null)
                {
                    Debug.WriteLine("depthImageFrame is null!");
                }
                else
                {
                    //Debug.WriteLine(inputDepthFrame);
                }
    
                if (strCurrHand != "left" || strCurrHand != "right")
                {
                    strCurrHand = trackingSystem.determineDominantHand(currSkeleton, inputDepthFrame);
                }
    
                //trackingSystem.setHand(strCurrHand);
    
                if (strCurrHand.Equals("left"))
                {
                    //Run a check-and-draw function
                    Debug.WriteLine("Tracking Left hand!");
                    //MainWindow.CurrentHand = checkAndDraw(data, strCurrHand);
                    //handTracker leftHand = new handTracker();
                    MainWindow.CurrentHand = "left";
                    leftHand.setHand("left");
                    //leftHand.setSkelData(currSkeleton.Joints);
                    leftHand.setTrackedSkeleton(currSkeleton);
                    leftHand.setDepthImageFrame(inputDepthFrame);
                    leftHand.passSkeleton(skeleton);
                    leftHand.setRuntime(kinectSensor);
                    leftHand.tracking();
                    dblUIHandX = leftHand.getHandX();
                    dblUIHandY = leftHand.getHandY();
                    blnValidHand = leftHand.getHandValidity();
                }
                else if (strCurrHand.Equals("right"))
                {
                    //Run a check-and-draw function
                    //MainWindow.CurrentHand = checkAndDraw(data, strCurrHand);
                    Debug.WriteLine("Tracking Right hand!");
                    //handTracker rightHand = new handTracker();
                    MainWindow.CurrentHand = "right";
                    rightHand.setHand("right");
                    //rightHand.setSkelData(currSkeleton.Joints);
                    rightHand.setTrackedSkeleton(currSkeleton);
                    rightHand.passSkeleton(skeleton);
                    rightHand.setDepthImageFrame(inputDepthFrame);
                    rightHand.setRuntime(kinectSensor);
                    rightHand.tracking();
                    dblUIHandX = rightHand.getHandX();
                    dblUIHandY = rightHand.getHandY();
                    blnValidHand = rightHand.getHandValidity();
                }
                else if (strCurrHand.Equals(""))
                {
                    //Debug.WriteLine("I don't know which hand is dominant!");
                    blnValidHand = false;
                    //MainWindow.CurrentHand = determineDominantHand(currSkeleton);
                }
    
                if (blnValidHand == true)
                {
                    Debug.WriteLine("dblUIHandX: " + dblUIHandX);
                    Debug.WriteLine("dblUIHandY: " + dblUIHandY);
                    //Debug.WriteLine("blnValidHand: " + blnValidHand);
                    uiInstance.setCursorX(dblUIHandX);
                    uiInstance.setCursorY(dblUIHandY);
                    uiInstance.updateInterface();
                }
                else
                {
                    MainWindow.strLastActiveHand = strCurrHand;
                    //uiInstance.startTween();
                }
    
            }

    determineDominantHand:

    public String determineDominantHand(Skeleton data, DepthImageFrame depthImageFrame)
            {
                Joint leftHand = new Joint();
                Joint rightHand = new Joint();
                Joint pelvis = new Joint();
                Joint neck = new Joint();
    
                Point leftHandPos = new Point();
                Point rightHandPos = new Point();
                Point pelvisPos = new Point();
                Point neckPos = new Point();
    
                String strCurrHand = "";
    
                leftHand = data.Joints[JointType.HandLeft];
                rightHand = data.Joints[JointType.HandRight];
                pelvis = data.Joints[JointType.HipCenter];
                neck = data.Joints[JointType.ShoulderCenter];
    
                leftHandPos = getConvertedPoint(leftHand, currentDepthImageFrame);
                rightHandPos = getConvertedPoint(rightHand, currentDepthImageFrame);
                pelvisPos = getConvertedPoint(pelvis, currentDepthImageFrame);
                neckPos = getConvertedPoint(neck, currentDepthImageFrame);
    
                /*foreach (Joint currJoint in data.Joints){
                    if (currJoint.JointType == data.Joints[JointType.HandLeft])
                    {
                        leftHand = currJoint;
                    }
                }*/
    
                /*leftHandPos = leftHand.Position;
                rightHandPos = rightHand.Position;
                pelvisPos = pelvis.Position;
                neckPos = neck.Position;*/
    
                //Debug.WriteLine("lefthandpos X: " + leftHandPos.X);
                //Debug.WriteLine("lefthandpos Y: " + leftHandPos.Y);
                //Debug.WriteLine("lefthandpos Z: " + leftHandPos.Z);
    
                if ((rightHandPos.X > neckPos.X) && (rightHandPos.Y < pelvisPos.Y))
                {
                    strCurrHand = "right";
                }
                else if ((leftHandPos.X < neckPos.X) && (leftHandPos.Y < pelvisPos.Y))
                {
                    strCurrHand = "left";
                }
    
                //strCurrHand = "right";
    
                return strCurrHand;
            }
    getConvertedPoint:

    private Point getConvertedPoint(Joint inputJoint, DepthImageFrame depthFrame)
            {
                double inputX = inputJoint.Position.X;
                double inputY = inputJoint.Position.Y;
                double convertedX;
                double convertedY;
                DepthImagePoint depthPoint = new DepthImagePoint();
                Debug.WriteLine("depthFrame: " + depthFrame);
                //Prints the string literal and nothing else, suggesting that depthFrame isn't getting successfully passed
    
                if (depthFrame == null)
                {
                    Debug.WriteLine("depthFrame perceived as null!");
                    //depthFrame = nui.DepthStream.
                    convertedX = 0;
                    convertedY = 0;
                }
                else
                {
                    depthPoint = depthFrame.MapFromSkeletonPoint(inputJoint.Position);
                    convertedX = depthPoint.X / depthFrame.Width;
                    convertedY = depthPoint.Y / depthFrame.Height;
                }
    
                //Debug.WriteLine("inputX: " + inputX);
                //Debug.WriteLine("inputY: " + inputY);
    
                
    
                return new Point(convertedX, convertedY);
            }

    By the time the code gets to getConvertedPoint, somehow the passed DepthImageFrame is null. If you guys need to see any more code, let me know.

    Tuesday, March 13, 2012 1:56 AM

Answers

  • From the code you provided, it looks like you only use the mapping functions from the DepthImageFrame, is that correct? These are available from the KinectSensor object so you wouldn't need to pass this around.

    public DepthImagePoint MapSkeletonPointToDepth (
             SkeletonPoint skeletonPoint,
             DepthImageFormat depthImageFormat
    )
    
    depthImageFormat typical values
    - DepthImageFormat.Resolution640x480Fps30
    - DepthImageFormat.Resolution320x240Fps30

    For maintainability of your code, I would leave the using statements and copy the byte[] data using the CopyPixelDataTo() function. The key is going to be for performance, keep your callback functions short. For anything intensive, spawn a new thread with a local copy of the data.

    Wednesday, March 14, 2012 7:02 PM

All replies

  • This is by design. You will have to copy out the data from the depth frame as fast as possible, as the memory allocated for that frame is only temporary. Have a look at this post for more information on using the depth information. 

    http://social.msdn.microsoft.com/Forums/en-US/kinectsdknuiapi/thread/639f76c2-5880-4435-8e3d-0fac745efed7
    http://social.msdn.microsoft.com/Forums/en-US/kinectsdknuiapi/thread/c39bab30-a704-4de1-948d-307afd128dab


    Tuesday, March 13, 2012 4:20 AM
  • I missed this one, you are calling OpenDepthImageFrame more than once in your callback. After the first time you call it, it will be processed and than it will be disposed and no longer available. Dispose is implicit by the using statement.

    http://msdn.microsoft.com/en-us/library/yh598w02.aspx

    If you don't use the using statement, be sure to explicitly call .Dispose() on the DepthImageFrame At the end of your function.

    using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
    {

    ...

    } <- gets disposed of

    ...

      Debug.WriteLine("No depth data found; application possibly took too much time");
                   
    }

               
    }

               
    if (blnHaveSkeletonData)
               
    {
                   
    DepthImageFrame currentDepthFrame = e.OpenDepthImageFrame();
    ...
     }//End blnHaveSkeletonData if statement

               
    if (blnHaveDepthData == true)
               
    {
                   
    using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
    ...


    Tuesday, March 13, 2012 6:24 PM
  • If I understand what you're saying, if I were to allocate it to a separate variable other than the "depthImageFrame" within the using, I should be able to work with that data?

    Edit: And/or would I be better off not using the 'using' functionality and instead just explicitly disposing of the DepthImageFrame at the end of AllFramesReady?

    I also wanted to add that I appreciate how responsive you guys are to calls for help.

    Tuesday, March 13, 2012 10:44 PM
  • From the code you provided, it looks like you only use the mapping functions from the DepthImageFrame, is that correct? These are available from the KinectSensor object so you wouldn't need to pass this around.

    public DepthImagePoint MapSkeletonPointToDepth (
             SkeletonPoint skeletonPoint,
             DepthImageFormat depthImageFormat
    )
    
    depthImageFormat typical values
    - DepthImageFormat.Resolution640x480Fps30
    - DepthImageFormat.Resolution320x240Fps30

    For maintainability of your code, I would leave the using statements and copy the byte[] data using the CopyPixelDataTo() function. The key is going to be for performance, keep your callback functions short. For anything intensive, spawn a new thread with a local copy of the data.

    Wednesday, March 14, 2012 7:02 PM
  • Awesome. Thanks so much, Carmine!
    Wednesday, March 14, 2012 11:36 PM