none
Kinect SDK InteractionStream : InvalidCastException with polling acquisition RRS feed

  • Question

  • Hello.

    I am trying to use the Kinect Interaction toolkit with a polling acquisition rather than an event acquisition of the Kinect stream.

    No problem when i use the Kinect interaction and event handlers to process the depth and the skeleton stream (polling=false in the code below).

    If i use a thread to request the stream (polling=true in the code below), i got the following error :

    Unable to cast COM object of type 'System. __ComObject' to interface type 'Microsoft. Kinect. Toolkit. Interaction. INuiInteractionStream'. This operation failed because the QueryInterface call on the COM component for the interface with IID  failed due to the following error: No such interface supported (Exception from HRESULT: 0x80004002 (E_NOINTERFACE)).

    The code i use :

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Windows;
    using System.Windows.Controls;
    using System.Windows.Data;
    using System.Windows.Documents;
    using System.Windows.Input;
    using System.Windows.Media;
    using System.Windows.Media.Imaging;
    using System.Windows.Navigation;
    using System.Windows.Shapes;
    using Microsoft.Kinect;
    using Microsoft.Kinect.Toolkit.Interaction;
    using System.Diagnostics;
    using System.IO;
    using System.Threading;
    
    
    namespace KinectInteraction
    {
        public partial class MainWindow : Window
        {
            KinectSensor nui;
            short[] depthPixels;
            Skeleton[] skeletons = new Skeleton[0];
    
            InteractionStream interactionStream;
            MDKinectInteractions kinectInteraction;
    
            UserInfo[] userInfos;
    
    
            void SensorDepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
            {
                using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
                {
                    if (depthFrame != null)
                    {
                        depthFrame.CopyPixelDataTo(depthPixels);
                        interactionStream.ProcessDepth(depthFrame.GetRawPixelData(), depthFrame.Timestamp);
                    }
                }
            }
    
    
            void OnSkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
            {
                if (null != nui)
                {
                    using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame())
                    {
                        if (skeletonFrame != null)
                        {
                            if (skeletons.Length != skeletonFrame.SkeletonArrayLength)
                            {
                                skeletons = new Skeleton[skeletonFrame.SkeletonArrayLength];
                            }
    
                            skeletonFrame.CopySkeletonDataTo(skeletons);
                            interactionStream.ProcessSkeleton(skeletons, nui.AccelerometerGetCurrentReading(), skeletonFrame.Timestamp);
                        }
                    }
                }
            }
    
    
            void Its_InteractionFrameReady(object sender, InteractionFrameReadyEventArgs e)
            {
                using (InteractionFrame frame = e.OpenInteractionFrame())
                {
                    if (frame != null)
                    {
                        if (userInfos == null)
                        {
                            userInfos = new UserInfo[InteractionFrame.UserInfoArrayLength];
                        }
    
                        frame.CopyInteractionDataTo(userInfos);
                    }
                    else
                    {
                        return;
                    }
                }
    
                foreach (UserInfo userInfo in userInfos)
                {
                    foreach (InteractionHandPointer handPointer in userInfo.HandPointers)
                    {
                        string action = null;
    
                        switch (handPointer.HandEventType)
                        {
                            case InteractionHandEventType.Grip:
                                action = "gripped";
                                break;
    
                            case InteractionHandEventType.GripRelease:
                                action = "released";
                                break;
                        }
    
                        if (action != null)
                        {
                            string handSide = "unknown";
    
                            switch (handPointer.HandType)
                            {
                                case InteractionHandType.Left:
                                    handSide = "left";
                                    break;
    
                                case InteractionHandType.Right:
                                    handSide = "right";
                                    break;
                            }
    
                            string getInteraction = "User " + userInfo.SkeletonTrackingId + " " + action + " their " + handSide + "hand.";
                            Console.WriteLine(getInteraction);
                        }
                    }
                }
            }
    
    
            void DoWork()
            {
                while (true)
                {
                    if (null == this.nui)
                    {
                        return;
                    }
    
                    using (DepthImageFrame depthFrame = this.nui.DepthStream.OpenNextFrame(0))
                    {
                        if (null != depthFrame)
                        {
                            depthFrame.CopyPixelDataTo(depthPixels);
                            interactionStream.ProcessDepth(depthFrame.GetRawPixelData(), depthFrame.Timestamp);
                        }
                    }
    
                    using (var skeletonFrame = this.nui.SkeletonStream.OpenNextFrame(0))
                    {
                        if (skeletonFrame != null)
                        {
                            if (skeletons.Length != skeletonFrame.SkeletonArrayLength)
                            {
                                skeletons = new Skeleton[skeletonFrame.SkeletonArrayLength];
                            }
    
                            skeletonFrame.CopySkeletonDataTo(skeletons);
                            interactionStream.ProcessSkeleton(skeletons, nui.AccelerometerGetCurrentReading(), skeletonFrame.Timestamp);
                        }
                    }
    
                    using (InteractionFrame frame = interactionStream.OpenNextFrame(0))
                    {
                        if (frame != null)
                        {
                            if (userInfos == null)
                            {
                                userInfos = new UserInfo[InteractionFrame.UserInfoArrayLength];
                            }
    
                            frame.CopyInteractionDataTo(userInfos);
                        }
                        else
                        {
                            return;
                        }
                    }
    
                    foreach (UserInfo userInfo in userInfos)
                    {
                        foreach (InteractionHandPointer handPointer in userInfo.HandPointers)
                        {
                            string action = null;
    
                            switch (handPointer.HandEventType)
                            {
                                case InteractionHandEventType.Grip:
                                    action = "gripped";
                                    break;
    
                                case InteractionHandEventType.GripRelease:
                                    action = "released";
                                    break;
                            }
    
                            if (action != null)
                            {
                                string handSide = "unknown";
    
                                switch (handPointer.HandType)
                                {
                                    case InteractionHandType.Left:
                                        handSide = "left";
                                        break;
    
                                    case InteractionHandType.Right:
                                        handSide = "right";
                                        break;
                                }
    
                                string getInteraction = "User " + userInfo.SkeletonTrackingId + " " + action + " their " + handSide + "hand.";
                                Console.WriteLine(getInteraction);
                            }
    
                        }
                    }
                }
                
            }
    
    
            public MainWindow()
            {
                InitializeComponent();
    
                bool polling = true;
    
                var index = 0;
                while (nui == null && index < KinectSensor.KinectSensors.Count)
                {
                    try
                    {
                        nui = KinectSensor.KinectSensors[index];
    
                        nui.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
                        nui.DepthStream.Range = DepthRange.Default;
                        nui.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;
                        nui.SkeletonStream.Enable();
    
                        depthPixels = new short[nui.DepthStream.FramePixelDataLength];
    
                        nui.Start();
    
                        kinectInteraction = new MDKinectInteractions();
                        interactionStream = new InteractionStream(nui, kinectInteraction);
    
                        if (!polling)
                        {
                            interactionStream.InteractionFrameReady += Its_InteractionFrameReady;
    
                            nui.DepthFrameReady += SensorDepthFrameReady;
                            nui.SkeletonFrameReady += OnSkeletonFrameReady;
                        }
                    }
                    catch (IOException ex)
                    {
                        nui = null;
                        Debug.Print(ex.StackTrace);
                    }
                    catch (InvalidOperationException ex)
                    {
                        nui = null;
                        Debug.Print(ex.StackTrace);
                    }
    
                    index++;
                }
    
    
                if (polling)
                {
                    Thread thread = new Thread(DoWork);
                    thread.Start();
                }
            }
        }
    
    
        public class MDKinectInteractions : IInteractionClient
        {
            public InteractionInfo GetInteractionInfoAtLocation(int skeletonTrackingId, InteractionHandType handType, double x, double y)
            {
                return new InteractionInfo
                {
                    IsPressTarget = true,
                    IsGripTarget = true,
                };
            }
        }
    }
    

    Thanks.

    Friday, June 28, 2013 3:30 PM

Answers

All replies

  • Ok. Problem solved with : http://stackoverflow.com/questions/12162818/loading-com-object-in-c-sharp-throws-exception-unable-to-cast-com-object-of-typ
    Monday, July 1, 2013 9:45 AM
  • Hi,

    How exactly did you solve the problem ?

    I had the same problem and I solve it by manually setting the Timestamp of Depth and Skeleton streams.


    Vincent Guigui Innovative Technologies Expert at OCTO Technology MVP Kinect

    Tuesday, July 2, 2013 9:23 AM
  • Hi.

    My problem (if I guess right) was that I created interactionStream in the constructor and I fed it with the depth and skeleton stream in an another thread (DoWork).

    So, I just construct the interactionStream in the DoWork method (called by a thread after) before the while instruction.

    How do you manually set the Timestamp ? Something like that ?

    long customTimestamp = depthFrame.Timestamp;
    
    //[...]
    
    interactionStream.ProcessDepth(depthFrame.GetRawPixelData(), customTimestamp);
    
    //[...]
    
    interactionStream.ProcessSkeleton(skeletons, nui.AccelerometerGetCurrentReading(), customTimestamp);

    I didn't dig further your method but this doesn't work for me because I still access interactionStream from an another thread and I think this is the reason of the exception occurring in my program.

    I think there must be a proper method to resolve my problem of using the Kinect Interaction dll with thread but I just start learning C#.



    Tuesday, July 2, 2013 3:09 PM
  • I did exactly what you wrote in order to get Depth and Skeleton streams in "sync".

    Accessing InteractionStream from a thread shouldn't be an issue.

    Keep me updated if you find another solution


    Vincent Guigui Innovative Technologies Expert at OCTO Technology MVP Kinect

    Tuesday, July 2, 2013 3:29 PM
  • I am not exactly sure what you are trying to do with polling vs event model? If you use AllFramesReady, you will get color, depth and skeletal. By using separate callbacks, you will find your color and depth our out of sync.

    Carmine Si - MSFT



    Tuesday, July 9, 2013 7:16 PM