Use "Controls Basics" example (or alternatives) for Windows mouse cursor control RRS feed

  • Question

  • Hello,

    I am trying to change the Windows mouse cursor position via Kinect Hand Tracking. Therefore I used the "BodyBasics" example and changed some code in the "Reader_FrameArrived" event:

    if (body.IsTracked) { Joint handRight = body.Joints[JointType.HandRight]; // sometimes the depth(Z) of an inferred joint may show as negative // clamp down to 0.1f to prevent coordinatemapper from returning (-Infinity, -Infinity) CameraSpacePoint position = handRight.Position; /* if (position.Z < 0) { position.Z = InferredZPositionClamp; }*/ position.Z = 1; // Coordinate Mapping //DepthSpacePoint colorSpacePoint = this.coordinateMapper.MapCameraPointToDepthSpace(position); ColorSpacePoint colorSpacePoint = this.coordinateMapper.MapCameraPointToColorSpace(position); // Set Mouse int x = Convert.ToInt32(colorSpacePoint.X); /* if (colorSpacePoint.X >= 360.0) { x = Convert.ToInt32(colorSpacePoint.X * (1920 / 512)); } else { x = Convert.ToInt32(1920 - (colorSpacePoint.X * (1920 / 512))); }*/ // = Convert.ToInt32(colorSpacePoint.X * (1920 / 320)); // Console.WriteLine(colorSpacePoint.X); int y = Convert.ToInt32(colorSpacePoint.Y); Console.WriteLine(x); Console.WriteLine(y); setMousePosition(x, y); }

    public void setMousePosition(int x, int y)
                SetCursorPos(x, y);

    The mapping is the problem here, especially because it seems that the z-coordinate of your hand will change the x and y coordinates. That´s why I set it to 1. Also I am working with an ColorSpacePoint. It´s working ok, but I don´t really want to move to the left or right to get to the top corner of the screen. So I think I need a multiplication factor.

    In addition I looked through the "Controls Basics" example. The HandPointer is working very nicely, but I couldn´t find the math for the screen mapping and I think it´s only working for WPF and XAML (?).

    Can anyone enlighten me pls.

    Thanks for the help


    Tuesday, September 23, 2014 12:59 PM

All replies

  • Hand Pointer in Controls Basics is using KinectCoreWindow's PointerMoved events under the covers. You can look at the pointer events sub sample to see how to get that raw data.

    Note, these pointer events only are raised to the current focused app. If you are attempting to support apps you control, this will work fine. It is not possible to use this data centrally to control many apps.

    Perhaps you can describe more details of what you are trying to do, so we can help steer you the right direction.

    Thanks, Rob Relyea

    Rob Relyea [MSFT]

    Tuesday, September 23, 2014 4:19 PM
  • You should watch the MVA Programming Kinect for Windows v2 Jump Start Chapter 7:

    What you need is a PHIZ -> Physical Interaction Zone. The PHIZ is like a virtual area around your natural Hand Position which represents your Screen space. So as you described you only want to move your Hand like 30cm to reach the border of the Screen.

    The SDK has a builtin PHIZ embedded in the Interactions, you can use it through class KinectCoreWindow. However it can only be used in a single application and not as a Service or Background application, so you won't be able to control the mouse when another application has the Focus. For that you will have to implement your own PHIZ.

    Tuesday, September 23, 2014 4:19 PM
  • @ Rob Relyea: I am developing a small background application primary to control Powerpoint slides with gestures for my university. This part is working quite good. I made a heuristic and a VGB approach. However I am thinking about adding a gesture which activates the general windows mouse cursor control via your right hand joint coordinates.

    @ Philippe Gackstatter: As written above, this will be a background application. I basically need to create a rectangle and my raised hand will be the middle point. Then I need to map the coordinates of my hand in this rectangle to my resolution, is this correct?

    Thanks again guys


    Tuesday, September 23, 2014 4:45 PM
  • Basically that's correct. But as in the MVA Video described the center should be located at an offset to your shoulder so the center can move along with the tracked Body. The Video describes in Detail how this can be implemented. Unfortunately it requires a little try and error to find good values for the size of the PHIZ and some clever logic to smooth the movements, as you don't want the Cursor to jump around. For the smoothing you can check this thread: 

    Another interesting document can be found here -> check page 80+.

    Tuesday, September 23, 2014 5:23 PM