none
Pupil cursor kinect RRS feed

  • Question

  • For my latest project i need to move the cursor only using the movement of my pupils(head stays still) with the help of my kinect. So far, i detected the face, created the mash, obtained the coordinates of the center of the right eye and move the cursor, but it's moving very slowly and i have to move my head a lot. The goal is that the small movements of my eyes to be detected over the whole area of screen.

    I researched the problem and i saw a lot of tips tips for C# where there is the ScaleTo method from the Coding4Fun library that does all the job. My project is implemented in C++ so i need another kind of algorithm to (i think so) create a window to viewport transformation or scaling/mapping method or whatever needs so i can realise a small movement of the pupils in order to move the cursor higher and wider. 

    Source snippet:

    //Get the coordinates for the 100 points detected
    HRESULT hr = pAAMRlt->Get2DShapePoints(&pPts2D,&pts2DCount);
    
    //get the x and y coordinates for the right eye
    int rightX = pPts2D[87].x;
    int rightY = pPts2D[87].y;
    
    RECT r = {0};
    GetWindowRect(GetDesktopWindow(), &r);
    INPUT mouseInput = {0};
    mouseInput.type = INPUT_MOUSE;
    mouseInput.mi.dx = (int) (rightX * 65535) / (r.right - r.left);
    mouseInput.mi.dy = (int) (rightY * 65535) / (r.bottom - r.top);
    mouseInput.mi.dwFlags = MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE;
    
    SetCursorPos(mouseInput.mi.dx,mouseInput.mi.dy);
    SendInput( 1, &mouseInput, sizeof( INPUT ) );
    Later edit: I will post the youtube link with the actual movement once my accound is verified.



    Tuesday, December 17, 2013 6:26 PM

Answers

  • If you want to move the cursor using Head position, you should first decide what is the optimal PHisical Interaction Zone (head movement limit in the real world).

    Ask yourself and mesure, from where to where you want to move your head.

    It could be:

    • a 60cmx40cm rectangle around Kinect field of view center (top-left coordinates:-0.30,0.20 ; bottom-right:0.30,-0.20)
    • or the same rectangle centered on a predefined origin (initial head position, head position during a calibration phase)

    Once the PhIZ is defined, you can compute the transformation (translation, scaling) to convert physical coordinates (in meters) into screen resolution.


    Vincent Guigui Innovative Technologies Expert at OCTO Technology Kinect For Windows MVP award

    • Marked as answer by Bogdan Floria Monday, December 23, 2013 6:39 AM
    Friday, December 20, 2013 11:30 AM

All replies

  • Given the resolution/quality and general size of a eye is going to be when Face Tracking will detect a person, using the Kinect color camera may not be enough. Keep in mind, you don't need to draw the area, since FaceTracking will give you the general area of the eye and you can calculate a rectangular region based on that.

    Given that region(~75x35 pixels) is that enough data and quality to detect a pupil? You might be able to determine left/right based on some semblance of white in the eye. Zooming in requires more resolution, contrary to what "CSI" is able to fake on TV.

    The only way to generate more resolution is to use a hi-resolution color camera. This will be available in the next version of Kinect, but for now you would need to use a separate camera. This introduces new calibration and mapping issues which are beyond the scope of what the SDK can do.


    Carmine Sirignano - MSFT

    Tuesday, December 17, 2013 9:04 PM
  • This is the video with how the project works so far: http://www.youtube.com/watch?v=xdfssNSFbY0

    PS: I need to do something like this: http://futuretechblog.com/?p=71  and i don't think he uses another camera, he uses only the kinect.

    There are a lot of samples that move the cursor with the help of the hand and there are done only with the kinect, i don't really undestand why i have to use another camera to realise the scaling to the monitor device.

    Thank you for your fast and complete response!

    • Edited by Bogdan Floria Wednesday, December 18, 2013 10:10 AM pm
    Wednesday, December 18, 2013 10:05 AM
  • The sample you mentioned (futuretechblog) is not using the pupil position but head movement and some facial movement (eyebrow).

    As Carmine said, current Kinect is not precise enough to detect pupil. The Facetracking SDK gives you general face mesh and external eye contour (eye lid, eye corner) but not the pupil.

    Other face tracking SDK may work better using Kinect and closer distance between user and Kinect.


    Vincent Guigui Innovative Technologies Expert at OCTO Technology Kinect For Windows MVP award

    Wednesday, December 18, 2013 2:54 PM
  • Now I fully understand! Thank you and thanks Carmine Si! But still i have one more question left. If i were to change the project goal, to only move the cursor with the movement of the head, how can i move the cursor with small movements of the head, because right now, you can see in my video, that i have to change the position of my head a lot(!!) so the cursor will move a quite small distance.

     In the samples for the movement of the cursor with the hand for c# i saw the ScaleTo method that scales according to the screen resolution of the computer. I think i need something similar(or point me in the right direction) so that i can change the cursor on the whole width and height of the computer with relatively small movements of the head(not like it is right now, in the uploaded video - http://www.youtube.com/watch?v=xdfssNSFbY0).

    Thank you again for your responses and for making me understand the concept!



    • Edited by Bogdan Floria Wednesday, December 18, 2013 5:47 PM misspelling
    Wednesday, December 18, 2013 5:47 PM
  • If you want to move the cursor using Head position, you should first decide what is the optimal PHisical Interaction Zone (head movement limit in the real world).

    Ask yourself and mesure, from where to where you want to move your head.

    It could be:

    • a 60cmx40cm rectangle around Kinect field of view center (top-left coordinates:-0.30,0.20 ; bottom-right:0.30,-0.20)
    • or the same rectangle centered on a predefined origin (initial head position, head position during a calibration phase)

    Once the PhIZ is defined, you can compute the transformation (translation, scaling) to convert physical coordinates (in meters) into screen resolution.


    Vincent Guigui Innovative Technologies Expert at OCTO Technology Kinect For Windows MVP award

    • Marked as answer by Bogdan Floria Monday, December 23, 2013 6:39 AM
    Friday, December 20, 2013 11:30 AM
  • Thank you for your precise answer! I defined the PHIZ  but now my problem is that i don't know how to translate the physical coordinates into screen resolution. Any hints?
    Monday, December 23, 2013 1:12 PM
  • To translate from one system to another, have a look at kinectSensor.CoordinateMapper class. Especially MapDepthToColor.

    Vincent Guigui Innovative Technologies Expert at OCTO Technology Kinect For Windows MVP award

    Monday, December 23, 2013 1:50 PM
  • Do you mean the 'NuiImageGetColorPixelCoordinatesFromDepthPixel' method in C++?
    Monday, January 27, 2014 12:22 PM
  • Yes, that one.

    Vincent Guigui Innovative Technologies Expert at OCTO Technology Kinect For Windows MVP award

    Monday, January 27, 2014 1:14 PM