How do I manage mouseover event using UI Automation?


  • Hello all,

    In first place I apoligize for my English, because I'm studying it now.

    I would like to know how do I manage mouseover event using UI Automation? I know that there is a handle in order to call a method when focus change to other application

       AutomationFocusChangedEventHandler afceh = new AutomationFocusChangedEventHandler(funcion2);

    but I need to call a function when mouse position changes over screen elements.

    Does anybody have the answer?

    Thursday, June 25, 2009 8:17 PM


All replies

  • This is not supported in UI Automation at this time.  The application itself knows when the mouse passes over it, but the client does not. 

    If you need to monitor what the mouse is doing, you might be able to use a low-level mouse hook:


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, June 26, 2009 3:55 PM
  • Hello Michael,

    In first place, thank you for your answer. I would like to ask other question. Do you think that using this hook I could get screen elements information in order to build a screen reader? Last week I have seen a video from MS Accessibility Developer Center where to build a screen reader using UI Automation is explained but I think it doesn't work because only controls focus change event.

    Thank you for your time and answers.

    Best Regards.

    Jesús Merino.
    Sunday, June 28, 2009 10:23 AM
  • Hi, Jesus,

    I'm not sure I understand your question.  It sounds like you want to build a screen reader where the reader reads what the mouse is moving over.  Is that right?

    If so, then yes, you can use UI Automation to do that.  The mouse hook will tell you the coordinates of the mouse pointer.  You can then use the UI Automation API function ElementFromPoint() to find the UI element at those coordinates, and ask the element for its name, role and other information.  Once you have that, you can write the rest of the code to read that information out loud.

    This posting is provided "AS IS" with no warranties, and confers no rights.
    Tuesday, June 30, 2009 3:44 AM
  • Hi Michael,

    It was my idea. I have started to do it yesterday.

    Thank you very very much for your time. I hope I can write you in future if I have problems.

    Best Regards,

    Tuesday, June 30, 2009 7:48 AM
  • Hello again,

    As said I have been working in order to study how to use low-level mouse hooks. I have got to obtain mouse's coordinates when it moves, but when I try to get an AutomationElement using FromPoint function an exception occurs. I paste the exception code:

    System.Runtime.InteropServices.COMException (0x8001010D): No se puede hacer una llamada de salida desde la aplicación que está ejecutando una llamada de sincronización de entrada. (Excepción de HRESULT: 0x8001010D (RPC_E_CANTCALLOUT_ININPUTSYNCCALL))
       en Accessibility.IAccessible.accHitTest(Int32 xLeft, Int32 yTop)
       en MS.Internal.AutomationProxies.Accessible.HitTest(Int32 x, Int32 y)
       en MS.Internal.AutomationProxies.MsaaNativeProvider.DescendantFromPoint(Int32 x, Int32 y, Boolean nullMeansThis)
       en MS.Internal.AutomationProxies.MsaaNativeProvider.System.Windows.Automation.Provider.IRawElementProviderFragmentRoot.ElementProviderFromPoint(Double x, Double y)
       en System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
       en MS.Internal.Automation.UiaCoreApi.CheckError(Int32 hr)
       en MS.Internal.Automation.UiaCoreApi.UiaNodeFromPoint(Double x, Double y, UiaCacheRequest request)
       en System.Windows.Automation.AutomationElement.DrillForPointOrFocus(Boolean atPoint, Point pt, UiaCacheRequest cacheRequest)
       en System.Windows.Automation.AutomationElement.FromPoint(Point pt)
       en GlobalHookDemo.MainForm.MouseMoved(Object sender, MouseEventArgs e) en C:\Documents and Settings\Jesús\Escritorio\globalhook_src\MainForm.cs:línea 128

    adçnd the code that is executed is the following:
    public void MouseMoved(object sender, MouseEventArgs e)
                System.Windows.Point p_x = new System.Windows.Point(e.X, e.Y);//e.X, e.Y);
                if (e.X % 5 == 0)
    //In the next line exception is thrown
     AutomationElement ae = AutomationElement.FromPoint(p_x); labelMousePosition.Text = String.Format("susito x={0} y={1} wheel={2} name={3}", e.X, e.Y, e.Delta, ae.Current.Name); LogWrite("nombrE: " + ae.Current.Name); } catch(Exception ex) { LogWrite("Se ha capturado una excepcion: " + ex.ToString()); } } }

    Any idea? I'm stuck.

    Thank you very much.

    Thursday, July 02, 2009 8:11 PM
  • Sorry for the late response -- we've been busy with Windows 7.

    The error code there is a good guide: you can't make outbound calls from within an inbound synchronous call.  (This rule exists to prevent deadlocks: A calls B and waits for a response, B calls back to A, and now you are stuck.)

    Try having MouseMoved() post a message back to your main message loop with the point information.  When the main loop gets around to processing the posted message, it will be OK to call out using AutomationElement.FromPoint().

    - Michael
    This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, July 23, 2009 4:08 PM
  • Hello Michael.

    I've found another solution creating and controlling new child process, but I will try your way too. Thank you very much. I'm proud of you answer my questions.

    Good luck with Windows 7.

    Best regards,

    Jesús Merino.
    Thursday, July 23, 2009 9:05 PM