none
UIA Text Pattern expectations and alternatives

    Question

  • The Text Pattern support in UIA would be extremely useful to me if more than WPF applications supported it.   The two most important for my target audience are Internet Explorer and Microsoft Word.   Could someone provide guidance about the prospects of Text Pattern support in future and alternatives for the present?

    It appears Word 2010 has zero support, in fact I can't find any way to get even rudimentary text out of a document via UIA.   Is this correct?   Is it likely to change?   Is there a best-practice on other approaches?   (The one I'm considering is a Word add-in that exposes text from the internal object model via some proprietary hack.)

    I stumbled across Michael's IE9HookSample which suggests Text Pattern support is coming to IE9.  I don't see any sign of it using Inspect against the beta build.  Is this something that is in a later CTP?  Is there some reason it would be accessible from an in-proc hook but not via Inspect?  Am I just doing something boneheaded?

    Thanks in advance,
    -Blake

    Wednesday, February 09, 2011 10:48 AM

All replies

  • Scratch one of those possible answers off the list.  The shiny new IE9 RC build doesn't show Text Pattern support when queried with Inspect.exe.

    Any insight?

    Saturday, February 12, 2011 12:23 AM
  • IEBlog has now made support official, but still no response here:

    http://blogs.msdn.com/b/ie/archive/2011/02/15/ie9-s-assistive-technology-interface-enhancements.aspx

    Time for further guesswork, it appears.

    Wednesday, February 16, 2011 12:02 AM
  • Hi, Blake,

    IE9 RC does support Text Pattern.  You can see this using my sample; for full reference, it is here:
    http://code.msdn.microsoft.com/ie9textscope

    The IE Accessibility tree has a lot of interior nodes; if you navigate up to the one that corresponds to the root of the "Internet Explorer_Server" HWND, you'll see the one with IsTextPatternAvailable = true.  My sample does demonstrate this.

    Please give that a try and reply back if you don't see it.

    Thanks,
    Michael


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, February 16, 2011 8:04 PM
  • Thank you, Michael.

    I was aware of your IE9TextScope sample (and mentioned in the first post in this thread).   Your psychic debugging was on the nose about my missing that I needed to navigate up the tree to find the pane that implements TextPattern.

    Having made it through that, I've turned up two or three other bugs.

    1) Using an out of process client, IE9's TextPattern is visible through the new COM interface but not through the function interface.   At least, that appears to be the underlying problem.  The managed code below works fine linked against your UIAComWrapper.dll but fails to find a TextPattern at all when linked against UIAutomationClient/UIAutomationTypes.dll.

    2) If IE's zoom is set to anything other than 100% (which is surely to be true for most low-vision users) then IUIAutomationTextPattern::RangeFromPoint mis-interprets the position by the zoom factor.  The same sample code below demonstrates that bug.

    3) Not officially supported, so less important, but your UIAComWrapper's version of AutomationElement seems to always give me a null .Current.ControlType.

      [StructLayout(LayoutKind.Sequential)]
      struct Win32Point { public int X, Y; }
      [DllImport("user32.dll", SetLastError = true)]
      [return: MarshalAs(UnmanagedType.Bool)]
      static extern bool GetPhysicalCursorPos(out Win32Point pt);

      static Point GetCursorPos() {
       Win32Point wpt;
       GetPhysicalCursorPos(out wpt);
       return new Point(wpt.X, wpt.Y);
      }

      static TextPattern GetTextPattern(this AutomationElement ae) {
       object o = null;
       while (ae != null && !ae.TryGetCurrentPattern(TextPattern.Pattern, out o))
        ae = TreeWalker.RawViewWalker.GetParent(ae);
       return o as TextPattern;
      }

      static int Main(string[] args)
      {
       var pt = GetCursorPos();
       Console.WriteLine("Cursor Point: {0}", pt);
       var ae = AutomationElement.FromPoint(pt);
       Console.WriteLine("AutomationElement Rect: {0}", ae.Current.BoundingRectangle);
       var tp = ae.GetTextPattern();
       if (tp == null) {
        Console.Error.WriteLine("No TextPattern found: {0}", ae.Current.LocalizedControlType);
        return 1;
       }
       var r = tp.RangeFromPoint(pt);
       r.ExpandToEnclosingUnit(TextUnit.Word);
       Console.WriteLine("Word Under Cursor: {0}", r.GetText(1024));
       return 0;
      }
     

    Thursday, February 17, 2011 1:27 AM
  • Regarding bug #2 above, closer examination shows IE9's RangeFromPoint calculation is both off by the zoom factor and an additional offset related to where the origin of the TextPattern lies compared to the origin of the client area.

    The less perfomant, but accurate work-around for me has been to simply iterate through the bounding rectangles of all the pattern's visible ranges and find the one that contains the point.  (With appropriate fallback for the case where none of them contain the point, of course.)

    Regards,
    -Blake

    Thursday, February 17, 2011 11:55 AM
  • Thanks, Blake.  Some color commentary up front: the IE9 Text Pattern's first goal is to replace the functionality that a GDI hook used to provide for IE: getting a copy of all of the output text data.  Your work-around is very much in this spirit: you get a copy of the visible text data and than analyze the copy, rather than using RangeFromPoint.   RangeFromPoint() turns out to be an interestingly ambiguous concept in IE -- for example, for any given bit of whitespace (say, off to the side), what does it mean to find the 'closest' range?  I think you might be wise to use the approach you have, which will eliminate any such ambiguity. 

    Issue #1: This is a known limitation with the managed-code UIA Client.  IE is implementing UIA via the IAccessibleEx interface (introduced with Windows 7)  The managed-code client (introduced with Windows Vista) simply doesn't know how to support IAccessibleEx.  Using the COM client is the right solution.  If my UiaComWrapper is helpful with this, I'm delighted.

    Issue #2: Thanks for mentioning this -- I'll refer it to our test team.

    Issue #3: That's surprising - I'll have to take a look when I get some spare cycles.  I thought I had a unit test for that, but if I don't, I will soon. 

    Thanks,
    Michael


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, February 17, 2011 4:33 PM