none
How to get all visible words of an TextPattern using UIAutomation?

    Question

  • I'm using UIAutomation to get all the words from an AutomationElement that are currently visible on the screen, including its text and its bounding rectangles.

    I already receive all visible lines using the TextPattern.GetVisibleRanges() method which returns an array of TextPatternRanges.

    In the next step, I try to split each line into words using the TextPatternRange.Move() method with TextUnit.Word. The problem is that I could not find an indicator that tells me when I reach the last word of the line. 

    I already tried comparing the endpoints of the TextPatternRanges of the line and the current word, their bounding rectangles or the length of the string containing the lines and the extracted words. But I still could not figure out when I reach the end of the line. That leads to an infinit loop in some lines because there is no indication when to end the line processing.

    // Traverse all (at least partially) visible lines
    var visibleLines = textPattern.GetVisibleRanges();
    List<string> testresults = new List<string>();
    string expectedText = "";
    foreach (var lineRange in visibleLines)
    {
    	// Select first word
    	var currentRange = lineRange.Clone();
    	currentRange.Move(TextUnit.Word, 1);
    	currentRange.Move(TextUnit.Word, -1);
    
    	expectedText += lineRange.GetText(-1);
    	int lineLength = lineRange.GetText(-1).Length;
    	int currentLength = 0;
    
    	// Traverse through all words of the line
    	int distance = -1;
    	System.Windows.Rect currentWordBounds = System.Windows.Rect.Empty;
    	System.Windows.Rect lastWordBounds = System.Windows.Rect.Empty;
    	while ((currentLength < lineLength) && (currentWordBounds.IsEmpty || !lastWordBounds.Equals(currentWordBounds)))
    	{
    		string wordText = currentRange.GetText(-1);
    		currentLength += wordText.Length;
    		var currentWordBoundList = currentRange.GetBoundingRectangles();
    		lastWordBounds = currentWordBounds;
    		currentWordBounds = currentWordBoundList.Length != 0 ? currentWordBoundList[0] : System.Windows.Rect.Empty;
    		if (!currentWordBounds.IsEmpty)
    			testresults.Add(wordText);
    
    		currentRange.Move(TextUnit.Word, 1);
    		distance = currentRange.CompareEndpoints(TextPatternRangeEndpoint.End, lineRange, TextPatternRangeEndpoint.End);
    	}
    }
    

    Using this code, the inner while-loop does not break at the end of the line in some cases although I compare the length of the extracted words and the length of the line as well as the their bounding rectangles. I also tried to compare the endpoints of the TextPatternRanges of the current word and the line using the TextPatternRange.CompareEndpoints() method, but it always returns -1 even for the last word of the line. This is probably caused by invisible characters such as \r\n, but I don't know how to get around this issue. Unfortunately, I could not find an example that shows how to achieve this.

    So my question is: How can I detect if I reached the end of a line TextPattern by using the Move method? Or which break condition should I use for the inner while loop, respectively? Or is there another, better way to split a line TextPatternRange into multiple word TextPatternRanges?


    Sunday, January 08, 2017 5:47 PM

All replies

  • Hi Mr Sparkle,

    According to your description, you want to get all the text content in TextPattern. In other words, you want to get the displayed text with UI Automation, is it right?

    Please refer to following thread which provide a sample code about get element text with UI Automation. Hope that can help you.

    http://stackoverflow.com/questions/23850176/c-sharp-system-windows-automation-get-element-text

    For your code snippet, we need your TextPattern sample to help us reproduce this issue in our side. Please provide a entire sample about your TextPattern to help us reproduce. Thanks.

    Best Regards,
    Weiwei


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, January 09, 2017 5:31 AM
  • Hi Weiwei,

    thank you for your reply. Acutally I don't just need the displayed text, but the text of each word together with its bounding rectangle.

    The example you posted just retrieves the text value of the whole document.

    I tried iterating though the TextPatternRanges returned by the TextPattern.GetVisibleRanges method, because the TextPatternRange provides both the text and the bounds of each word. But the iteration ends up in an infinite loop because I cannot find a working breaking condition for the loop.

    The complete example looks like this:

    AutomationElement element = AutomationElement.FromPoint(new System.Windows.Point(x, y));
    object pattern;
    element.TryGetCurrentPattern(TextPattern.Pattern, out pattern);
    TextPattern textPattern = pattern as TextPattern;
    
    if (textPattern != null)
    {
    	// Traverse all (at least partially) visible lines
    	TextPatternRange[] visibleLines = textPattern.GetVisibleRanges();
    	List<string> testresults = new List<string>();
    	string expectedText = "";
    	foreach (var lineRange in visibleLines)
    	{
    		// Select first word
    		TextPatternRange currentRange = lineRange.Clone();
    		currentRange.Move(TextUnit.Word, 1);
    		currentRange.Move(TextUnit.Word, -1);
    
    		expectedText += lineRange.GetText(-1);
    		int lineLength = lineRange.GetText(-1).Length;
    		int currentLength = 0;
    
    		// Traverse through all words of the line
    		int distance = -1;
    		System.Windows.Rect currentWordBounds = System.Windows.Rect.Empty;
    		System.Windows.Rect lastWordBounds = System.Windows.Rect.Empty;
    		while ((currentLength < lineLength) && (currentWordBounds.IsEmpty || !lastWordBounds.Equals(currentWordBounds)))
    		{
    			string wordText = currentRange.GetText(-1);
    			currentLength += wordText.Length;
    			System.Windows.Rect[] currentWordBoundList = currentRange.GetBoundingRectangles();
    			lastWordBounds = currentWordBounds;
    			currentWordBounds = currentWordBoundList.Length != 0 ? currentWordBoundList[0] : System.Windows.Rect.Empty;
    			if (!currentWordBounds.IsEmpty)
    				testresults.Add(wordText);
    
    			currentRange.Move(TextUnit.Word, 1);
    			distance = currentRange.CompareEndpoints(TextPatternRangeEndpoint.End, lineRange, TextPatternRangeEndpoint.End);
    		}
    	}
    }
    
    


    Monday, January 09, 2017 2:31 PM
  • Or to rephrase my question: Is it possible to get the exact word at a specified position, e.g. under the mouse pointer?
    Monday, January 09, 2017 9:17 PM
  • Hi Mr Sparkle,

    This forum is discuss Visual Studio WPF/SL Designer, Visual Studio Guidance Automation Toolkit, Developer Documentation and Help System, and Visual Studio Editor.

    Since your problem is related to UI Automation, I help you move this thread to Automation forum to get a better help. Thanks for your understanding.

    Best Regards,
    Weiwei


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Wednesday, January 11, 2017 6:36 AM
  • Or to rephrase my question: Is it possible to get the exact word at a specified position, e.g. under the mouse pointer?

    What kind of control is exposing the text through UI Automation?
    Wednesday, January 11, 2017 10:33 AM
  • @Weiwei: Unfortunately, I couldn't post in the UIAutomation Forum. It was not listed in the forum list when I created this thread. Could you please move the thread over there?

    @RLWA32: It should work with any control that exposes the TextPattern, but I tested it with the Code Editor of Visual Studio.

    Tuesday, January 17, 2017 12:56 AM
  • Hi Mr Sparkle,

    I have moved this thread to Windows Desktop Development for Accessibility and Automation forum, which forum often discuss UIAutomation problem.

    Best Regards,
    Weiwei


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, January 17, 2017 3:09 AM