How to get the exact word or text on the screen when I pass x-y coordinates

Question

I am making a Windows Store App.

I have it currently so that when I hover over a word, I get a feedback. Each word that is important is in a separate TextBlock and I have a hover event associated with it. But instead of the hover event what I want to do is send x-y coordinates and based on that get the word/text that is on the screen. 

Eg: I have the following sentence on screen as individual textblocks. "The quick brown fox"

<textblock>The</textblock><textblock>quick</textblock> <textblock>brown</textblock> <textblock>fox</textblock>

If I pass the coordinates x, y as some value and if it matches the textblock of "quick", it should display a feedback somewhere else on the screen saying that the word at position (x,y) is "quick". If there is no textblock for the x,y coordinates provided the feedback should be that no word detected.

I am developing in C# and XAML

I am fairly new and learning Windows Store App development. This may be something easy but I have spent hours trying to figure it out and failed. I appreciate any help!

Answer 1

You can use VisualTreeHelper to find the control at a given point, but getting the specific text may not be possible. The information is exposed through UI Automation, but Windows Store apps act only as servers (providing the info) not as clients (consuming the info). It may be possible to get directly to the TextPattern from within the app itself though.

Answer 2

Thank you I will look into VisualTreeHelper

Answer 3

Hello again Rob,

I used VisualTreeHelper and that was great help. Now in order to get the word I need to put individual words in its own textblock because if I put each word in individual runs I am unable to get the value for that run. Here is what I have used:

private bool DoesPointContainElement(Point hitTestPoint, string elementName, UIElement referenceFrame)
        {
            IEnumerable<UIElement> elementStack =
              VisualTreeHelper.FindElementsInHostCoordinates(hitTestPoint, referenceFrame);
            foreach (UIElement item in elementStack)
            {
                HitTestElement.Text += " "+(String)item.ToString();
                FrameworkElement feItem = item as FrameworkElement; //cast to FrameworkElement, need the Name property
                DependencyObject current = VisualTreeHelper.GetChild(item, 0);
                if(current!=null)HitTestElement.Text += " child: " + (String)current.ToString();
                if (feItem != null)
                {
                    if (feItem.Name.Equals(elementName))
                    {
                        return true;
                    }
                }
            }
            // elementName was not in this stack 
            return false;
        }

void OnEntered(object sender, PointerRoutedEventArgs e)
        {
            if (e.Pointer.PointerDeviceType != PointerDeviceType.Mouse)
                return;

            
            this.count++;
            PointerPoint pp = e.GetCurrentPoint(null); // or pass an element for relative coords

            double x = pp.Position.X;
            double y = pp.Position.Y;
            Point location;
            location.X = x;
            location.Y = y;
            string hello = r1.Text;
            //showInfo.Text = hello;
            //showInfo.Text = "Location: " + this.count + "--> x,y: " + location.ToString();
            string testElement = "Text1";
           if (DoesPointContainElement(location, testElement, mainpage))
            {
                showInfo.Text = "Location: " + this.count + "--> x,y: " + location.ToString() + (String)r1.Text;
            }
           //this.PointerEntered -= OnEntered;
           //e.Handled = true;

        }

<Grid x:Name="Grid_Text" HorizontalAlignment="Left" VerticalAlignment="Top" Margin="17,76,0,0" Grid.Row="1" Width="922" Height="74">
            <TextBlock x:Name="Text1" Grid.Column="0" IsHitTestVisible="True" TextAlignment="Justify" Style="{StaticResource EbookTextStyle}" VerticalAlignment="Top" Height="60" FontFamily="Global User Interface" HorizontalAlignment="Left" Width="881">
        		<Span x:Name="s1">
        			<Run x:Name="r1" Text="Here"/>
        		</Span>
        		<Span x:Name="s2">
        			<Run Text=" is"/>
        		</Span>
        		<Span x:Name="s3">
        			<Run Text=" Edward"/>
        		</Span>
            </TextBlock>
        </Grid>

is there anyway I can get the return value of DoesPointContainElement as a x:Name of the run?

---

Anuj