locked
Obtaining list of icons on toolbar and on desktop RRS feed

  • Question

  • Hello, 

    I would like to use UIA API to write a piece of code that performs the following:

    1. maps all the programs icons that are on the TaskBar, and on the dekstop

    2. get the location of each icon on the screen  (pixel X-Y coordinates)

    3. get the name (or relevant .exe file) of each icon

    Im completely new to UIA API so I'd really appreciate any guidance on where to start.
    Any piece of code that performs something alike, or other relevant examples will be mostly appreciated

    thanks!

    Wednesday, August 22, 2012 6:03 PM

Answers

  • Hi, I don’t think UIA can be used to find the executables associated with the desktop icons or taskbar icons. So if those exes are necessary, you may need take another approach. But if UIA is still of interest, this is what I’d do…

    Run the latest Inspect tool that’s a part of the Windows SDK. Inspect uses the native-code UIA client API to access data that’s shown on the screen. So any results that Inspect shows can also be accessed by your own app if it uses the UIA API.

    I pointed Inspect to the Recycle Bin icon, and got the following results…


    On the left side, I can see the UIA tree of elements, with the Recycle Bin icon as an item in a list of icons on the desktop. So the first step would be for your app to programmatically access the list of icons. You never want to do a find through all elements below the root UIA element, (that’s the “desktop” pane element at the top of the tree”). So I would do a find through only the direct children of that root element, in order to find the pane element that’s the grandparent of the list element for the icons. I clicked on the pane element in Inspect and looked at its properties, and unfortunately it doesn’t expose much useful data. (Its name’s empty, and it has no AutomationId.) I noticed that its class name is “WorkerW”, so if this is the only direct child of the root which has this ClassName, then you could find the element using that.

    If you look at the code snippets I included at http://social.msdn.microsoft.com/Forums/en-US/windowsaccessibilityandautomation/thread/9c41eb58-754a-4bfd-8f5e-693d0bc48525/#bfbe32e2-e789-4c9f-9891-ae02cb1aacf2, you can see how I found a direct child of the root based on an element’s AutomationId. So you would do something like…

    VARIANT varProp;
    varProp.vt = VT_BSTR;
    varProp.bstrVal = SysAllocString(L”WorkerW”);
     
    IUIAutomationCondition *pCondition; 
    hr = pUIAutomation->CreatePropertyCondition(UIA_ClassNamePropertyId, varProp, &pCondition);
    if (SUCCEEDED(hr))
    {
        hr = pElementRoot->FindFirst(TreeScope_Children, pCondition, &pElementApp);

    By doing this, you’d get the pane element which is the grandparent of the list you’re after.

    You then want to find the list itself. You could use an IUIAutomationTreeWalker to get the first child of the WorkerW pane, and then the first child of that, but TreeWalker’s are slow given that they involve a cross-process call with each UIA call made. So I would probably do a FindFirst() again, using a UIA_ControlTypePropertyId of UIA_ListControlTypeId. (The FindFirst() would take TreeScope_Descendants, because the list isn’t a direct child of the pane element you retrieved earlier.)

    Having got the list element, you then want certain properties of all the listitems beneath the list element. If you point Inspect to the Recycle Bin icon again, on the right side you see all the properties being exposed by the element. This includes a BoundingRectangle and a Name. So could call FindAll() from the list pane to find all direct children of the list element, and also set a up a cache request, to say that when UIA returns the collection of listitems, the BoundingRectangle and Name properties are to be cached with the results. This means that when you access those properties, no cross-process calls are being made at that time. This means a lot of data is being gathered in a single cross-process call, and that’s great for performance.

    In fact, I’m pretty sure you could optimize this further if you want to:

    (i) Call FindFirst() from the root element to get the pane which is the grandparent of the list.
    (ii) Call FindAllBuildCache() and say you want all elements whose control type is listitem, and you want the Name and BoundingRect properties cached with that call.

    You’d then iterate through the array of results and call CachedBoundingRectangle() to get the bounding rectangles of the elements.

    So there are different approaches to accessing the data you want. It’s really a case of looking at the results shown in Inspect, and picking an approach that you’re comfortable with and is performant enough for your needs. (You’d take the same approach for the icons on the taskbar.)

    I’d recommend you take a look at my sample up at http://code.msdn.microsoft.com/Windows-7-UI-Automation-9131f729. This accesses all hyperlinks shown in a browser, and caches the Name and BoundingRects of the hyperlink elements. So it probably makes a lot of the same calls that your app would make.

    Thanks,

    Guy

    • Marked as answer by Lior.writeme.a Wednesday, August 29, 2012 2:10 PM
    Thursday, August 23, 2012 2:23 PM

All replies

  • Hi, I don’t think UIA can be used to find the executables associated with the desktop icons or taskbar icons. So if those exes are necessary, you may need take another approach. But if UIA is still of interest, this is what I’d do…

    Run the latest Inspect tool that’s a part of the Windows SDK. Inspect uses the native-code UIA client API to access data that’s shown on the screen. So any results that Inspect shows can also be accessed by your own app if it uses the UIA API.

    I pointed Inspect to the Recycle Bin icon, and got the following results…


    On the left side, I can see the UIA tree of elements, with the Recycle Bin icon as an item in a list of icons on the desktop. So the first step would be for your app to programmatically access the list of icons. You never want to do a find through all elements below the root UIA element, (that’s the “desktop” pane element at the top of the tree”). So I would do a find through only the direct children of that root element, in order to find the pane element that’s the grandparent of the list element for the icons. I clicked on the pane element in Inspect and looked at its properties, and unfortunately it doesn’t expose much useful data. (Its name’s empty, and it has no AutomationId.) I noticed that its class name is “WorkerW”, so if this is the only direct child of the root which has this ClassName, then you could find the element using that.

    If you look at the code snippets I included at http://social.msdn.microsoft.com/Forums/en-US/windowsaccessibilityandautomation/thread/9c41eb58-754a-4bfd-8f5e-693d0bc48525/#bfbe32e2-e789-4c9f-9891-ae02cb1aacf2, you can see how I found a direct child of the root based on an element’s AutomationId. So you would do something like…

    VARIANT varProp;
    varProp.vt = VT_BSTR;
    varProp.bstrVal = SysAllocString(L”WorkerW”);
     
    IUIAutomationCondition *pCondition; 
    hr = pUIAutomation->CreatePropertyCondition(UIA_ClassNamePropertyId, varProp, &pCondition);
    if (SUCCEEDED(hr))
    {
        hr = pElementRoot->FindFirst(TreeScope_Children, pCondition, &pElementApp);

    By doing this, you’d get the pane element which is the grandparent of the list you’re after.

    You then want to find the list itself. You could use an IUIAutomationTreeWalker to get the first child of the WorkerW pane, and then the first child of that, but TreeWalker’s are slow given that they involve a cross-process call with each UIA call made. So I would probably do a FindFirst() again, using a UIA_ControlTypePropertyId of UIA_ListControlTypeId. (The FindFirst() would take TreeScope_Descendants, because the list isn’t a direct child of the pane element you retrieved earlier.)

    Having got the list element, you then want certain properties of all the listitems beneath the list element. If you point Inspect to the Recycle Bin icon again, on the right side you see all the properties being exposed by the element. This includes a BoundingRectangle and a Name. So could call FindAll() from the list pane to find all direct children of the list element, and also set a up a cache request, to say that when UIA returns the collection of listitems, the BoundingRectangle and Name properties are to be cached with the results. This means that when you access those properties, no cross-process calls are being made at that time. This means a lot of data is being gathered in a single cross-process call, and that’s great for performance.

    In fact, I’m pretty sure you could optimize this further if you want to:

    (i) Call FindFirst() from the root element to get the pane which is the grandparent of the list.
    (ii) Call FindAllBuildCache() and say you want all elements whose control type is listitem, and you want the Name and BoundingRect properties cached with that call.

    You’d then iterate through the array of results and call CachedBoundingRectangle() to get the bounding rectangles of the elements.

    So there are different approaches to accessing the data you want. It’s really a case of looking at the results shown in Inspect, and picking an approach that you’re comfortable with and is performant enough for your needs. (You’d take the same approach for the icons on the taskbar.)

    I’d recommend you take a look at my sample up at http://code.msdn.microsoft.com/Windows-7-UI-Automation-9131f729. This accesses all hyperlinks shown in a browser, and caches the Name and BoundingRects of the hyperlink elements. So it probably makes a lot of the same calls that your app would make.

    Thanks,

    Guy

    • Marked as answer by Lior.writeme.a Wednesday, August 29, 2012 2:10 PM
    Thursday, August 23, 2012 2:23 PM
  • Thank you very much for the detailed answer

    I wrote some lines of code and managed to build a list of the taskbar icons.
    The code I wrote is in C#, not C++.

    I tried to do the same for the desktop icons as you explained. Looking for a FirstChild of the root element with the ClassName "WorkerW" returned null.
    So I called FindAll(...) on the root element, with Condition.TrueCondition, build a list of all the direct children's ClassName. Surprisingly, the ClassName of the ""pane that holds the desktop wasn't "WorkerW", but it was "SysListView32". I knew to find it according to its location between two other children I could recognize with confident in the inspect tool.

    So , I have few questions:

    1. Is there an explanation to this difference? Im trying to find the most reliable way to perform this task, so this is important...

    2. Is there any difference in the API features between C++ and C#?  Anything I should take into consideration?

    3. What is the difference between the "Cache" methids and the regular ones. Are they better for performance? How do they behave differently?

    4. Is there a ready to use project, or an API, that can record the user's interaction with the GUI elements?

    Thanks,
    Lior

    Here is a printscreen of inspect and the list I built:

    


    lior

    Wednesday, August 29, 2012 1:49 PM
  • I mistakenly posted the printscreen when the inspect tool shows the "ToolBar" pane and not the "Desktop" pane. So you cant see there that the classname shown by the inspect tool is "WorkerW".

    Here is a new one:


    lior

    Wednesday, August 29, 2012 1:53 PM
  • I further investigated this.

    The inspect tool shows that the root element has:

    a child named ""pane, with ClassName "WorkerW"
    a grandchild named ""pane with ClassName "SHELLDLL_DefView"
    and ggrandchild named ""list with ClassName "SysListView32"

    the last one is the parent of all the desktop icons.

    When I try to implement this path with code, it seems that the two middle nodes do not exist, and the ggrandchild who's the parent of all desktop icons is a direct child of the root element. 
    So what I did currently, I wrote the following code to retrieve the desktop icons list:


                AutomationElement aeDesktop = AutomationElement.RootElement;
                AutomationElement elemTemp;
                elemTemp = aeDesktop.FindFirst(TreeScope.Children, new PropertyCondition(AutomationElement.ClassNameProperty, "SysListView32"));
    
                //get collection of all the elemnts of icons on the taskbar
                AutomationElementCollection colect = elemTemp.FindAll(TreeScope.Children, Condition.TrueCondition);
                IEnumerator enumer = colect.GetEnumerator();
                String tmpStr, Str;
                Str="List Of Taskbar Icons:\n";
                while (enumer.MoveNext())
                {
                    elemTemp = (AutomationElement)(enumer.Current);
                    tmpStr = (String)(elemTemp.GetCurrentPropertyValue(AutomationElement.NameProperty));
                    Str += tmpStr + "\n" ;
                }
                MessageBox.Show(Str);

    So what is going on here?...

    Thanks, 

    Lior


    lior

    Wednesday, August 29, 2012 2:10 PM
  • Hi lior,

    If you’re using C# to call UIA, the first thing to consider is whether you’re using the managed UIA API that’s included in the .NET framework, or the native-code UIA API that’s included in Windows. The managed UIA API can be very useful for many scenarios, but there are some situations where it can’t access all the data that’s accessible through the native-code UIA API. I only use the native-code UIA API. The Inspect SDK tool also uses the native-code UIA API. If you’re currently using the managed code UIA API, I’d suggest moving to the native-code UIA API, and see if the results are different. (That might not help, but I do think it’s best to be using the same version of the API that Inspect uses if you’re specifically looking for the same elements that are reported in Inspect.)

    There are a couple of ways you can call the native-code UIA API from C#, and I’ve discussed these at http://social.msdn.microsoft.com/Forums/en-US/windowsaccessibilityandautomation/thread/c3f142e1-0624-4ec5-a313-482e72d5454d.

    Regarding the “cache” functionality in UIA, this is all about performance. In my sample up at http://code.msdn.microsoft.com/Windows-7-UI-Automation-9131f729, my client app retrieves many hyperlink elements from a browser, and it also wants the name and bounding rects of the hyperlinks. It could find the hyperlinks first, and then for each one, go back to the browser and ask for the name and then the bounding rect. If there hundreds of hyperlinks, then that approach would result in hundreds of cross-process calls. Cross-process calls are slow, and so this approach would take a long time. By using a cache, a client can tell UIA to get all the hyperlinks, all the names and all the bounding rects in a single cross-process call. This means the action might take a fraction of a second instead of many seconds. My sample shows code for using a cache and also taking the same action without a cache, eg…

    if (fUseCache)
    {
        hr = pElementBrowser->FindAllBuildCache(TreeScope_Descendants, pCondition, pCacheRequest, &_pElementArray);
    }
    else
    {
        hr = pElementBrowser->FindAll(TreeScope_Descendants, pCondition, &_pElementArray);
    }

    So for a few simple operations, you might feel it’s acceptable to not use a cache. But if you’re getting multiple elements and multiple properties, then using a cache can keep the performance great for users of your client app.

    I’ve not used tools for recording the user’s actions. This is a separate topic from UIA, as UIA is a framework for making UI data accessible between applications rather than tracking user action. But maybe the Coded UI Test feature in Visual Studio could be useful to you here.

    Thanks,

    Guy

    Friday, August 31, 2012 2:08 PM