locked
could image icon on win32 tree control item node be identified by UI automation? RRS feed

  • Question

  • Hi all,

    I notice that in http://msdn.microsoft.com/en-us/library/ms743384.aspx, it says that TreeItem could have zero or more child node such as image or checkbox.For standard win32 CTreeCtrl, we could set imagelist so that each treeitem could have a image icon.But now i could not find a child image node for each treeitem by using UISpy control view.

    So i wonder that could such image icon be identified by UI automation automatically? if not, what is situation fitting for the link says that treeitem could have image or checkbox child node? 

    If i want to find such a image child for CTreeControl,  could it possible to create a provider for it?

    is there anyone could give some help? thank you!

    alan
    • Edited by pipibility Wednesday, June 3, 2009 2:34 AM
    Wednesday, June 3, 2009 2:10 AM

Answers

  • Interesting question, pipibility. 

    In the Microsoft-supplied implementation of UI Automation for tree controls, we chose not to expose the images for a non-checkbox tree view.  In most cases, the images are decoration -- they do not supply additional information, and so would just be confusing to, say, a screen reader.  The screen reader cannot tell the customer what the image is.  The Accessibility guideline is that decorations like treeview images should not carry any information that is not available in another way, such as through the name of the tree node.

    For a checkbox tree view, the image means either checked or unchecked, and so is exposed through the toggle pattern.

    You asked if it were possible to create a provider for this.  If you can change the code that contains the tree view, you can certainly override the provider.  Our MSDN documentation explains how to do this.  However, I think you'll still run into the same problem: there is no way to expose the image in UIA in a way that provides any useful information. 

    If you are trying to write test automation, it might be better to use UIA to locate the tree item's rectangle on screen, then take a snap shot of that rectangle and compare it to a bitmap for what you think the image on the tree item should be.

    - Michael


    This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by pipibility Thursday, June 4, 2009 12:35 AM
    Wednesday, June 3, 2009 5:45 PM

All replies

  • Interesting question, pipibility. 

    In the Microsoft-supplied implementation of UI Automation for tree controls, we chose not to expose the images for a non-checkbox tree view.  In most cases, the images are decoration -- they do not supply additional information, and so would just be confusing to, say, a screen reader.  The screen reader cannot tell the customer what the image is.  The Accessibility guideline is that decorations like treeview images should not carry any information that is not available in another way, such as through the name of the tree node.

    For a checkbox tree view, the image means either checked or unchecked, and so is exposed through the toggle pattern.

    You asked if it were possible to create a provider for this.  If you can change the code that contains the tree view, you can certainly override the provider.  Our MSDN documentation explains how to do this.  However, I think you'll still run into the same problem: there is no way to expose the image in UIA in a way that provides any useful information. 

    If you are trying to write test automation, it might be better to use UIA to locate the tree item's rectangle on screen, then take a snap shot of that rectangle and compare it to a bitmap for what you think the image on the tree item should be.

    - Michael


    This posting is provided "AS IS" with no warranties, and confers no rights.
    • Marked as answer by pipibility Thursday, June 4, 2009 12:35 AM
    Wednesday, June 3, 2009 5:45 PM
  • Thank you, Michael.


    So it seems that i have to overwrite the tree view provider in order to show the rectangle for the image.Because i have a customize tree control which is derived from CTreeCtrl and it has some operations on the image.
    Thursday, June 4, 2009 12:35 AM
  • Yes, I think that's right.  If you supply your own tree view provider, you can represent the image as a separate node.
    This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, June 4, 2009 3:49 AM
  • If it is a win32 (or vb6, c++) tree control this actually isn't too complicated depending on what you really want.  If you need this for .Net, you are better off making a server side provider.

    Michael is right you probably need a client side provider, but all win32 controls have a rich API so all you need to do is dig around to figure out the magic functions.  (We rewrote many of the default UIAutomation providers due to lack of functionality, so I know where you are coming from).

    The secret to the game is that the treeview in win32 world doesn't have a pointer to the image displayed by tree items.  Instead it has the offset index from an image array.  9 times out of 10 all you really want to know is which icon is in use, so this works great for you since the system can tell you which is which without even doing image comparison!

    *If you really want the image, then it is vastly easier to use win32 to take a screenshot of the window handle at the position of where the image should be relative to the bounding rectangle UIAutomation will give you.

    The best place to start is by using spy++ to get the win32 class handle for the actual control, so you know for sure what you are interacting with and to get the criteria for how your provider will take ownership.  If you need this for a derivative of TreeView, then the magic definitions you need are:

       Public Enum WinMessages As Int32
            TV_FIRST = &H1100
            TVM_GETNEXTITEM = (TV_FIRST + 10)
            TVM_EXPAND = (TV_FIRST + 2)
            TVM_GETITEMRECT = (TV_FIRST + 4)
            TVM_GETCOUNT = (TV_FIRST + 5)
            TVM_GETINDENT = (TV_FIRST + 6)
            TVM_SETINDENT = (TV_FIRST + 7)
            TVM_SELECTITEM = (TV_FIRST + 11)
            TVM_GETITEM = (TV_FIRST + 12)
            TVM_SETITEM = (TV_FIRST + 13)
            TVM_EDITLABEL = (TV_FIRST + 14)
            TVM_ENSUREVISIBLE = (TV_FIRST + 20)
            TVM_GETITEMSTATE = (TV_FIRST + 39)
       End Enum
    
       Public Enum TreeViewItemState As Int32
            TVIS_FOCUSED = &H1
            TVIS_SELECTED = &H2
            TVIS_CUT = &H4
            TVIS_DROPHILITED = &H8
            TVIS_BOLD = &H10
            TVIS_EXPANDED = &H20
            TVIS_EXPANDEDONCE = &H40
            TVIS_OVERLAYMASK = &HF00
            TVIS_STATEIMAGEMASK = &HF000
            TVIS_USERMASK = &HF000
        End Enum
    
        Public Enum TreeViewItemMask As Int32
            TVIF_TEXT = &H1
            TVIF_IMAGE = &H2
            TVIF_STATE = &H8
            TVIF_SELECTEDIMAGE = &H20
            TVIF_EXPANDEDIMAGE = &H200
        End Enum
    
        Public Enum TreeViewElement As Int32
            TVE_COLLAPSE = &H1
            TVE_EXPAND = &H2
            TVE_TOGGLE = &H3
        End Enum
    
        Public Enum TreeViewNavigation As Int32
            TVGN_ROOT = &H0
            TVGN_NEXT = &H1
            TVGN_PREVIOUS = &H2
            TVGN_PARENT = &H3
            TVGN_CHILD = &H4
            TVGN_FIRSTVISIBLE = &H5
            TVGN_NEXTVISIBLE = &H6
            TVGN_PREVIOUSVISIBLE = &H7
            TVGN_DROPHILITE = &H8
            TVGN_CARET = &H9
        End Enum
    
        <StructLayout(LayoutKind.Sequential)> Public Structure TVITEM
            Public mask As UInt32
            Public hItem As IntPtr
            Public state As UInt32
            Public stateMask As UInt32
            Public pszText As IntPtr
            Public cchTextMax As Int32
            Public iImage As Int32
            Public iSelectedImage As Int32
            Public cChildren As Int32
            Public lParam As Int32
        End Structure

    Just keep in mind two things.  While running with administrator rights, you will need to allocate memory on the other process for the structure objects you get back from send message.  Calling send message with your own process's memory won't work because the TreeView control won't have the ability to access it!  The second thing to keep in mind is that you need to set the TVMask correctly to include everything you want on your call or the TreeView won't populate the TVItem structure.

    Saturday, June 20, 2009 1:46 AM
  • Hi Starrider1,

    Am working to get the rectangle of Treenode. I am able to get Handle for treeView and my Treenode from different process but not rectangle. 
    I also tried allocating memory on the other process but result was not as expected. 

    Below is the way I tried. Can you please help me in getting treenode bounding rectangle.  

          [DllImport("kernel32.dll", EntryPoint = "VirtualAllocEx")]
            private static extern IntPtr VirtualAllocEx(IntPtr hProcess, IntPtr lpAddress, int dwSize, uint flAllocationType, uint flProtect);
            [DllImport("kernel32.dll", EntryPoint = "OpenProcess")]
            private static extern IntPtr OpenProcess(uint dwDesiredAccess, [MarshalAs(UnmanagedType.Bool)] bool bInheritHandle, int dwProcessId);
            [DllImport("user32.dll", EntryPoint = "GetWindowThreadProcessId")]
            private static extern int GetWindowThreadProcessId(IntPtr hWnd, ref int lpdwProcessId);
    
     [DllImport("kernel32.dll", EntryPoint = "WriteProcessMemory")]
            [return: MarshalAs(UnmanagedType.Bool)]
            private static extern bool WriteProcessMemory(IntPtr hProcess, IntPtr lpBaseAddress, ref RECT lpBuffer, int nSize, ref int lpNumberOfBytesWritten);
    
    
            [DllImport("user32.dll", EntryPoint = "SendMessageW")]
            private static extern IntPtr SendMessageW(IntPtr hWnd, uint Msg, int wParam, IntPtr lParam);
    
        [DllImport("kernel32.dll", EntryPoint = "ReadProcessMemory")]
            [return: MarshalAs(UnmanagedType.Bool)]
            private static extern bool ReadProcessMemory(IntPtr hProcess, IntPtr lpBaseAddress, ref RECT lpBuffer, int nSize, ref int lpNumberOfBytesRead);
    
      [DllImportAttribute("kernel32.dll", EntryPoint = "VirtualFreeEx")]
            [return: MarshalAs(UnmanagedType.Bool)]
            private static extern bool VirtualFreeEx(IntPtr hProcess, IntPtr lpAddress, uint dwSize, uint dwFreeType);
    
    //--------------------------------------------//
    bool success = false;
                IntPtr hWnd = TreeNodeHandle;
                RECT rect = new RECT();
    
    
                int procId = 0;
                GetWindowThreadProcessId(treeViewHandle, ref procId);
                IntPtr hProc = OpenProcess(PROCESS_VM_OPERATION | PROCESS_VM_READ | PROCESS_VM_WRITE, false, procId);
    
                RECT nRect = new RECT();
                nRect.left = TreeNodeHandle.ToInt32();
    
                IntPtr tvNodeRectPtr = VirtualAllocEx(hProc, IntPtr.Zero, Marshal.SizeOf(nRect), MEM_COMMIT, PAGE_READWRITE);
                int testc = 0;
                WriteProcessMemory(hProc, tvNodeRectPtr, ref nRect, Marshal.SizeOf(nRect), ref testc);
    
                SendMessageW(treeViewHandle, TVM_GETITEMRECT, 1, tvNodeRectPtr);
                int abcd = 0;
                ReadProcessMemory(hProc, tvNodeRectPtr, ref nRect, Marshal.SizeOf(nRect), ref abcd);
    
                VirtualFreeEx(hProc, tvNodeRectPtr, 0, MEM_RELEASE);
              
    
                RECT tvRect = new RECT();
                GetWindowRect(treeViewHandle, ref tvRect);
    You help would be highly appreciable. 


    Monday, August 22, 2016 5:34 AM