Invoke a button using UI Automation of Metro App

Question

Hi,

How do I invoke a "button click" event in a metro UI app using UI Automation library in Windows 8? Using UIAutomation library & ActivationManager Class, I could launch the metro app. However, because we could not debug the metro UI automation, I am not able to check if I have got the handles of the window (my app window) and a button in it. my requirement is launch a metro app, click on a button in it.

I understand that I need to get the root element with the app name and then find firstchild with the name "button name" and invoke it (click it). but how do I do that? Can anyone help me with this?

Kindly help.

Thanks,

Veena.

---

Cheers, Veena.

Answer 1

Hi, Could anyone please help me with this. I need to invoke a click event on a button on a metro app using UIAutomation.Thanks.

---

Cheers, Veena.

Answer 2

Hi Veena, the steps required to use UI Automation (UIA) to invoke a button shown in a Metro style app are similar to those for invoking a button in a desktop app.

The first thing I’d do is point the Inspect SDK tool at the UI you’re interested in. That will show you the accessible properties exposed by your app, including for the UIA element that represents the app itself, and also for the element representing your button. It should also show you that the button supports the UIA InvokePattern, which means it can be programmatically invoked. You’ll be using some of the accessible properties reported by Inspect, in your own client app which invokes the button.

Some of the things you might want to try are...

1. Get the UIA root element.

Note that CLSID_CUIAutomation8 is used here, (that was introduced in Windows 8).

IUIAutomation *pUIAutomation = NULL;
hr = CoCreateInstance(CLSID_CUIAutomation8, NULL, CLSCTX_INPROC_SERVER, IID_PPV_ARGS(&pUIAutomation));
if (SUCCEEDED(hr))
{
    IUIAutomationElement *pElementRoot = NULL;
    hr = pUIAutomation->GetRootElement(&pElementRoot);

2. Find the UIA element representing your app.

When you find elements, use the AutomationId property for the element if there is one. Otherwise you could use the Name property, but realize that that will probably be different on different languages.

VARIANT varProp;
varProp.vt = VT_BSTR;
varProp.bstrVal = SysAllocString(<The AutomationId as reported in Inspect>);

IUIAutomationCondition *pCondition; 
hr = pUIAutomation->CreatePropertyCondition(UIA_AutomationIdPropertyId, varProp, &pCondition);
if (SUCCEEDED(hr))
{
    hr = pElementRoot->FindFirst(TreeScope_Children, pCondition, &pElementApp);

3. Find the element representing your button.

Again, use the AutomationId if available, otherwise you could use the name.

Note: Use the TreeScope_Descendants scope here because you want to search the entire app for the button. Never use TreeScope_Descendants when searching beneath the root UIA element. (That’s why TreeScope_Children was used above.)

hr = pElementApp->FindFirst(TreeScope_Descendants, pCondition, &pElementButton);

4. Get the InvokePattern from the element representing the button, and invoke it.

IUIAutomationInvokePattern *patternInvoke = NULL;
hr = pElementButton->GetCurrentPatternAs(UIA_InvokePatternId, IID_PPV_ARGS(&patternInvoke));
if (SUCCEEDED(hr) && (patternInvoke != NULL))
{
    hr = patternInvoke->Invoke();

    patternInvoke->Release();
    patternInvoke = NULL;
}

Hopefully something like that should achieve what you need. My sample at http://code.msdn.microsoft.com/Windows-7-UI-Automation-9131f729 does some of the things above. But that sample works with a desktop app, and so finds the element representing the app from the hwnd. That app is C++. For an equivalent C# app, check out http://code.msdn.microsoft.com/Windows-7-UI-Automation-0625f55e. In fact I have a bunch of UIA samples listed at http://code.msdn.microsoft.com/site/search?f%5B0%5D.Type=User&f%5B0%5D.Value=Guy%20Barker%20MSFT. None of these access Metro style apps, but most of the UIA action related to interacting with elements is valid for Metro style apps.

By the way, all these samples were built using the Windows 7 UIA API, and so use CUIAutomation. UIA Client apps running on Windows 8 will want to use CUIAutomation8.

Thanks,

Guy

Answer 3

Hi Guy,

Thanks for the post:)

I could get the Root Element. But my problem has always been getting the control of the App window. I use the following code:

VARIANT varProp;
varProp.vt = VT_BSTR;
varProp.bstrVal = SysAllocString(L"CameraCaptureUI CS sample");    //I had to use the Name (as given by inspect.exe) because my app does not have an automation id.
hr = pAutomation->CreatePropertyCondition(UIA_AutomationIdPropertyId, varProp, &pCondition);   //This is successful

hr1 = pElementRoot->FindFirst(TreeScope_Children, pCondition, &pElementApp);   //Here, pElementApp is NULL though hr = S_OK.

No matter how I try, I do not seem to get the Metro App window:( Am I doing something wrong here?

I am writing a win32 console application. From what I understand, I think that my console application is not able to find the metro app window while enumerating the desktop. I could see the app is launched from my application. Is there something special I need to do here? Please let me know.

Thanks,

Veena.

---

Cheers, Veena.

Answer 4

Hi Veena, I don't know if this is the only reason why you're not getting the results you expect, but there is one thing I noticed...

You said that you used the Name property because the UI that you're trying to access doesn't expose an AutomationId. (Unfortunately a fair bit of UI doesn't expose an AutomationId.) But in the call to CreatePropertyCondition(), you're passing in UIA_AutomationIdPropertyId. So if you're actually using the code snippet above, you're trying to find an element whose UIA_AutomationIdPropertyId is "CameraCaptureUI CS sample". If so, try using UIA_NamePropertyId in the call to CreatePropertyCondition() instead.

Thanks,

Guy

Answer 5

try the  TreeScope_Subtree instead of TreeScope_Children.

Answer 6

Does anyone has sample code in C# to automate the button click for Windows 8 Modern UI using UI Automation? Somehow I am able to lauch the application but not getting the window handle using UIA. It would be great help if I can get a sample code in C#. Thanks.

Answer 7

Hey any one succeeded with this ? Am not able to get the window handle after launching the metro app . 

I am also able to launch the application through a Cpp code but not getting the window handle using UIA.

i.e. after launching the app using CPP code i would like to use UIA in C# . But i am not able to get the window handle or say control on the launched windows 08 (metro) app.

It would be great help if I can get a sample code in C#.

Thanks

Answer 8

Hi,

Did you get any further with this?

Anyone know how to do UI Automation testing with the Windows Phone 8 Emulator? 

I understand from the documentation that Coded UI Tests are not supported but not having the Premium or Ultimate versions I cannot try this?

Sure we can move most code to MvvM

Thanks

Answer 9

Hi Guy,

            Thanks for your post.But i am still unable to invoke a button using this.I have a few queries:

                 1.Will this procedure given by you work after launching the app by Activate Application method of IApplicationActivationManager?

                 2. varProp.bstrVal = SysAllocString(<The AutomationId as reported in Inspect>);   

                     Automation Id which we need to place here is the same Automation Id which i have used in launching the application?

                 3.I just want to invoke a button of my application so do i need to input button's Automation Id for invoking it?

I have tried everything but except launching the app i am unable to do anything.Can you please provide a complete/partial automation code to us for a sample application like "Hilo" (Sample provided by Microsoft - http://code.msdn.microsoft.com/windowsapps/Hilo-C-sample-b53fd433) ?

Anything you provide will be a great help for us.Please reply.

Thanks in Advance,

Shobhit Garg