locked
How to use Vnext build: ordered tests, distribute test, collect results RRS feed

  • Question

  • Hello,

    Please help me sovle this issue if possible. Firstly, I'd like to show you my scenario:

    - I had a server machine with TFS 2015 on-premises

    - I'd configured a simple Vnext build to run tests with a .dll file

    My concerns are:

    1. How can I use Vnext build to execute the tests in that .dll file in ordered? Because we're following XAML build and MTM can trigger tests in ordered (associated coded UI tests). But I could not find any solution to do that with Vnext.

    2. Within a Vnext build definition, can I distribute test case 1 on machine A and test case 1 (the same) on machine B in parallel mode? For example, I want to test function login on both Internet Explorer and Firefox at a same time with only one test case script.

    3. Does Vnext support users to publish results back to MTM (associated coded UI tests)? After running a Visual Test, I'd just opened the results from TFS web page. I could not find them on MTM as TFS 2013 does with XAML build. Because I've designed test cases on MTM and I want to track the overall results from MTM.

    Thanks and best regards,

    Tuc Nguyen,




    • Edited by Tuc Nguyen Tuesday, December 22, 2015 9:15 AM
    Monday, December 21, 2015 4:19 AM

Answers

  • Hi Tuc,

    Thank you for posting here.

    About run tests in order in Vnext build , you could add an Ordered Test file in your test project and run it in Vnext Build.
    1.Add an Order Test file in your test project and use it to define the testing order.
    2.In your build template, add  an Visual Studio Test. Change the Test Assembly like the picture below.

    About run test on multiple machines, you could use Visual Studio Test Agent Deployment step to deploy tests on different machine at the same time.
    1.In TFS web access, click TEST--Machines, click the plus button.
    2.Add the machines you want to run tests in one group.

    In VNext build, it doesn’t support publish results back to MTM. You could submit this feature request: http://visualstudio.uservoice.com/forums/121579-visual-studio. The Visual Studio product team is listening to user voice there.

    • Proposed as answer by John QiaoModerator Thursday, December 24, 2015 2:01 AM
    • Marked as answer by Tuc Nguyen Tuesday, December 29, 2015 9:44 AM
    Wednesday, December 23, 2015 2:23 AM
    Moderator
  • Hi Tuc,

    Thank you for coming back.

    #1. The .orderedtest file can not combine with “Test Filter criteria” just as you said.

    #2. Here is an article about your requirement: https://msdn.microsoft.com/en-us/library/mt270062.aspx?f=255&MSPPError=-2147217396.

    But I found it can't run tests on both machines even though machine1 and machine2 were added into one test machine group. If you have tried the method too and find the same issue, you could submit this feedback to Microsoft Connect feedback portal:  https://connect.microsoft.com/VisualStudio. Microsoft engineers will evaluate them seriously. Thank you for your understanding.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place. <br/> Click <a href="http://support.microsoft.com/common/survey.aspx?showpage=1&scid=sw%3Ben%3B3559&theme=tech"> HERE</a> to participate the survey.



    Friday, December 25, 2015 2:31 AM
    Moderator

All replies

  • Hi Tuc,

    Thank you for posting here.

    About run tests in order in Vnext build , you could add an Ordered Test file in your test project and run it in Vnext Build.
    1.Add an Order Test file in your test project and use it to define the testing order.
    2.In your build template, add  an Visual Studio Test. Change the Test Assembly like the picture below.

    About run test on multiple machines, you could use Visual Studio Test Agent Deployment step to deploy tests on different machine at the same time.
    1.In TFS web access, click TEST--Machines, click the plus button.
    2.Add the machines you want to run tests in one group.

    In VNext build, it doesn’t support publish results back to MTM. You could submit this feature request: http://visualstudio.uservoice.com/forums/121579-visual-studio. The Visual Studio product team is listening to user voice there.

    • Proposed as answer by John QiaoModerator Thursday, December 24, 2015 2:01 AM
    • Marked as answer by Tuc Nguyen Tuesday, December 29, 2015 9:44 AM
    Wednesday, December 23, 2015 2:23 AM
    Moderator
  • Hi Tingting,

    Thank for your support. But I'm still stucked with issue #2:

    #1. I can now trigger a VS test on Vnext build to call .orderedtest file as you said. Just for your information, .orderedtest file cannot combine with "Test Filter criteria" and I think it is not recommanded.

    #2. I'd tried your solution before but it didn't work because: "We do distribution on a per test container basis ie if you have 2 machines and 2 test dlls we will execute all tests from testdll1 on machine1 and the rest from testdll2 on machine2." (http://blogs.msdn.com/b/allendm/archive/2015/07/20/remote-test-execution-using-team-foundation-server-2015-rc2-and-beyond.aspx)

    So is there a way to call just only one .dll or .orderedtest file and distribute to multiple machines? Or is it possible to call VS test (build step) many times in parallel mode?

    #3. I'll post my suggestion there.

    Merry Christmas,

    Tuc Nguyen

    Thursday, December 24, 2015 9:00 AM
  • Hi Tuc,

    Thank you for coming back.

    #1. The .orderedtest file can not combine with “Test Filter criteria” just as you said.

    #2. Here is an article about your requirement: https://msdn.microsoft.com/en-us/library/mt270062.aspx?f=255&MSPPError=-2147217396.

    But I found it can't run tests on both machines even though machine1 and machine2 were added into one test machine group. If you have tried the method too and find the same issue, you could submit this feedback to Microsoft Connect feedback portal:  https://connect.microsoft.com/VisualStudio. Microsoft engineers will evaluate them seriously. Thank you for your understanding.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place. <br/> Click <a href="http://support.microsoft.com/common/survey.aspx?showpage=1&scid=sw%3Ben%3B3559&theme=tech"> HERE</a> to participate the survey.



    Friday, December 25, 2015 2:31 AM
    Moderator
  • Hi @Tuc, 

    i am not sure if you already find it out , but i do it in this way, where the Test Drop Location its my Complete Project and in side i have a folder called Executions with all the Ordered Test. Hope it helps. Best regards

    Tuesday, July 5, 2016 8:59 AM