locked
How can I change the order of the execution of tests in an automated test build? RRS feed

  • Question

  • I have a build which executes the test Coded UI tests. When I run the build, I've noticed that the tests are executed in the alphabetical order of the name of the test methods.

    I tried setting the order in MTM but later came across this which states

    Please note this only applies to the manual tests but not automated tests. For automated tests, the order you set here will not be respected during test execution

    Is there any way that I could specify the order in which the tests are to be executed?

    • Moved by Oscar_Wu Wednesday, October 5, 2016 3:01 AM related to TFS
    Tuesday, October 4, 2016 1:15 PM

All replies

  • Hi asifrr,

    This forum is discussing and asking questions about using the new Visual Studio Test Explorer, Unit Testing providers and general Unit Testing, fakes, stubs and mocking techniques.

    Your issue may be related to TFS, I will move this thread to corresponding forum for a professional answer.

    Sincerely,

    Oscar


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place. Click HERE to participate the survey.

    Wednesday, October 5, 2016 2:58 AM
  • Thanks, Oscar
    Wednesday, October 5, 2016 6:01 AM
  • which version of TFS/VS are you using?

    take a look at ordered test https://msdn.microsoft.com/en-us/library/ms182631.aspx

    Wednesday, October 5, 2016 6:09 AM
  • I'm using VS 2015 but I'm not sure of the version of TFS as it's set up on the client's side.

    I did try an ordered test but I couldn't figure out how to get it to execute as part a build execution. Do I have to create a separate test case in TFS for that and link the two? 

    Wednesday, October 5, 2016 8:30 AM
  • Hi,

    First add a ordered test file in your project to define the order all test method. Secondly, in the build definition, for the Test Assembly parameter, change " **\*test*.dll; " to " **\*test*.orderedtest; ". Then TFS will find the ordered test file and run it.  


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, October 6, 2016 1:59 AM
    Moderator
  • Hi,

    I can't seem to find the option to specify the test assembly file specification. Let me try to give you an idea of how the build has been set up at the moment.

    The solution I'm working on is a solution with a Coded UI test project which tests a WPF application. I'm using a coded approach rather than recorded test via the test builder. For this, a gated check-in build was created using a build process template that was is followed by all teams. The following are the build process parameters:


    There is another build which is triggered at a schedule time. This build runs the test and follows a different build process template. The following are the build process parameters:

    The following are the lab process settings:

    In the Build section of the Lab Process Settings, the gated check-in build is selected as the build definition.

    This set up has been followed since before I joined. If there's a better approach to do this, I'm open to suggestions as well. My priority for now though is to get the test to execute in an order that I specify rather than being executed in alphabetical order of the name of the test project.

    Thursday, October 6, 2016 8:20 AM
  • Hi,

    It seems that you're using your custom build template not the default build process template. And from the picture you posted above, there's no step about test. The default template is like the picture below. And I think you'd better refer to the default template to add the test step to your build template.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, October 7, 2016 2:28 AM
    Moderator
  • Hi,

    Should step be included in the test run build? Or the gated check in build?

    Friday, October 7, 2016 4:50 AM
  • Hi,

    Should step be included in the test run build? Or the gated check in build?

    What does this mean?

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, October 7, 2016 7:30 AM
    Moderator
  • Looks like you are using lab default template and refer test suites for running the test cases.

    Can you try creating a new test suite in the test plan that you use using MTM and add the test cases one by one that you want to run, in the order you wanted it to be run?

    Edit the lab process template - Test - > Select test suites -> add the new suite and kick off the build


    • Edited by nmathew Friday, October 7, 2016 12:58 PM
    Friday, October 7, 2016 12:57 PM
  • Hi,

    Should step be included in the test run build? Or the gated check in build?

    What does this mean?

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Previously you said, "I think you'd better refer to the default template to add the test step to your build template."

    Since I have two builds (one being a gated check-in for the code and the other to run tests on a scheduled basis), I was wondering which of these two build should I add the test step to.

    Monday, October 10, 2016 5:22 AM
  • Looks like you are using lab default template and refer test suites for running the test cases.

    Can you try creating a new test suite in the test plan that you use using MTM and add the test cases one by one that you want to run, in the order you wanted it to be run?

    Edit the lab process template - Test - > Select test suites -> add the new suite and kick off the build


    Hi nmathew,

    This is how I've been doing it so far. I have a different test plan for the automated tests. Once I script a certain test case, I add this test case to the automation test plan.

    Monday, October 10, 2016 5:25 AM
  • I understand. I've encountered the same problem couple of months back. I found the order of test cases you define doesn't take effect in a test suite for automated tests. It's the order how you add the test case in a test suite is honored and not necessarily in test plan. Anyway good luck!
    • Proposed as answer by Tingting MOModerator Tuesday, October 11, 2016 3:06 AM
    • Unproposed as answer by asifrr Tuesday, October 11, 2016 9:13 AM
    Monday, October 10, 2016 12:10 PM