locked
Need to control test class execution RRS feed

  • Question

  • We are using TFS automated test agent to run our tests.  We have multiple test classes that contain multiple test methods.  We are using .vsmdi list files to run our tests.  One thing we have noticed is that the TestMethods are randomly chosen to run between each test class by the Test Agent.  This is causing a problem with our tests that have [ClassInitialize] attributes.  The [ClassInitialize] is only run once when the first test in the class is run, however if the Test Agent then selects a method from a different class to run and then returns to a TestMethod in the first test class, the [ClassInitialize] is not run because it was already done for the class instance.

    Is there any way that we can configure our tests so that when a TestMethod in a class is run that the test agent does not jump to another test class until all the TestMethods in the class have been executed?  I don't really care about the order in which the test methods are run, but I want all of the methods in a class to run before the test agent moves on to the next test class.

    Friday, March 27, 2015 3:14 PM

Answers

  • Hi Ken Varn,

    As far as I know, it is hard for us to really control the test methods' execution between different class no matter locally or remotely.

    Generally we use the ordered test to control the order.

    Some features like the Test Category or Test playlist, they are used for group tests, but I think it is not the solution you want to get.

    Maybe you could submit a user voice here:

    http://visualstudio.uservoice.com/forums/121579-visual-studio.
    The Visual Studio product team is listening to user voice there. You can send your idea there and people can vote.

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Monday, March 30, 2015 8:42 AM

All replies

  • Hi Ken Varn,

    As far as I know, it is hard for us to really control the test methods' execution between different class no matter locally or remotely.

    Generally we use the ordered test to control the order.

    Some features like the Test Category or Test playlist, they are used for group tests, but I think it is not the solution you want to get.

    Maybe you could submit a user voice here:

    http://visualstudio.uservoice.com/forums/121579-visual-studio.
    The Visual Studio product team is listening to user voice there. You can send your idea there and people can vote.

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Monday, March 30, 2015 8:42 AM
  • For various reasons, we have avoided using the Ordered Test lists for running our tests.  The VS 2012 Ordered Test editor is very cumbersome to use and does not provide as good of a user interface in comparison to the legacy test lists (.vsmdi).  We have well over 500 test cases spread around 20 test classes.  It is very difficult to manage all of those test cases with the Ordered Test editor. 

    Besides, we like the randomization of the tests, but we also need to make sure that each test class is in a known state before the test methods in that class are executed.  Since each of our test classes has their own set of known states that may interfere with other test class states, we need to make sure that the test class is initialized properly.  Our initialization is time consuming, so we don't want to do that in the [TestInitialize] section because this gets run before every [TestMethod].  This is why we prefer the randomization to complete all tests within each test class respectively in random order. 


    • Edited by Ken Varn Monday, March 30, 2015 12:43 PM
    Monday, March 30, 2015 12:41 PM
  • Hi Ken Varn,

    Thanks for your friendly response.

    Like my previous suggestions, if ordered test is not the way you want to get, maybe you could think about the Test Category or Test playlist(VS2013), so it could help you group your tests for different classes. For example, when you run your tests, you could run the specific category or playlist.Of course, I know it is just a workaround for you.

    Please submit a feature request here: http://visualstudio.uservoice.com/forums/121579-visual-studio. If you submit it, please share me the link here, I will help you vote it.

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, March 31, 2015 2:25 AM
  • That would be great if MSTest supported Test Playlists, but that does not appear to be the case.
    Tuesday, March 31, 2015 4:20 PM
  • That would be great if MSTest supported Test Playlists, but that does not appear to be the case.

    @ Ken,

    Playlist feature is in the VS IDE, I mean that we could use it in test explorer window, but it doesn't support the command line(mstest). Test List has been removed in VS2013/VS2012. 

    But for the previous main issue, I'm afraid that it has a limitation for it now.

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, April 1, 2015 2:40 AM