none
MSTest Executes Test Cases Parallely??? RRS feed

  • General discussion

  • Hello everyone,

    I heard that MStest does parallel test case execution using multithreading concepts in contrast to NUnit's sequential execution.But I dont feel the parallel execution of test cases when i observed the test case execution progress since only one test case is in progress at any point of time and all others are in pending state. Hence the below questions

    1. If we execute test cases from visual studio test view or test list editor , will MStest bydefault exceute them sequentially(one after another)?
    2. If the default execution behavior is sequential, what is so special about ordered test except specific ordering within the test cases?
    3. Is it the same behavior with MSTest command line mode of test case execution?
    4. How to run test cases parallely if at all MSTest supports?

    Appreciate your comments as soon as possible

    Thanks & Regards
    Samba

    Friday, June 1, 2012 10:45 AM

All replies

  • If we execute test cases from visual studio test view or test list editor , will MStest bydefault exceute them sequentially(one after another)?

          Yes they will be executed serially, not necessarily in required sequence.

    If the default execution behavior is sequential, what is so special about ordered test except specific ordering within the test cases?

          Capability to execute tests in predefined sequence.

    Is it the same behavior with MSTest command line mode of test case execution?

          Yes.

    How to run test cases parallely if at all MSTest supports?

          See  Executing Unit Tests in parallel on a multi-CPU/core.

    -Tejas.

    Friday, June 1, 2012 11:06 AM
  • Hell Tejas,

    Thanks for your quick response.

    Iam able to verify parallel test case execution on a dummy test project.It works perfectly based on the parallelTestCount value set in the Local.testsettings file.

     I updated parallelTestCount value  in TraceAndTestImpact.testsettings file instead of Local.testsettings file of the dummy project.Interestingly , test case execution became sequential.

    What is the difference between Local.testsettings and TraceAndTestImpact.testsettings , though both of them are providing similar UI and features?

    I am actually working on database unit test project where unit test cases are written for procedures and functions.I didn't find Local.testsettings in my solution Items folder.So I enabled parallel test execution by editing TraceAndTestImpact.testsettings file, I couldn't experience the parallel execution of test cases instead they are executed sequentially.

    1.Is parallel test execution supported for database unit test cases?

    2.Why Local.testsettings file not available for database unit test solution?

    3.Does it make sense to carry parallel test execution for database business logic?(due to parallel read/writes to database)

    Thanks for your focus on this issue to help me

    Regards

    Samba

    Tuesday, June 5, 2012 8:51 AM
  • Hi,

    I haven't used database unit tests.

    Testsettings file contains various attributes which are used during test execution(roles/dignostics data adapters etc). You can have multiple testsettings file in a solution having different values of these atributes so that you just have select the required one when you execute tests and need not changes these attributes repeatedly.

    What is the difference between Local.testsettings and TraceAndTestImpact.testsettings , though both of them are providing similar UI and features?

    Since both files are same structure, the UI and features are same. The only difference is in the values of the attributes. You use 'Local' to execute tests locally and other to execute tests on remote machine(s).

    2.Why Local.testsettings file not available for database unit test solution?

    It should be available when you create Test Project. The testsettings files are common to solution. You can add an existing file to the solution or create a new one. See Create Test Settings to Run Automated Tests from Visual Studio and How to: Apply Test Settings from Microsoft Visual Studio.

    3.Does it make sense to carry parallel test execution for database business logic?(due to parallel read/writes to database)

    It makes a sense for testing business logic until there are not many deadlocks. You are not much concerned about performance and hence a little delay is okay.

    -Tejas.

    Wednesday, June 6, 2012 4:12 AM
  • Hello Tejas,

    Appreciate your answers to the above questions.

    1.Yes, Local.testsettings file is available for database unit test projects as well.In my case ,it was excluded from solution.

    2.I verified parallel executon of database unit tests by setting parallelTestCount value in Local.testsettings.

    Here Iam more worried about the question3.Here is the detailed information

    We created multiple test projects based on number of schemas available in database.Each test project consists of two test classes for procedures and functions respectively.We are using data generation plans for every schema and its being invoked in the ClassInitialize method of each test class.For every test method we have certain pretest conditions to ensure execution of business logic which means the pretest condition modifies the data present in tables.

    With this being the situation, how can parallel execution of db test cases give accurate results.Theoritically the tables might be undergoing simultaneous inserts/deletes/updates which may effect test case execution.

    Another important observation is the parallel execution is giving the expected results when i run the tests from test view and when i execute the tests from test list editor i am finding more number of failed tests.

    What is the difference between test view and test list editor and Test->Run->All Tests in Solution mode of test execution?

    The last question is really frustating as I dont have clue why test results are not consistant.

    Thanks & Regards

    Samba

    Wednesday, June 6, 2012 4:39 AM
  • Hi Samba,

    With this being the situation, how can parallel execution of db test cases give accurate results.Theoritically the tables might be undergoing simultaneous inserts/deletes/updates which may effect test case execution.

    If the test cases are not independent such that the changes in one can not affect another, you should go for sequential execution.

    What is the difference between test view and test list editor and Test->Run->All Tests in Solution mode of test execution?

    There is no difference in execution except the order of the tests. Check whether tests are executed in same order and still giving different results? (I don't think it would). Because the database conditions at the time of test case execution plays important role in your tests, I would suggest you to go for sequential execution of tests in a predefined order. You can use Ordered Test for this purpose. See Ordered Tests Overview. Ordered test contains multiple tests and gaurantees execution in defined order.

    -Tejas.


    • Edited by TejasJ Wednesday, June 6, 2012 7:47 AM
    Wednesday, June 6, 2012 7:46 AM
  • Hi Tejas,

    My observatons are

    Test View:Test cases are loaded and executed in alphabetical order.

    if schema names are A,B,C etc... the respective test methods are executed in the following order

    A.Method1()

    B.Method1()

    C.Method1()

    Test List Editor:

    There is no specific order of test method execution.

    Regards

    Samba

    Wednesday, June 6, 2012 8:08 AM