locked
MSTEST - How to stop test execution on first test failure? RRS feed

  • Question

  • Hi,

    I have created number of unit tests using SpecFlow + MSTEST and they are assigned to the same category, say "SMOKETEST". I use MSTEST to run all "SMOKETEST" test cases. How do I stop test execution on first test failure when they were run overnight unattended?


    Thanks,
    Steven

    Monday, January 5, 2015 8:54 PM

Answers

  • Hi Jack,

    I have tests that fail when executed with others but pass individually. The reason I'm introducing StopOnFailure because I want to debug a test at the state of environment when it failed. Continue executing remaining test cases changes the environment makes it hard to debug.

    Your recommendation to stop certain process, such as MSTEST.exe, may work but I went ahead and did a similar approach.

    My approach may not be perfect but suits my need.

    What I did was 
    1) Create a static boolean variable defaults to true. Set this value to false as soon as 1 test fails
    2) Create a global [BeforeScenario] or [BeforeFeature] method and checks this static boolean variable value. If false, do Assert.Fail. As soon as 1 test fails, all remaining tests will fail at [BeforeScenario] or [BeforeFeature] method before the scenario gets executed. 
    3) I have also introduced another static variable that turns this "StopOnFailure" feature on and off. Its value can be set via config file (e.g. App.config). This way, you could run all tests over night uninterrupted, or stops on failure when you want to troubleshoot. 

    Hope others may find this useful.

    • Marked as answer by xiaohundan Tuesday, January 6, 2015 10:14 PM
    Tuesday, January 6, 2015 9:00 PM

All replies

  • Hi Steven,

    We often use the Assert.Fail method in our tests to stop one test.

    https://social.msdn.microsoft.com/Forums/en-US/f7e89111-5cdc-440d-84c8-960c34034b85/how-to-make-a-coded-ui-test-be-set-to-failed-when-an-exception-is-caught-using-trycatch-blocks?forum=vsautotest

    But as far as I know, it would still run other tests in the same category.

    Not very sure that how you run your tests overnight, but one idea is that maybe you could use the method "[TestCleanup]", since every tests finished, they will call the "[TestCleanup]" method, so maybe you could stop certain process which are related to the test tool coercive if one test failed, so other tests couldn't run without this process/tool. Just one idea, maybe it could provide a path for you. 

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, January 6, 2015 11:21 AM
  • Hi Jack,

    I have tests that fail when executed with others but pass individually. The reason I'm introducing StopOnFailure because I want to debug a test at the state of environment when it failed. Continue executing remaining test cases changes the environment makes it hard to debug.

    Your recommendation to stop certain process, such as MSTEST.exe, may work but I went ahead and did a similar approach.

    My approach may not be perfect but suits my need.

    What I did was 
    1) Create a static boolean variable defaults to true. Set this value to false as soon as 1 test fails
    2) Create a global [BeforeScenario] or [BeforeFeature] method and checks this static boolean variable value. If false, do Assert.Fail. As soon as 1 test fails, all remaining tests will fail at [BeforeScenario] or [BeforeFeature] method before the scenario gets executed. 
    3) I have also introduced another static variable that turns this "StopOnFailure" feature on and off. Its value can be set via config file (e.g. App.config). This way, you could run all tests over night uninterrupted, or stops on failure when you want to troubleshoot. 

    Hope others may find this useful.

    • Marked as answer by xiaohundan Tuesday, January 6, 2015 10:14 PM
    Tuesday, January 6, 2015 9:00 PM
  • Glad to know that you resolved it.

    Have a nice day,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, January 7, 2015 1:42 AM