locked
Multiple Runs of the same test case imapcts performance RRS feed

  • Question

  • Hi,

    I have set of 23 test cases that I execute using Mstest.

    The time taken to execute the test case is around 1 hr 20 mins.

    However if I need to run the 23 test cases for 5 iterations back to back then for each iteration the time duration differs.

    The last iteration would have taken around 2 hrs 50 mins to complete instead of 1 hr 20 mins.

    It has been observed that after 2 iterations if the machine is restarted then it takes 1 hr 20 mins for  the rest iterations .

    Since it is not possible to restart the machine during each iteration ,is there any better way to overcome this issue



    Thursday, April 24, 2014 1:36 PM

Answers

  • Hi Hajira,

    Is it possible that you have identified a problem in the system under test, namely; a memory leak?

    I mention this because you say that restarting the machine sets the duration back to the original value and this is a classic example of a memory leak performance issue.

    So despite this not being a performance test, it has appeared to uncover a performance problem which is always a win :)

    In which case it would make sense to log the problem as a bug with your team as normal. You could potentially overcome the issue for now by including code to terminate the application being tested after each iteration (and obviously, code to initate the application before each iteration). This is at least better than restarting the machine between iterations.

    Hope this helps,

    • Proposed as answer by Amanda Zhu Friday, April 25, 2014 3:28 AM
    • Marked as answer by Amanda Zhu Friday, May 2, 2014 10:10 AM
    Thursday, April 24, 2014 2:45 PM
  • Hi,

    Not sure about your test scenario so I can’t confirm this issue is not because of Coded UI Test itself.

    I only can give you some suggestions to check whether we can improve the performance of test playback.

    Please check the problematical controls’ Search Configurations in Coded UI Test Editor to make sure the AlwaysSearch option is not set. Using AlwaysSearch will make that UI test playback engine does not use a cached control to perform any search action.

    And please disable ‘Smart Match’ and ‘Wait For Ready Level’ in the test case to check the result. For detailed information, please see: Configuring Playback in Visual Studio 2010

    In addition, in order to check what caused the issue, you also can check the coded UI test log after test run. Please reference: Analyzing Coded UI Tests Using Coded UI Test Logs

    Best regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    • Proposed as answer by Amanda Zhu Friday, May 2, 2014 1:31 AM
    • Marked as answer by Amanda Zhu Friday, May 2, 2014 10:10 AM
    Monday, April 28, 2014 7:13 AM

All replies

  • Hi Hajira,

    Is it possible that you have identified a problem in the system under test, namely; a memory leak?

    I mention this because you say that restarting the machine sets the duration back to the original value and this is a classic example of a memory leak performance issue.

    So despite this not being a performance test, it has appeared to uncover a performance problem which is always a win :)

    In which case it would make sense to log the problem as a bug with your team as normal. You could potentially overcome the issue for now by including code to terminate the application being tested after each iteration (and obviously, code to initate the application before each iteration). This is at least better than restarting the machine between iterations.

    Hope this helps,

    • Proposed as answer by Amanda Zhu Friday, April 25, 2014 3:28 AM
    • Marked as answer by Amanda Zhu Friday, May 2, 2014 10:10 AM
    Thursday, April 24, 2014 2:45 PM
  • If this turns out to be memory leak issue ,I would be more than happy .

    But I just want to be double sure that this is not because of coded UI  in any way .

    Actually after each test case I am closing the application and restarting it again.

    My observation during test execution has been that sometimes even when the control is visible coded UI would just wait and wait for some time  before performing the actions and the same actions would be quite fast when machine has been restarted.

    For e.g :In case if the code has been written to wait for a WPF Window and to enter the password,even when the window is available coded ui still waits .It would take around 1 Min to enter the credentials and this starts happening  in the 3rd iteration of test case execution.

    Friday, April 25, 2014 11:00 AM
  • Hi,

    Not sure about your test scenario so I can’t confirm this issue is not because of Coded UI Test itself.

    I only can give you some suggestions to check whether we can improve the performance of test playback.

    Please check the problematical controls’ Search Configurations in Coded UI Test Editor to make sure the AlwaysSearch option is not set. Using AlwaysSearch will make that UI test playback engine does not use a cached control to perform any search action.

    And please disable ‘Smart Match’ and ‘Wait For Ready Level’ in the test case to check the result. For detailed information, please see: Configuring Playback in Visual Studio 2010

    In addition, in order to check what caused the issue, you also can check the coded UI test log after test run. Please reference: Analyzing Coded UI Tests Using Coded UI Test Logs

    Best regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    • Proposed as answer by Amanda Zhu Friday, May 2, 2014 1:31 AM
    • Marked as answer by Amanda Zhu Friday, May 2, 2014 10:10 AM
    Monday, April 28, 2014 7:13 AM