locked
Generate a Run Comparison Load Test Report in Excel: Change 'Baseline' and 'Comparison run'

    Question

  • Hi,

    When I am creating a 'Run Comparison' Load Test Report in VS2010, it opens up a wizard in Excel 2010 and I can successfully create a report using my two load test runs (ran on separate environments). 

    However, for reasons I cannot understand, the generated Excel report insists on incorrectly classing my two runs in the generated report as 'Baseline' and 'Comparison Run' in the first tab '1.1. Runs'.

    The first load test performed I would like to label as 'Comparison Run' (as it's the environment we will move the application under test to), the second test is the baseline load test in the current BAU test environment, but these classifications are incorrect in the report - the run that I want classed as 'Baseline' is 'Comparison Run' and vice versa. I'm happy with the stats and cool charts that the report is generating, but unfortunately if I distribute it, it will cause a lot of confusion because all of the stats are the opposite way round than they should be.

    Is there a way of correctly classifying my runs, so that the generated report will display the correct stats?  I would like to achieve this without having to create a new custom report, as I am unsure of how to do that.

    Thursday, March 01, 2012 1:16 PM

Answers

  • Hi James,

    Glad to receive your replyJ

    I also try to repro this issue, and I try to change the Run Type in “1.1 Runs”, as the screen shot provided by you, my chart displayed in “1.2 Test Comparison” didn’t change anything.

    I did some research about this issue, but I don’t find any MSDN documents about set the Run Type in the load tests. As my understanding, the current version doesn’t support it, so as my first reply, you could submit the suggestions and ideas: http://visualstudio.uservoice.com/forums/121579-visual-studio. Thanks for your understanding.

    Sincerely,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    Wednesday, March 07, 2012 5:50 AM
    Moderator
  • Thanks for the assistance Jack. 

    I also tried and failed to find any documentation about this. In addition to you reproducing the issue, I have also tried running my baseline load test before my comparison load test. I can confirm that the report correctly recognises the baseline and comparison under this scenario. Therefore all the stats and charts are now being generated correctly.

    So basically it was a learning issue for me in the future to always run my baseline before my comparison run. It would be nice to have a way to configure this though. Therefore I have raised this as a suggestion using the link you provided. I would appreciate anybody reading this to follow the link http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/2677399-allow-changing-the-run-type-of-a-load-test-run and vote if you would find this feature useful. 

    Thanks,

    James

    • Marked as answer by James_OC Tuesday, March 13, 2012 2:38 PM
    Tuesday, March 13, 2012 2:37 PM

All replies

  • Hi James,

    Thank you for posting in the MSDN forum.

    Do you mean that you want to correct the “Baseline” or “Comparisons Run” in Run Type (without creating a new custom report) like the following screen shot?

    This page gives a little info about the 2 runs which were selected. It seems that the state of the last load test result is the "Comparisons Run”.


    But unfortunately if I distribute it, it will cause a lot of confusion because all of the stats are the opposite way round than they should be.

    Do you mean that you just double click it and change the Run Type in 1.1 Runs?

    If so, as my understanding, if we change it, I think we will get the opposite information.

    Maybe the next version can achieve it. You could submit the suggestions and ideas: http://visualstudio.uservoice.com/forums/121579-visual-studio. Microsoft engineers will evaluate them seriously, thanks for your understanding.

    Best Regards,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    Monday, March 05, 2012 3:36 AM
    Moderator
  • Hi Jack,

    Thanks for the reply. Your screenshot looks absolutely correct with what I am trying to change in the generated comparison report. 

    The issue with me manually editing the information in that screenshot, is that it does not change the run type in the other sections of the report. For example, the tab '1.2 Test Comparison' displays the following chart (I've omitted my test names)

    Well because the report has got my baseline and comparison mixed up, unfortunately the lovely chart above and the rest of the stats in the report are misleading (as I know that this is actually displaying the performance being marginally worse in my comparison run - so these aren't "performance improvements" at all!). So as you can see, it is not quite as simple as performing a quick find and replace. 

    Are we saying that this comparison report of two load tests, always treats the most recent run as the "Comparison Run" and there is no way to configure that as a setting? I must admit I would find that a bit hard to believe as it would seem like a major limitation to always need to run a comparison load test after a baseline. I am about to do another few run throughs of the load scenario and will make sure to do my 'baseline' load test before comparison, so I should be able to check whether that makes a difference. Sure, if this is the case I will definitely suggest it on the link provided.

    Thanks,

    James 

    Tuesday, March 06, 2012 4:19 PM
  • Hi James,

    Glad to receive your replyJ

    I also try to repro this issue, and I try to change the Run Type in “1.1 Runs”, as the screen shot provided by you, my chart displayed in “1.2 Test Comparison” didn’t change anything.

    I did some research about this issue, but I don’t find any MSDN documents about set the Run Type in the load tests. As my understanding, the current version doesn’t support it, so as my first reply, you could submit the suggestions and ideas: http://visualstudio.uservoice.com/forums/121579-visual-studio. Thanks for your understanding.

    Sincerely,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    Wednesday, March 07, 2012 5:50 AM
    Moderator
  • Thanks for the assistance Jack. 

    I also tried and failed to find any documentation about this. In addition to you reproducing the issue, I have also tried running my baseline load test before my comparison load test. I can confirm that the report correctly recognises the baseline and comparison under this scenario. Therefore all the stats and charts are now being generated correctly.

    So basically it was a learning issue for me in the future to always run my baseline before my comparison run. It would be nice to have a way to configure this though. Therefore I have raised this as a suggestion using the link you provided. I would appreciate anybody reading this to follow the link http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/2677399-allow-changing-the-run-type-of-a-load-test-run and vote if you would find this feature useful. 

    Thanks,

    James

    • Marked as answer by James_OC Tuesday, March 13, 2012 2:38 PM
    Tuesday, March 13, 2012 2:37 PM