locked
What is the best way to compare reports? RRS feed

  • Question

  • Many of my coded UI tests involve running reports with various inputs and comparing the report output against a previous run of the report, using the same inputs,  to insure that it is the same.

    As far as i know the only way to compare the report output is to get the inner text of the html report and compare the strings, original version vs. latest version. If an error occurs I use the Assert.AreEqual function to output the strings into the test results so I can get an idea of the issue.

    When I run these coded ui Tests from MTM, the error message output of the Assert.AreEqual statement is truncated to 512 characters. So I have to run the test(s) in debug mode to find out what the differences might be.

    Is there a better way to compare report outputs ?

    I really appreciate any insight.

    Tuesday, December 11, 2012 11:16 PM

Answers

  • It appears that there is no better way to compare the output of a report than to compare the innertext of the original (recorded) report vs the the text of the report as it is rendered in the latest version of the application under test.

    That said, I now understand that the innertext property of an html report can change with the browser version. So the issue then becomes testing against all supported browser versions and properly defining the expected results. That's another thread.

    Thanks for your help.

    • Marked as answer by OneHip51 Tuesday, December 18, 2012 6:39 PM
    Tuesday, December 18, 2012 6:39 PM

All replies

  • I would consider comparing the two strings for equality. If not equal then remove the common front parts and do the assertion on the remaining text. It may be useful to report the length of the removed front part. I am thinking of code based on the following outline:

    if ( a != b ) {
      int len = a.length < b.length ? a.length : b.length;
      for ( int ii=0; ii<len; ii++) {
        if( a[ii] != b[ii] ) {
          assert.areequal(a.substring(ii),
               b.substring(ii),
               false,
               string.format("After {0} same characters", ii));
    }}}
    
    You will need to extend this handle the wierd cases such as where one string is the prefix of the other

    Regards

    Adrian

    Wednesday, December 12, 2012 10:12 AM
  • I am currently doing an assert.AreEqual(expected,actual) the question is, is there a better way to compare two html reports than by getting the inner text and comparing the new against some older expected value.

    Since MTM sees fit to truncate the error message that is thrown, if they are not the same then I can't see the returned result to see what the problem might be. I have to rerun the test in debug mode to do that.

    Thursday, December 13, 2012 12:45 AM
  • Thanks for Adrian’s help.

    Hi OneHip51,

    Thank you for posting in the MSDN forum.

    Would you mind letting us know more information about this issue? Do you mean that you are running the Coded UI test in the MTM? You want to get the reports when you running it in the MTM, am I right?

    As far as i know the only way to compare the report output is to get the inner text of the html report and compare the strings, original version vs. latest version. If an error occurs I use the Assert.AreEqual function to output the strings into the test results so I can get an idea of the issue.

    If we run the tests with the VS IDE, I feel that your idea is perfect. We could get the information in the test result. Maybe You could submit this feature request: http://visualstudio.uservoice.com/forums/121579-visual-studio. The Visual Studio product team is listening to user voice there. 

    Not the MTM expert, but if you are running it in the MTM, I suggest you post this issue to the MTM forum for better support. Thanks for your understanding.

    Best Regards,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Thursday, December 13, 2012 7:09 AM
    Moderator
  • OneHip51, my apologies. When I read your first message I saw two questions and I only answered one of them.

    For the question of is there a better way of comparing html reports than comparing their inner texts I did not give any answer because I could not think of any better method.

    For question about comparing two strings where the output from the assert.areequal truncates, I tried to provide an answer that would mean you do not need to rerun the test under the debugger.

    Another possibility is to add code based on these lines just before the assert:

    Console.WriteLine("====== Actual text: ======");
    Console.WriteLine(expected);
    Console.WriteLine("====== Expected text: ======");
    Console.WriteLine(actual);
    Console.WriteLine("====== End ======");

    Rather than "Console" you might find another output stream is better for your tests, see this blog:

    http://blogs.msdn.com/b/gautamg/archive/2009/12/08/logging-a-message-in-test-result-as-part-of-an-automated-test.aspx

    Regards

    Adrian

    Thursday, December 13, 2012 9:25 AM
  • Hi OneHip51,

    If there are some replies helping you to resolve your problem, you could mark the answer then I will close this thread, if not please post the latest situation of this issue. Thanks for your understanding.

    Sincerely,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Tuesday, December 18, 2012 2:29 AM
    Moderator
  • It appears that there is no better way to compare the output of a report than to compare the innertext of the original (recorded) report vs the the text of the report as it is rendered in the latest version of the application under test.

    That said, I now understand that the innertext property of an html report can change with the browser version. So the issue then becomes testing against all supported browser versions and properly defining the expected results. That's another thread.

    Thanks for your help.

    • Marked as answer by OneHip51 Tuesday, December 18, 2012 6:39 PM
    Tuesday, December 18, 2012 6:39 PM