locked
Unit tests always pass in Debug but randomly (seems) fail in Release mode RRS feed

  • Question

  • I’m facing very confusing and puzzling behavior of VS 2010 SP1 Unit Test facility. When solution is built in Debug configuration unit tests 100% pass. When solution is built in Release configuration and I Run all unit tests some of them (looks like random set and number) fail! But when I try to debug them they pass again. Run them again – they fail. Debug them again – they pass. I’m stuck!

    Small solution demonstrating the problem is shared here:  https://skydrive.live.com/?cid=6f39718f92854399&id=6F39718F92854399!285

    And it does not seem to be connected to Code Coverage – I have code coverage turned off in test configuration.

    Please help!

     

    Thank you! Konstantin

    Thursday, January 12, 2012 3:30 AM

Answers

  • I think I got to the bottom of it.

     

    The tricky part of those unit tests is that to verify trace library functionality they use the same trace library. Example:

     

     Should.Error(

                    () => first.ShouldBeSameAs(second),

                    "() => first should be same as System.Object but was System.Object"

                );

     

    How many assertion checks are here? Actually TWO: one is inside and supposed to fail and throw and another is outside which supposed to catch and pass, i.e. not throw. In release mode both were throwing and the test was failing. The outer assertion was throwing because the message of the inner exception does not match expected pattern and actually looks like garbage.

     

    Why we get garbage in Release?

    There is method GetStackFrameForOriginatingTestMethod which walks the stack looking for the first stack frame which does not belong to trace library itself. It determine if particular method frame belongs to the library by checking the method type custom attribute [TraceServicesMethods]. All involved types in the library are indeed marked with that attribute.

     

    Now look at this code:

     

     Should.Error(

                    () => first.ShouldBeSameAs(second),

                    "() => first should be same as System.Object but was System.Object"

                );

     

    Exception and therefore stack walk happen from within ShouldBeSameAs method highlighted. Both Error method and ShouldBeSameAs method belong to the class marked with [TraceServicesMethods], so it looks good, BUT look at that ShouldBeSameAs method:

     

    public static void ShouldBeSameAs(this object actual, object expected)

            {

                actual.AssertInternal(Is.SameAs(expected), actual, expected);

            }

     

    Obviously compiler sees it as the perfect candidate for inlining! See where it goes?

     

    Stack walk is designed to stop at ShouldBeSameAs, but when the code is optimized this method (and corresponding stack frame) are not even there, so stack walk sees anonymous method without [TraceServicesMethods] attribute and only stops at Error method (or more precisely it goes right above it and then steps back). Then it starts to calculate relevant message strings based on that wrong location and basically get garbage.

     

    Obvious solution is to add compiler attribute [MethodImpl(MethodImplOptions.NoInlining)] to all relevant methods and I have verified that it fixes all failures in Release mode.

     

    Forrest, if you agree with my conclusions please feel free to close the issue now.

     


    Thank you! Konstantin
    Friday, January 20, 2012 1:35 AM

All replies

  • Konstantin,

    I'm able to reproduce the problem you described, but I don't quite understand what caused the problem either.  I try to get other people involved on this question, please wait some time.  If you are extremly urgent, you could contact Profession Support:  http://support.microsoft.com/default.aspx?id=fh;en-us;offerprophone

    Best Regards,


    Forrest Guo | MSDN Community Support | Feedback to us

    Monday, January 16, 2012 3:41 AM
  • Forrest, thank you for your attention! This problem keeps bothering me and actually blocking a number of people from accepting VS Unit Test facility for production purposes. We all appreciate you looking further into it.
    Thank you! Konstantin
    Monday, January 16, 2012 3:56 AM
  • I think I got to the bottom of it.

     

    The tricky part of those unit tests is that to verify trace library functionality they use the same trace library. Example:

     

     Should.Error(

                    () => first.ShouldBeSameAs(second),

                    "() => first should be same as System.Object but was System.Object"

                );

     

    How many assertion checks are here? Actually TWO: one is inside and supposed to fail and throw and another is outside which supposed to catch and pass, i.e. not throw. In release mode both were throwing and the test was failing. The outer assertion was throwing because the message of the inner exception does not match expected pattern and actually looks like garbage.

     

    Why we get garbage in Release?

    There is method GetStackFrameForOriginatingTestMethod which walks the stack looking for the first stack frame which does not belong to trace library itself. It determine if particular method frame belongs to the library by checking the method type custom attribute [TraceServicesMethods]. All involved types in the library are indeed marked with that attribute.

     

    Now look at this code:

     

     Should.Error(

                    () => first.ShouldBeSameAs(second),

                    "() => first should be same as System.Object but was System.Object"

                );

     

    Exception and therefore stack walk happen from within ShouldBeSameAs method highlighted. Both Error method and ShouldBeSameAs method belong to the class marked with [TraceServicesMethods], so it looks good, BUT look at that ShouldBeSameAs method:

     

    public static void ShouldBeSameAs(this object actual, object expected)

            {

                actual.AssertInternal(Is.SameAs(expected), actual, expected);

            }

     

    Obviously compiler sees it as the perfect candidate for inlining! See where it goes?

     

    Stack walk is designed to stop at ShouldBeSameAs, but when the code is optimized this method (and corresponding stack frame) are not even there, so stack walk sees anonymous method without [TraceServicesMethods] attribute and only stops at Error method (or more precisely it goes right above it and then steps back). Then it starts to calculate relevant message strings based on that wrong location and basically get garbage.

     

    Obvious solution is to add compiler attribute [MethodImpl(MethodImplOptions.NoInlining)] to all relevant methods and I have verified that it fixes all failures in Release mode.

     

    Forrest, if you agree with my conclusions please feel free to close the issue now.

     


    Thank you! Konstantin
    Friday, January 20, 2012 1:35 AM
  • Hi

    I have similar problem. I have 4 unit tests. When running in debug mode they all pass. When I run them one by one in release mode they all pass. But when I run them all 4 together from the test list editor, one or two randomly fail.

    This is driving me crazy.

    The tests run a lot of code, allocates large chucks of memory and exercise both c++, managed c++ and a c# module. The managed c++ is used as a wrapper of the c# module that is used to read am XML configuration file and translate it to an unmanaged data structure that is used as input for the real test.

    It seems not to be the use of both c++, managed c++ and c# that causes trouble - we have a lot of unit test running this way without any trouble. But when we also uses a lot of heap memory is seems to go wrong.

    Please give hints of what to do.


    Jens Mose Pedersen

    Friday, December 14, 2012 2:22 PM