locked
Unit test pass in debug mode but fails sometime in release mode RRS feed

  • Question

  • Hi

    I use VS2010 and have a number of unit tests in one module. When I run the test in debug mode they all run. When I run the test one by one they all pass. But when i run the tests together from the test list editor they randomly fail in release mode. It seems timing dependent! Sometimes test pass on my local machine and fail on the build server!

    The tested code does not have any static variables. Each test load a configuration from an XML file, do a lot of computation and then the results are checked.

    Will all tests when started as a group of tests run at the same time?

    What are the difference between debug and release mode for the tests?

    My code does not have any if-debug settings or conditional compile.

    I have even tried to set up a global mutex around the test to avoid any influence from any shared memory or variable I have missed, but that did not change the result.

    Any ideas of what to do - wait for VS2012?


    Jens Mose Pedersen

    Tuesday, December 18, 2012 9:20 AM

Answers

  • Ohh my god. I am so ashamed. By help of a colleague I just found an unitialized class member variable.

    The debug code must have initialized it for me. And that is the reason the tests worked in debug mode. Now tests work.

    Sorry for the trouble.

    Regards Jens


    Jens Mose Pedersen

    • Marked as answer by Jens Mose Wednesday, December 19, 2012 1:47 PM
    Wednesday, December 19, 2012 1:47 PM

All replies

  • You say they 'randomly fail'. The exception that causes the test failure will probably provide hints...

    If you are not using static variables, my guess is it is probably because some resource is being held across tests whose disposal is dependent on garbage collection. I have encountered exactly the above symptoms when a test uses an XmlReader or opens a file but without the using(...) pattern and therefore never disposes/closes it; the second test tries to access the file and it is locked (sometimes!) from the previous test. These leaks are not always easy to find :-)

    In this case, the apparent random failures are because of the non-deterministic nature of the garbage collection and therefore the release of any resources (files etc.) acquired in the first test.

    If you start multiple tests, by default, MsTest will run them sequentially unless you have configured it to behave otherwise: put trace statements in the TestInitialize and TestCleanup methods to dump the TestContext.FullyQualifiedClassName and TestContext.TestName properties and ensure they pair up. There should be no interleaving of tests.

    The above is just a shot in the dark, but 'dangling' resources acquired in previous tests have always been the cause of any random failures I have encountered.

    Grey Ham


    Blog: http://www.havecomputerwillcode.com/blog

    Tuesday, December 18, 2012 9:47 AM
  • Thank you for some hints. However please clarify a few things for me.

    My code uses a C# module with a managed C++ wrapper to read my XML configurations files. My code under test and the test stubs are plain C++.

    You mention "without the using(...) pattern" - what do you mean with that?

    I don't think the problem is in the C#/C++ managed code, but how can I control the garbage collection?

    Another question. "by default, MsTest will run them sequentially unless you have configured it to behave otherwise". Where do you control the configuration of the test sequences?

    When I make a new instance of my object under test and the test stubs used for each test and I don't have any static variables, how could one test run influence the other?


    Jens Mose Pedersen

    Tuesday, December 18, 2012 11:36 AM
  • Hi,

    OK: this has opened up a can of worms: there is little I can do from here: it's almost certainly not a testing problem you have; it's a programming problem. 

    The using statement. Please see: http://msdn.microsoft.com/en-us/library/yh598w02.aspx. If the object being used implements IDisposable, it is called automatically when the code block completes / exits. 

    To all intents and purposes, you should not 'control' the garbage collection: you should write your code so that all resources are freed correctly after use. 

    Parallel testing in MsTest: more information here: http://stackoverflow.com/questions/3917060/how-to-run-unit-tests-mstest-in-parallel

    With regards your last question: they will affect each other if they take  a resource that is not explicitly freed (ie: a file on disk). Doesn't matter if your tests are isolated, they still need to clean up after themselves if subsequent tests rely on the same resources. 

    It sounds like the problems you are facing here are not testing problems - they are C#/.Net programming problems. 

    I am out of ideas. The only thing I can suggest is to make sure that your destructors are being called deterministically on your test object (C#, Managed C++, Plain C++) when your test object goes out of scope in your test. 

    Good luck!

    Grey Ham



    Blog: http://www.havecomputerwillcode.com/blog

    Tuesday, December 18, 2012 9:42 PM
  • Hello Jens,

    Thank you for your post.

    Hope you can provide us more information to narrow down this issue:

    1. Can all unit tests run successfully in Release mode with VS2010 on your local machine?
    2. Did the random failure in Release mode only on the builder server?
    3. What do you specify about the Debug/Release mode? The Configuration of Build pane or the Configuration of Debug pane in the testsettings?

    If yes 1 and 2, I am afraid that the issue may be more related to Builder server.

    If some tests also failed in Release mode with VS2010 on your local machine, you can provide us some related warning or error messages so that we can further look at this issue.

    Will all tests when started as a group of tests run at the same time?

    I think that the tests will run sequentially rather than at the same time.

    If you have other environments, I suggest that you can try the same scenario on another machine to check if the issue is related to your test environment.

    If I have misunderstood anything, please feel free to let me know.

    Best regards,


    Amanda Zhu [MSFT]
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Wednesday, December 19, 2012 9:58 AM
  • Hello Amanda

    Just a few words about my system first. It is a rather large embedded application written mostly in C++. Basically it takes an XML parameter file, reads the parameters and use these parameters for calculation of HW setup and finally transfers the HW setup to the HW. The program consists of a lot of individual control blocks each with an abstract interface so the module easily can be stubbed of during unit tests. It is one of these control blocks I want to unit test. I use the real XML file reader that is made as a managed C++ wrapper around a C# module for reading the XML and marshal this to unmanaged data structures. This works in a lot of unit tests both in debug and release mode. The access to the HW is stubbed of with a special module that simulates the real HW. This also work for a lot of units tests.

    Now the module that causes problems during test is kind of special as it uses a lot of memory on the heap, including aligned memory for intrinsic calculations. The calculation includes calculation of up to 1000000 squareroots.

    I have 4 specific test runs in my unit test class. When I run each of these test on my local mashine they all pass in debug mode. No matter if I run them one by one or start them together from the Test View or the Test List Editor.

    When I run the tests in release mode they also pass when I run them one by one. But when I run them together from the Test View I sometime exerience that one or more of the tests fail.

    The way the test fail is typically that some internal assert or limit check inside the unit under test fails. The failure seems to be timing dependent. Right now I have enabled some debug prints during my test an then it does not fail!

    The unit under test is a C++ class and is in the test created with a unique pointer. The tested class has no static variables. The other modules and stubs used during the test are also classes created with unique pointers.

    I have also tried to run my application with Intels inspector tools looking for memory leaks or use of unallocated memory and found no problems. Unfortunately this tool can not be used during unit tests.

    So back to your questions. As it seems like the test is timing dependent when the test are run as a block of tests it sometimes fails on the local machine and sometimes on the build server.

    The tests always pass in debug mode, both locally and on the build server.

    I am not sure what you mean with configuration of Build pane or debug pane. I compile my unit under test and all other modules to debug or to release. I use "run" unit test for testing in release mode.

    I really cannot see how one test can influence on the execution of another when the whole unit and and all the stubs are created from scratch in each test.

    Regards Jens


    Jens Mose Pedersen

    • Marked as answer by Jens Mose Wednesday, December 19, 2012 1:43 PM
    • Unmarked as answer by Jens Mose Wednesday, December 19, 2012 1:47 PM
    Wednesday, December 19, 2012 12:41 PM
  • Ohh my god. I am so ashamed. By help of a colleague I just found an unitialized class member variable.

    The debug code must have initialized it for me. And that is the reason the tests worked in debug mode. Now tests work.

    Sorry for the trouble.

    Regards Jens


    Jens Mose Pedersen

    • Marked as answer by Jens Mose Wednesday, December 19, 2012 1:47 PM
    Wednesday, December 19, 2012 1:47 PM