locked
TEST_CLASS_INITIALIZE methods are not running correctly RRS feed

  • Question

  • I have several native C++ test classes in one project.  Each class has its own TEST_CLASS_INITIALIZE and TEST_CLASS_CLEANUP methods.  I am expecting the first test class' TEST_CLASS_INITIALIZE method to run, then all of the first test class' tests should run, then the first test class' TEST_CLASS_CLEANUP method to run.  I then expect the second test class' TEST_CLASS_INITIALIZE method to run, followed by the second test class's tests, then the second class' TEST_CLASS_CLEANUP.  In hierarchical form:

    TEST_CLASS_INITIALIZE ( A )
       TEST_METHOD ( A1 )
       TEST_METHOD ( A2 )
       ...
    TEST_CLASS_CLEANUP ( A )
    TEST_CLASS_INITIALIZE( B )
       TEST_METHOD ( B1 )
       TEST_METHOD ( B2 )
       ...
    TEST_CLASS_CLEANUP ( B )

    Instead, both test class' TEST_CLASS_CLEANUP methods run after all of the initialize and test methods run, in the opposite order in which their respective initializers ran.

    TEST_CLASS_INITIALIZE ( A )
       TEST_METHOD ( A1 )
       TEST_METHOD ( A2 )
       ...
    TEST_CLASS_INITIALIZE ( B )
       TEST_METHOD ( B1 )
       TEST_METHOD ( B2 )
       ...
    TEST_CLASS_CLEANUP ( B )
    TEST_CLASS_CLEANUP ( A )  !  This should have run just prior to TEST_CLASS_INITIALIZE ( B )

    Is what I am seeing the correct execution?  It does not match the behavior of other test frameworks I've worked with.

    Regards,

    Bob Schmidt


    Bob

    Thursday, January 15, 2015 10:17 PM

Answers

  • Hi Bob,

    Based on the result, if we doesn't see/view A, or only see the A, actually every test method still run normally.

    The real issue would be related to the class cleanup.

    http://blogs.msdn.com/b/ploeh/archive/2007/01/06/classcleanupmayrunlaterthanyouthink.aspx

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, January 20, 2015 8:47 AM
  • You are welcome. Bob!

    Actually I also test it in the C# unit test project, it still has this issue. The class cleanup really run later.

    If you got a workaround, if possible, please share us in your new reply, so it would be beneficial for other members who get the same issue.

    If there are any replies which are helpful for you, please also mark them as the answers.

    >> From the comments it appears I'm not the only one who thinks that running all of the cleanup methods at the end isn't quite right.

    About this issue, you can submit this feedback to Microsoft Connect feedback portal: http://connect.microsoft.com/VisualStudio/feedback/CreateFeedback.aspx, Microsoft engineers will evaluate them seriously. Thanks for your understanding.

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, January 21, 2015 2:28 AM

All replies

  • Hi Bob,

    Actually one important issue is that it is not the TestCleanUp or TestInitialize Attribute, it is the ClassCleanUp or ClassInitialize attribute, they are different attributes.

    ClassCleanUp: Identifies a method that contains code to be used after all the tests in the test class have run and to free resources obtained by the test class.

    TestCleanUp: Identifies a method that contains code that must be used after the test has run and to free resources obtained by all the tests in the test class.

    So one attribute just used for all tests finished, and another one just used for one test finished every times. I think you could get different result if you use the TestCleanUp or TestInitialize Attribute.

    Reference:

    http://blogs.msdn.com/b/densto/archive/2008/05/16/using-a-base-class-for-your-unit-test-classes.aspx

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, January 16, 2015 7:50 AM
  • Thanks for your response, Jack.

    I think we are saying the same thing, with one difference - you are talking about function attributes and I'm talking about the macros VS supplies for doing native code unit testing.  I would expect that the attribute ClassInitialize would map to the macro TEST_CLASS_INITIALIZE, and the attribute ClassCleanUp would map to the macro TEST_CLASS_CLEANUP.

    Using the attributes, the hierarchy (for C#) is

    AssemblyInitializeAttribute
        ClassInitializeAttribute
            TestInitializeAttribute
                TestMethod
            TestCleanupAttribute
            TestInitializeAttribute
                TestMethod
            TestCleanupAttribute
        ClassCleanupAttribute
        ClassInitializeAttribute
            TestInitializeAttribute
                TestMethod
            TestCleanupAttribute
            TestInitializeAttribute
                TestMethod
            TestCleanupAttribute
        ClassCleanupAttribute
    AssemblyCleanupAttribute

    Similarly, the hierarchy for the macros should be

    TEST_MODULE_INITIALIZE
        TEST_CLASS_INITIALIZE
            TEST_METHOD_INITIALIZE
                TEST_METHOD
            TEST_METHOD_CLEANUP
            TEST_METHOD_INITIALIZE
                TEST_METHOD
            TEST_METHOD_CLEANUP
        TEST_CLASS_CLEANUP
        TEST_CLASS_INITIALIZE
            TEST_METHOD_INITIALIZE
                TEST_METHOD
            TEST_METHOD_CLEANUP
            TEST_METHOD_INITIALIZE
                TEST_METHOD
            TEST_METHOD_CLEANUP
        TEST_CLASS_CLEANUP
    TEST_MODULE_CLEANUP

    Instead, I am seeing all of the TEST_CLASS_CLEANUP functions running after the last method in all classes have been run.

    I am going to try rewriting the tests using attributes directly, to see if there is a problem with the macro.

    Regards,

    Bob


    Bob

    Friday, January 16, 2015 3:09 PM
  • My attempt to rewrite using attributes directly failed, possibly because MS has not implemented attributes as an extension to standard C++.  I don't have time to fight it.  I did write a test program that illustrates what I am seeing.  Here's the code:

    #include <fstream>

    #include "CppUnitTest.h"

    using namespace std;
    using namespace Microsoft::VisualStudio::CppUnitTestFramework;


    namespace daq_mipstest
    {
        ofstream  *p_log;

        TEST_MODULE_INITIALIZE ( module_initialize )
        {
            p_log = new ofstream ();
            p_log->open ( "c:\\temp\\log.txt" );

            *p_log << "module_initialize" << endl;    
        }

        TEST_MODULE_CLEANUP ( module_cleanup )
        {
            *p_log << "module_cleanup" << endl;    

            p_log->close ();
        }

        TEST_CLASS (  TEST_TEST_1 )
        {
            TEST_CLASS_INITIALIZE ( class_initialize )
            {
                *p_log << "   class_initialize 1" << endl;
            }

            TEST_CLASS_CLEANUP ( class_cleanup )
            {
                *p_log << "   class_cleanup 1" << endl;    
            }

            TEST_METHOD_INITIALIZE ( test_initialize )
            {
                *p_log << "      test_initialize 1" << endl;    
            }

            TEST_METHOD_CLEANUP ( test_cleanup )
            {
                *p_log << "      test_cleanup 1" << endl;    
            }

            TEST_METHOD ( test_1 )
            {
                *p_log << "         test_1 1" << endl;    
            }

            TEST_METHOD ( test_2 )
            {
                *p_log << "         test_2 1" << endl;    
            }
        };

        TEST_CLASS (  TEST_TEST_2 )
        {
            TEST_CLASS_INITIALIZE ( class_initialize )
            {
                *p_log << "   class_initialize 2" << endl;
            }

            TEST_CLASS_CLEANUP ( class_cleanup )
            {
                *p_log << "   class_cleanup 2" << endl;    
            }

            TEST_METHOD_INITIALIZE ( test_initialize )
            {
                *p_log << "      test_initialize 2" << endl;    
            }

            TEST_METHOD_CLEANUP ( test_cleanup )
            {
                *p_log << "      test_cleanup 2" << endl;    
            }

            TEST_METHOD ( test_1 )
            {
                *p_log << "         test_1 2" << endl;    
            }

            TEST_METHOD ( test_2 )
            {
                *p_log << "         test_2 2" << endl;    
            }
        };
    }

    Here's the output:

    module_initialize
       class_initialize 2
          test_initialize 2
             test_2 2
          test_cleanup 2
          test_initialize 2
             test_1 2
          test_cleanup 2
       class_initialize 1
          test_initialize 1
             test_2 1
          test_cleanup 1
          test_initialize 1
             test_1 1
          test_cleanup 1
       class_cleanup 1
       class_cleanup 2
    module_cleanup

    Ass you can see, "class_cleanup 2" is not being executed after "test_cleanup 2", but rather just prior to "module_cleanup".  I don't think this is the correct behavior.  It should execute after "test_cleanup 2".

    Regards,

    Bob.


    Bob

    Friday, January 16, 2015 3:49 PM
  • Hi Bob,

    As my understanding, it is not the test issue, the real issue is that how we run the four tests.

    To make it clear, I change the method name:

    TEST_METHOD ( test_1 )//Change name to Test1_2
             {
                 *p_log << "         test_1 2" << endl;    
             }
    
             TEST_METHOD ( test_2 )//Change name to Test_2_2
             {
                 *p_log << "         test_2 2" << endl;    
             }

    For example, if you just run test _1 and test_ 2, I think you could get the result like these:

    module_initialize
       class_initialize 1
          test_initialize 1
             test_1 1
          test_cleanup 1
          test_initialize 1
             test_2 1
          test_cleanup 1
       class_cleanup 1
    module_cleanup

    If you just run test_1_2 and test_2-2:

    module_initialize
       class_initialize 2
          test_initialize 2
             test_1 2
          test_cleanup 2
          test_initialize 2
             test_2 2
          test_cleanup 2
       class_cleanup 2
    module_cleanup

    But if run the test methods with other way, I think it would have the "wrong" result. For example, If I run test_1 and test_1_2. I will get the result:

    module_initialize
       class_initialize 2
          test_initialize 2
             test_1 2
          test_cleanup 2

       class_initialize 1
          test_initialize 1
             test_1 1
          test_cleanup 1
       class_cleanup 1
       class_cleanup 2
    module_cleanup

    But one issue is that why you use the same module_initialize and module_cleanup to output the text for different class, if you want to use different assemblies, I'm afraid that this sample doesn't meet the requirements.

    The real way is that we would use two test projects in the same solution with two log.txt output results.

    You could get the project sample here:

    https://onedrive.live.com/embed?cid=55E83A59BF9AE3EB&resid=55E83A59BF9AE3EB%21578&authkey=AD9GBAJMhQM1BcI

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Monday, January 19, 2015 6:14 AM
  • Hi Jack,

    You are correct.  If I only run the tests for one class (as per your first two examples), I get the results I expect - the same results you listed.

    Where we diverge is when you start talking about different assemblies.  I don't want these tests in different assemblies.  I purposely put them in the same test project, in the same test DLL.

    In my real code, I don't use the module setup and cleanup functions, but I do use the class setup and cleanup member functions.  I have multiple classes in a project, and I want to test all of these classes in the associated test project.

    I tried moving the classes into different files within the test project (this time using VS 2013), and got the same (bad) results.  I have to put the tests for each class into different projects (like you did) in order to get the cleanup routines to run at the proper time relative to each other.

    I don't want to have a separate project for each class' tests in order to get the desired results.  That's just silly.

    Thanks for your help.

    Best regards,

    Bob


    Bob

    Tuesday, January 20, 2015 1:50 AM
  • Hi Bob,

    Based on the result, if we doesn't see/view A, or only see the A, actually every test method still run normally.

    The real issue would be related to the class cleanup.

    http://blogs.msdn.com/b/ploeh/archive/2007/01/06/classcleanupmayrunlaterthanyouthink.aspx

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, January 20, 2015 8:47 AM
  • Jack,

    The blog you just posted was exactly what I was looking for before I started this thread; I just didn't find it.  From the comments it appears I'm not the only one who thinks that running all of the cleanup methods at the end isn't quite right.

    I have figured out a way to work around the issue, which was causing serious memory leaks and heap corruption.

    Thank you for all of your help over the past several days.

    Regards,

    Bob


    Bob

    Tuesday, January 20, 2015 3:07 PM
  • You are welcome. Bob!

    Actually I also test it in the C# unit test project, it still has this issue. The class cleanup really run later.

    If you got a workaround, if possible, please share us in your new reply, so it would be beneficial for other members who get the same issue.

    If there are any replies which are helpful for you, please also mark them as the answers.

    >> From the comments it appears I'm not the only one who thinks that running all of the cleanup methods at the end isn't quite right.

    About this issue, you can submit this feedback to Microsoft Connect feedback portal: http://connect.microsoft.com/VisualStudio/feedback/CreateFeedback.aspx, Microsoft engineers will evaluate them seriously. Thanks for your understanding.

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, January 21, 2015 2:28 AM