locked
Tests pass when change in dependency means they should fail RRS feed

  • Question

  • Using .NET 4.0, VS 2010 and MSTest - in recent weeks I have observed cases where a dependency assembly in the same solution has changed (e.g. a DAL or a Framework library) and that tests of code that depend on these that should fail, in fact pass. It's only if the code that is being tested is touched in some way do the tests start to fail due to the change in the dependency. This is happening  both on developer PCs (WinXP VS2010) and the TFS TeamBuild server (Windows Server 2008 R2).

    It seems that something in the way MSTest runs, or even the framework itself means there`s a cache somewhere (possibly of the JIT-compiled code dependencies) and this cache isn't updated with the new dependency code.

    Anyone seen this or can suggest possible cures or workaround?

    Wednesday, December 7, 2011 11:27 AM

All replies

  • Hi J.Denning,


    If you right click the “TestProject” in the Solution Explorer, select “Rebuild”, then run it, whether it can run normally?


    Thanks,

     


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us
    Thursday, December 8, 2011 8:23 AM
  • No, Rebuild and Clean have no effect whatsoever. I've manually removed all my bin and obj directories from my local solution folder hierarchy and see where that takes me (those folders would also include the TestResults folders). Furthermore this is happening on the TFS Team Build which is effectively a full Rebuild every time.

    I suspected this in a previous project (that used VS2010 and .NET3.5) but couldn't easily identify the scenario nor reproduce it but it feels as if the new version of the codebase is not being loaded - this is crucial for refactoring where no new tests or changes to outer functionality would occur but underlying changes might actually have a problem that is not detected.

    BTW I am using Deployment in my test settings if that has any bearing on it although it is just to include a config file used by one of the tests.


    • Edited by J.Denning Thursday, December 8, 2011 10:23 AM
    Thursday, December 8, 2011 10:22 AM
  • Hi J.Denning,

    Sorry for my delay.

    1.       We should make sure that whether the previous dependency code is called, so you call debug your unit test with “Step Into”.

    2.       How about add the Deployment item with the following way?

    (1)    Test->Windows->Test View

    (2)    Right click the unit test in Test View, select Properties, and then use the “Deployment Items” to check it.

    If still no help, you could create a similar simple APP, please also attach your Visual Studio project, you can upload it to the sky driver, and then share the download link in your post. We try to run it with our PC.

    Thanks,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    Monday, December 12, 2011 9:08 AM
  • Hi Jack

    I think whenthe test is run under debug it forces something to use the latest code as all breakpoints in the dependency code are hit and the failure is encountered (but I shouldn`t have to run all the tests under the debugger...).

    I am not understanding what your reference to Deployment items means - do you mean make sure all the dependency assemblies are included as deployment items even though all the dependency code is in the solution so the solution build should output new versions of these assemblies?

    Cheers

    James

    Wednesday, December 14, 2011 7:16 PM
  • Hi J.Denning,

    Sorry for my misunderstanding.

    At first, I think that if the dependency code isn’t called, it will generate this issue.

    The step2 is only another way to run tests with dependency item. It is similar to use the Local.testsettings to load the assemblies. Like this blog http://blogs.msdn.com/b/vstsqualitytools/archive/2010/01/09/assembly-resolution-for-unit-tests.aspx.

    Would you mind creating a similar simple APP? Please also attach it, you can upload it to the sky driver, and then share the download link in your post. We will download and check it. Or you could send it to V-jake at Microsoft dot com. Thanks for your understanding. 

    Sincerely,


    Jack Zhai [MSFT]
    MSDN Community Support | Feedback to us

    Thursday, December 15, 2011 3:44 AM
  • Hi Jack

    I'll spend some time over the Xmas period on an accurate and consistent way to reproduce the problem and get the sample to you.

    Thanks

    James

    Monday, December 19, 2011 5:23 PM