locked
Tests pass in IDE but fail with MsTest.exe

    Question

  • We have an issue running our Unit Tests on our Team Build Server.

    The tests fail and the returned error is just : "Test host process exited unexpectedly."
    No log, no trace, no dump, no entry in the eventlog, no exception trapped by any attached debugger...

    Trying to investigate this issue on a workstation, we have found that the Unit Tests only pass when run from the IDE.
    They always fail when run from a command line, using MsTest.exe (as actually done by the Build Server)

    Using Process Monitor, we have monitored both the Devenv.exe and msTest.exe while running the Unit Tests.
    The big difference is that MsTest.exe does not copy all the assemblies required to run the tests in the TestResult's "\Out" folder. Devenv.exe does it !

    Consequence : when VsTestHost.Exe want to load the missing assembly, it tries for 11 seconds and finaly fail (after about 72000 "QueryOpen" on the missing assembly).

    After failing, VsTestHost.exe read the registry key DbgJITDebugLaunchSetting (value 2), but does not start any debbuger ???
    Next, VsTestHost.exe does some "Thread Exit", a "Process Exit" and close all the open files.


    What can be the difference between Devenv and MsTest regarding the copy of the required assemblies in the "\out" folder ?
    Ie.; how is MsTest.exe computing which assembly must be copied in this "\out" folder...

    Thx a lot in advance for any help.

    Valery.


    PS. : Using Process Explorer, we have suspended the VsTestHost.Exe, copyed the missing assemblies manually into the "\out" folder and then, the tests passed...
    Monday, February 25, 2008 4:22 PM

Answers

  • Hi Valery,

     

    Steve is right.  You need to specify the TestRunConfig file you have in the IDE on the command-line with the /runconfig option.

     

    That file is automatically used in the IDE, but must be specified from command-line since you aren't executing in context of a loaded solution and project.  I hope that helps.

     

    Cheers,

    David

    Wednesday, February 27, 2008 1:37 AM
    Owner
  • Valery,

    In your testrunconfig file, in the Deployment tab, specify any additional assemblies that are required.  You can also uncheck the 'Enable deployment' checkbox.  If you are running Smart device tests, they must use deployment.

    Hope this helps,


    Steve - www.platinumbay.com
    Monday, February 25, 2008 4:33 PM
  • Thx a lot for the tip. I was indeed not using the testrunconfig on the command line.

    I have tried to add the missing assemblies in the testrunconfig file but then another error occurred. A warning actually:
    Warning: conflict during test run deployment: deployment item 'xxx.dll' directly or indirectly referenced by the test container 'xxx.unittesting.dll' cannot be deployed to 'yyy.dll' because otherwise the file 'xxx.dll' would overwrite deployment item 'xxx.dll' directly or indirectly referenced by deployment item 'yyy.dll' specified by the test run configuration.

    However, the good news is that a MS consultant working part-time in our team solved the problem yesterday.

    This consultant is realizing a UI framework for our project and he needed to call one of our components (We develop our own  framework on top of the .Net framework). To do so, he simply added a file reference onto our assemblies in his VS.Net project. His solution was still compiling but at the runtime, he got the error : "file not found". The missing file was actually one of the assemblies also missing while running our unit tests from the command line. And looking in his bin folder, I could see that all the "not found" assembly was actually well available there; so his problem was as ununderstandable as our.

    Because the "not found file" was searched by a "Factory" to instanciate some classes, he decided to update all the references onto the 'Factory' component in his and our projects (He wanted to use a older version of this Factory).

    This "Factory" component is not one of the assemblies missing during our unit Tests.  But it is called in order to load these missing assemblies. All the assemblies "missing" during our Unit Tests are indeed only containing "providers" that are instantiated via the Factory (I.e.: late binding). We added references onto these "providers" assemblies in our all projects but we actually never declare directly any object using types from these assemblies. The references are only there to force the compiler to copy the assemblies in the bin folder.

    Since the "reference update", the VS.Net Solution of our MS consultant runs fine (He possibly had a problem with the Factory ? However, we don't find any significant difference in our source code).

    But as a side effect, our Unit Tests pass as well when run from the command line! We don't see why changing the references onto the Factory assembly force now MsTest to copy the "Providers" assemblies in his out folder !?!? So we have concluded that some references where "corrupted"...

    Notice that our Tests pass even if we don't use the testrunconfig file on the command line (However, the instrumentation is correctly also done in order to do the code coverage ! So I guess there is some info available for MsTest in the metadata ???). Here is the exact command line used :
    C:\Program Files\Microsoft Visual Studio 9.0\VC>mstest /testmetadata:"C:\...\Configuration.vsmdi"  /resultsfile:c:\temp\results.trx /testlist:repository

    Wednesday, February 27, 2008 11:02 AM

All replies

  • Valery,

    In your testrunconfig file, in the Deployment tab, specify any additional assemblies that are required.  You can also uncheck the 'Enable deployment' checkbox.  If you are running Smart device tests, they must use deployment.

    Hope this helps,


    Steve - www.platinumbay.com
    Monday, February 25, 2008 4:33 PM
  • Hi Valery,

     

    Steve is right.  You need to specify the TestRunConfig file you have in the IDE on the command-line with the /runconfig option.

     

    That file is automatically used in the IDE, but must be specified from command-line since you aren't executing in context of a loaded solution and project.  I hope that helps.

     

    Cheers,

    David

    Wednesday, February 27, 2008 1:37 AM
    Owner
  • Thx a lot for the tip. I was indeed not using the testrunconfig on the command line.

    I have tried to add the missing assemblies in the testrunconfig file but then another error occurred. A warning actually:
    Warning: conflict during test run deployment: deployment item 'xxx.dll' directly or indirectly referenced by the test container 'xxx.unittesting.dll' cannot be deployed to 'yyy.dll' because otherwise the file 'xxx.dll' would overwrite deployment item 'xxx.dll' directly or indirectly referenced by deployment item 'yyy.dll' specified by the test run configuration.

    However, the good news is that a MS consultant working part-time in our team solved the problem yesterday.

    This consultant is realizing a UI framework for our project and he needed to call one of our components (We develop our own  framework on top of the .Net framework). To do so, he simply added a file reference onto our assemblies in his VS.Net project. His solution was still compiling but at the runtime, he got the error : "file not found". The missing file was actually one of the assemblies also missing while running our unit tests from the command line. And looking in his bin folder, I could see that all the "not found" assembly was actually well available there; so his problem was as ununderstandable as our.

    Because the "not found file" was searched by a "Factory" to instanciate some classes, he decided to update all the references onto the 'Factory' component in his and our projects (He wanted to use a older version of this Factory).

    This "Factory" component is not one of the assemblies missing during our unit Tests.  But it is called in order to load these missing assemblies. All the assemblies "missing" during our Unit Tests are indeed only containing "providers" that are instantiated via the Factory (I.e.: late binding). We added references onto these "providers" assemblies in our all projects but we actually never declare directly any object using types from these assemblies. The references are only there to force the compiler to copy the assemblies in the bin folder.

    Since the "reference update", the VS.Net Solution of our MS consultant runs fine (He possibly had a problem with the Factory ? However, we don't find any significant difference in our source code).

    But as a side effect, our Unit Tests pass as well when run from the command line! We don't see why changing the references onto the Factory assembly force now MsTest to copy the "Providers" assemblies in his out folder !?!? So we have concluded that some references where "corrupted"...

    Notice that our Tests pass even if we don't use the testrunconfig file on the command line (However, the instrumentation is correctly also done in order to do the code coverage ! So I guess there is some info available for MsTest in the metadata ???). Here is the exact command line used :
    C:\Program Files\Microsoft Visual Studio 9.0\VC>mstest /testmetadata:"C:\...\Configuration.vsmdi"  /resultsfile:c:\temp\results.trx /testlist:repository

    Wednesday, February 27, 2008 11:02 AM
  • Just notice that we have now another issue Sad

    V.
    Tuesday, March 04, 2008 3:22 PM
  • The problem came back this morning.

     

    here is the command line used to reproduce our problem :

    MSTest /testmetadata:"OurSolution.vsmdi" /runconfig:"OurSolution.testrunconfig" /resultsfile:"C:\Temp\TestResults"

     

    We were able to solve the problem applying the Steve's advice:

     

     .NETicated! wrote:
    You can also uncheck the 'Enable deployment' checkbox.

     

    Thx a lot !

     

    V.

    Monday, March 17, 2008 1:25 PM
  • Hi Valery,

    I am running in to the exact same issue as you did. I've tried running Process Monitor and Dependency Walker to find out what were the missing assemblies, but I've been having no luck. Is there something specific I can look for, or anything else that I can try?

    I've turned off the Deployment Items in the LocalTestRunConfig but that didn't fix it. 
    Wednesday, June 03, 2009 8:30 PM
  • Hi There,

    I came across the similar problem.

    I am using OrderedTest, and running from batch file which executes all the test cases listed in that ordered test.

    Now If I run OrderedTest from Visual Studio environment, it run all cases, but if I run them from command line, it executes only few cases, and abort the run. When I look at the result, I see "Pending" status for not executed tests.

    I am getting this error message: "Test host process exited unexpectedly."

    This is my batch file looks like:

    ----------------------------------------------------

    CLS

    ECHO off

    ECHO Starting Daily Test Run... Press any key to continue OR CTRL+C to Stop

    PAUSE

    cd "C:\project\regressiontesting" 

    "C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"  /runconfig:LocalTestRun.testrunconfig


    cd "C:\project\regressiontesting\bin\Debug" 

    "C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\MSTest.exe"  /TestContainer:DailyTestRun.orderedtest /Test:DailyTestRun

    ----------------------------------------------------

    With help of above comments, I added runconfig file into batch file. But it doesn't seem to work. 

    Please note: I didn't include metadata because when I included it, it started to running all cases in the project, and not selected once.

    Can anyone please help how to make batch file more reliable? Thanks in anticipation.

    Wednesday, March 31, 2010 11:09 PM