BVT's without Team System Tester Edition - Possible?

Question

I'm trying to create a Team Build that will run all my unit tests as part of the build.  The problem is though, when going through the wizard it allows me to select the vsmdi file, but there is no test lists available in the drop-down (and since I don't have team tester edition I don't think I can create new test lists).  We can still run the tests in our Team Software Dev Edition, is there a way to make a Team Build that will also run all the tests as part of the build process without purchasing Team Tester Edition?

Answer 1

You can install either VSTS for Testers or VSTS for Developers on the build machine to run your tests. 

Here's an excerpt from the licensing whitepaper.

As part of the build process, Team Foundation Server may run quality tests and/or analysis on the precompiled or compiled code. These tests rely on functionality found within Team System client products, typically within the Team Edition for Software Developers or Team Edition for Software Testers products. These products may be installed on the build machine by licensed users of those products, as long as they are not directly used by any individuals who are not licensed for those products. Team Foundation Server will gather results of the build as well as any quality tests or analysis and deposit them in the Team Foundation Server data repository.

You can find the licensing details at http://www.microsoft.com/downloads/details.aspx?familyid=1FA86E00-F0A3-4290-9DA9-6E0378A3A3C5&displaylang=en.

I would expect that you could create new test lists with the VSTS for developers.

Buck

Answer 2

I hope you're correct.  In Team Tester you can choose Create New Test List... from the Test menu.  However, this option does not appear in the test menu in VSTS For Developers.

Could somebody shed some light on this, or tell me of an alternative way to create a Test List that I can use with Team Build from VSTS For Developers?

Answer 3

I'm going to move this to the VSTS - Developers forum to make sure you get an accurate answer.

Buck

Answer 4

Thanks for the feedback - I've actually been pushing hard for something like this since we first hit the problem in the beta days, so this will help me build a stronger case.  Until then, I've been told the official recommendation for Visual Studio 2005 is to install [Team Build+Team Dev+Team Test] or [Team Build+Team Suite]. 

Eric Jarvi
http://blogs.msdn.com/ejarvi

 

 

 

Answer 5

Eric,

Just to clarify.  You are saying that if the only version of VS we own is VSTS For Software Developers, then it is not possible at this time to create a Team Build that will run our tests as part of the build process.

Is that correct?

Thanks,

Dylan

Answer 6

Bump!

Just to clarify.  You are saying that if the only version of VS we own is VSTS For Software Developers, then it is not possible at this time to create a Team Build that will run our tests as part of the build process.

Is that correct?

I am in the same boat.  Tell me Microsoft isn't so insane that they don't expect some developers to wear more than one hat.  Let me know if someone found a way around this.

Thanks
James Coleman
Agency.com

Answer 7

Does anyone have an answer for people with VSTS for Developers?

Answer 8

You need to a specific list created within the vsmdi file for Team Build's wizard to pick it up. The only Editions that will create this special list are the Test Edition and the Team Suite Edition.

Users can install the Dev Edition and the Test Edition on the same machine.

<rant>This is the most insane thing I have ever heard. MS allows developers to code their own unit tests with the Developer Edition but they aren't allowed to schedule those tests to be run with Team Build....OMG! </rant>

-Ken

Answer 9

You can now run tests as part of your build without using .vsmdi files.  See http://blogs.msdn.com/buckh/archive/2006/11/04/how-to-run-tests-without-test-metadata-files-and-test-lists-vsmdi-files.aspx for details.

Buck

Answer 10

Great add-on. Probably the best TFS add-on I have seen. Or maybe that’s because of all the frustration I have had with the fact that users using TSFD could not push their tests to the nightly build.

 

Anyways I have one question for you. I probably just can’t follow directions.

 

I installed everything and it all works. Except I no longer have any code coverage. Is that by design or did I miss something?  Here is that data I entered in my test Build.proj. ( I only have unit tests that compile to this one ddl and they all run in the build fine, just no cc.)

<TestContainerInOutput Include="projTestfoo.dll" />

 

 

Thanks,

Joe

Answer 11

You need to specify a .testrunconfig file (which has coverage turned on) to get code coverage with team build.

Answer 12

Can somebody provide some details about how we specify a .testrunconfig file if we are no longer using a vsmdi?  Do we still need to use a vsmdi file in a <MetaDataFile> element in addition to the <TestContainerInOutput> tag?  If so can sombody provide a sample of the relevant portion of the TfsBuild.proj file?

 

Thanks,

Dylan

Answer 13

I tried doing this in my TfsBuild.proj:

 

<MetaDataFile Include="$(SolutionRoot)\ShopFloorTeamBuild\ShopFloorTeamBuild.vsmdi">

<!--<TestList>BVTList</TestList>-->

</MetaDataFile>

<TestContainerInOutput Include="ShopFloorTests.dll" />

I was thinking that I could specify the vsmdi in the <MetaDataFile> tag but without any <TestList> tags specified just so it would know where to find the testrunconfig.  Then use the <TestContaininerInOutput> tag to specify which tests I want run as part of the build.  What actually happened is the whole suite of tests ran twice, the first time they ran with the testrunconfig file and all passed (not sure why all the tests run if I didn't specify a <TestList>).  Then they all ran again only without the proper config specified in my testrunconfig, and most of them failed.  The build report says that 257/257 tests passed, but the run tests step failed.  The build log revealed that the full 257 tests were actually run twice.

What is the proper way to specify a testrunconfig file to use?  Ideally I'd just like to be able to do something like:

<TestContainerInOutput Include="ShopFloorTests.dll" ConfigFile="$(SolutionRoot)\ShopFloorTeamBuild\localtestrun.testrunconfig" />

Answer 14

You'll need to specify a .testrunconfig file by setting the RunConfigFile property in a PropertyGroup in your tfsbuild.proj file.

 

Buck

Answer 15

I've been struggling with this issue for the last couple of days. It looks like the RunConfigFile property in a PropertyGroup works file as long as you have a single .testrunconfig file that works for all solutions.

 

Is there any way to specify a different .testrunconfig file for each of the solutions/test.dll's (that the TestToolsTask allows) given that there is nolonger a reference to these in the .vsmdi files?

 

Still using TFS 2005 at this point, will consider 2008 if that's the only way to fix this problem.

 

Thanks,

David