locked
Code Coverage and Unit Testing Test Results RRS feed

  • Question

  • Hi

    We are trying to configure a project with check-in policies that makes Unit Testing and Code Coverage of X% mandatory. 

    When we were writing the policies to achieve the same, we found that the code coverage results and the Unit Testing results are stored locally and not on the TFS Server. 

    If it has to be stored on the TFS Server, then it has to be published.

    We actually wanted to validate the unit testing and the code coverage results against the results stored on the TFS server (Basically compare the existing and the previous results against a value set in the check-in policy).  We do not find anything that would help us with such a scenario.

    Regards
    Saturday, September 24, 2005 11:59 AM

Answers

  • WHile there is no automatica way to get the code coverage data into TFS other than as part of a build (When a Team Build completes, it can automatically publish those results), You should be able to compare local numbers with remote numbers. Just write your plugin to query Team Foundation server for the numbers and compare with the local version.

    You will need to make sure that you compared per assembly, rather than overall, since a given set of tests etc my only instrument one assembly depending on end user selection.
    Saturday, September 24, 2005 5:25 PM
  • You can publish code coverage results against a particular team build of your TFS project. For example, for a build 20050924 of MyBuild build type, you can locally run code coverage against that build and publish the test results against that build. Do not forget to check the code coverage check box when you publish the results.

    Once you have that, you can then do a checin policy plugin and query TFS for test results and verify the numbers.

    -Munjal

    Sunday, September 25, 2005 7:02 AM

All replies

  • WHile there is no automatica way to get the code coverage data into TFS other than as part of a build (When a Team Build completes, it can automatically publish those results), You should be able to compare local numbers with remote numbers. Just write your plugin to query Team Foundation server for the numbers and compare with the local version.

    You will need to make sure that you compared per assembly, rather than overall, since a given set of tests etc my only instrument one assembly depending on end user selection.
    Saturday, September 24, 2005 5:25 PM
  • You can publish code coverage results against a particular team build of your TFS project. For example, for a build 20050924 of MyBuild build type, you can locally run code coverage against that build and publish the test results against that build. Do not forget to check the code coverage check box when you publish the results.

    Once you have that, you can then do a checin policy plugin and query TFS for test results and verify the numbers.

    -Munjal

    Sunday, September 25, 2005 7:02 AM
  • I have a feeling that there is lot more to this, lets take a case

    I have a project xyz, and this project has two files a.cs and b.cs.  Now, we have four developers working.  Two on a.cs and two on b.cs.  Each one have developed their own unit test cases for each of their file/methods.

    Now Developer 1, who is working on a.cs runs the test cases against his file/methods and publishes the unit test case results and the code coverage results. 

    Same with Developer 2 

    When the developer (1 or 2) tries to check in, we try to retrieve the test results for the project/file and we need to retrieve the results of all the run of 'unit test cases' of a.cs.  We have not found that the published results do not have a method name.

    ie.,

    First identify the artifact (a.cs) being checked in
    if it is a.cs, then Identify the test cases for the artifact
    Identify the test runs for the artifact and retrieve the results for the test runs
    Identify the test run for a specific method in the artifact
    Identify the perfomance for the specific method/entire artifact

    This I dont think would be possible in the current implementation.  There is currently no way to identify the testrun ID for an artifact. 

    In the case of a team build, I would probably have the same testrun id for multiple projects testcases.  I am assuming this since, the testrunid is associated to a build.

    Please do correct me if I am wrong.

    I have not found any good documentation on how to retrieve data for the published test results.  Could you please direct me to good documentation on the test interface(s) of VSTS.

    Regards

    Sunday, September 25, 2005 12:28 PM
  • While you are correct that there is no simple way to do this -- through some extension it maybe possible for you to achive it through some additional technology.

    From the Operational Store (that is the Team Build Database) you can retrieve a directory that contains the Tests that were run, the deployment items, and coverage (if it was published for that run). With this directory you can load the .coverage files and query the deep coverage information as needed.

    Additionally, within the operational store we store the test name. To me, at least with my understanding, your remaning problem is to work out which test cases are relevant for your product code. This relationship could be created through creation of a custom attribute which you could examine in your checkin policy.

    I assume your ultimate goal here is to ensure that coverage of new code remains high?

    Saturday, October 1, 2005 11:53 PM