Is it possible to programmatically access code coverage data?

Question

Hi,

I am working with the unit testing system in Team System for Dev. So far it's been great except for one small issue: programmatic access to results. Our build process (a custom solution that has evolved over the years and is unlikely to be swapped out in the near future) runs our unit tests using mstest.exe. The resulting build email would ideally show both test results (pass/fail, failure details) and code coverage details.

At this point the build extracts pass/fail and failure detail information by running an xslt on the trx file. This is not ideal, but acceptable for the time being.

The more serious issue we have is that currently the only way we seem to be able to view the code coverage information is through the VS IDE. The code coverage information can be exported from the IDE to produce a nice XML file.

What we would like to do is either access all the test information (including coverage) through some object model provided for this purpose or to be able to export code coverage data to xml programmatically or via an automatable command line tool. Is this possible, and, if so, how?

Thanks

Answer 1

Uffff it would be great, I'm was trying also to access this data (also unit testing results) for a custom checkin policy based on percentage of code covarage (for the moment just for demo purposues) but couldn't find anything, maybe it's just trying to see about artifacts and their Uris, but I couldn't find anything

Answer 2

I just found this: http://blogs.msdn.com/ms_joc/articles/406608.aspx 

It looks like the section at the end should allow us to load and manipulate the data (albeit I haven't actually made it WORK yet ;)

Significant portion:

using System;

using System.Collections.Generic;

using System.Text;

using Microsoft.VisualStudio.CodeCoverage;

 

// You must add a reference to Microsoft.VisualStudio.Coverage.Analysis.dll

 

namespace CoverDump

{

       class DumpProgram

       {

              static void Main(string[] args)

              {

                     // Create a coverage info object from the file

                     CoverageInfo ci = CoverageInfoManager.CreateInfoFromFile("myfile.coverage");

 

                     // Ask for the DataSet.  The parameter must be null

                     CoverageDS data = ci.BuildDataSet(null);

 

                     // Write to XML

                     data.Lines.WriteXml("mylines.xml");

              }

       }

}

Answer 3

Hmmm it seems good, I will try to check tomorrow, I don't have here TFS

Answer 4

Specifically, I get

Microsoft.VisualStudio.CodeCoverage.CoverageCreateException was unhandled
  Message="Error when creating coverage info: Error loading symbol file. Symbol and binary files should be in the same folder as the coverage file or on the symbol path: D:\Code\Gemini\Public Access\Web Server\Web Site\Bin\ACS.Web.Logging.dll."
  Source="Microsoft.VisualStudio.Coverage.Analysis"
  StackTrace:
       at Microsoft.VisualStudio.CodeCoverage.CoverageInfoManager.ReadModData(String fqPath, UInt32 imagesize, UInt32 linktime, UInt32 blockCount)
       at Microsoft.VisualStudio.CodeCoverage.CoverageInfoManager.CreateInfoFromFile(String filename)
       at ConsoleApplication1.Module1.Main() in C:\Documents and Settings\rods\Local Settings\Application Data\Temporary Projects\ConsoleApplication1\Module1.vb:line 18
       at System.AppDomain.nExecuteAssembly(Assembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()

At the CoverageInfoManager.CreateInfoFromFile call, even if I copy the code coverage file into the same directory that all the assemblies and instrumented pdb files are in.

So close...

 

Answer 5

Finally found the missing link in John Cunningham's second article (http://blogs.msdn.com/ms_joc/articles/495996.aspx). Partway through he mentions setting the SymPath and ExePath; Once this is done the CoverageInfo loads up without error :)

The original article (based on beta 2) calls for data.Lines.WriteXml but in the production version data.WriteXml appears to be what we want.

This is the code that works in my test project (Imports Microsoft.VisualStudio.CodeCoverage):

Dim ci As CoverageInfo

Dim data As CoverageDS

' Create a coverage info object from the file

CoverageInfoManager.SymPath = "D:\Code\Gemini\Public Access\Web Server\Unit Tests\TestResults\rods_TALLGUY 2006-02-28 10_02_48\Out\"

CoverageInfoManager.ExePath = "D:\Code\Gemini\Public Access\Web Server\Unit Tests\TestResults\testusr_TALLGUY 2006-02-28 10_02_48\Out\"

ci = CoverageInfoManager.CreateInfoFromFile("D:\Code\Gemini\Public Access\Web Server\Unit Tests\TestResults\testusr_TALLGUY 2006-02-28 10_02_48\Out\data.coverage")

' Ask for the DataSet. The parameter must be null

data = ci.BuildDataSet(Nothing)

' Write to XML

data.WriteXml("D:\mylines.xml")

 

Answer 6

Hmmm nice sample

Answer 7

I was going to reply to this thread earlier, but in the meantime you seem to have answered your own question . Joc's article is the best place to start with if you want to either programatically collect or analyze code coverage data. Just post back here if you have any more issue with the code coverage tool.

 

Thanks,

Ian

Answer 8

These posts have been very helpful, but for some reason, my xml file is empty. All that is in there is the following:

<?xml version="1.0" standalone="yes"?>
<CoverageDSPriv />

Any ideas?

Answer 9

Nevermind, I found my mistake. Instead of calling "data.WriteXml()" I was calling "data.Lines.WriteXml()". I think I was messing around and forgot to change it back. Sorry, but thanks again for these posts.

Answer 10

my excuses. it's a nice sample indeed. but i have one question: is there any ways to get *.coverage file from the *.xml file created by this snippet above?

Answer 11

After getting the XML, what is the best way, programmatically, to go over the file and get relevant information form it?

Can't I do the same using the CoverageDS class without generating the xml? Is it necessary?

I want, for example to analyze the coverage of specific binaries, some them up and get a final result.

 

Thanks,

Dafna

 

Answer 12

I did the same:

 

CoverageInfoManager.ExePath = ExePath;

CoverageInfoManager.SymPath = SymPath;

// Create a coverage info object from the file

CoverageInfo ci = CoverageInfoManager.CreateInfoFromFile(CoverageDataPath);

// Ask for the DataSet. The parameter must be null

CoverageData = ci.BuildDataSet(null);

 

and got the exception:

Error when creating coverage info: Error loading symbol file. Symbol and binary files should be in the same folder as the coverage file or on the symbol path: C:\Users\dafnaf\Documents\Visual Studio 2005\Projects\CodeCoverageAnalizer\BasicCoverageTest\bin\Debug\BasicCoverageTest.exe.

 

 

What am I doing wrong?

 

Would really appriciate the help.

 

Thanks.

Dafna

Answer 13

Hi Dafna,

 

We'll handle this issue internally.  Once resolved, we'll post the findings back here.

 

Regards,

Chris

 

Answer 14

Hi Dafna,
Please check for reference in your project you need to have following references in your project to get it working
1) C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\Microsoft.DbgHelp.manifest
2) C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.Coverage.Analysis.dll

Once these reference are added to project following lines creates XML

 using Microsoft.VisualStudio.CodeCoverage;

            CoverageInfoManager.SymPath = @"D:\Code\Gemini\Public Access\Web Server\Unit Tests\TestResults\rods_TALLGUY 2006-02-28 10_02_48\Out\ ";

            CoverageInfoManager.ExePath = @"D:\Code\Gemini\Public Access\Web Server\Unit Tests\TestResults\rods_TALLGUY 2006-02-28 10_02_48\Out\ ";
            // Create a coverage info object from the file
            CoverageInfo ci = CoverageInfoManager.CreateInfoFromFile(@"D:\Code\Gemini\Public Access\Web Server\Unit Tests\TestResults\rods_TALLGUY 2006-02-28 10_02_48\Out\ mytest.coverage");

           
            // Ask for the DataSet
            // The parameter must be null
            CoverageDS data = ci.BuildDataSet(null);

            // Write to XML
            data.ExportXml(@"c:\mylines.xml");

Thanks and Regards,
Nishant

Answer 15

I am following the discussion on this thread. I am a Unix guy and have been able to generate code coverage results using 'gcov' very successfully. I am trying to use MSVC9 Team Server edition 2008 to instrument and generate code coverage data for my executable and succeeed so far. I have my code coverage file as C:\mycoverage.coverage.

I am a NOT AT ALL FAMILAIR with Microsoft technologies person.

I also need to export my code coverage data from coverage file to XML or HTML.

I am so dumb with MS technologies here that I don't even know in what language you ate discussing the above example.

Can you please provide a walk through of writing a program in the right language to create the executable that converts data from coverage file to XML

Thanks