locked
Inconsistent Code Coverage Results RRS feed

  • Question

  • Hi All,

     

    I've been having some issues with code coverage over the past few weeks and I was hoping that someone here might be able to shed some light on the situation.

     

    I've been tasked with gathering coverage information from our test environment, which consists of 4 web and 4 application servers. All dlls are generated on a build machine, instrumented, re-signed (if necessary) and then deployed to all of the servers. A script runs on all of the servers which kicks off coverage and then stops it again at some later time after testing has been done.

     

    Initially I was getting good results every time while building everything in debug mode. I later began building everything in release mode with debug symbols because of issues with conditional debug statements, which is when the problems started.

     

    The problem I am experiencing is that (seemingly at random) when I try to import the results into Visual Studio I get the cryptic error message: "Error when querying coverage data: Invalid symbol data." I've read that this can be caused by inconsistent dll versions, but since I am doing a clean build + deploy each time, I know this is not the case. The strangest part is that while all 4 web+app server pairs have the exact same set of assemblies, I will be able to import coverage from some and not others.

     

    This is all being done on Server 2003 Standard SP2.

     

    One other thing I noticed is that in SP2, vsinstr spits out the message:

     

    "Warning VSP2014 : The resulting instrumented image appears to be invalid, and may not run." when instrumenting assemblies, although the seem to run fine. Could this be part of the problem? In SP1, the only message that was displayed was "Warning VSP2013 : Instrumenting this image requires it to run as a 32-bit process.  The CLR header flags have been updated to reflect this."

     

    Thanks in advance.

    Thursday, August 2, 2007 3:17 PM

Answers

  • Hi Guillermo,

     

    Since there seem to be so many things that could potentially be breaking here, I decided to attack one thing at a time. I explicitly built a test assembly for x86 and then attempted to run vsinstr against it on the build machine like so:

     

    vsinstr /coverage c:\testasm.dll

     

    Which results in the following:

     

    Successfully instrumented file c:\testasm.dll.
    Warning VSP2014 : The resulting instrumented image appears to be invalid, and may not run.

     

    If I do not use the /coverage flag, I get no warning.

     

    This machine has VS2005 Team Edition for Software Developers w/SP1 installed. Operating system is 2003 Standard w/SP2.

     

    I decided to experiment on my own machine, which is running VS2005 Team Suite w/SP1 for Vista installed. Operating system is Vista Business.

     

    This time, the exact same assembly was instrumented with no warnings. Out of curiosity, I checked the version numbers of vsinstr and VSCover80.

     

    On the 2003 machine, they were 8.0.50727.762.

    On the Vista machine, they were 8.0.50727.867.

     

    If I copy the Microsoft Visual Studio 8\Team Tools\Performance Tools directory over to the 2003 machine, the instrumentation succeeds with no errors.

     

    Were there significant changes between these versions or were they Vista-only fixes? Unfortunately I cannot experiment with actual coverage testing using the newer version at the moment, but if there are significant differences between these versions it would be good to know.

     

    Thanks,

     

    George 

    Thursday, September 20, 2007 9:17 PM

All replies

  •  

    I should probably also add that all of these instrumented assemblies are running under IIS 6.
    Friday, August 3, 2007 2:41 PM
  • Any ideas on this? I really wish the BuildDataSet method on CoverageInfo would throw a more useful exception than "Invalid symbol data."

     

    Wednesday, August 15, 2007 9:21 PM
  • Hi George,

     

    Are you still looking for an answer to this?

     

    thanks

    Guillermo Serrato

    VS Team Test

    Monday, September 17, 2007 6:11 PM
  • Hi Guillermo,

     

    I have not had much time recently to work on this particular problem, so I am not much further along in my own investigation then when I initially posted the question. I am, however, definitley still looking for an answer. Let me know if you have any questions or need more specific details about the issue; I'll be more than happy to provide answers/information.

     

    Thanks,

     

    George

     

    Tuesday, September 18, 2007 8:33 PM
  • Hi George,

     

    I would recommend you take a look at these other threads:

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2011715&SiteID=1

    http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=512702&SiteID=1

     

    Also, looking at the warning that you show, it is a warnint to having an invalid PE header in the dll. It seems to me that it could be related to the fact that you are using 64 bit binaries.

     

    There's also issues with the combination of release and deploy.

     

    Were you using SP1 before also?

     

    Hope this helps

    Guillermo Serrato

    VS Team Test

     

    Tuesday, September 18, 2007 9:47 PM
  • Hi Guillermo,

     

    Since there seem to be so many things that could potentially be breaking here, I decided to attack one thing at a time. I explicitly built a test assembly for x86 and then attempted to run vsinstr against it on the build machine like so:

     

    vsinstr /coverage c:\testasm.dll

     

    Which results in the following:

     

    Successfully instrumented file c:\testasm.dll.
    Warning VSP2014 : The resulting instrumented image appears to be invalid, and may not run.

     

    If I do not use the /coverage flag, I get no warning.

     

    This machine has VS2005 Team Edition for Software Developers w/SP1 installed. Operating system is 2003 Standard w/SP2.

     

    I decided to experiment on my own machine, which is running VS2005 Team Suite w/SP1 for Vista installed. Operating system is Vista Business.

     

    This time, the exact same assembly was instrumented with no warnings. Out of curiosity, I checked the version numbers of vsinstr and VSCover80.

     

    On the 2003 machine, they were 8.0.50727.762.

    On the Vista machine, they were 8.0.50727.867.

     

    If I copy the Microsoft Visual Studio 8\Team Tools\Performance Tools directory over to the 2003 machine, the instrumentation succeeds with no errors.

     

    Were there significant changes between these versions or were they Vista-only fixes? Unfortunately I cannot experiment with actual coverage testing using the newer version at the moment, but if there are significant differences between these versions it would be good to know.

     

    Thanks,

     

    George 

    Thursday, September 20, 2007 9:17 PM
  •  

    Hey George,

     

    The Vista changes should not change anything other than vista related issues. Actually for vista you install SP1 then the Vista part of SP1 so it would be interesting to know why it is behaving like this. I will let you know if I find out about anything. Hope your workaround can let you do your work.

     

    thanks

    Guillermo Serrato

    VS Team Test

    Wednesday, October 10, 2007 12:18 AM
  • I am also getting this issue (using TSD 2005).  Could this be an issue that was fixed in Suite but not Developer??

     

    Are there any workarounds??

     

    Monday, October 15, 2007 7:13 PM
  • I went through the same steps and was able to reproduce this 'warning' message. I have the exact same build environment; one hosted on a windows xp pro sp2 box, the other with windows 2003 server r2. Running on the winxp box I get no instrumentation errors. Running on windows 2003 server r2 I get the VS2014 warning.

    This test was run on a very simple solution, one assembly and one test project, with a single unit test.

    I have set the compile options many different ways and verified that under every different type of compile option (release|debug;x86|all;pdbonly|default;debugsymbols on|off) I still get the VSP2014 error. Any news on if this is going to be ‘fixed’ for 2005, or maybe in 2008?

    Tuesday, October 30, 2007 12:23 AM
  • I had the same problem, so I did the copy from a developer Vista box to the Win2K3 Build Server.

     

    Now I see this:

     

    Code Block
    C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v8.0\TeamBuild\Microsoft.TeamFoundation.Build.targets : warning : Code coverage instrumentation warning while processing file xxxxx.dll:
    C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v8.0\TeamBuild\Microsoft.TeamFoundation.Build.targets : warning : Warning VSP2013 : Instrumenting this image requires it to run as a 32-bit process.  The CLR header flags have been updated to reflect this.

     

     

    I then found this about the VSP2013 errors: http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=2185627&SiteID=1

     

    I changed the build to target the x86 platform as per that post and the tests now work and there are no VSP warning messages.

     

    The only problem I now have is that I'm seeing a message in the build indicating that "Coverage results are out of date. Results merge is in progress...".  The message is still there when the build has completed.

     

    - Richard

    Thursday, November 1, 2007 1:02 AM
  • Hey Guillermo Serrato,

     

    I am also facing the same problem when instrumenting the a test binary using the vsinstr .

    OS : win 2003,  version of vsinstr.exe: is 8.0.50727.762 built by: SP(DDBLD307-dlab) 

     

    D:\Program Files\Microsoft Visual Studio 8\Team Tools\Performance Tools>vsinstr /Coverage "D:\MSIT\Learning\Video's\Code Coverage\SayHello - Code Coverage - VB amd CS\SayHello_CS\Bin\TestBO.dll"
    Microsoft (R) VSInstr Post-Link Instrumentation 8.0.50727 x86
    Copyright (C) Microsoft Corp. All rights reserved.

    File to Process:
       D:\MSIT\Learning\Video's\Code Coverage\SayHello - Code Coverage - VB amd CS\S
    ayHello_CS\Bin\TestBO.dll --> D:\MSIT\Learning\Video's\Code Coverage\SayHello -
    Code Coverage - VB amd CS\SayHello_CS\Bin\TestBO.dll
    Original file backed up to D:\MSIT\Learning\Video's\Code Coverage\SayHello - Cod
    e Coverage - VB amd CS\SayHello_CS\Bin\TestBO.dll.orig

    Successfully instrumented file D:\MSIT\Learning\Video's\Code Coverage\SayHello -
     Code Coverage - VB amd CS\SayHello_CS\Bin\TestBO.dll.
    Warning VSP2014 : The resulting instrumented image appears to be invalid, and ma
    y not run.

    TestBO.dll is simple Dotnet DLL, i have tried various option of platform target and for each case got the error.

    Could you please if this was any with issue vsinstr and is it Fixed.

    thanks,

    Prabhakar

    Monday, April 21, 2008 10:23 AM