locked
Code Coverage does not run on the build server. RRS feed

  • Question

  • Hello Our Product has chosen Visual Studio's Unit Testing and Code Coverage Tools.

    I have impressed my manager with the reports and a small utility that compiles the reports of multiple coverage xml files.

    Now that it has become time to automate and deploy to a buidl server I am running inot problems.

    The Build Server is TeamCity

    I did install Visual Studio Premium 2012 onto the build server.

    The tests do run and are imported into TeamCity

    However if I take the generated code coverage fiel offf the build system and import it it says it yielded empty results.

    When I look in the event log of the build server I see three entries.

    1.

    The TraceLog Profiler in process vstest.executionengine.exe logger not running after being started, command line "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\IntelliTrace\11.0.0\IntelliTrace.EXE" run /n:"vstest.executionengine.exe_00001994_01d0f21442cde117" /cp:"C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\VisualStudio\11.0\TraceDebugger\Settings\qzjik1ir.xxv" /f:"vstest.executionengine_00001994_150918_151642.iTrace". 

    2.

    TraceLog Profiler failed initialization, process vstest.executionengine.exe

    3.

    .NET Runtime version 4.0.30319.0 - Loading profiler failed.  The profiler COM object was instantiated, but the profiler failed during its initialization callback.  Profiler CLSID: '{b19f184a-cc62-4137-9a6f-af0f91730165}'.  HRESULT: 0x80004005.  Process ID (decimal): 6548.  Message ID: [0x2505].

    To clarify this is vstest and the microsoft code coverage technology.

    I am not debugging a custom profiler I am using a MS product.

    Please help.

    Thank you for your time.

    -Christopher Buteau

    Friday, September 18, 2015 1:42 PM

All replies

  • Update 

    I created an msbuild script that runs vstest.console.exe explicitly.

    It did this using the CodeTaskFactory and a build Task that managed a process.

    This generated coverage data.

    I debugged the script in a VS Command Line window.

    When I plugged it into the TeamCity step I got the same errors in my event log and an empty coverage file.

    Thank you for your time.

    -Christopher Buteau

    Friday, September 18, 2015 6:26 PM
  • Hi OnlyKingmob,

    >> However if I take the generated code coverage fiel offf the build system and import it it says it yielded empty results.

    Please provide the detail steps.

    To analyze code coverage in the build service, we change Type of run settings to CodeCoverageEnabled.

    # Using Code Coverage to Determine How Much Code is being Tested

    https://msdn.microsoft.com/en-us/library/dd537628(v=vs.110).aspx

    What’s the result of tests during the build? Are there any exception during test?

    Regards

    Starain


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, September 21, 2015 2:46 AM
  • I am not using the CodeCoverageEnabled switch.

    I am using a runsettings file with code coverage enabled.

    https://msdn.microsoft.com/en-us/library/jj635153.aspx

    The Tests all pass on the build.

    There are errors in the event log which is the only hint (besides the empty report).

    I provided details about these errors in the opening post.

    The detailed steps are as follows.

    ================

    In TeamCity.

    I configure a step to build the tests solution.

    Following it I configure a step to run Visual Studio Tests with the VSTEst engine for Visual Studio 2012.  I configure it to point at a runsettings file.

    The tests pass and the log file reports an attachment which is a test results report.

    When I copy said report to my development system it reports empty results.

    Monday, September 21, 2015 11:37 AM
  • The detailed steps.

    On a TeamCity Server.

    I have a build configuration.

    It's first step is to do a NuGet refresh based on a solution file.

    It's second step does a build based on the same solution.

    The third step executes tests using the Visul Studio Tests Runner using the VSTEST engine.

    The tests run and the log file reports an attachment.

    [14:50:33][Step 3/4] Attachments:
    [14:50:33][Step 3/4] C:\BuildAgentThree\work\1d051bdcea5ceb79\3DSpaceConnector\Features\UnitTesting\SolidWorks.PDSAddin\Testing.Plugin\TestResults\85d59892-3c0c-432b-b0e4-7cd8a6b66261\SYSTEM_STO-JANSON-SWK 2015-09-21 14_50_24.coverage

    I copy said attachment to my development machine and it reports empty results.


    • Edited by OnlyKingmob Monday, September 21, 2015 6:59 PM
    Monday, September 21, 2015 1:23 PM
  • Hi OnlyKingmob,

    >> I have a build configuration.

    Is it a build definition? If not, what is it?

    >> It's second step does a build based on the same solution.

    Do you mean build solution through Visual Studio?

    >> The third step executes tests using the Visul Studio Tests Runner using the VSTEST engine

    Do you mean you just execute tests though Visual Studio?

    Regards

    Starain


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Tuesday, September 22, 2015 5:31 AM
  • On TeamCity it is a configuration.

    Did you miss the TeamCity part? or are you just trying to point out TFS has CI abilities.

    I am not building through visual studio.

    I believe it points the solution at msbuild.

    No it is not executing from Visual Studio.

    It appears to be looking up the vstest.console.exe and executing that.

    [19:47:23][Step 3/4] VSTest reporting tests in real-time
    [19:47:23][Step 3/4] VSTest executable: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe

    The output claims it generated a report but it is empty.

    [19:48:19][Step 3/4] Attachments:
    [19:48:19][Step 3/4] C:\BuildAgentThree\work\1d051bdcea5ceb79\3DSpaceConnector\Features\UnitTesting\SolidWorks.PDSAddin\Testing.Plugin\TestResults\2b9cd960-95c1-4d3c-865d-130a6690f367\SYSTEM_STO-JANSON-SWK 2015-09-21 19_48_09.coverage

    So to summarize.

    When I execute the tests in visual studio I see a coverage report.

    When it is executed on the CI system (TeamCity) the report file is generated but it reports empty results.

    Can you actually help me or should I open a Microsoft Connect issue?

    Tuesday, September 22, 2015 12:08 PM
  • Hi OnlyKingmob,

    What’s the result if you try it in other machine?

    What’s the result if you run the test through VSTest.Console.exe with this command?

    vstest.console.exe MyTestAssembly.dll /EnableCodeCoverage 

    Regards

    Starain


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, September 23, 2015 2:18 AM
  • Other machines work fine.

    I use the runsettings file instead of the /EnableCodeCoverage switch.

    vstest.console.exe MyTestAssembly.DLL /Settings:MyTestAssembly.runsettings

    I have an msbuild script which runs this command.

    It works from the command line and generates a report.

    But I plugged it (the script) into a build step and it generates an empty report and I get the errors in the Event Log.

    Wednesday, September 23, 2015 11:52 AM
  • Hi OnlyKingmob,

    Is the TeamCity a test tool and you have issue when do test through that tool?

    Regards

    Starain


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, September 24, 2015 2:24 AM
  • TeamCity is a CI system.

    And yes the issue occurs when running the tests in that tool.

    The tests run but the coverage report is always empty.

    It is looking like we will evaluate another tool.

    Friday, October 2, 2015 7:06 PM
  • Hi,

    I don’t familiar with TeamCity tool, since the issue occurs when running the tests in that tool, I think it is related to that tool. I suggest that you could contact TeamCity support engineer or post it to TeamCity site/forum.

    Regards

    Starain


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, October 5, 2015 1:16 AM
  • TeamCity is running the command correctly.   It is applying the runsettings file I created.

    A report is being generated it is empty.

    That doesn't sound like TeamCity's fault at all.

    It sound like the code coverage tool cannot be automated.


    Monday, October 5, 2015 2:27 PM
  • Hi,

    >>TeamCity is running the command correctly.

    But you said TraceLog Profiler failed initialization, process vstest.executionengine.exe.

    Tuesday, October 6, 2015 2:13 AM
  • Was there a question in there?

    TeamCity was obviously running the command correctly because the tests actually ran and it was generating a report (albeit empty).

    I assumed the problem had to do with the initialization errors I kept seeing in the event log.

    I never resolved it and it looks like we might use dotTrace instead.

    Wednesday, October 7, 2015 7:18 PM
  • Hi,

    Could TeamCity run as administrator?

    Regards

    Starain


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, October 8, 2015 8:38 AM
  • It is running as a service under system.

    My understanding was system was pretty high in the ladder of user privileges.

    I am not going to waste cycles getting an admin account to apply to see if works.

    MS Code Coverage works on the desktop but fails on the build server.

    Wednesday, October 21, 2015 5:57 PM