locked
VS2012 Code Coverage - including attributes? RRS feed

  • Question

  • I've recently migrated to VS2012 and have been looking at the new way of specifying code coverage settings using the .runsettings file and I'm a bit confused about how all the includes/excludes work together.

    One project in my solution only has unit tests for 2 classes, so I don't want the code coverage looking at the rest of this project.  What I'm attempting to do is have this assembly in the Exclude node of ModulePaths so that it isn't picked up for the most part, then have an Include node under Attributes for a new attribute which I can put on the 2 classes I want covered.  However, as soon as I do this my code coverage drops right down and only a few classes are included.  The same behaviour is seen if I change the name of the attribute in the Include node to one which doesn't exist, leading me to believe it either doesn't accept this approach or that I'm doing it wrong.

    Is it actually possible to use the Include node in nodes other than ModulePaths?  It seems possible to set up conflicting settings, for example excluding a file using ModulePath but then including a method in this assembly using the Function node.  How are these nodes processed?  Do you have to set up every assembly you want included in the ModulePaths node first?  If so, what's the point in the Include node under Functions in the example on this page?
    Thursday, August 23, 2012 7:41 PM

Answers

  • Hello Martyn,

    According to the link you provided in the initial post, you can see the following statement:

    If <Include> is empty, then code coverage processing includes all assemblies (.dll and .exe files) that are loaded and for which .pdb files can be found, except for items that match a clause in an <Exclude> list.

    So I think you are right that you can see the code coverage result when letting Include empty.

    In addition, on my computer, if the attribute, function or modulepath does not exist or not be used, the code coverage result is empty.

    Maybe it is the reason from the link you provided above:

    If there is an error in a regular expression, such as an unescaped and unmatched parenthesis, then code coverage analysis will not run.

    If you want to exclude other 98 classes, as my personal point of view, you have to add Exclude manually.

    Because you include its dll, it means that you include all classes of the project.

    I know it is not the real solution for this issue, to help you resolve this issue, you could submit this feature request:

    http://visualstudio.uservoice.com/forums/121579-visual-studio

    The Visual Studio product team is listening to user voice there. You can send your ideas/suggestions there and people can vote. Thanks for your understanding.

    Best regards,


    Amanda Zhu [MSFT]
    MSDN Community Support | Feedback to us

    • Marked as answer by Amanda Zhu Thursday, August 30, 2012 4:01 AM
    Wednesday, August 29, 2012 11:06 AM

All replies

  • Hi Martyn.S,

    Thank you for posting in the MSDN forum.

    Based on your description, do you put your unit test project assembly in the Exclude node of ModulePaths? If so, I think that this assembly will be excluded from code coverage analysis, no matter what you do in an Include node under Attributes or Functions later.

    Actually, the Include node can be used out of ModulePaths. You can use an Include node under Attributes or Functions. But in order to make it sense, you need to include your assembly in ModulePaths node first.

    You wrote that “It seems possible to set up conflicting settings, for example excluding a file using ModulePath but then including a method in this assembly using the Function node.” Actually, if you set up your assembly you want included in the ModulePaths node first, it will benot a conflict setting.

    And using Include and Exclude nodes under Function node provide a flexible way to let us filter the methods you want to collect code coverage data.

    I hope this will be helpful to you.

    Best regards,


    Amanda Zhu [MSFT]
    MSDN Community Support | Feedback to us

    Friday, August 24, 2012 3:48 AM
  • Thanks for the reply.

    From what I gather then ModulePaths is the main section that defines what assemblies will and won't be included, then the other sections allow further filtering of this.  I assume that using Include under the other nodes has the same behaviour as in ModulePaths in so far as if anything is present this becomes the only thing included?  ie, if I have nothing under ModulePaths to imply everything and a single attribute under the Attributes/Include node then only code with this attribute should be covered and the rest of the solution should be ignored?  I kind of see this behaviour as when I add an attribute most of my coverage drops out, however not all of it does.  Even adding the name of an attribute that doesn't exist still gives some code coverage which I don't understand?  I would have expected none if I'm specifying an attribute which doesn't exist and therefore won't be present on any code.

    Ideally what I'd like is to have full code coverage for the majority of my solution, but within one assembly just the two classes.  I don't see this being possible as it would be special casing one assembly (unless I went through and added ExcludeFromCodeCoverage to every other class in there).  It's not a big problem at all, I can just exclude this whole assembly like when I was using VS2010 but I just thought that with this new settings file it may have been possible.

    Friday, August 24, 2012 10:51 AM
  • Hello Martyn,

    If you have nothing under ModulePaths to imply something and a single attribute under the Attributes/Include node, you will see the Empty results generated in Code Coverage results window.

    Based on your description, do you mean that you see some corresponding code coverage when you include an attribute which does not exist in test class?

    In addition, I would like to know if you don’t want to see the code coverage result of some classes which are not test classes. If so, you can include the .dll under ModulePaths like this:

    <ModulePaths>

     <Include>

    <ModulePath>.*UnitTestProject1.dll$</ModulePath>

        </Include>

    </ModulePaths>

    Then under Functions node, you can exclude some classes you don’t want to collect code coverage like this:

    <Functions>

      <Exclude>

          <Function>^UnitTestProject1\.Class1\..*</Function>

    </Exclude>

     </Functions>

    Note that you need to care the regular expression. If there is an error in a regular expression, such as an unescaped and unmatched parenthesis, then code coverage analysis will not run.

    I hope this will be helpful to you.

    Best regards,


    Amanda Zhu [MSFT]
    MSDN Community Support | Feedback to us

    Tuesday, August 28, 2012 6:54 AM
  • If you have nothing under ModulePaths to imply something and a single attribute under the Attributes/Include node, you will see the Empty results generated in Code Coverage results window.

    This is what I was expecting to see, however, using either of the following setups my results are not empty - a few thousand blocks (very few in comparison to the whole solution) show up as covered, loads more show up as not covered, and many namespaces/assemblies are missing entirely (how they all should be).

    <ModulePaths>
        <Include>
        </Include>
    </ModulePaths>

    <Attributes>
        <Include>
            <Attribute>^AttributeThatDoesntExist$</Attribute>
        </Include>
    </Attributes>

    ---

    <ModulePaths>
        <Include>
        </Include>
    </ModulePaths>

    <Attributes>
        <Include>
            <Attribute>^AttributeThatExistsButIsntUsed$</Attribute>
        </Include>
    </Attributes>

    Basically, I can type anything into an Attribute/Include node and still get coverage.

    Regarding what I'd like covered, assume my solution has 30 non-test projects (I'm not interested in code coverage for the test classes).  Assume each of these 30 projects have 100 classes.  I'd like 29 of the projects to have full coverage enabled, and the last one to only have 2 of it's classes covered.  In other words, 2902 classes should be covered, 98 shouldn't.  I therefore can't exclude project 30 from coverage entirely as the 2 classes I want covered won't be, but at the same time I can't add the ExcludeFromCodeCoverage attribute to my other 98 classes as this would take forever.

    I hope this makes sense!

    Tuesday, August 28, 2012 11:17 PM
  • Hello Martyn,

    According to the link you provided in the initial post, you can see the following statement:

    If <Include> is empty, then code coverage processing includes all assemblies (.dll and .exe files) that are loaded and for which .pdb files can be found, except for items that match a clause in an <Exclude> list.

    So I think you are right that you can see the code coverage result when letting Include empty.

    In addition, on my computer, if the attribute, function or modulepath does not exist or not be used, the code coverage result is empty.

    Maybe it is the reason from the link you provided above:

    If there is an error in a regular expression, such as an unescaped and unmatched parenthesis, then code coverage analysis will not run.

    If you want to exclude other 98 classes, as my personal point of view, you have to add Exclude manually.

    Because you include its dll, it means that you include all classes of the project.

    I know it is not the real solution for this issue, to help you resolve this issue, you could submit this feature request:

    http://visualstudio.uservoice.com/forums/121579-visual-studio

    The Visual Studio product team is listening to user voice there. You can send your ideas/suggestions there and people can vote. Thanks for your understanding.

    Best regards,


    Amanda Zhu [MSFT]
    MSDN Community Support | Feedback to us

    • Marked as answer by Amanda Zhu Thursday, August 30, 2012 4:01 AM
    Wednesday, August 29, 2012 11:06 AM
  • Thanks for the response, I'll have a look at submitting the feature.
    Sunday, September 2, 2012 11:53 AM