none
TestCategory attributes of xUnit tests do not end up in TRX file on TFS 2015.2

    Question

  • Hello,

    I try to achieve something like the following: on test methods, I would like to specify traits, which reflect e.g. the test-case number on TFS, which this unit test is "implementing". When I then run and execute the tests on TFS, I would like to have the .TRX file reflecting these specified traits as TestCategory elements, to be able to display this column in VS test results or on TFS. However, these attributes are not transformed/mapped to the TRX file.

    When I have MSTest unit tests in my tests, the TestCategory attribute becomes mapped into the TRX result file and later on can be displayed like the following:

    In the TRX file, the corresponding section can be found in the section of the XML.


    After reading the following post on the ALM blog, I actually assumed, that the xUnit Trait attribute becomes mapped to the TestCategory attribute of MSTest and then is also available in the TRX file.

    See: https://blogs.msdn.microsoft.com/visualstudioalm/2013/06/11/part-3-unit-testing-with-traits-and-code-coverage-in-visual-studio-2012-using-the-tfs-build-and-the-new-nuget-adapter-approach/


    As it seems, this is not done. I checked some of the code of the test adapter of xUnit, but have not found anything, which would map the trait to a test-category item. 

    I already tried traits with the name: "TestCategory", "Category", "Trait", etc.

    I opened a new xUnit issue on github, but got redirected to the TFS community, as the xUnit Test Adapter seems to do the right things. See https://github.com/xunit/xunit/issues/1052 

    Can somebody probably explain to me, what is happening here?

    Thanks for any help regarding this!

    Best regards,

    Harald

    Tuesday, December 20, 2016 10:09 AM

All replies

  • Hi Harald,

    After reading your description, here is my opinion:

    First, I could not use mstest or vstest.console to run the xUnit test method in my side, please share me you command. I could only use xunit.console to run the xUnit test method, just link this command line:

    xunit.console "C:\Users\Administrator\Documents\Visual Studio 2015\Projects\UnitTestProject1\UnitTestProject1\bin\Debug\UnitTestProject1.dll"

    >>I already tried traits with the name: "TestCategory", "Category", "Trait", etc.

    If you want to see TestCategory property in trx file, please refer to the following step:

    1). Add TestCategory attribute before TestMethod, just like the following code:

            [TestMethod]
            [TestCategory("123")]
            public void TestMethod1()
            {
    
    
            }

    2). Then with Mstest to run this test like the following command:

    mstest /testcontainer:UnitTestProject1.dll /category:"123"

    3). Then open trx file in VS, we could see the testcategory property, please refer to the following screenshot:

    But how to implement it with xunit.console?

    If I misunderstand your issue, please feel free to let me know. If you could implement it in VS, and TFS could be able too.

    Sincerely,

    Oscar


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.



    Wednesday, December 21, 2016 5:57 AM
    Moderator
  • Hello Oscar,

    thank you very much for your fast reply. 

    You cannot run xUnit tests with the mstest runner nor the vstest.console directly, unless you specify the test-adapter path. But running the tests is not the problem. The problem really is, that xUnit [Trait] attributes applied on xUnit tests are not mapped to the [TestCategory] in the TRX files, as actually documented by MS. The Test Adapter of xUnit does forward these attributes to the vstest.console and MSTest Engine, but they do not show up in the TRX file afterwards. I need xUnit Traits, because I have xUnit tests, not MSTests.

    For TFS, the test adapter path gets "injected" automatically. On VS, you need to install the xUnit.net Runner for VS. My generated command there for the Test Step looks like the following.

    vstest.console.exe "D:\TfsData\Build\DefaultAgent\_work\7\s\_bin\x64\Release\IF.XYZ._Tests.dll" "D:\TfsData\Build\DefaultAgent\_work\7\s\_bin\x64\Release\IF.XYZ.Acceptance._Tests.dll" "D:\TfsData\Build\DefaultAgent\_work\7\s\_bin\x64\Release\IF.XYZ.Application._Tests.dll"  /InIsolation /Platform:x64 /logger:trx /TestAdapterPath:"D:\TfsData\Build\DefaultAgent\_work\7\s"

    According to the xUnit guys, all traits are pushed to the VS API, but they do not end up in the TRX file afterwards. Please feel free to also have a look at the original issue on GitHub for xUnit, which I created: https://github.com/xunit/xunit/issues/1052

    Best regards,

    Harald

    Wednesday, December 21, 2016 7:37 AM
  • Hi Harald,

    I think I could repro your issue in my side, here is my test method:

      

    [Fact]
    [Trait("TFS-TC","1234")]
    [Trait("Category","Trait-TC-1234")]
    [Trait("TestCategory","TestXunit")]
    public void xUnit_TestMethod()
    {
      Assert.Equal(1,1);
    }

         

    When I run it from command line with Test Adapter:

    "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" "C:\Users\Administrator\Documents\Visual Studio 2015\Projects\UnitTestProject1\UnitTestProject1\bin\Debug\UnitTestProject1.dll" /logger:trx /TestAdapterPath:"."

    I get the following result:

    I think vstest.console not support the attribute from the xUnit by now.

    Against this issue, I help you submitted a User Voice to the VS connect, I think engineers from Microsoft will evaluate this issue seriously.

    You could vote and comment it, and would you please mark it as answer, which could help other who has the same issue as yours to vote and comment it.

    https://visualstudio.uservoice.com/forums/121579-visual-studio-ide/suggestions/17508760-vstest-console-not-support-add-attribute-of-xunit

    Sincerely,

    Oscar


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    Wednesday, December 21, 2016 8:52 AM
    Moderator