locked
How do I make multiple results from my custom TestMethodAttribute to show in Test Explorer? RRS feed

  • Question

  • I have made a custom test method attribute, that uses enumeration of embedded resources to generate multiple results for the same test like this: 

    [AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
    public sealed class ResourcesTestAttribute : TestMethodAttribute
    {
        readonly string resourcesPrefix;
    
        public ResourcesTestAttribute(string resourcesPrefix)
        {
            this.resourcesPrefix = resourcesPrefix ?? throw new ArgumentNullException(nameof(resourcesPrefix));
        }
    
        public override TestResult[] Execute(ITestMethod test)
        {
            Assembly assembly = test.MethodInfo.DeclaringType.Assembly;
            TestResult[] results = assembly.GetManifestResourceNames()
                .Where(name => name.StartsWith(this.resourcesPrefix))
                .Select((name, index) => {
                    var result = test.Invoke(new object[] { ReadResource(assembly, name) });
                    result.DatarowIndex = index;
                    result.DisplayName = name;
                    return result;
                    })
                .ToArray();
            return results;
        }
    
        static string ReadResource(Assembly assembly, string name)
        {
            using (var reader = new StreamReader(assembly.GetManifestResourceStream(name))) {
                return reader.ReadToEnd();
            }
        }
    }

    You could simplify Execute to

    new []{1, 2}.Select(i => test.Invoke(new []{i})).ToArray()

    for the purpose of the example.

    Now the problem I have is when I run a test method marked with [ResourcesTest], Visual Studio only reports the first result (e.g. for int 1) in the Test Explorer window, and claims the test passed despite test for int 2 failing.

    How do I make Visual Studio show results for each data input separately? If it is not possible, can I at least make the test fail when it fails on either of datas without manually merging test results?

    Sunday, February 11, 2018 8:53 AM

Answers

  • I have evaluated data-driven tests as an option, before even trying the custom attribute, and decided, that it is much less convenient for my case. The data would have to go into a DB or XML file, which are harder to work with. I need to edit my test data often in a text editor.

    So I guess I'll have to stick to manual TestResult merging. Here's the snippet:

    static TestResult Merge(TestResult into, TestResult result)
    {
        into.DebugTrace += "\n\n" + result.DebugTrace;
        into.Duration += result.Duration;
        into.LogError += "\n\n" + result.LogError;
        into.LogOutput += "\n\n" + result.LogOutput;
        into.Outcome = result.Outcome == UnitTestOutcome.Failed ? UnitTestOutcome.Failed
            : result.Outcome == UnitTestOutcome.Inconclusive ? UnitTestOutcome.Inconclusive
            : into.Outcome;
        into.TestContextMessages += "\n\n" + result.TestContextMessages;
        into.TestFailureException = into.TestFailureException ?? result.TestFailureException;
        return into;
    }

    Now I just do

    .Aggregate(new TestResult {
          DisplayName = test.TestMethodName,
          Outcome = UnitTestOutcome.Passed },
        (accum, result) => Merge(into: accum, result: result))


    • Edited by lostmsu Monday, February 12, 2018 8:35 AM
    • Proposed as answer by Fletch Zhou Tuesday, February 13, 2018 6:43 AM
    • Marked as answer by lostmsu Tuesday, February 13, 2018 8:27 AM
    Monday, February 12, 2018 8:35 AM

All replies

  • >>How do I make Visual Studio show results for each data input separately? 

    Unfortunately, right now there is no particular functionality or feature in VS could make it directly, but based on your description, I suggest you use data-driven test to implement it:

    https://msdn.microsoft.com/en-us/library/ms182527.aspx?f=255&MSPPError=-2147217396

    http://codeclimber.net.nz/archive/2008/01/18/How-to-simulate-RowTest-with-MS-Test/

     

    Regards,

    Fletcher

    This reply includes a third-party website, MS does not guarantee the safety of any third-party's website.


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    • Edited by Fletch Zhou Monday, February 12, 2018 8:30 AM
    Monday, February 12, 2018 8:28 AM
  • I have evaluated data-driven tests as an option, before even trying the custom attribute, and decided, that it is much less convenient for my case. The data would have to go into a DB or XML file, which are harder to work with. I need to edit my test data often in a text editor.

    So I guess I'll have to stick to manual TestResult merging. Here's the snippet:

    static TestResult Merge(TestResult into, TestResult result)
    {
        into.DebugTrace += "\n\n" + result.DebugTrace;
        into.Duration += result.Duration;
        into.LogError += "\n\n" + result.LogError;
        into.LogOutput += "\n\n" + result.LogOutput;
        into.Outcome = result.Outcome == UnitTestOutcome.Failed ? UnitTestOutcome.Failed
            : result.Outcome == UnitTestOutcome.Inconclusive ? UnitTestOutcome.Inconclusive
            : into.Outcome;
        into.TestContextMessages += "\n\n" + result.TestContextMessages;
        into.TestFailureException = into.TestFailureException ?? result.TestFailureException;
        return into;
    }

    Now I just do

    .Aggregate(new TestResult {
          DisplayName = test.TestMethodName,
          Outcome = UnitTestOutcome.Passed },
        (accum, result) => Merge(into: accum, result: result))


    • Edited by lostmsu Monday, February 12, 2018 8:35 AM
    • Proposed as answer by Fletch Zhou Tuesday, February 13, 2018 6:43 AM
    • Marked as answer by lostmsu Tuesday, February 13, 2018 8:27 AM
    Monday, February 12, 2018 8:35 AM
  • I see, manual TestResult merging seems more convenient for compared to Data-driven test.

    Your could mark your reply as answer which could help others as well.

    Your understanding and cooperation will be grateful.

     

    Regards,

    Fletcher


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    • Edited by Fletch Zhou Tuesday, February 13, 2018 6:30 AM
    Tuesday, February 13, 2018 6:30 AM