Data-Driven Unit Tests only execute the last DataRow from the specified DataSource. RRS feed

  • Question

  • I'm experiencing a very strange issue where data-driven tests execute only the last data row from the data source specified in the data source parameter.  However, this only happens every other time I run a data-driven test.  For example, the 1st, 3rd, 5th, etc. time I execute the test after opening the solution, all data rows in the data source are executed.  The 2nd, 4th, 6th, etc. time I execute the test, only the last data row is executed.  The data source is a csv file but I've also reproduced this behavior with an xml file.

    The IDE is VS2013 Ultimate Update 1.  I have verified this same behavior on two other developer machines using the same source code and solution file.  The output in the VS Test Explorer supports this observation because it only provides output for the last data row.  Also, I have forced the first data row in the data source to result in a failed assertion, which will result in an overall failed test.  The last data row, however, contains data that will result in a valid assertion.  As a result, every other time I run the test, without changing any test logic or the data source, the test alternates between passing and failing.

    Here's some screen captures detailing the issue. As you can see this test took 80ms and executed several data rows; with data row 0 failing, resulting in an overall failed unit test. I apologize for the blurs, but this still details the relevant information.

    Now, I simply execute all the tests in this TestClass again, you'll have to take my word for it that the highlighted test is the same test that failed in the previous screen capture, and the test now passes.  Note that the entire test took 5ms and only output detail for data row 13 is present.

    To further my confusion, this only happens within Visual Studio.  If I run the data-driven test using vstest.console.exe or mstest.exe in the command prompt everything executes as it should.  I have tried to repro this behavior in a small, contained test harness but I've had no luck doing so.  The solution that exhibits this behavior is medium/large size with 50+ projects consisting of native C++, mixed mode C++/CLI, C#, and VB.NET.  However, I don't see what the contents on the solution itself would have to do with the behavior of vstest.executionengine.  The tests are being run in the x86 engine and I have tried running them with the "Keep Test Execution Engine Running" option both on and off with no change in behavior.

    If I cannot get these tests running consistently my team is going to have to abandon the use of the data-driven unit test features of the VS unit testing framework.

    Thursday, May 8, 2014 9:34 PM

All replies

  • Hi,

    “The solution that exhibits this behavior is medium/large size with 50+ projects consisting of native C++, mixed mode C++/CLI, C#, and VB.NET.  ”

    Before you run the tests again, please end the task: vstest.executionengine.x86.exe from task manager.

    If no help, since  the issue occured with the complex test scenation, we can't repro it on our side, I suggest submitting this feedback to Microsoft Connect feedback portal: http://connect.microsoft.com/VisualStudio/feedback/CreateFeedback.aspx, Microsoft engineers will evaluate them seriously.  After you submit the feedback, you can post the link here which will be beneficial for other members with the similar issue.

    Best regards,

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    • Edited by Amanda Zhu Wednesday, May 14, 2014 6:41 AM edit
    Friday, May 9, 2014 9:18 AM
  • I have also observed this behaviour - on alternate runs of a test the whole set of data row scenarios are executed, on the next only the last.

    We say our tests are playing tennis =/

    It's not funny, though.

    Tuesday, September 1, 2015 1:05 PM