locked
VStest.console.exe Query for a Data Row Result DURING Test Execution Status RRS feed

  • Question

  • I started with the following thread and was asked to create a new thread. 

    I have the following test that reads a .csv as a data source.

            /// <summary>
            /// Summary description for Test
            /// </summary>
            [TestCategory("LongTest"),
             TestCategory("Visual Studio 2013"), TestMethod]
            [DeploymentItem("data.csv")]
            [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV",
                "|DataDirectory|\\data.csv",
                "data#csv",
                DataAccessMethod.Sequential),
            DeploymentItem("data.csv")]
    
            public void Create_And_Build_All_Templates()
            {
                testmethodname = "Create And Build All Templates ";
                LaunchVisualStudio2013();
            }

    When I run the test from VStest.console.exe, I see the following:

    vstest.console.exe

    /testcasefilter:"TestCategory=LongTest"

    /settings:"C:\testing\CodedUI.testsettings"

    /logger:TRX

    /logger:CodedUITestLogger

    C:\AppTests\CodedUITest.dll Microsoft (R) Test Execution Command Line Tool Version 12.0.31101.0 Copyright (c) Microsoft Corporation. All rights reserved. Running tests in C:\testing\bin\debug\TestResults Starting test execution, please wait...

    I want to report on the status of the iterations DURING the test run from VStest.console.exe, like how the test explorer window does this. 


    How can I achieve the output below (notice the (Data Row ) values) ?

    vstest.console.exe
    
     /testcasefilter:"TestCategory=LongTest"
    
     /settings:"C:\testing\CodedUI.testsettings"
    
     /logger:TRX
    
     /logger:CodedUITestLogger
    
     C:\AppTests\CodedUITest.dll
    Microsoft (R) Test Execution Command Line Tool Version 12.0.31101.0
    Copyright (c) Microsoft Corporation.  All rights reserved.
    
    Running tests in C:\testing\bin\debug\TestResults
    Starting test execution, please wait...
    Test Passed - Create_And_Build_All_Templates (Data Row 1)
    Test Passed - Create_And_Build_All_Templates (Data Row 2)
    Test Failed - Create_And_Build_All_Templates (Data Row 3)
    Test Passed - Create_And_Build_All_Templates (Data Row 4)


    Ian Ceicys





    • Edited by Ian Ceicys Tuesday, April 14, 2015 2:30 PM
    Tuesday, April 14, 2015 2:24 PM

Answers

  • Hi Jack, I've gone ahead and created the uservoice suggestion here:
    http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/7660938-provide-vstest-console-exe-test-explorer-real-ti

    I am a bit confused about your reply, is it possible to output the message to the output window, or is it NOT possible?

    Could you please provide a code sample with a console app (as you suggested) that would do the following: "you could read the file in a general app like the Console app, and then it could output the message to the Output window with the Debug.WriteLine() method if you debug this Console app when the test is running."

    Thanks for your help in understanding the technical details for a possible solution.


    Ian Ceicys

    Wednesday, April 22, 2015 1:04 PM
  • I will help you vote it, Ian! Thanks for your feedback and suggestion.

    Have a nice day,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    • Marked as answer by Ian Ceicys Wednesday, May 6, 2015 6:16 PM
    Friday, April 24, 2015 2:23 AM

All replies

  • Hi Ian Ceicys,

    I use the VS2013 ultimate version.

    Like the following screen shot, I run a simple data driven unit test, I get the result here:

    If I run it with command line, I get the result like the screen shot 2:

    >>How can I achieve the output below (notice the (Data Row ) values) ?

    Do you mean that you don't want to get the "Data Row 0"?

    If so, I'm afraid that you need to ignore the Row 0 with custom code in your test method, I mean that you could run your test from the row 1, and then add a new row 5.

    Reference:

    https://social.msdn.microsoft.com/Forums/en-US/9664862a-4302-4754-a915-2beceff364d5/how-to-stop-sequential-execution-of-data-driven-method-in-coded-ui?forum=vstest

    If I have misunderstood this issue, please feel free to let me know.

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.



    Wednesday, April 15, 2015 10:20 AM
  • Jack,

    You're screenshot shows the results AFTER the test has completed, and I am trying to get the status of the results DURING the test execution. 

    Take a codedui test that runs for 90 mins but has 300+ iterations, at an hour into the test I want to be able to find if it is on row 80 or row 188 or row 288. 

    If you want an example make a unit test that uses thread.sleep and replace the wait time with a value from the csv file. 

              Thread.sleep(TestContext.DataRow["WaitTime"].ToString());

    Use the following as the values:

    WaitTime,

    10000

    5000

    10000

    5000

    10000

    Again how can I get the status of the results i.e the Row value for a data driven test, DURING the test execution?

    Just to be clear I am using Visual Studio 2013 Update 4


    Ian Ceicys







    • Edited by Ian Ceicys Wednesday, April 15, 2015 2:24 PM
    Wednesday, April 15, 2015 11:45 AM
  • Hi Ian,

    >>Take a codedui test that runs for 90 mins but has 300+ iterations, at an hour into the test I want to be able to find if it is on row 80 or row 188 or row 288. 

    My understanding is that if the specific test doesn't finish, it would be hard for us to get the test result "Pass/failed".

    If you want to get the row value during the test is running, the real issue is that where you want to get the row value, if you want to get the output in command line, I'm afraid that it doesn't support now, if you want to get the value like my first screen shot with "Output link" in Test Explorer window. I think we could achieve it with custom code in coded UI test project. Or you could output the row value to a txt file or others file, so you could know which row was using in your test when the whole test is not finished with all rows value.

    Reference:

    http://blogs.msdn.com/b/gautamg/archive/2009/12/08/logging-a-message-in-test-result-as-part-of-an-automated-test.aspx  

    I found that the "/detail" command line with MSTest.exe, maybe we could get some output information with it.

    https://msdn.microsoft.com/en-us/library/ms182489.aspx#detail 

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Thursday, April 16, 2015 5:19 AM
  • Jack, if you can provide a code sample for getting the row value during the test run from the Test Explorer Window that would be great. I don't have to use the command line but it was my understanding that in order to work with the test execution status, vstest.console is the only option. 

    I looked into the /detail command line option and it doesn't appear to an option for vstest.console, and that's why I had to adapt the customer logger code from here: https://github.com/ppiotrowicz/AwesomeLogger

    The link you posted only prints the status information to the output context and that gets printed to the console at the end of the run, while I'd like to know which data row I am currently running, during the run.

    If there is a way to extend Test Explorer to report back which data row is currently being executed I'm interested in learning about who to write such a plug-in!

    In doing some limited research this looks promising:

    http://stackoverflow.com/questions/23016174/how-do-i-programatically-determine-how-many-unit-tests-have-been-executed


    Ian Ceicys



    • Edited by Ian Ceicys Monday, April 20, 2015 3:31 PM
    Thursday, April 16, 2015 10:45 AM
  • Hi Ian,

    Thanks for your friendly response.

    >>(1)The link you posted only prints the status information to the output context and that gets printed to the console at the end of the run, while I'd like to know which data row I am currently running.

    >>(2)In doing some limited research this looks promising:

    http://stackoverflow.com/questions/23016174/how-do-i-programatically-determine-how-many-unit-tests-have-been-executed

    Like the link in (2), actually I have the same idea in my previous reply, for example, the test calls all row in database sequentially, so the row number is 0,1,2,3...

    Based on the link provided by you, I change some code, so the test method runs one time, the database will be called one row, so "times=Rows".

    public static int _executedTests = 0; public static int _passedTests = 0; public void IncrementTests() { _executedTests++; } public void IncrementPassedTests() { _passedTests++; } [TestInitialize] public void TestInitialize() { IncrementTests(); Console.WriteLine("Total tests Row executed: {0}", _executedTests); } ...Test Method... [TestCleanup] public void TestCleanup() { if (TestContext.CurrentTestOutcome == UnitTestOutcome.Passed) { IncrementPassedTests(); } Console.WriteLine("Total passed tests: {0}", _passedTests); }

    So we could get the result like the following screen shot.

    Is it what you want to get? Maybe it could provide a path if you want to output the value to a text file or others.

    >>If there is a way to extend Test Explorer to report back which data row is currently being executed I'm interested in learning about who to write such a plug-in!

    Sorry for that actually we didn't support the extension/plug-in issue, maybe you could get some idea from the VSX forum here:

    https://social.msdn.microsoft.com/Forums/vstudio/en-US/home?forum=vsx

    Or you could submit a feature request here:

    http://visualstudio.uservoice.com/forums/121579-visual-studio.
    The Visual Studio product team is listening to user voice there.

    Sincerely,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, April 17, 2015 2:41 AM
  • Jack, again the results are printed to the std. out console AFTER the test data row has been completed. Is there a way to query VSTest.console and find out which test\data row is being executed so it can be written out to the console DURING the test run? 

    I put together the following screencast showing the issue: 
    http://www.screencast.com/t/IrxxfhGlzD

    Also here is the github repo with the source code that I included in the screen cast:

    https://github.com/ianceicys/VisualStudioSamples2015

    Take a look at LongRunningDataDrivenTest.sln

    Unit Test

        [TestClass]
        public class UnitTest
        {
            public static int _executedTests = 0;
            public static int _passedTests = 0;
    
            public void IncrementTests()
            {
                _executedTests++;
            }
    
            public void IncrementPassedTests()
            {
                _passedTests++;
            }
    
            [TestInitialize]
            public void TestInitialize()
            {
                IncrementTests();
                Console.WriteLine("Total tests Row executed: {0}", _executedTests);
    
            }
    
            [TestCleanup]
            public void TestCleanup()
            {
                if (TestContext.CurrentTestOutcome == UnitTestOutcome.Passed)
                {
                    IncrementPassedTests();
                }
    
                Console.WriteLine("Total passed tests: {0}", _passedTests);
            }
    
    
    
            private TestContext testContextInstance;
            /// <summary>
            /// Long Running Test
            /// </summary>
            [TestCategory("Long Running Test Example"),TestMethod]
            [DeploymentItem("data.csv")]
            [DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV",
            "|DataDirectory|\\data.csv",
            "data#csv",
            DataAccessMethod.Sequential),
            DeploymentItem("data.csv")]
            public void LongRunning_TestMethod_Takes_Over_30_Seconds_to_Run()
            {
                WaitTime(Convert.ToInt32(TestContext.DataRow["WaitTime"].ToString()));
            }
            public void WaitTime (int waittime)
            {
                Thread.Sleep(waittime);
            }
           public TestContext TestContext
            {
                get { return testContextInstance; }
                set { testContextInstance = value; }
            }
        }

    data.csv

    WaitTime,
    13000
    19000
    10000
    11000
    15000

    Ian Ceicys







    • Edited by Ian Ceicys Monday, April 20, 2015 9:51 PM
    Monday, April 20, 2015 3:29 PM
  • Hi Ian,

    Thanks for your friendly response.

    >>Jack, again the results are printed to the std. out console AFTER the test data row has been completed. Is there a way to query VSTest.console and find out which test\data row is being executed so it can be written out to the console DURING the test run? 

    Actually it is also the real issue I also think about. Even if I use the wait event in my test code (in Test initialization), so before every test starts running, it would run the test initialization. But it seems that it is by design, the test results just ouput after all tests finished in test explorer window. So I'm afraid that it is hard for us to visit them one by one druning running in test explorer window. 

    Not the development members, just one idea/workaround is that we could output the inforamtion to a custom text file, so even if the coded UI test doesn't finshed, the row num still would input to the file.

    For example, we define a file in test [TestInitialize], and then output the row number to the file in  [TestCleanup]. So we could visit it in the file since the test explorer window just shares them after the whole test finished.

    If it still no help, I really suggest you submit a feature request.

    http://visualstudio.uservoice.com/forums/121579-visual-studio. If you submit it, please share me the link here, I will help you vote it.

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Tuesday, April 21, 2015 3:40 AM
  • Jack I was able to output the information to a file in the [TestInitialize] and [TestCleanup], however I am unable to the read the file and output the information via stdout into the output window.

    I still just see the following:

    ------ Run test started ------
    ========== Run test finished: 1 run (0:01:08.2694549) ==========


    When I want to output is 


    ------ Run test started ------
    Test Executed on Row 1 for 13 seconds Test Executed on Row 2 for 19 seconds Test Executed on Row 3 for 10 seconds ========== Run test finished: 1 run (0:01:08.2694549) ==========

    If your suggested workaround "works" how can I read the file and then output the information via stdout into the output window?

    I will go ahead and file a feature request but this "by design" seems to be designed poorly.


    Ian Ceicys

    Tuesday, April 21, 2015 12:37 PM
  • Hi Ian,

    >>however I am unable to the read the file and output the information via stdout into the output window.

    The test projects really have many differences from the normal/general apps like the console app or others.

    For example, we could use the method Debug.WriteLine() to output the message to the Output window for general app.

    Reference:

    http://stackoverflow.com/questions/7461808/how-to-console-writeline-from-testmethod

    http://stackoverflow.com/questions/9466838/writing-to-output-window-of-visual-studio

    But in test projects like the coded UI tests or the unit tests, the result would output to the Test Explorer windows. 

    So I'm afraid that we couldn't output the message to the output window in the test project derectly unless you could read the file in a general app like the Console app, and then it could output the message to the Output window with the Debug.WriteLine() method if you debug this Console app when the test is running.

    >>I will go ahead and file a feature request but this "by design" seems to be designed poorly.

    If you submit a feature request, please share me the link here, I will help you vote it.

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, April 22, 2015 3:18 AM
  • Hi Jack, I've gone ahead and created the uservoice suggestion here:
    http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/7660938-provide-vstest-console-exe-test-explorer-real-ti

    I am a bit confused about your reply, is it possible to output the message to the output window, or is it NOT possible?

    Could you please provide a code sample with a console app (as you suggested) that would do the following: "you could read the file in a general app like the Console app, and then it could output the message to the Output window with the Debug.WriteLine() method if you debug this Console app when the test is running."

    Thanks for your help in understanding the technical details for a possible solution.


    Ian Ceicys

    Wednesday, April 22, 2015 1:04 PM
  • Hi Ian,

    >>I am a bit confused about your reply, is it possible to output the message to the output window, or is it NOT possible?

    Sorry for that I didn't describe this issue clearly, I mean that we couldn't achieve it in Test project as the tester, but we could achieve it with the development app as the developer.

    >>Could you please provide a code sample with a console app (as you suggested) that would do the following: "you could read the file in a general app like the Console app, and then it could output the message to the Output window with the Debug.WriteLine() method if you debug this Console app when the test is running."

    Just a simple sample:

    The text file has the result from your unit test project like:

    The row 1 The row 2 The row 3

    A simple console app which is used to read the text file.

    public static void Main()
            {
                try
                {
                    using (StreamReader s = new StreamReader("Test.txt"))
                    {
                        String Row= s.ReadToEnd();
    
                        Debug.WriteLine(Row);
                        Console.WriteLine(Row);
                    }
                }
                catch (Exception e)
                {
                    Console.WriteLine("The file could not be read:");
                    Console.WriteLine(e.Message);
                }
            }

    Debug the console app with (F5). We could get the output message here.

    Not very sure that whether it is a workaround for you. Since you have submit a user voice for this issue, I will help you vote it.

    Have a nice day,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    Thursday, April 23, 2015 1:57 AM
  • Jack,

    Ah now I understand what you meant by using a console app. That's not really the result I am looking for. 

    I put together the following mock up of what I'm looking for. Ideally I want to query the state of the tests while they are running calculate a percent complete and a time remaining. I put the feature request via uservoice as you requested. 

    Mock up of real time testing


    Ian Ceicys

    Thursday, April 23, 2015 6:35 PM
  • I will help you vote it, Ian! Thanks for your feedback and suggestion.

    Have a nice day,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    • Marked as answer by Ian Ceicys Wednesday, May 6, 2015 6:16 PM
    Friday, April 24, 2015 2:23 AM