locked
TDD xUnit test crashing on WinRT component in the VS test executor, but running individual tests pass. RRS feed

  • Question

  • I'm having a problem where my my tests are failing using the xUnit runner for and xUnit built for testing WinRT applications.

    **The odd thing is that individual tests are succeeding if ran individually.**

    These tests are running in the Visual Studio Test Explorer window.

    The error message is : 

    ------ Run test started ------
    Updating the layout...

    Registering the application to run from layout...

    Deployment complete. Full package name: "22f638ca-e318-4849-bac8-1f6d381c0896_1.0.0.0_x86__nsmz8dbv8bkn8"

    The active Test Run was aborted because the execution process exited unexpectedly. To investigate further, enable local crash dumps either at the machine level or for process vstest.executionengine.appcontainer.x86.exe. Go to more details: http://go.microsoft.com/fwlink/?linkid=232477
    ========== Run test finished: 0 run (0:00:05.7153326) ==========



    Now, the interesting thing is the test is performed on a UI panel.  In order to test resizing behaviour and layout I need it to be added to the visual tree.

    The test stripped down to only the lines needed to reproduce the crash looks like this :


            public IAsyncAction ExecuteOnUIThread<TException>(DispatchedHandler action)
            {
                return CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, action);
            }

            [Fact]
            public async Task WheelPanel_Can_Be_Hosted__attempt_crashes_tester()
            {
                Grid grid = null;

                await ExecuteOnUIThread<ArgumentException>(() =>
                {
                    Window nonVisibleMainWindow = Windows.UI.Xaml.Window.Current;
                    grid = new Grid { Width = 800, Height = 800 };
                    nonVisibleMainWindow.Content = grid;

                    Assert.True(true);
                });
            }

    Of course the real intended test would then add the custom panel to the grid and then call UpdateLayout() or other functions.... but this is all that is needed to cause the crash.

    I also have dump logs from the vstest.executionengine.appcontainer.x86.exe process....

    The tail of the log looks like this : 

    [--== snip ==--]

    V, 7100, 10, 2013/08/01, 12:47:04.551, 287501339, vstest.executionengine.appcontainer.x86.exe, TestExecutorExtensionManager: Loading executor Xunit.Runner.VisualStudio.VsTestRunner
    V, 7100, 10, 2013/08/01, 12:47:04.551, 287501401, vstest.executionengine.appcontainer.x86.exe, TestExecutorService: Loaded the executors
    V, 7100, 10, 2013/08/01, 12:47:04.552, 287503111, vstest.executionengine.appcontainer.x86.exe, TestExecutorService: Loaded the settings providers
    I, 7100, 10, 2013/08/01, 12:47:04.552, 287503171, vstest.executionengine.appcontainer.x86.exe, TestExecutorService: Loaded the extensions
    E, 7100, 8, 2013/08/01, 12:47:04.555, 287513737, vstest.executionengine.appcontainer.x86.exe, Solution directory 'C:\Users\root\documents\visual studio 2012\Projects\mangosteen\' does not exists. Please make sure solution directory specified in runsettings exists and have read permissions for directory.
    V, 7100, 8, 2013/08/01, 12:47:04.555, 287514216, vstest.executionengine.appcontainer.x86.exe, TestExecutorService: PopulateTestRunContext: Executor process is expected to shutdown after the test run.
    V, 7100, 8, 2013/08/01, 12:47:04.555, 287514292, vstest.executionengine.appcontainer.x86.exe, TestExecutorService: StartTestRun: Created the test run context IsDataCollectionEnabled=False, AreTestCaseLevelEventsRequired=False. 
    V, 7100, 8, 2013/08/01, 12:47:04.556, 287518236, vstest.executionengine.appcontainer.x86.exe, TestExecutorService: StartTestRun: Started the test run.
    V, 7100, 8, 2013/08/01, 12:47:04.556, 287518297, vstest.executionengine.appcontainer.x86.exe, TestExecutorWebService: Handled StartTestRun.
    I, 7100, 12, 2013/08/01, 12:47:04.566, 287552603, vstest.executionengine.appcontainer.x86.exe, TestDiscoveryManager: Discovering tests from sources C:\Users\root\documents\visual studio 2012\Projects\mangosteen\Mangosteen.Test\bin\x86\Debug\Mangosteen.Test.dll
    V, 7100, 12, 2013/08/01, 12:47:04.570, 287563945, vstest.executionengine.appcontainer.x86.exe, TestExecutionManager.RunTests: Running tests for executor://xunit.codeplex.com/VsTestRunner
    V, 7100, 12, 2013/08/01, 12:47:05.952, 292163402, vstest.executionengine.appcontainer.x86.exe, TestExecutionManager.RunTests: Completed running tests for executor://xunit.codeplex.com/VsTestRunner

    Setting up the xUnit test project requires a different fork of xUnit at this time and a bit esoteric folder layout, but I've done dozens of other tests with no problem.  If you need a full reproduction solution I can provide it, as well as the xUnit source code.  The test appears to run through the source code of xUnit with no problems.... at least that I can see.

    I can also provide the trace of a test that passes for comparison, but I don't see anything very different except that the long continues well past this point.

    Thanks!

    -Jeremy White


     


    Thursday, August 1, 2013 8:13 AM

Answers

  • Hello,

    Since individual tests are able to run successfully from Visual Studio, I think the crash issue should be not in your test itself. How do you run tests together from VS? In an ordered test?

    As my experience, the crash sometimes is related to VS/test environment. In order to check if the crash issue is related to your VS/test environment, I suggest that you try the following steps to see if they can help:

    Please open Windows Explorer, and navigate to <Visual Studio Installation Path>\Common7\IDE

    • Devenv.exe /SafeMode: Launches the IDE in safe mode loading minimal windows.
    • Devenv.exe /ResetSettings: Restores the IDE's default settings, optionally resets to the specified VSSettings file.
    • Devenv.exe /ResetAddin: Removes commands and command UI associated with the specified Add-in.
    • Devenv.exe /ResetSkipPkgs: Clears all SkipLoading tags added to VSPackages.

    If no help, you can try the same scenario on another machine to check the result if possible.

    Best regards,


    Amanda Zhu
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    • Marked as answer by Amanda Zhu Thursday, August 8, 2013 11:45 AM
    Sunday, August 4, 2013 7:50 AM

All replies

  • Hello,

    Thank you for your post.

    Based on your description, you failed to run xUnit tests in Visual Studio 2012.

    If xUnit means xUnit.net, and you use xUnit.net runner, I am afraid that your issue is out of support range of our forums since xUnit.net is from CodePlex and everything in CodePlex is not Microsoft product, so there is no support for it.

    I suggest that you can consult your issue through starting a discussion on http://xunit.codeplex.com/discussions like this discussion: http://xunit.codeplex.com/discussions/450105 for better response and solutions.

    Thank you for your understanding.

    Best regards,


    Amanda Zhu
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Friday, August 2, 2013 3:10 AM
  • Amanda, 

    Sorry if the previous question was lost in translation.

    I'm working on the xUnit source code.

    I'm writing my own custom test runner (based off of xUnit) using the plugin architecture provided by visual studio.  I don't know if you realize this but the runner architecture has provisions to call out to custom runners.

    The failure isn't occurring in xUnit, its failing in Visual Studio.  In your test runner (after it calls into my extension and my extension returns.)

    In process vstest.executionengine.appcontainer.x86.exe.

    *After* the runner exits xUnit.  And I've stepped through xUnit to verify that it isn't part of the problem as much as possible.

    As I don't have the ability to look at the Visual Studio source code other then by using your fairly hidden debug log dumping method, I'm stuck.

    Again, it isn't in xUnit, its in the runner that has a plugin architecture to run custom tests.

    There aren't many(any?) test runners out there other then the one in Visual Studio that handle Windows Store apps at all so this is likely related to issues surrounding that.

    Again, I'm looking for any way to determine why vstest.exectutionengine.appcontainer.x86.exe (part of visual studio) might be crashing.  The error message is almost non-existent.

    Edit : I'll move the code to the vs test runner to see if there is any different behaviour I can detect.

    Edit : The VS test unit doesn't crash.

    Ideally I would like to be able to give someone a project that they can run and perhaps they can tell me what xUnit that later crashes the Visual Studio test runner?

    As far as I can tell the xUnit plugin conforms to the meager documentation I have on how to write a unit test runner.



    Friday, August 2, 2013 8:18 AM
  • Hello,

    Since individual tests are able to run successfully from Visual Studio, I think the crash issue should be not in your test itself. How do you run tests together from VS? In an ordered test?

    As my experience, the crash sometimes is related to VS/test environment. In order to check if the crash issue is related to your VS/test environment, I suggest that you try the following steps to see if they can help:

    Please open Windows Explorer, and navigate to <Visual Studio Installation Path>\Common7\IDE

    • Devenv.exe /SafeMode: Launches the IDE in safe mode loading minimal windows.
    • Devenv.exe /ResetSettings: Restores the IDE's default settings, optionally resets to the specified VSSettings file.
    • Devenv.exe /ResetAddin: Removes commands and command UI associated with the specified Add-in.
    • Devenv.exe /ResetSkipPkgs: Clears all SkipLoading tags added to VSPackages.

    If no help, you can try the same scenario on another machine to check the result if possible.

    Best regards,


    Amanda Zhu
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    • Marked as answer by Amanda Zhu Thursday, August 8, 2013 11:45 AM
    Sunday, August 4, 2013 7:50 AM
  • Hello,

    I am marking useful reply as answer. If the suggestion can’t help you, you can come back and unmark it as answer. If you have anything unclear, please feel free to come back and ask me.

    Best regards,


    Amanda Zhu
    MSDN Community Support | Feedback to us
    Develop and promote your apps in Windows Store
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Thursday, August 8, 2013 11:45 AM