locked
Timeouts in test runs using Moles RRS feed

  • Question

  • Hi,

    I'm using TFS to control a build process, and we're getting intermittant errors in tests that use the Moles Test Adapter. The test run will show something like:

     

     Timeout    Client.Tests.ApplicationTests.ApplicationStartup_RunningInBrowser_DoesNotCheckForUpdatesRunsBootstrapper
    Error Client.Tests.ApplicationTests.ApplicationStartup_RunningOutOfBrowser_ChecksForUpdateDoesNotRunBootstrapper
    Error Client.Tests.ApplicationTests.ApplicationStartup_RunningOutOfBrowser_NoUpdate_RunsBoostrapper
    Error Client.Tests.ApplicationTests.ApplicationStartup_RunningOutOfBrowser_Update_ShowsPopUpMessageOnly
    Error Client.Tests.ApplicationTests.Constructor_SetsUpEventHandlers
    Error ServiceTests.AuditUnitTests.GetUsername_EdgeCaseNameWithDomain_ReturnsName
    Error ServiceTests.AuditUnitTests.GetUsername_EdgeCaseNameWithoutDomain_ReturnsName
    Error ServiceTests.AuditUnitTests.GetUsername_LongNameWithDomain_ReturnsTruncatedName
    Error ServiceTests.AuditUnitTests.GetUsername_LongNameWithoutDomain_ReturnsTruncatedName
    Error ServiceTests.AuditUnitTests.GetUsername_ShortNameWithDomain_ReturnsName
    Error ServiceTests.AuditUnitTests.GetUsername_ShortNameWithoutDomain_ReturnsName
    Error ServiceTests.AuditUnitTests.SetAuditTrailOnAmend_DisablesLegacyConfirmations
    Each of the Error'd tests has a failure message resembling:

     

     The test adapter 'MolesAgentAdapter' threw an exception while running test 

    'ApplicationStartup_RunningOutOfBrowser_ChecksForUpdateDoesNotRunBootstrapper'.

    Object reference not set to an instance of an object.


    In addition to the failed tests, there are a large number of passed tests (>1000) that don't use the Moles Adapter, and some other Moles tests (>50 in other test assemblies) that do use the adapter and pass. We're using MSTest to control the test run, with the following command and args (I've removed some of the test assemblies for brevity):

     

    Command: "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\MSTest.exe"
    Command Args: "/nologo" "/usestderr"
    "/searchpathroot:C:\Builds\8\TestBuild\Binaries"
    "/resultsfileroot:C:\Builds\8\TestBuild\TestResults"
    "/testSettings:C:\Builds\8\TestBuild\Sources\Local.testsettings"
    "/publish:http://tfsserver:8080/tfs/DefaultCollection"
    "/publishbuild:vstfs:///Build/Build/TestBuild_20110105.5"
    "/teamproject:app" "/platform:Any CPU" "/flavor:Debug"
    "/testcontainer:C:\Builds\8\TestBuild\Binaries\Client.Infrastructure.Tests.dll"
    "/testcontainer:C:\Builds\8\TestBuild\Binaries\Client.Tests.dll"
    "/testcontainer:C:\Builds\8\TestBuild\Binaries\ServiceTests.dll"
    "/testcontainer:C:\Builds\8\TestBuild\Binaries\Helpers.Tests.dll"
    "/testcontainer:C:\Builds\8\TestBuild\Binaries\Service.Confirmation.Tests.dll"
    "/testcontainer:C:\Builds\8\TestBuild\Binaries\Service.Contracts.Tests.dll"
    "/testcontainer:C:\Builds\8\TestBuild\Binaries\Service.DataAccess.Tests.dll"

     

    The TFS server is hosted on an x64 Windows 2008 R2 machine, but the test run is forced to run in x86 and we're using the x86 version of Moles (v0.94.50921.0). We're running TFS 2010 (v30319.01) and VS2010 Ultimate (v10.0.30319.1). When the tests are run locally from within Visual Studio 2010, the tests don't time out and usually complete in <5s. The server timeout is set to 1m.

     

    Using ProcMon, I've noticed that MSTest spawns a number of sub-processes of Microsoft.Moles.VsHost.x86.exe. It looks like the Error'd tests are limited to a single sub-process (so other Moles tests running in a different process will complete successfully even if one process times out). Unfortunately I've been unable to find the line that a test halts on when timing out (and it's not the same test each time).

    Do you know what could be causing these problems?

     

    Please let me know if you need any more information about the environment or setup, or if you'd like me to run additional diagnostics or logging. We're running MSTest under NCover to generate coverage data, but the timeouts can also occur when running MSTest from the command line on our build server (so I'm hoping that it's not an NCover problem!). We don't get any coverage data for tests run using the Moles Adapter, but I'm aware that NCover are looking at this.

    Thanks for writing such a useful tool!

     

    Best,

    Andy

     

    Wednesday, January 5, 2011 11:09 AM

All replies

  • Hi Andy,

    Moles will spin a different process per test assembly. If your Moles tests are spread multiple test assemblies, Moles will spin a new process for each of them. However, this should not influence the timeout behavior.

    Could you try to do a run with NCover enabled to see if that changes anything? Otherwise, you can add this attribute to enable additional logging:

      [assembly: MolesAssemblySettings(Diagnostic = true)]

     

    The 'Test Run Report' should contain relevant messages if any.


    Jonathan "Peli" de Halleux - Try Pex online at www.pexforfun.com!
    Monday, January 10, 2011 10:26 PM