Test run aborts unexpectedly

Question

We are developing and maintaining a large project in .Net using VS2008 and TFS. We have a build agent set up, which has been working reliably for years doing nightly builds with a full test run. The test run comprises some 5800 test cases and normally takes 45 minutes to complete.

Lately we have been finding that the test run has been aborting part way through - never at the same test but almost always after about 30 minutes. I have also been finding this when running the tests from within VS2008 (but not consistently so).

The test run failure error message is

Warning: conflict during test run deployment: deployment item 'C:\Users\neil\Documents\Visual Studio 2008\Referenced Assemblies\Third Party Dlls\log4net\2.0\debug\log4net.dll' directly or indirectly referenced by C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\Test Projects\MatrixBLLTestProject\bin\x86\Release\MatrixBLLTestProject.dll cannot be deployed to 'log4net.dll' because otherwise the file 'C:\Users\neil\Documents\Visual Studio 2008\Referenced Assemblies\Third Party Dlls\log4net\2.0\debug\log4net.dll' would overwrite deployment item 'C:\Users\neil\Documents\Visual Studio 2008\Referenced Assemblies\Third Party Dlls\log4net\2.0\release\log4net.dll' directly or indirectly referenced by C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\Test Projects\Workflow and UI-Windows TestProject\bin\x86\Release\WorkflowAndUIWindowsTestProject.dll.
Warning: conflict during test run deployment: deployment item 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\log4net.dll' directly or indirectly referenced by the test container 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\matrixblltestproject.dll' cannot be deployed to 'log4net.dll' because otherwise the file 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\log4net.dll' would overwrite deployment item 'C:\Users\neil\Documents\Visual Studio 2008\Referenced Assemblies\Third Party Dlls\log4net\2.0\release\log4net.dll' directly or indirectly referenced by C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\Test Projects\Workflow and UI-Windows TestProject\bin\x86\Release\WorkflowAndUIWindowsTestProject.dll.
Warning: conflict during test run deployment: deployment item 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixdaltestproject\bin\x86\release\MatrixBLL_Accessor.dll' directly or indirectly referenced by the test container 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixdaltestproject\bin\x86\release\matrixdaltestproject.dll' cannot be deployed to 'MatrixBLL_Accessor.dll' because otherwise the file 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixdaltestproject\bin\x86\release\MatrixBLL_Accessor.dll' would overwrite deployment item 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\MatrixBLL_Accessor.dll' directly or indirectly referenced by the test container 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\matrixblltestproject.dll'.
Warning: conflict during test run deployment: deployment item 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\workflowtestproject\bin\x86\release\Workflow_Accessor.dll' directly or indirectly referenced by the test container 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\workflowtestproject\bin\x86\release\workflowtestproject.dll' cannot be deployed to 'Workflow_Accessor.dll' because otherwise the file 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\workflowtestproject\bin\x86\release\Workflow_Accessor.dll' would overwrite deployment item 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\Workflow_Accessor.dll' directly or indirectly referenced by the test container 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\matrixblltestproject.dll'.
Warning: Test Run deployment issue: Failed to get the file for deployment item 'MatrixUserTable' specified by the test 'WorkflowAndUIWindowsTestProject.MatrixUserTableTests.MatrixUserTable_ForceInsert_called_DoForceInsert': System.IO.FileNotFoundException: Could not find file 'C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\MatrixUserTable'.
Warning: conflict during test run deployment: deployment item 'UI - Windows.exe' specified by the test 'UI___WindowsTestProject.CommonWinListUITest.AlphaKeyPressed_ResultsInCorrectAction' cannot be deployed to 'log4net.dll' because otherwise the file 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\log4net.dll' would overwrite deployment item 'C:\Users\neil\Documents\Visual Studio 2008\Referenced Assemblies\Third Party Dlls\log4net\2.0\release\log4net.dll' directly or indirectly referenced by C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\Test Projects\Workflow and UI-Windows TestProject\bin\x86\Release\WorkflowAndUIWindowsTestProject.dll.
Warning: conflict during test run deployment: deployment item 'MatrixBLL.dll' specified by the test 'MatrixBLLTestProject.MatrixSample_Test6.MatrixSample_GetNewResult_Returns_A_MatrixSampleResult_From_NewMatrixSampleTest_With_NewResult_When_ComponentRowsAndColumns_Match' cannot be deployed to 'log4net.dll' because otherwise the file 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\log4net.dll' would overwrite deployment item 'C:\Users\neil\Documents\Visual Studio 2008\Referenced Assemblies\Third Party Dlls\log4net\2.0\release\log4net.dll' directly or indirectly referenced by C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\Test Projects\Workflow and UI-Windows TestProject\bin\x86\Release\WorkflowAndUIWindowsTestProject.dll.
Warning: Test Run deployment issue: Failed to get the file for deployment item 'UserTableMetaData' specified by the test 'WorkflowAndUIWindowsTestProject.MatrixUserTableTests.UserTableMetaData_GetFactory_Called_GetUserTableFactory': System.IO.FileNotFoundException: Could not find file 'C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\UserTableMetaData'.
Warning: Test Run deployment issue: Failed to get the file for deployment item 'MatrixBllDalSharedTestProject\TestData\ITableAdapter1.csv' specified by the test 'MatrixBllDalSharedTestProject.ITableAdaptertests.ITableAdapter_GetFieldByOrdinalIntersectionGetsCorrectFieldByName_2': System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\MatrixBllDalSharedTestProject\TestData\ITableAdapter1.csv'.
Warning: conflict during test run deployment: deployment item 'Workflow.dll' specified by the test 'WorkflowAndUIWindowsTestProject.SampleManagementWorkflowTest.GetApprovalStatusTest_OptionNotSet' cannot be deployed to 'log4net.dll' because otherwise the file 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\log4net.dll' would overwrite deployment item 'C:\Users\neil\Documents\Visual Studio 2008\Referenced Assemblies\Third Party Dlls\log4net\2.0\release\log4net.dll' directly or indirectly referenced by C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\Test Projects\Workflow and UI-Windows TestProject\bin\x86\Release\WorkflowAndUIWindowsTestProject.dll.
Warning: conflict during test run deployment: deployment item 'MatrixDAL.dll' specified by the test 'MatrixDALTestProject.ObjectADOWriterTest2.DeleteWithVersionTest' cannot be deployed to 'log4net.dll' because otherwise the file 'c:\users\neil\documents\visual studio 2008\projects\matrixroot\matrix5\test projects\matrixblltestproject\bin\x86\release\log4net.dll' would overwrite deployment item 'C:\Users\neil\Documents\Visual Studio 2008\Referenced Assemblies\Third Party Dlls\log4net\2.0\release\log4net.dll' directly or indirectly referenced by C:\Users\neil\Documents\Visual Studio 2008\Projects\MatrixRoot\Matrix5\Test Projects\Workflow and UI-Windows TestProject\bin\x86\Release\WorkflowAndUIWindowsTestProject.dll.
Test run exceeded the specified timeout of 1800 seconds.

From this I deduce that

1. The reason for the test run abort after 30minutes is the timeout mentioned at the end.

2. The timeout is occurring because one or more of the file copy operations is being blocked, probably because of a lock. it looks like one test is maintaining a lock when another test is trying to copy over the file (it is not the tests themselves doing the copying, but is it possibly the test harness ??)

But why? This isn't consistent at all, and some of the above error messages do not make sense (look at the source and target file names).

Also, this is an error report from a local VS2008 test run. When the build agent test run aborts we get no error such message, and nothing similar  I can find in the logs.

Is there an MSTest guru who can cast some light?

TIA

---

Neil Haughton

Answer 1

Hi Neil,

Thank you for posting in the MSDN forum.

Do you mean that if you create a new Test project without team build and run it locally, it has the same issue?

Based on the warnings, it seems that it is related to the deployment items.

Please try to add the deployment items in the .testrunconfig file, see http://msdn.microsoft.com/en-us/library/ms182480.aspx for more information.

Best Regards,

---

Jack Zhai [MSFT]
MSDN Community Support | Feedback to us

Answer 2

Hi Jack,

Thanks for responding.

Please hold my hand a little on this one. There are a few things in your reply I don't quite understand:

if you create a new Test project without team build and run it locally...

I'm not sure what you mean by 'Test project'. Do you mean a new VS project in the solution, comprising classes with the TestClass atrtribute and methods with the TestMethod attribute, etc etc, or do you mean a new test run config? 

What do you mean by "create...without team build" ? I odn't understand this.

Most of our tests in this solution (there are over 3300 test methods, 5580 test cases in all) have been created 'by hand', but some have been created initially by using the 'Create Unit Test' rt-click option, and those ones tend to have the DeploymentItem attribute on the test method. The hand-crafted ones (ie just a method in a test class decorated with the TestMethod attribute) tend not to. Mostly this makes no difference to whether they run or not. Should it?  Deployment is switched on in the test run config and we only have a few additional deployment files specifically listed, items that are outside the solution but which certain tested code will need to have available to run correctly (product licence file, that sort of thing) and normally that is all that is required.

Since my original post I have noticed that the error refers to both the debug and release folders for log4net.dll. I suspected that this might be causing an issue, and corrected the reference error in the solution so all projects consistently reference the release version, but this hasn't made any significant difference. We still have the problem.

One question that puzzles me is if this is a timeout issue (as appears to be the case) why does the test framework abandon all further test execution, instead of just aborting the current test and proceeding with the next one? Could the answer to that provide a further clue to the reason for the timeout occurring?

TIA.

---

Neil Haughton

Answer 3

Hi Neil,

Sorry about my misunderstanding about this issue.

I feel that you are run your tests with Team Foundation Server - Build Automation. Sorry again.

It seems that you run many unit tests at the same time, if we run it with few tests, whether
it has this issue?

Error: Test run exceeded the specified timeout of 1800 seconds

About how to resolve this issue, please see: http://social.msdn.microsoft.com/Forums/is/vststest/thread/68c06265-bd5d-445e-9805-c27aa090b6e9. Hope it can help.

Have a nice day,

---

Jack Zhai [MSFT]
MSDN Community Support | Feedback to us

Answer 4

Jack,

Thanks for your help, but you were not mistaken. We are indeed using  Team Foundation Server - Build Automation, but also running the test list locally.

My experience is that when the problem occurs, an aborted test run usually occurs after 700 - 1100 completed tests (it is not consistent). I've tried splitting the test list into smaller test lists (of around 500 tests each) but that did not solve the problem. Up until recently we have been repeatedly running a test list of over 5000 test cases without this problem.

How does the test run config "Test Run Timeout" parameter work? I have just reduced this to 5 minutes and run the test list locally, and the run aborted after 10 minutes. I checked in the amended test run config and scheduled a new CI build on the Build Agent, and it too aborted after 10 minutes. Colleagues report that even with this parameter left at the default 30 minutes, sometimes the test run successfully completes after 40-50 minutes, and sometimes it aborts much earlier. Can you tell me what is the correct expected effect of this parameter on a test run execution? It doesn't seem to do what it says 'on the box'! Does this parameter actually work in VS2008/TFS?

TIA

---

Neil Haughton

Answer 5

Hi Neil,

Glad to receive your reply.

As this article here, setting time limits for tests and test runs has these effects:

1. When the duration of a test run reaches its time-out value, the test run is aborted, even if some tests have not been run.
2. When the duration of an individual test reaches its time-out value, the execution of that test stops, and the test is marked as Timeout. The test run continues with the subsequent test.

As my understanding, the test run time is related to many factors, like the longer of the code, the configuration of the computer, the CPU Usage and so on, for example, if your system is very busy, if we run the test, I afraid that it will take a long time.

So about how to set the Test Run Timeout, we should think over many factors, about your issue, I think if we run an aborted test alone, it will work normally. The default value of the Test Timeouts is 30 minutes, since you run so many tests at the same time, so I think you would increase the time to check it in your side.

Sincerely,

---

Jack Zhai [MSFT]
MSDN Community Support | Feedback to us

Answer 6

Hi Jack,

I understand.

What we have now found (and this confused us) is that the test run timeout period starts when the first test starts executing, not when the test run starts. There is sometimes quite a long pause between the test run starting and the first test being executed, and this accounts for the discrepancy.

Anyway, by setting the test run timout period to a couple of hours the problem has now been overcome.

Thanks for your help. Unfortunately the link you posted points to your own machine 

file:///C:/Users/v-jake/Desktop/How%20to:%20Set%20Time%20Limits%20for%20Running%20Tests

so I cannot access the document. Can you possibly post a publicly accessible link that I can use?

Lastly, is there a limit on the number of tests that can be executed in a test run?

---

Neil Haughton

Answer 7

Hmm, you are right:)

I have corrected it, glad to hear that your problem has been resolved.

In addition, it seems that there is no limit on the number of unit tests that can be executed at the same time. But as this issue, the timeout can impact it.

Have a nice day,J

---

Jack Zhai [MSFT]
MSDN Community Support | Feedback to us