Tests passing locally, "not executed" on build server

Question

Hi,

      I am currently running into an issue with Unit tests passing locally, but getting reported "failed" test cases on our build servers. Locally, I am running win7 VS2013 test agent and the build servers are running win8 VS2013 test agent.

Firstly, all our tests are passing on our local machines, so we are having trouble reproducing the problems on the build server. Secondly, on the build servers, the tests are reporting that they aren't being executed, but based on the console output as well as some coverage information reported, we can see that the tests are actually running (which is a mystery to us).

We have attempted the tests locally and all our local machines will pass, but all our server machines will not pass certain tests. Also we have attempted to rollback to a previous state on the build server that was passing, but even that has not worked. We have already attempted a clean build/rebooting of the build machines, but nothing has helped so far as the tests are still reported as "not executed" (it looks like the tests are in fact executing).

anyone running into this issue or have a solution or a suggestion on where to look?

-L

Answer 1

Hi L Chen,

Thank you for posting here.

Could you please share your build error log here.

Are you using TFS build to build your tests? It seems that you are using TFS XAML build. If so, do you use the labDefaultTemplate to call test agents or using a test setting file to call test agents?

On the other hand, your test agents and build agent are on the same machine? If they are not the same, have you tested this project could be run on test agent machine?

---

We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click HERE to participate the survey.

Answer 2

Hi,

thank you for the reply. Unfortunately I cannot provide the build log due to proprietary reasons, however yes we are using TFS XAML and using test setting files. I should also mention that there are no build errors. However the tests results will show "Not Executed" and the Console will report "The active Test Run was aborted because the execution process exited unexpectedly." The console output will print an indication that the tests did run through completely though, so something like this:
TestSetup()

UnitTest()

TestTearDown()

Our builds and test agents are on the same machine as well.

However, after further debugging, I believe the issue revolves around a singleton class. We are seeing some odd behaviors calling some methods in some test suites. For example, the singleton class we are using is used by several test suites, but will only fail for a few test cases. We have even removed all but one test case that calls one method of the singleton. We did this for two different Unit Tests and it passes for one unit test, but not the other.

I should mention, we recently rolled out a new windows 8 patch (which isn't supposed to affect visual studios), but these problems did appear after the patch updates. We believe it might have unearthed an underlying issue, but we don't know what exactly. Have you seen this behavior before?

-L

Answer 3

Hi,

What’s the result if you run tests manually through VSTest.Console.exe tool on test agent machine?

What’s the code like? Please provide the sample code.

Regards

Starain

---

We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click HERE to participate the survey.

Answer 4

Hi all,

Update : We have pinpointed the issue with our implementation and it turns out it our testers picked up the problem rather than our live environment probably due some luck on our part. We were having issues with our critical section implementation with the use of smart pointers. Our critical section was being freed before the destructor was called in our singleton class. We're not sure what changed the order in the cleanup as we have not had any issues for months/years, but we have made our deallocations explicit, which solved our problems.

Thanks for the help.

-L

Answer 5

Hi L,

Thank you for sharing your experience here.  You could mark your last reply as the answer for this thread.  It will be very beneficial for other community members having the similar questions.

Thank you.

---

We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click HERE to participate the survey.