Running ordered tests without test metadata

Question

Our environment is VS 2010 SP1/TFS 2010 SP1.

Since test metadata files have been removed from VS 11 we are converting our tests not to use them. One problem we ran into is ordered tests. While they run locally, on build sefver we are getting "Cannot find the test 'SomeTest' with storage Test.dll". Looks like the storage attribute of TestLink element in .orderedtest file must point exactly to the Dll with the test.

If the ordered test is ran as a part of .vsmdi file then everything works fine.

Is this a bug in MSTest? How do we work around it?

Answer 1

Hi Alex,

Thank you for posting in the MSDN forum.

I’m not an expert about the TFS, but I have involved the TFS expert, when we run this test, if we select the test metadata (.vsmdi) setting in the TFS, the way is that:

TFS->MSTest->.vsmdi ->dll->TestA TestB…  

If you remove the.vsmdi file, it would not find the dll file.

Actually, it is different when we run the tests locally without the test lists("Tests Not in a List" in the screen shot) with MSTest (We just use the MSTest). We often run it like the following steps.

MSTest-> .dll->TestA TestB

Like the following screen shot, if you use the “Lists of Tests”, when we run the test in TFS, we need to use the .vsmdi file. Open the .vsmdi file with Notepad, we will get that it was used to call the “Test List”.

A msdn document about how to use MSTest.exe: http://msdn.microsoft.com/en-us/library/ms182489.aspx. Hope it can help.

Best Regards,

---

Jack Zhai [MSFT]
MSDN Community Support | Feedback to us

Answer 2

Jack,

I cannot find an answer to my question. We need to be able to run tests without .vsmdi, only .orderedtest.

Answer 3

Hi Alex,

Glad to receive your reply.

We need to be able to run tests without .vsmdi, only .orderedtest.

If so, you could cut the .orderedtest in the “Lists of Tests” and paste it in the “Tests Not in a List” like the following screen shot, then run the test with MSTest /testcontainer: XX. orderedtest. Like the following screen shot.

Best Regards,

---

Jack Zhai [MSFT]
MSDN Community Support | Feedback to us

Answer 4

Jack,

I know how to run .orderedtest. The problem is running them on build server. PLease read my original post.

Answer 5

Hi Alex,

Glad to receive your reply.

We need to make sure that:

1. Are you running the test with TFS build?
2. If you are using the TFS build. Which way are you using in the following screen shot? “Test assembly file specification (recommended) or (Test metadata file(.vsmdi))”?

If you want run tests without .vsmdi, you could select the “Test assembly file specification (recommended)” and add the .orderedtest like the above screen shot.

If I have misunderstanding anything, make free feel to let us know.

Have a nice day,

---

Jack Zhai [MSFT]
MSDN Community Support | Feedback to us

Answer 6

Hi Alex,

It seems that my reply no help, if so, would you mind providing us the detailed information?

If you are using the TFS build, I will involve the TFS build expert to see this issue, but we hope that you could provide us more information or you could share us a screen shot about this issue, so we could resolve this issue as soon as possible. Thanks for your understanding.J

Have a nice day,

---

Jack Zhai [MSFT]
MSDN Community Support | Feedback to us

Answer 7

Please see my original post for details. If the level of details is not enough you need to tell me exactly what kind of details you want.

Answer 8

Hi Alex,

Glad to receive your reply.

As my previous reply:

We need to make sure that:

1. Are you running the test with TFS build?
2. If you are using the TFS build. Which way are you using in the following screen shot? “Test assembly file specification (recommended) or (Test metadata file(.vsmdi))”?

Since as you said that you are running them on build server, so I’m not very sure whether it is a VSTS-Testing issue or a TFS-build issue. It will be better you could share us a screen shot about your issue.

We need to be able to run tests without .vsmdi, only .orderedtest.

If you want to run the orderedtest without .vsmdi in the TFS build process template, you could use the “Test assembly file specification (recommended)”, and add the orderedtest like the screen shot in my third reply. 

Best Regards,

---

Jack Zhai [MSFT]
MSDN Community Support | Feedback to us

Answer 9

I'm moving exactly in same problem you're experiencing. Were you able to advance on running ordered test from a TFS Builder agent without using vsmdi? My case is we're planning to move shortly toward new TFS 11 and several of our tests are running based on a set of ordered tests, quite similar to yours. Since TFS 11 doesn't support vsmdi, we need to find an alternative way to execute these tests in an ordered way.

---

http://www.codinghorror.com/blog/2007/03/the-works-on-my-machine-certification-program.html

Answer 10

We ended up copying .orderedest container to the output directory.