VS/MsTest does not refresh tests automatically if they generated by a SingleFileGenerator

Question

Hi,

I have a SingleFileGenerator that can generate MsTest test classes from another source. (The SingleFileGenerator is invoked whenever I save the source file by Visual Studio. Also VS takes care of saving the generated test classes to the behind file.)

It works generally great, but there is a strange situation (which is quite frustrating):
1. I some tests already generated.
2. I run all tests in the solution. -> works fine
3. I modify something in my source file, that results in a new test method in the test class.
4. I save the file, the SingleFileGenerator is generating the updated test class with the new method.
5. I build (even rebuild) and again run all tests in the solution. -> the newly added test method is not included!

It seems that the test engine did not realize that the test class has been updated. Build/rebuild does not help, the only way to run the new test is to a) touch the generated file manually and save b) click the "refresh" button on the "Test View" window.

My suspicion is that the test runner is somehow triggers the refresh event only when the test class file is interactively changed (someone types something in), and not when the Visual Studio updates anyhow the file content.

Is there any way to fix / workaround this issue?

---

Gaspar Nagy / TechTalk

Answer 1

Hello Gaspar,

You also may post your issue on the Visual Studio Extensibility Forum to see if any community members have came across this issue and could offer some suggestions.

---

Please remember to mark the replies as answers if they help and unmark them if they provide no help.
Send us any feedback you have about the help from MSFT at [e-mail].

Answer 2

Yeah, I was also thinking on that.

I did it now, here is the VSX post: http://social.msdn.microsoft.com/Forums/en-US/vsx/thread/b88c8968-2528-409b-a229-11469fb2f798

---

Gaspar Nagy / TechTalk

Answer 3

Gaspar,

How VS updates tests:
- When build happens: when a test project is built we refresh tests from the build output (dll)
- When you run tests (such as run all tests in solution): we check whether test project needs compilation and if it does we will built it and refresh tests from the build output
- Manual click on refresh button in Test View/Test List Editor

It looks like the generator tool either is not setting the timestamp sooner than before it was run, or VS doesn't think that the project needs compilation.

When your generator runs, does it cause VS to realize that the generated file it dirty (so that '*' appears in the doc title)?
If you build the project after generator runs, does VS actually build the source?
What happens if you modify generated file manually (so that '*' appears in the doc title) and then runs all tests in solution without click on refresh button?

Thanks,
Michael Koltachev
Visual Studio Team Test

Answer 4

Hello Michael,

Thank you for the reply.

The code generation is not done directly by my code, but through the IVsSingleFileGenerator interface of Visual Studio. This interface has a Generate method, that I implement. Here I only need to return the content of the file to be generated (as text), and everything else (creating, updating and saving the generated file) is handled by the Visual Studio.

The SingleFileGenerator infrastructure works in a way, that the generation is invoked when the main file (where the generator is registered to) is saved. Because of that, Visual Studio automatically saves the generated files, so it won't be dirty.

The timestamp update works properly, I think, because the build porcess realizes the change and rebuilds the project if I regenerate the file.

If I make any manual change in the generated file, than the test runner refreshes the file content properly, and I don't have to push the refresh button manually.

If that helps, I can create a simplified SingleFileGenerator, that shows the problem, and I can send it to you for deeper investigation.

As a solution, I was thinking on firing some event from the generator (maybe with a timer, if nothing else helps) that makes the MsTest collection refresh. Can you maybe help me with a hint, how to trigger this refresh programatically?

Thx,
Gaspar

---

Gaspar Nagy / TechTalk

Answer 5

FYI: the question was answered by Chao Kuo on the VSX forum. He has provided a code to refresh the tests from code after generation that works.

---

Gaspar Nagy / TechTalk

Answer 6

Thank you for sharing the solution here.

---

Please remember to mark the replies as answers if they help and unmark them if they provide no help.
Send us any feedback you have about the help from MSFT at [e-mail].