locked
Ordered Test Becomes Non-Runnable After Closing VS IDE

    Question

  • A valid and runnable Ordered Test becomes invalid and Non-Runnable after closing the VS IDE and then re-opening it.

    The “Non-runnable Error” is: “Cannot find the test 'XXXXXXX' with storage 'bin\debug\YYYYYY.dll'.”

    The only way we've been able to correct this problem is by removing the Unit Test (that VS thinks is invalid) from all of the Ordered Tests and then immediately adding it back. This solves the problem until you close the IDE again. But manually modifying every Ordered Test in our project every time the IDE is loaded is not a usable workaround for our team.

    Is there a fix or workaround that is permanent?

     

    Thanks,

    -Matt

    Friday, June 23, 2006 5:02 PM

Answers

  • I've worked with Matt on creating a repro of the problem. This is actually a bug that happens when ordered tests have a specific combination of items in them, and at the same time a specific combination of test names. I've logged a bug against this and I'll work on getting it fixed for next version.

    Thanks,
    David Gorena Elizondo
    [MSFT] VSTS

    Friday, June 23, 2006 11:18 PM

All replies

  • Hello Matt

    This can be happening for several reasons, for instance

    1. If after adding a unit test to an ordered test, you change the name of your unit test, then the ordered test will become not-runnable.
    2. If after closing the IDE, you move your unit test to another location, then the ordered test will become not-runnable as well.

    Can you please elaborate more on your scenario, letting me know step by step what you're doing? That will definitely help me in understanding the problem.

    Thanks,
    David Gorena Elizondo
    [MSFT] VSTS

    Friday, June 23, 2006 6:39 PM
  • It's neither #1 or #2, or at least we are not physically performing either of those steps.

    Here is what we are doing:

    1) Open VS IDE with a valid project.

    2) Notice that all Ordered Tests are Non-Runnable

    3) For each Ordered Test, Remove Unit Test "X" (that is causing the problem) from the Ordered Test's list. *No Renaming, just removing from the Ordered Test's list.*

    4) Immediately re-add the Unit Test "X" back to each Ordered Test. *Again, no changing anything else.*

    5) Absolutely nothing has changed from a functional perspective.

    6) Notice that all Ordered Tests are now Runnable

    7) Save Changes.

    8) Close VS IDE.

    9) Immediately repeat from step #1 *Don't change a thing*

    It happens over and over, every time we start the IDE; even after we've corrected the error and saved the changes.

     

    Friday, June 23, 2006 7:02 PM
  • In Step #4, how are you re-adding unit test "X" ? Are you creating a new test using the "Add new test" dialog?

    If you look at your files on disk, what is the difference between the unit tests at the point when you see your ordered tests runnable, and the point where you see them not-runnable? For example, look at your project hierarchy in disk, and you should have something like testprojectX\testprojectX\unittest1.cs; is unittest1.cs in the same location when you see the ordered test not-runnable?

    Finally, can you repro that scenario with a newly created project?

    Thanks,
    David Gorena Elizondo
    [MSFT] VSTS
    Friday, June 23, 2006 7:56 PM
  • I've worked with Matt on creating a repro of the problem. This is actually a bug that happens when ordered tests have a specific combination of items in them, and at the same time a specific combination of test names. I've logged a bug against this and I'll work on getting it fixed for next version.

    Thanks,
    David Gorena Elizondo
    [MSFT] VSTS

    Friday, June 23, 2006 11:18 PM
  • The same thing happens to me, also with load tests.

    Is there yet a way to fix this problem?

     

    Thanks,

    Ricardo

    Wednesday, September 13, 2006 5:09 PM
  • I think I've found a workaround for this:

    I had a folder named webtests with webtests in it, a folder loadtests with loadtests and a folder orderedtests with ordered tests.

    I created the ordered and load tests on the root of the project and now they seem to be working fine, so far. (the xml of the oredered test had "../webtests/test.webtest" like paths, which led me on trying this, don't know if this has to do with the actual problem but may be a hint).

    As for the repeated tests on the test list from where the tests are added to the ordered test, deleting the TestResult folder where VS2005 stores test results seems to alleviate this problem most times.

    I hope this may help someone having this same problem.

     

    Happy coding :P

    Ricardo

     

     

     

     

    Wednesday, September 13, 2006 6:00 PM
  • Hi,

    my problem is very closely related to this one. Currently, I'm calling a few ordered tests from a .bat file, all of which are executed with the mstest command.


    Please help.

    Ashik

    Thursday, December 27, 2007 11:59 AM
  • We're having the same issue - we have over 500 unit tests and would really like to make the running of them more automated. Our tests must run in a particular order and whenever we introduce OrderedTests the tests come up as non-runnable.

    Is there any workaround you can offer? We have tried puttint the OrderedTests in a separate Unit Test Project. This did not help.

    Sunday, July 20, 2008 9:40 PM
  • A hotfix would be very helpful.

     

    Monday, August 11, 2008 9:03 PM