locked
Stop VS from deploying test assemblies with VS 2013 Update 3 RRS feed

  • Question

  • We have setup our Coded UI test framework so that all of the test binaries are built into a subfolder under the product under test.  This allows us to use relative paths to launch the product as well as do string resource lookups from our tests to find localized controls and messages.  The following show the structure when debugging locally:

    Bin

     +-Debug (Contains the product exes and their dependencies)

            +--Test (Contains CUI Test dlls & their dependencies)

            +-- en/fr/etc.  (localized resources)

    When we made the switch from VS 2010 to VS 2013, I tried switching over to using the runsettings file instead of testsettings, however, it seemed to insist on copying and running the test DLLs under the TestResults\...\Out folder and we lost reliable access to the product assemblies, so we continued to use the testsettings file. 

    Today, I installed VS 2013 Update 3 but now when I run our tests referencing our unmodified testsettings file, the test DLLs are copied to the TestResults\...\Out folder and they are referenced from there.   This is causing our test code to fail when trying to load string from the product resource assemblies. 

    Any ideas on how to avoid copying these file and running them from the TestResults location? The following is the testsettings file which had been working in VS 2013 without any updates applied:

    <?xml version="1.0" encoding="UTF-8"?>
    <TestSettings name="Local" id="a16ca785-42f0-43c2-bb8b-6bd15c781cca" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
      <Description>These are default test settings for running tests locally from VS or using the RunUITest command script.</Description>
      <Deployment enabled="false" />
      <Execution>
        <Timeouts runTimeout="42300000" testTimeout="600000" />
        <TestTypeSpecific>
          <UnitTestRunConfig testTypeId="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b">
            <AssemblyResolution>
              <TestDirectory useLoadContext="true" />
            </AssemblyResolution>
          </UnitTestRunConfig>
        </TestTypeSpecific>
        <AgentRule name="LocalMachineDefaultRole">
        </AgentRule>
      </Execution>
      <Properties />
    </TestSettings>

    I’ve also again tried using a runsettings file without any luck.  The following is the MSTest section from the file where I tried to disable deployment:

      <!-- MSTest adapter -->

      <MSTest>

        <!-- See http://msdn.microsoft.com/en-us/library/jj635153.aspx for more information -->

        <ForcedLegacyMode>false</ForcedLegacyMode>

        <IgnoreTestImpact>false</IgnoreTestImpact>

        <!--<SettingsFile></SettingsFile>-->

        <KeepExecutorAliveAfterLegacyRun>false</KeepExecutorAliveAfterLegacyRun>

        <CaptureTraceOutput>true</CaptureTraceOutput>

        <MapInconclusiveToFailed>true</MapInconclusiveToFailed>

        <CaptureTraceOutput>false</CaptureTraceOutput>

        <DeleteDeploymentDirectoryAfterTestRunIsComplete>false</DeleteDeploymentDirectoryAfterTestRunIsComplete>

        <DeploymentEnabled>false</DeploymentEnabled>

        <InProcMode>false</InProcMode>

      </MSTest>

    Any help on a resolution would be greatly appreciated.

    Thanks.


    >>>DJS>>>


    • Edited by DJS-MTS Wednesday, August 13, 2014 9:15 PM
    Wednesday, August 13, 2014 9:14 PM

Answers

All replies

  • Hi DJS-MTS,

    >> This is causing our test code to fail when trying to load string from the product resource assemblies. 

    Could you tell me how you load string from the product resource assemblies? Please also share us the detailed error message in your side?

    >>Stop VS from deploying test assemblies with VS 2013 Update 3

    Do you mean that you wouldn’t like to copy the assemblies or files to you out folder?

    If yes, you could try to run test using VS2013 IDE, it will not copy the assemblies or files to you out folder.

    As far as I know, the out folder is the actual deployment folder, to which assemblies and other deployment files are copied when the test run starts. So if you deployed some assemblies or files using testsettings file, it will default copy these assemblies or files to you out folder when you run the test.

    However, there have some differences in VS2010 and VS2013.

    In VS2010, if you deployed some assemblies or files using testsettings file and then run test using VS IDE or Mstest.exe, it will copy the assemblies or files to you out folder.

    In VS2013, if you deployed some assemblies or files using testsettings file and then run test using VS IDE, it will not copy the assemblies or files to you out folder. Since we run test using VS2013 IDE, the TestResults folder will not save any test results. So it will not also copy the assemblies or files to you out folder. But if you run test using mstest.exe, it will also copy the assemblies or files to you out folder.

    Therefore, if you run your tests using Mstest.exe, it will default copy the assemblies or files to you out folder, so if you call mstest.exe in your run setting file, it would have the same result. But if you call your test setting file using VS2013 IDE, it will not copy the assemblies or files to you out folder.

    If I have misunderstood this issue, please feel free to let me know.

    Best Regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place. Click HERE to participate the survey.


    • Edited by Tina-Shi Friday, August 15, 2014 7:31 AM
    Friday, August 15, 2014 7:31 AM
  • Thanks for the reply.

    DJS> Could you tell me how you load string from the product resource assemblies? Please also share us the detailed error message in your side?

    We reference the strings directly by adding a reference to the product DLL which contains the string resources, we then just reference them through the string resource classes directly in the test code to get names and values when necessary while running our tests.

    DJS> Do you mean that you wouldn’t like to copy the assemblies or files to you out folder?

    Yes, we don’t want to deploy any assemblies to the Out folder.  We have a large number of assemblies in the product so their projects are all configured to build directly into a common Bin\Debug or Bin\Release directory.  The test assembly projects are configured build into a sub-directory of the project build directory, so either Bin\Debug\UITest or Bin\Release\UITest.  The Deployment option is disabled in the .testsettings file, so we are not deploying anything in addition to the test DLLs.  I have setup a custom assembly resolver for the test code which will find look for the product assemblies either in the parent directory or the language specific sub-directory of the parent for our references to the string resources.

    We where never able to move to VS2013 for another issub, but prior to installing VS2013 update 3, running from the VS2013 IDE or MSTest.exe from the command line, we wouldn’t get any files copied to the TestResults\...\Out folder.  The test assembly would be loaded from the Bin\Debug\UITest folder as expected and they would find the product assemblies perfectly using the assembly resolver and relative paths.

    DJS> Please also share us the detailed error message in your side?

    I’ve just booted my test VM get the error messages and I’m rather puzzed.  Now when I run the tests, I’m not seeing the behavior described.  I do not see any of the test assemblies in the out directory and I’m back to seeing the  Update 2 bug that I reported at:

    https://connect.microsoft.com/VisualStudio/feedback/details/893888/ui-test-fails-after-vs-2013-update-2-install

    I’m not sure what’s changed.  The only thing I did was try and create a runsettings file in the project.  Reverthing that file and the updated .sln file didn’t bring back the original behavior. 

    Frustrating, but this appears to be a non-issue now.  Now if we can get a resolution to the the bug listed above, we can get caught up on the VS updates.

    Thanks.

    >>>DJS>>>


    Monday, August 18, 2014 3:22 PM
  • Hi DJS,

    >> Frustrating, but this appears to be a non-issue now.  Now if we can get a resolution to the bug listed above, we can get caught up on the VS updates.

    It means that the error message issue has been resolved, am I right?

    If yes, what is the real issue now and could you please tell me detail message about your issue?

    >>We where never able to move to VS2013 for another issub, but prior to installing VS2013 update 3, running from the VS2013 IDE or MSTest.exe from the command line, we wouldn’t get any files copied to the TestResults\...\Out folder. 

    Do you mean that you wouldn’t get any files copied to TestResults\...\Out folder before you didn’t install VS2013 update3? Or you wouldn’t get files copied to TestResults\...\Out folder after you install VS2013 update3.

    According to your description, I try to some operations in my side. I find that when we used the testsettings file or deployment attribute method to deploy file and run tests in VS2013 IDE, it will not copy the any files toTestResults\...\Out folder.

    However, when we used the testsettings file or deployment attribute method to deploy file and run tests using Mstest.exe, it will default copy the files to TestResults\...\Out folder.

    Please tell me more detail message about your issue now so that we will further help you solve this issue.

    Thanks for your understanding.

    Best Regards.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place. Click HERE to participate the survey.


    Wednesday, August 20, 2014 12:26 PM
  • Tina-Shi,

    Thanks for your reply.  I have not seen any issues with deployment since my initial work with it.  I'm not sure what I did to change the behavior.  I did try changing over to a run settings file again but didn't get too far with that.  The next time I booted the VM to try and get more info, it worked as it had prior to Update 3.

    The other issue that I'm having with Update 3 also existed in Update 2 and doesn't appear to have been fixed.  It looks like the VS UITest assembly resolver is throwing an exception while our application under test is running.  I filed the following connect bug but haven't heard any update on it in a while.

    https://connect.microsoft.com/VisualStudio/feedback/details/893888/ui-test-fails-after-vs-2013-update-2-install

    Thanks for your help here.


    >>>DJS>>>

    Wednesday, August 20, 2014 12:48 PM
  • Hi DJS,

    Thanks for your friendly response.

    You know that one thread for one issue, as you said that the first issue has been resolved, if so, I will mark and close this case. For the new issue, if possible, please open a new case in our forum, our team members would follow up your new issue.

    >> The other issue that I'm having with Update 3 also existed in Update 2 and doesn't appear to have been fixed.  It looks like the VS UITest assembly resolver is throwing an exception while our application under test is running.  I filed the following connect bug but haven't heard any update on it in a while.

    https://connect.microsoft.com/VisualStudio/feedback/details/893888/ui-test-fails-after-vs-2013-update-2-install

    For the new issue, as you already submit one feedback on the Microsoft Connect site, let's focus on the case on that channel, the senior members would provide more professional support there.

    Thanks for supporting our work.J

    Have a nice weekend,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place. <br/> Click <a href="http://support.microsoft.com/common/survey.aspx?showpage=1&scid=sw%3Ben%3B3559&theme=tech"> HERE</a> to participate the survey.

    Friday, August 22, 2014 8:26 AM