DataSource and DataDirectory issues

Question

I have added an xml file to my solution at the Solution Level called DataSources.xml.   The file is visible in the Solution Explorer in the Solution Items virtual folder. For certain unit tests, I add a DataSource attribute to create data-driven tests to run once for each row in the xml file.

Here is the attribute I add to the tests:

[DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "|DataDirectory|\\DataSources.xml", "Datasource", DataAccessMethod.Random)]


When I run the tests from the command line via mstest, the xml file is located and the tests execute properly.  However, when I run the tests within the VS2008 IDE, they fail with the error:

"The unit test adapter failed to connect to the data source or to read the data. For more information on troubleshooting this error, see "Troubleshooting Data-Driven Unit Tests" (http://go.microsoft.com/fwlink/?LinkId=62412) in the MSDN Library.
Error details: Object reference not set to an instance of an object."

Is the DataDirectory different for the command line execution versus running within the IDE?   What is the proper way to set up a data-driven test to utilize a file included in the solution so that it will work both via command line and within the IDE?

The reason I can't hard-code a path to the file is that the solution is checked out of source control by QA, cleaned and compiled, then the tests are run via command line on the newly compiled solution.  The files are checked out and executed with different paths depending on the system they are checked out on.

Thanks!

Answer 1

Hello,

 

This should work from within the IDE as long as you are deploying your xml file. Are you currently deploying it? You can do this using [DeployItem] attribute on the test itself, or through your testrunconfig file (in your solution directory)

 

Thanks,

David Gorena Elizondo

[MSFT] VSTT

Answer 2

What is the relative path from a project within a solution to a solution level item in the root solution folder?  I added deploy item for the xml file and the tests still fail from the IDE and pass from the command line.  I tried:

[DeploymentItem("..\\..\\DataSources.xml")]
[DeploymentItem("..\\DataSources.xml")]
[DeploymentItem("DataSources.xml")]

I can't hardcode the path (c:\blah\DataSources.xml) because our QA team checks out the tests and runs them.  they might not check it out to the same exact path.  I need it to be relative.

I already had the deployment item in my testrunconfig file:

<Solution Directory>\\DataSources.xml

What can I try next?

Thanks!

Answer 3

Hi Patrick

 

You can try the following workaround:

1. Configure the test run config file to deploy DataSources.xml as you already did. After this, there's no need to add DeploymentItem attribute for DataSources.xml.
2. In the unit test, set DataSource attribute to [DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "DataSources.xml", "Datasource", DataAccessMethod.Random)]

 

This can work if you run tests from IDE. When you run tests from command line, please call MSTest with testmetadata and testlist switch. MSTest read the config file and copy deployment items to the Out folder. Then the test framework should be able to find data source.

 

I look forward your test results of the workaround.

Answer 4

The tests still fail in the IDE and pass on the command line.  The current setup is as you specified, testrunconfig has the deployment information and each test has the DataSource attribute.

Here is another wrinkle, though, if it helps debug this.   If I run the 6 attributed DataSource tests by themselves within the IDE, they execute and pass.  But, if I run them in the same run as 200 other tests in the solution, they fail.

Could something in one of the other tests be making the xml file unreachable?  Only those 6 tests use the datasource.

Answer 5

Patrick

 

This is weird. And I got a few more questions:

1. You mean all tests can pass if you run them from command line?
2. Are all your unit tests in the same test project?  If so, do other test projects contain data driven tests? Make sure their data source file name is not the same.
3. If you just run tests that don't use DataSources.xml in Visual Studio IDE, can them pass?

Answer 6

1.  Yes, all tests pass when run from the command line.  All tests pass in the IDE (selecting every test in the Test View window) except for the 6 DataSource tests.

2.  All of the unit tests are in the same Solution, but they are in 15 different projects.

3.  I'm not quite sure what you mean here.  If I run the non-DataSource tests in the IDE they pass.  If I remove the datasource from the datasource tests (hardcode data), they pass.

Answer 7

Any other suggestions?  Can I try something else to provide you with more information?

Answer 8

Patrick

 

I'm sorry for not following up timely because I'm not sure why the datasource is corrupted by other tests and cause unit test adapter failed. Actually, IDE and command line tool use the same engine to run tests. By creating some simple projects, could you please figure out how we can repro this issue? This can be very helpful for the troubleshooting.

Answer 9

I had a similar error when using VSTT's dataSources element in VS2008.  Error msg was:

"Unit Test Adapter threw exception: Object reference

not set to an instance of an object.."

 

Turns out this was caused by a connectionString misspelling in the dataSources <add> element.  Following is corrected:

<connectionStrings>

<add name="DatabaseUnderTest" connectionString="Data Source=sabdrkprfsql01;Integrated Security=SSPI;Initial Catalog=Br3_Main;" providerName="System.Data.SqlClient" />

</connectionStrings>

<microsoft.visualstudio.testtools>

<dataSources>

<add name="WebpageDataSource" connectionString="DatabaseUnderTest" dataTableName="vWebpageCheckedIn" dataAccessMethod="Sequential"/>

</dataSources>

</microsoft.visualstudio.testtools>

 

Answer 10

As a sort of closure to this thread, I was never able to get DataSource and |DataDirectory| to properly work together. 

If you read this Bill, perhaps you can add a bug to get a better error for the failed connection to a database.  It isn't clear if the file doesn't exist, if it is inaccessible due to permissioning issues, if the path doesn't exist, or if the file is fine and a table inside the file has a problem.  A more precise error message would be greatly appreciated.

In the end, I just stopped including the XML file in my solution and put it out on a server share that should be accessible for any of my test runs.  I hate to have an external dependency like that, but MSTS just wasn't cooperating. 

Another change request suggestion would be to allow the DataSource attribute to use embedded resources.  That way, I could build the XML file into the solution and still use it for data-driven unit testing.

Answer 11

Hi Patrick

Thanks for and suggestion. Customer's need is the first priority when we design our products. You are encouraged to file this suggestion to Visual Studio Connect. And you can also add to the watchlist of this suggestion. We appreciate for your feedback.

Answer 12

Maybe this could work for you, it worked for me:
http://www.sidesofmarch.com/index.php/archive/2008/12/29/mstest-deploymentitem-and-the-frustrating-relativepathroot-setting

Answer 13

I hit this same issue with VS 2010 Ultimate B2. 

 

 

//[DeploymentItem("TestAutomation\\XMLFile.xml"), DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "|DataDirectory|\\XMLFile.xml", "UserLogin", DataAccessMethod.Sequential), TestMethod]

[

 

DeploymentItem("TestAutomation\\XMLFile.xml"), DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", @"c:\users\dirkste\documents\visual studio 10\Projects\TestAutomation\TestAutomation\XMLFile.xml", "UserLogin", DataAccessMethod.Sequential), TestMethod]

The hardcoded path works in the ide just fine but I use the relative path (|DataDirectory|)I get the same error Patrick mentions.

Answer 14

Hello,

In my dll refrence I have changed the "Microsoft.VisualStudio.QualityTools.UnitTestFramework" refrence from 8.0 to 9.0 and it solved my my problem.

Thanks!

Answer 15

Add the following attributes to the test method:

[DeploymentItem("..\TestSolutionDir\TestProjedtDir\TestDataFolder\TestAutomationSpreadsheet.xlsx")]

[DataSource("System.Data.Odbc", "Dsn=Excel Files;dbq=|DataDirectory|\TestAutomationSpreadsheet.xlsx", "SpreadsheetTabName$", DataAccessMethod.Sequential)]

The |DataDirctory| variable is defined by the system when it runs the test. The Deployment item copies the spreadsheet there. You point to the spreadsheet and to the tab with in the spreadsheet that the data is coming from. Right-click on the tab to rename it to something easy to remember

Answer 16

Hello everybody:

I want to share what I'm doing to use custom datasource files when I want to execute from my VSTS IDE as well as using MSTest command Line.

Basically, I use the DataSource annotation in my unit test like this |DataDirectory|\Config\Datasource.xml. I works when it's runned from IDE.

When I need to execute the test from command line using MSTest I change the Deployment settings from the test.settings file like this:

<

 

Deployment

>

<

 

DeploymentItem filename="Config\Datasource.xml"

/>

</

 

Deployment>

With a custom class I copy the Datasource.xml file to the execution directory just before run the MSTest command. As you can see, It's not necessary to change the Unit Test.

private

 

 

const string CONFIG_FILE = "Datasource.xml"

;

 

 

private const string CONFIG_LOCATION = "Config\\Datasource.xml";

...

 

string

 

 

destination = Path.Combine(Directory.GetParent(Assembly

.GetExecutingAssembly().Location).FullName, CONFIG_LOCATION);

 

 

File.Copy(Path.Combine(ResultPath.Get(context), CONFIG_FILE), destination, true);

 

I hope this will help you

JUAN CAMILO RINCON

http://tfs2010testlibraries.codeplex.com/

 

Answer 17

I could tell two possible approaches from my experience:

1) If you want the datasources to work from the IDE:

-make sure that you have added the deployment item in the testsettings/testrunconfig

2) If you want to make sure that the datasources work from the command line:

-make sure that the copy to output property of the file is set to 'copy always'

-add a deploymentitem attribute with the relative path specific to your output folder (bin\debug), for eg if the file is present in bin\debug\test\file.xml, then add DeolpymentItem("test\\file.xml")

Thanks,

Anuj

Answer 18

I had the same problem, running on Windows 7 (not on XP).

I solved it changing the CopyLocal property to true of the reference to the Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll on the References folder at the test project.

Pedro Herrarte

---

Devjoker

Answer 19

For me, the issue was the deployment was not happening, even though I had the attribute 

        [DeploymentItem(@"SearchTests\SearchTestData.xml")]

on my method. 

I thought this took care of the "make sure that you have added the deployment item in the testsettings/testrunconfig" comment above. 

If you are having this same problem, check in the bin/Debug folder. If the XML file isn't there, you aren't deploying. I did that manually and my tests started running.

Obviously, you don't want to manually deploy that file; the solution is to open the Local.testsettings. In VS10, this will open up a properties page. You want the Deployment section. There's a checkbox right at the top labeled "Enable Deployment." By default, that's turned off. Turn it on, and away you go.

If you're more of an XML editing type, the solution is not obvious. The default, deployment disabled, means that this is a child of the <TestSettings> root:
  <Deployment enabled="false" /> 

You'd think that you just change "enabled" to "true" and you're good. But that doesn't work; if you turn it on with the dialog interface, it simply deletes the <Deployment> node entirely. Probably want to file a bug on that one; it should check the enabled attribute of the <Deployment> node, not just the *existence* of a deployment node.

Hope this saves someone else the frustration & lost time I experienced. Data-driven testing is such a huge win for software quality, it's a shame it's not easier to implement (hint to the VSTesting team: please feel free to copy mbUnit's approach, at least to get more people started with DD unit testing. Yes, the data-source driven approach you have now is better in the long run, but it's hard to get working).

Answer 20

Hi Patrick,

You need right click your XML file, choose "Properties".

Change the "Copy to Output Directory" build action  to "Copy Always".

Hope this will solve your problem.

Thanks.

Nathan Jin.