locked
How to get code coverage from Web Tests? RRS feed

  • Question

  • Hi,
    I'd like to get code coverage information from Web Tests. I configure the testo configuration in order to instrument the Web Site. I Run my web test but no coverage information was generated.
    Is there any additional Step?

    Thanks

    Wednesday, May 4, 2005 8:31 PM

Answers

  • Lucas, based on the information you've given me it sounds like your project looks something like the following:

    Solution
      c:\....\website Project
         default.aspx 
            OnButtonClick calls method in code.vb
      Class Library Project
         code.vb
      Test Project
         web test

    When you run the web test you aren't getting any code coverage data.  This looks like a bug in our code.  We aren't doing something correct with web tests.  I was able to do this successfully several times, but I have to shut down the IDE after each run otherwise the test fails.

    Can you create a unit test to exercise your code?  That should work.  You can either create a test for the class library, or you can create a test for the method that calls into it but doing this requires you to have the code under App_Code because it's part of the ASP project.

    Sorry for the inconvenience.  If I come up with other ideas for you I'll let you know.

    Chris
    Tuesday, May 10, 2005 9:53 PM
  • Gerhardo, things work fine for me for unit tests + code coverage on an ASP.NET web service.

    NOTE: If you are using a non-console Terminal Server session, code coverage may not work properly.

    Here are the scenarios that worked for me:

    Web Service and Test Project in same solution:
    1. Create a new HTTP web service.
    2. Right click a method and click Create Tests...
    3. In the test project that is created, verify that unit tests run fine by opening the test view and running a unit test.
    4. Open test run configuration by double-clicking the localtestrun.testrunconfig node in Solution Explorer
    5. Under code coverage, check the box for the web service dll. Save and Close
    6. Run the unit test again
    7. Once the test results window shows that the test run is done, right click in the test results window and click Code Coverage Results
    8. See the code coverage results

    Web Service and Test Project in different solutions:
    1. Create a new HTTP web service.
    2. Open another instance of Visual Studio and create a new test project
    3. Add a web reference to the web service created in step 1.
    4. Change the code for TestMethod1 to test a method from the web service DLL
    For example, I added a "using TestProjectX.localhost" at the top and changed the code to:

        Service s = new Service();
        if (!s.HelloWorld().Equals("Hello World"))
        {
            Assert.Fail("HelloWorld() returned an incorrect value");
        }


    5. Verify that the unit test works fine by going to the test view and running the unit test
    6. Open test run configuration by double-clicking the localtestrun.testrunconfig node in Solution Explorer
    7. Under code coverage, check the box for the test project dll. Save and Close
    8. Run the unit test again
    9. Once the test results window shows that the test run is done, right click in the test results window and click Code Coverage Results
    10. See the code coverage results

    I am not sure why it didn't work for you; one possibility is if you were using a non-console TS session. If you were not, please let me know as I will like to investigate more then.

    Hope this helps. Don't hesitate to shoot more questions. Thanks for your patience.
    Tuesday, May 10, 2005 11:18 PM

All replies

  • Lucas,

    I don't have an answer for you, but I ran into a similar problem. Hopefully two of us having a similar problem will attract the attention from somebody at Microsoft.Smile

    I'm having the same issue with a Web Service. I added a Web reference to my unit test DLL. I gave me the option to instrument the DLL in its original location (virtual directory under IIS). I reset IIS, and instrumented the code from within the test manager. That produced no output.

    I then instrumented the code using vsinstr and started the monitor with vsperfmon before running the unit test. It created a coverage file alright, but when I opened it for analysis, contained no data.

    What's the trick to getting this to work with web stuff?
    Friday, May 6, 2005 4:07 AM
  • I tried to figure out this yesterday but build I had some different issue that prevented me to run web tests. I sent an e-mail to get some answer to original post but I haven't got response yet. I will update it when I get answer.
    Generally to collect code coverage you need to tell to replace binaries you are collecting code coverage with instrumented bits. For binaries to be intrumented successfully for code coverage they also has to have symbol information or symbol files. Did you have symbol files or did you get any errors running vsinstr tool?

    Thanks,
    Bata
    Friday, May 6, 2005 6:17 PM
  • A few question regarding what you are trying to do:

    • What do you mean by "adding a web reference to the unit test DLL"? Do you mean you added the test project to the references to the web site?
    • "It gave me the option to instrument the DLL in its original location" - what dialog gave you such an option and what did you do to bring up the dialog?
    • How did you instrument the code from test manager?


    NOTE that even if you instrument the binaries manually, ASP is going to recompile and will replace your instrumented binaries with uninstrumented binaries. So you will not get any data. This is probably what you are seeing. To get data with ASP projects, you need to configure it to perform instrumentation after the compilation step.

    Friday, May 6, 2005 9:32 PM
  • How can I do that? ("perform instrumentation after the compilation step")

    Thanks
    Monday, May 9, 2005 3:20 PM
  • The reason I asked you the few questions I asked is to understand exactly how you are compiling and launching the ASP web page.

    When browsing to a dynamic web page like ASP, the code is compiled on the fly for the first time and is cached so that the second time it is not required to compile.

    MunjalD

    Monday, May 9, 2005 5:43 PM
  • OK. The first two questions are not for me (I didn't post that sentences).
    I created a Web Site (using file system), then I created a Test Project and added a WebTest that exercise the web site.
    I tried to enable Code Coverage for the Web site through Test Run configuration, selecting the Web Site folder for instrumentation (it appears in the list available for instrumentation).
    But nothing happened and code coverage is not collected.

    What should I do to make Web Test code coverage available?

    Thanks
    Monday, May 9, 2005 6:00 PM
  • 1) Did the test succeed?
    2) Do you have code in the App_Code folder?
    3) Did you look at the Run Details to see if there was any error/warning from code coverage?
    4) What is the text in the code coverage results window when you expand the node?  Does it say there was an error, or no data collected?
    5) Can you instrument the test project as well as the web site and run the test.  You should at least get data on the test project.  If you get data, that shows the monitor is collecting data.
    6) Are you doing anything special with the website?  Are you running it as a different user or in a different context of any kind?

    Chris

    Monday, May 9, 2005 7:09 PM
  • Gerhardo, have you solved your issues?  If not, is this thread helping you?  If this thread is diverging from your problem, you might want to start another thread so we can help resolve your issues directly.
    Monday, May 9, 2005 7:42 PM
  • I have a web service dll, MyWebService.Dll, with a method Foo.. To test MyWebService.Dll, I created a standalone test project, UnitTest.dll. I added a web reference to UnitTest.dll pointing it to MyWebService.Dll. The proxies to call MyWebService.dll were created properly.

    I ran the unit test and verified that it worked. Afterwards, I went to 'Test', Edit Configuration and turned on code coverage for MyWebService.Dll. I tried instrumenting it once with the 'Instrument in place' checkbox turned on and once with the 'Instrument in place' checkbox turned off.

    Both times a code coverage file was created. When I analyzed it, it said that no information had been collected.

    I then reset IIS and instrumented the DLL from the command line. I started coverage from the command line and then ran my unit test again. I turned code coverage off in the test configuration, as it had already been started from the command line.

    The test ran fine. I stopped the code coverage collection and looked at the coverage file. The result was the same. The file was created, but no information collected. I used process explorer from sysInternals to check whether the worker process was using the instrumented DLL. It was. The DLL it had loaded was twice the size of the regular DLL.

    What am I missing? The actual message is: None of the instrumented DLLs were used.

    The worker process had actually created a copy of the MyWebServices.dll in the temporary asp.net folder. I checked the size of those DLLs in memory and they were the same size as instrumented web service dll. Is the symbol file supposed to located in the same place as the DLL at run-time? the pdb file was not copied over to the temporary asp.net directories.

    Tuesday, May 10, 2005 2:06 AM
  • 1) Yes, The Test Succeed
    2) No. I have code in the Bin folder because my site reference another project (class library,that have one class with one shared method)
    3) There is no information about code coverage.
    4) "Empty results generated: none of the instrumented binary was used. Look at test run details for any instrumentation problem."
    5) I added the Test assembly and the class library project (referenced by my site) for instrumentation. Run the test again. It collected coverage information just for the class library (not the Test assembly or the Web Site)
    6) Nothing special. From the click event of a button placed in the Default.aspx I call a shared (VB) method that is part of a class that belongs to a Class Library. Everything running with ASP.Net Developer Server (using a fixed Port).

    Thanks in advance
    Tuesday, May 10, 2005 1:37 PM
  • Lucas, based on the information you've given me it sounds like your project looks something like the following:

    Solution
      c:\....\website Project
         default.aspx 
            OnButtonClick calls method in code.vb
      Class Library Project
         code.vb
      Test Project
         web test

    When you run the web test you aren't getting any code coverage data.  This looks like a bug in our code.  We aren't doing something correct with web tests.  I was able to do this successfully several times, but I have to shut down the IDE after each run otherwise the test fails.

    Can you create a unit test to exercise your code?  That should work.  You can either create a test for the class library, or you can create a test for the method that calls into it but doing this requires you to have the code under App_Code because it's part of the ASP project.

    Sorry for the inconvenience.  If I come up with other ideas for you I'll let you know.

    Chris
    Tuesday, May 10, 2005 9:53 PM
  • Gerhardo, things work fine for me for unit tests + code coverage on an ASP.NET web service.

    NOTE: If you are using a non-console Terminal Server session, code coverage may not work properly.

    Here are the scenarios that worked for me:

    Web Service and Test Project in same solution:
    1. Create a new HTTP web service.
    2. Right click a method and click Create Tests...
    3. In the test project that is created, verify that unit tests run fine by opening the test view and running a unit test.
    4. Open test run configuration by double-clicking the localtestrun.testrunconfig node in Solution Explorer
    5. Under code coverage, check the box for the web service dll. Save and Close
    6. Run the unit test again
    7. Once the test results window shows that the test run is done, right click in the test results window and click Code Coverage Results
    8. See the code coverage results

    Web Service and Test Project in different solutions:
    1. Create a new HTTP web service.
    2. Open another instance of Visual Studio and create a new test project
    3. Add a web reference to the web service created in step 1.
    4. Change the code for TestMethod1 to test a method from the web service DLL
    For example, I added a "using TestProjectX.localhost" at the top and changed the code to:

        Service s = new Service();
        if (!s.HelloWorld().Equals("Hello World"))
        {
            Assert.Fail("HelloWorld() returned an incorrect value");
        }


    5. Verify that the unit test works fine by going to the test view and running the unit test
    6. Open test run configuration by double-clicking the localtestrun.testrunconfig node in Solution Explorer
    7. Under code coverage, check the box for the test project dll. Save and Close
    8. Run the unit test again
    9. Once the test results window shows that the test run is done, right click in the test results window and click Code Coverage Results
    10. See the code coverage results

    I am not sure why it didn't work for you; one possibility is if you were using a non-console TS session. If you were not, please let me know as I will like to investigate more then.

    Hope this helps. Don't hesitate to shoot more questions. Thanks for your patience.
    Tuesday, May 10, 2005 11:18 PM
  • That's right. The estructure of my Solution is the one you described and the problem is that no data coverage is collected from the ASP.Net application.

    I could get code coverage from Unit Test exercising the Class LIbrary, no problem about that.
    The second thing you said are the ASP.Net Unite Test, right?

    We have to wait for a fix of this problem. Next CTP? :)

    Thanks.

    Thursday, May 12, 2005 5:40 PM
  • Hello MunjalD,

    Unfortunately, when I attempt to follow your example in #1 above, I get the result that was originally reported.

    I have a solution with a Web Service project and a test project. The unit test I am running is a data driven unit test.

    I enabled Code Coverage in a new test configuration and set it as the one to be run. I ticked the checkbox for the Web Service in the list of assemblies. When  I run the unit test, the test succeeds, but no code coverage is generated and I get the following error: "Empty results generated: None of the instrumented binary was used."

    I see nothing from my test results that would explain this error.

    What would you suggest I do next?
    Regards

    Edit: Note, to followup with Gerhardo's post below, I am running ASP.NET within VS, using Cassini. I haven't tried running on IIS.
    Tuesday, May 17, 2005 4:13 PM
  • Hi Munja,

    Thanks for the reply.  If I follow your steps above, the 'Hello World' example works fine. There are two major differences in our setup however: 1) We don't use the hosted ASP.NET process but go through IIS directly. 2) We have no code behind our *.asmx file. It simply points to a compiled class library.

    I assume you used Visual Studio to either create or open a new Web site for the "Hello World" example. In other words, you didn't go through IIS to run it but had Visual Studio or some external process host ASP.NET.

    When I change your example above and update the web reference to point to IIS, it doesn't work. To do so create a virtual directory in InetMgr and host your little 'Hello World' example under IIS. Update your web reference in your test project to point to \\localhost\YourvirtualDirectory.

    Once I do this, I can no longer get code coverage. I tried the command line tools to instrument the DLL and start the monitoring process as well as the GUI provided in Visual Studio 2005. No code coverage was collected.
    I also copied the DLL to the temporary directory off of Windows\Microsoft.Net\.... folder along with the symbol files, etc. Same result

    In other words, your scenario only works if you have Visual Studio host ASP.NET.


    There is another issue that we've been grappling with:
    We have no code in the web service project. It simply points to a class library. Our *.asxm file looks like this:<%@ WebService Language="vb" Class="WebServicesClassLibrary.Service" %>

    Even if you open the web site in Visual Studio and don't go through IIS, this does not generate code coverage for the WebServicesClassLibrary. The only way we could get that to work is to create a Service.vb class and have code behind the *.asxm file. The dummy class inherits from the Serivce.vb class exposed by the WebServicesClassLibrary.Service class. That worked for the 'Hello World' example, but when we tried it on more involved examples, the code coverage tool ended up with a bunch of errors when we were tyring to look at the data. We also had to restart the hosted ASP.NET process every time we ran a test. Otherwise it would throw exceptions.

    To make the tool really useful to us it has to
    a) be able to let us instrument a DLL used by ASP.NET when ASP.NET is hosted under IIS
    b) be able to collect coverage information from class libraries used by Web services without associating dummy classes with the *.asmx file that then inherit from the class that we are actually interested in which is located in a class library.

    We are not using non-console TS sessions and anything like that. If you have the time, I'd like to see if you can get code coverage to work when you create a class library  for your 'Hello World' sample and host ASP.NET under IIS.

    Thanks.

    Tuesday, May 17, 2005 5:54 PM
  • Michael, I am able to repro the problem you are reporting - creating a simple web service with unit tests and enabling code coverage is not collecting data. I am not sure how I was able to make it work the first time I replied to this post. Hence, I am investigating further. Also, Gerhardo says even he was able to succesfully run my Hello World scenario. Gerhardo, are you still able to get code coverage data for an ASP.NET Web Service from withing Visual Studio?

    We are working on fixing the issue; and I am also trying to find a work around for you. I will let you know as soon as I find the work around.

    Gerhardo, we are still investigating the issues you are facing. We know there are a few known issues with code coverage with ASP.NET projects, but we want to make sure we find out any work-around for you as well if it is possible.

    Thank you both for reporting the issues and your continued patience.

    Munjal
    (Miciosoft)
    Wednesday, May 18, 2005 9:17 PM
  • Gerhardo,

    Well, we have the opposite problem then. We do not host our web service in IIS yet - we get no test code coverage from VS Web Server.

    Regards,
    Thursday, May 19, 2005 2:14 PM
  • Michael,

    It seems a bit sporadic going through VS Studio as well. Can you take Cassini out of the mix and create a new Web site from within Visual Studio. Create your simple web service and then use the wizard to create a corresponding test project. New sites created through Visual Studio will use the little web server that comes with the framework.

    When we follow these steps, it usually works. There seems to be a problem of IIS or Casini not using the instrumented DLLs which is alleviated when you use their WebDev.WebServer.exe. It lives underneath the windows\framework path and is used as the default when you create a new web site from within visual studio.

    As long as you use it and have code behind your *.asmx file instead of just pointing to a class library it should work.

    You can rig up your project to use the WebDev Webserver as well, but it may be quicker to go through the motions and create a simple 'Hello World' example from scratch.
    Thursday, May 19, 2005 2:41 PM
  • Code covering a asp.net app with a web test was the frist thing I tried to do with vs2005. It is my first experience with VS so I didnt succeed and dont have a clue why it doesnt work other than its beta.

    Maybe this quote can help users with a better understanding than me :

    from http://www.cenqua.com/clover.net/doc/faq.html

    How do I Clover an ASP.NET application .

    As with the previous question, you need to have ASP.NET unload the AppDomain it is using for your webapp. When that unload occurs, the coverage is written out. The tricky part is getting ASP.NET to unload your webapp. This can be done by either deploying the unClovered version of your webapp or deleting the Clovered assemblies. ASP.NET will notice the change in the webapp assembly and unload the Clovered version.
    Sunday, May 22, 2005 10:06 AM
  •  Gerhardo wrote:
    Michael,

    There seems to be a problem of IIS or Casini not using the instrumented DLLs which is alleviated when you use their WebDev.WebServer.exe. It lives underneath the windows\framework path and is used as the default when you create a new web site from within visual studio.


    I thought Cassini *was* WebDev.Webserver.exe. I'm not sure I understand how to disable one for the other, from looking at the build options. I have created a web site within VS to test the web service, so I think, from your instructions, I'm already doing what you say.

    I still have the same issue as to the code coverage. Thanks!

    Regards,
    Michael Shorten
    Monday, May 23, 2005 1:59 PM
  • I have resolved the same issue today (unable to see coverage data for webservice using Magellan).  Try these steps, or similar approach:

    1. Instrument webservice.dll (webservice.dll.instr) and add it to the bin folder
    2. Remove  webservice.dll from bin folder
    3. Rename webservice.dll.instr to webservice.dll
    4. Run test
    5. Run utility to collect coverage data
    6. View coverage results

    Thursday, June 9, 2005 4:52 AM
  • Thanks! But how do you find the instrumented webservice dll. I don't have a bin folder under the site - and I don't find that dll anywhere else with the unit test dll.

    Regards,
    Friday, June 10, 2005 5:51 PM
  • Not sure I understand your problem.  You should be able to compile your web service project and create dll for it, correct?  Then instrument dll using utility bbcover (Magellan).  Then perform the steps above.  Am I missing something?

    Thursday, June 16, 2005 4:41 AM
  • Yes, I was unaware of this tool "bbcover" - guess I have something new to go lookup.

    Thanks!
    Monday, June 20, 2005 3:36 PM
  • How did you guys were able to generate a project.dll. I just created a web service project in vs 2005 in my localhost web server. I look at the folder that was generated from VS 2005 in the Inetpub wwwroot folder and don't see any dlls. The only files I saw in my wwwroot folders are: service.asmx, web.config and two application folders (App_Code which contain the service.cs and App_Data).  Can somebody help me out. Thanks
    Thursday, April 20, 2006 8:26 PM
  • Howdy,

    I am looking for bbcover.   I have looked in the VS SDK (version 3.0 released September 2006).  The search engines are void of a hit on this tool.  Can someone please point me the way?

    Thanks, Johnathan

    Wednesday, January 17, 2007 9:27 PM
  •  

    Hi micheal

     

    actually u want to set the Local test run configuration

     

    Just u want to go test->Edit run test configurations-> Localtestrun configuration

     

    and then select code coverage in that select the dll of ur application and click on

     

     

    And now run the tests ..u will find the coverage...reply if u have any clarification

    Wednesday, November 7, 2007 10:30 AM
  • Thursday, May 21, 2009 11:27 AM