locked
Combine the Results of Many Test Runs? RRS feed

  • Question

  • I may be going about this in completely the wrong way. If so, please tell me.

    I'm a developer who has been asked to run a set of manual tests. I've had to interrupt the test runs several times for various reasons.

    I'd like to be able to combine the results into a single report showing my progress. Otherwise, I'll have to paste together "n" different test runs. Is there a tool or technique to do this?
    John Saunders
    WCF is Web Services. They are not two separate things.
    Use WCF for All New Web Service Development, instead of legacy ASMX or obsolete WSE
    Use File->New Project to create Web Service Projects
    Friday, February 12, 2010 1:33 AM

Answers

  • Hi John,

    A few things.

    First, I think you should give a look to our VS2010 Release Candidate. We've done a ton of work to improve the manual testing experience. If you'll have to be doing this long term, then you should know about our new direction and how that will benefit you. For more info on our 2010 release visit our team blog at http://blogs.msdn.com/vstsqualitytools.

    Second, in the VS2008 and before world, test runs were represented as xml files (.trx). They cannot be merged from Visual Studio, but they can be published to a Team Foundation server against a specific build. The build report will give you an all up report of the testing. You can also create warehouse reports of the testing against a particular build.

    Last, I think I have heard about utilities to merge several .trx files together for an all up report that have been created by enthusiasts. I can't think of any names of the tools, but perhaps someone in this community can refer us to a good tool for you.

    Yours,
    David
    David R. Williamson Development Manager Microsoft Visual Studio Team Test
    Saturday, February 13, 2010 12:28 AM

All replies

  • Hi John

    I may get you wrong, but the term manual test in VSTT is captured in a text document or a Microsoft Word document that lists the steps, which is performed by a human being. To the best of my knowledge, there is no such a tool within VSTT to facilitate the test report but test process.
    You can see Manual Tests Overview at http://msdn.microsoft.com/en-us/library/ms182616.aspx.

    Hope others have better ideas.

    Thanks.

    Figo Fei
    MSDN Subscriber Support in Forum
    If you have any feedback on our support, please contact msdnmg@microsoft.com 


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Friday, February 12, 2010 6:48 AM
  • Right. See http://msdn.microsoft.com/en-us/library/ms182619.aspx, where it discusses running a manual test.

    The rich text (.mht) file is displayed. Above it is an area for comments, and a pass/fail radio button, along with an "Apply" button. One reads the text and performs the manual steps. Any comments are noted. Finally, either pass or fail is clicked, followed by "Apply".

    Clicking "Apply" completes the test as a Pass or a Fail.

    I had a list of manual tests to execute. If I had been able to run them all, I would have had a single test run that I would have included in my report. Since I had to break the test run into about 7 runs, I was hoping to be able to combine them all into a single report.

    I found that there's an "All" value in the dropdown used to select a loaded test run. This comes close to what I wanted. The remaining issue is that, when I found a test I couldn't run, it was left as "Not executed". When I succeeded in running the test on a subsequent test run, the same test would be in that test result as a "Pass" or "Fail". When displayed using "All", the same test displays as both "Not executed" and "Pass" or "Fail". This is confusing. I had hoped to be able to combine the test results in a way that would allow me to edit out the "Not executed" cases when they are subsequently run as "Pass" or "Fail".

    John Saunders
    WCF is Web Services. They are not two separate things.
    Use WCF for All New Web Service Development, instead of legacy ASMX or obsolete WSE
    Use File->New Project to create Web Service Projects
    Friday, February 12, 2010 6:22 PM
  • Hi John,

    A few things.

    First, I think you should give a look to our VS2010 Release Candidate. We've done a ton of work to improve the manual testing experience. If you'll have to be doing this long term, then you should know about our new direction and how that will benefit you. For more info on our 2010 release visit our team blog at http://blogs.msdn.com/vstsqualitytools.

    Second, in the VS2008 and before world, test runs were represented as xml files (.trx). They cannot be merged from Visual Studio, but they can be published to a Team Foundation server against a specific build. The build report will give you an all up report of the testing. You can also create warehouse reports of the testing against a particular build.

    Last, I think I have heard about utilities to merge several .trx files together for an all up report that have been created by enthusiasts. I can't think of any names of the tools, but perhaps someone in this community can refer us to a good tool for you.

    Yours,
    David
    David R. Williamson Development Manager Microsoft Visual Studio Team Test
    Saturday, February 13, 2010 12:28 AM
  • I'll try the 2010 RC if my employer will permit it. What functionality in this area is available against TFS 2008? It will obviously take a while after RTM before we upgrade our TFS servers to 2010.

    Also, can you say more precisely which reports I should look at?

    John Saunders
    WCF is Web Services. They are not two separate things.
    Use WCF for All New Web Service Development, instead of legacy ASMX or obsolete WSE
    Use File->New Project to create Web Service Projects
    Monday, February 15, 2010 2:11 AM
  • I have written a simple tool to merge TRX files, it is described here http://mytechfinds.com/articles/software-development/8-csharp/28-trx-merger

    Hope this helps!!
    Tuesday, March 2, 2010 2:12 PM