How do you use Pex in Visual Studio? Your Input Is Needed!
- In continuation of our previous thread about licensing questions (http://social.msdn.microsoft.com/Forums/en-US/pex/thread/52763686-3eb6-47cf-950d-21b2baa07423), we would like to get some input from everyone how you are using Pex, and how you would like to use it.
Please copy&paste and fill out the questions below.
1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no
2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other
3. How many people are in your team?
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?
Any other comments are appreciated as well, about the license terms, or required Visual Studio versions.
Thanks,
Your Pex team
All Replies
- Visual Studio
2008 Professional
Uno
Usually use NUnit.
Had used PEX briefly before. Was trying to use it again. Overall, I like the idea - although It really seems like there could be a window (tabbed document similar to a code window or a form designer window) that serves as a home page for your test projects.
The overall experience however has been quite poor for what I'm trying to accomplish.
My first attempt to create a PEX test failed with the error of (too long a filename) - I clicked the "Submit to Microsoft" in the dialog and this errored with a "Permission denied" and popped up an alert box stating to manually e-mail the error to "email address goes here". I Click okay, The dialog box showing the error message details disappears.
Second attempt to create the project was successful and after what I thought was a few tad winded dialogs I had something going. However it failed because the class that I was trying to generate tests for was a generic and it did not bother to prompt me whether i wished to provide types for the generic parameters.
In this case that is forgiveable, for the only reason being that in this case I am abusing generic types, however I still think that this is something PEX should support.
So i went to my class which inherits from the generic class and i right clicked on it and chose "create pex test stubs".
Well it seemed like it did stuff but it errored with the EXACT same error above. So apparently it was getting stuck on the prior creation and everything else was hanging up.
After trying a few more things i finally gave up, deleted the project, then decided maybe i should give it a clean start. So I recreated the project, this time no IO Path too long exception. I went straight to the inherited class and created my tests. It created the test methods
However it only generated tests for methods that were implemented in the. My immediate thought was right click in the PEX explorer on the class and look for an option to test base inherited methods. No go.
It seems that any attempt to re-update PEX stubs or generate new PEX stubs for other classes results in another project created every single time. Right now I have about 5 test projects after deleting another 4-6 prior
On top of that, after finding the "Generate tests for inherited members" hidden in the settings, removing all method name filters, getting a 7th test project, looking at the methods generated - it did not find any of the inherited methods
I'm running Visual Studio 2008 service pack 1 with .net 3.5 SP1 (all latest and greatest updates) and I would like to know whether this is "expected behavior"
To my way of thinking you should associate a single project with a single PEX Test project. Choosing to create PEX stubs in a class you've prior created test stubs should not request for a new PEX project like it is. It should update the existing test.
Same with adding new classes within the same project.
To my way of thinking the "Settings" that hides the option to "Generate tests for inherited members" should be more visible.
As for moving forward, assuming that there is some weird bug with my version of visual studio
I would personally like to see a "home page" of sort similar to the way a form designer, xsd designe, code window , etc
It shows all of your classes, perhaps in a structure very similar to the way Class Designer does.
There would be dropdowns/type ins that you can use to filter by (filter by names, filter by inherit from, , filter by method names, etc
then the option to checkmark individual items, or select all with one click. option to easily specify whether inherited members should be generated .
This home page would also serve as a command centre to run tests, see an overview of which tests were erroring and jump to the results display from there. - Hi,
1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no
only (A) from Visual Studio
2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other
(A) Visual Studio Professional
3. How many people are in your team?
one
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?
Mostly using (B) the wizard generated PUTs but sometimes also adding own PUTs.
Regards,
Alexander - Hi,
>> 1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no
Mostly A, but sometimes also B
I don't know of any other way to use it, so no to C
>> 2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other
B.) Team System
>> 3. How many people are in your team?
6, but I'm the only one that uses Pex (yet)
>> 4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?
Mostly B, generated by the VS interface for the wizard.
But once I add new functions to my assemblys under test, usaly run Pex on the base Project and let it add the new tests by saving the testresults.
I also sometimes write Puts, but that's currently rater the exception.
Once I have Pex generated Puts and tests for them, I will start writing more Puts
>>Any other comments are appreciated as well, about the license terms, or required Visual Studio versions.
Well I first tried Pex with VS2008, but since my Project is to big this didn't work very well.
After that I tried VS2010, which works a lot better (much lesser OOM crashes!)
I also tried the Commandline Version, but I must say I like the VS Version better, since this one writes the test into generated files that automatically linked to the file with the contained PUTs.
Regards,
Alexander Wurzinger - 1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no1a. We use Stubs extensively and find it *very* helpful.
2. Which version of Visual Studio are you using? B) Team System 2008
3. How many people are in your team? Between 4 and 10.
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs? - 1. A) From VS
2. A) Professional
3. 5 people in our team
4. Write PUTs.
Tech, life, family, faith: http://judahgabriel.blogspot.com - 1. Are you using Pex already?
A) From Visual Studio
2. Which version of Visual Studio are you using?
B) Team System
3. How many people are in your team?
12
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?
B) only use the Wizard-generated PUT stubs. Writing PUTs is a time cosuming task for our team :( - 1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no- D) no
2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other- A) Professional
3. How many people are in your team?- 20
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?- A) Parameterized Unit TestsThere are a couple of barriers to entry for my team:1. This is an ms research project with no guarantees that it will ever make it out of the research phase (that I know of). It is completely fair and understandable - it is just hard to see a product a) that might never see the light of day b) may have associated costs down the road that I am not yet aware of2. Pex (for commercial use at least) requires Visual Studio Team System. The organization I work for does not use this software and due to the cost of the upper-echelon VS systems never will. I'm assuming this is an architectural issue, and if so completely understand it - it just makes it a no-go for our team (and i'm guessing many others). - 1. Are you using Pex already?- (D) no
2. Which version of Visual Studio are you using?- (B) Team System
3. How many people are in your team?- 5
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?- (A) Parameterized Unit Tests
MVP Visual Basic 1. Are you using Pex already? A) From Visual Studio
2. Which version of Visual Studio are you using? B) Team System
3. How many people are in your team? 5
4. Only use the Wizard-generated PUT stubs?
Someone already mentioned this but it takes some persuasion to have Pex re-use an existing test project when the time comes to go back into the tests after a sprint or two to synchronize the unit tests. Still hitting 80-90% coverage, very effective tool for achieving code coverage.
Regards,
ofc.- 1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no
D -- much of our code is in C++/CLI and inaccessible from Pex. Please support C++/CLI.
2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other
B
3. How many people are in your team?
20
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?
Would do 4 if possible. See #1. - 1. A) for algorithmic types of code.
2. B)
3. 2
4. only A) - 1. Are you using Pex already?
- Yes, from Visual Studio.
2. Which version of Visual Studio are you using?
- Team System (the beta)
3. How many people are in your team?
- Just me (personal projects)
4. Do you write ...
- I use Pex (in combination with contracts) as a bug-finding tool more than a test-making tool. I have it explore a set of functions, then I review the errors (and sometimes check the return values).- Edited byStrilanc Sunday, September 13, 2009 4:36 PMfixing lost linebreaks
1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no
A)From Visual Studio
2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other
A) 2008 Professional
3. How many people are in your team?
There are four .net developers in this location. There are handfuls of .net developers in other subsidiaries and geographical locations working for this company.
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?
So far, B) only use the Wizard-generated PUT stubs.
It is difficult enough to convince management here (fortune 500 company) that the handful of .net developers should even have a recurring MSDN professional subscription. We evaluated Team System a year ago and it was decided the cost was unjustifiably too high. There's no way, especially in today's economy and a "it still stings" 30% reduction in our labor force that they'll bite on an incredibly more expensive IDE and/or subscription.
Pex is a GREAT idea and I'd love to get our team(s) using it (as even in simply evaluating how well it will work has helped me tremendously). That said, if it's decided to only support TFS for commercial use of Pex then it'll never get past my desk. Even an option of making Pex a separately purchased add-in for commercial users stands a far better chance of seeing it used for us than making an IDE we've already said "no" to a prerequisite.- 1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no
D
2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other
A
3. How many people are in your team?
On my immediate team, around 8. In our IT department, about 40.
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?
C) Neither. We use NUnit, but our unit tests do not use PUT yet. Our current tools don't support them. However, from my own testing experience:
1) I can see where they could be useful, and
2) How a tool like Pex could help plug holes or expose bugs in my own testing. - Answers Inline
1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no
[iconst] No, we are looking for a tool very much like TypeMock that would help you write stubs for any kind of class/method (including non-virtual).
We would be VERY interested to use this for our internal software.
2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other
[iconst] We are using Visual Studio Professional 2008.
3. How many people are in your team?
[iconst] 20 developers
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?
[iconst] See the answer to the first question.
1. Are you using Pex already? A) From Visual Studio, B) the command-line, C) other, D) no
yes, From vs 2010. i have never used the other options
2. Which version of Visual Studio are you using? A) Professional, B) Team System, C) other
2008 pro, 2010 team system (beta1)
3. How many people are in your team?
im the only one that only do code, but ~4 others code as well as do physical engineering, im the only one using pex
4. Do you write A) Parameterized Unit Tests (PUTs), or B) only use the Wizard-generated PUT stubs?
i mostly use the wizard generated ones and then modify them.- I would really like to use PEX.
I've had a look at the licensing and had a brief email conversation with Nikolai to evaluate. My conclusion is unfortunate. It's not worth my time and effort to evaluate this remarkable looking tool. When it gets launched, it may well be locked away from me inside of Team System.
Here's a few details.
My question after having a look at the licensing and distributions:
"I'm interested in testing out PEX. (I'm currently considering testing inference engines.)When I look at the licensing it gets too hard.
1) I see the commercial/non-commercial issue conflated with Team/Pro versions of VS. I find that pretty annoying. I'm not using Team and don't expect to. This suggests that it may only be released for team. If there is a good chance of that can you let me know. (It it is team only PEX is not for me.)
2) The commercial/non-commercial distinction seems illusory to me. (The situations I think of, including in Academic situations could all be argued both ways.) My problem is that I can evaluate PEX, find it useful then want to use it in a project that may one day become commercial. If there are limits on that there is no point in even testing it. I'm not going to spend resources on legal advice. Any idea when these ambiguities will be resolved? (With a time on that I can consider when I might be interested in evaluation.)
The idea looks great.
I hope these encumbrances don't harm development."
I now believe that it's likely that this will be locked up inside Team System.
I never expect to use TS, and for good reasons which I won't cover here. I really long for the old days when Microsoft meant it when it said Developers first. (Remember Balmer's dance?)
Now there is a bad case of SKU'itis. That horrible situation where mental contortions are made to justify artificial product differentiation (which, in this case looks antagonistic to many Visual Studio users).
I wish you very good luck with what appears to be a truly great product. (I don't now envisage ever driving it myself.)
To answer one of your questions. How do I test.
I've followed testing for a long time. I looked at Marc Clifton's original posts on some ideas and was delighted when Peli came up with MBUnit. (I still use it and performance with the latest version (3.1) is really great.) My coding had altered a lot over the years. Now I tend to develop something, get it right, build code generation tools then move on. My code generation uses, my home grown WinForms app (this has most active material at present), I have also actively tested good old fashioned XSLT and CodeSmith recently and expect to soon properly try out T4 (and maybe integrate it into my own templating tool).
I have recently turned a corner with tests (a lot of my tests aren't unit tests, they use databases and have "side effects", I find SQL Server CE to be very useful for testing purposes, it's XCopy capable!!). From now tests will increasingly be created from my XML definitions of projects. In other words the development process will incorporate "meta test code writing" and less hands on test code writing. (Test isn't even quite the right word, it's more like "testification" a combination of test and specification.)
Some parts of my work involve what boil down to mathematical functions. Code tests have serious limitations, so I produce pictorial graphs of functions (out of my test routines actually) which I inspect and which tell me more in 30 seconds than looking at a hundred little green dots. (Unless I find a realistic way to use tools to analyse functions and result sets, I may be with this visual test processing for a long time.)
I hope that's useful to you.
My best wishes for the PEX project. May it serve the whole developer community well.- Edited byMikeGale Friday, September 25, 2009 12:31 AMFound latest version of Gallio/MBUnit which performs great.
- Hi there
1. Are you using Pex already? A) From Visual Studio
2. Which version of Visual Studio are you using? A) Professional
3. How many people are in your team?
one
4. Do you write A) Parameterized Unit Tests (PUTs) yet still in practicing
Best wishes
Nick - 1. Are you using Pex already? Both A AND B. A From Visual Studio on our dev machines AND B. the command-line on our Continuous Integration server.
2. Which version of Visual Studio are you using? B. Team System
3. How many people are in your team? Four.
4. Do you write: B. Only use the Wizard-generated PUT stubs? And specifically just the detours for mocking the SharePoint API.
My blog: http://rapidapplicationdevelopment.blogspot.com/ - 1. From VS.
2. VS 2008 Professional.
3. 5
4. Right now we're just experimenting, and we've used PUTs.
Tech, life, family, faith: http://judahgabriel.blogspot.com
