locked
How much gets run when I "Pex" a method RRS feed

  • Question

  • I am just about to try to have Pex start "exploring" some of my real code.

    I read in the tutorial this statement:

    What does it mean to Run Pex Explorations? Pex will run your code, possibly many times with different inputs. Don’t run Pex on code that could launch real rockets!

    So, what gets run when pex explores?  does it complete any calls in the current method?  Just some?

    Here are some examples to get at what I am asking:

    public bool CheckInput(UserDataContract request)
    {
        bool inputIsEmpty = (string.IsNullOrEmpty(request.Input));

        if (!inputIsEmpty )
        {
             if ((request.Input.Length >= 6) && ((request.Input.Length <= 20)))
            {
                // 1.  Call to another method in the same class
                string changedInput = UpdateInputInSomeWay(request.Input);
                result.ChangedInput = changedInput ;

                try
                {
                    /// 2. Call to a static method in a different class
                    MyClass.MyStaticMethod(request.ChangedInput);
                   
                }
                catch (ApplicationException exception)
                {
                    throw new ApplicationException("Error Message", exception);
                }
            }
            else
            {
                result.ChangedInput = request.Input;
            }

            // 3. API Call to unmanged code
            int result = NetUserSetInfo(SERVER, userName, 1, ref userinfo, out output);
           
        }
    }


    Of the numbered comments (1, 2 and 3) which calls will be made?
    Friday, February 12, 2010 4:33 PM

Answers

  • > If I run Pex on AcceptUserUpdate, will the explorations be calling SaveUserToDatabase?
    Yes. (If the bounds are set high enough and Pex doesn't stop execution early.)

    > Is there any way to say "don't make that call"?  Stop at ParseUserUpdates and just mole out a return with true and a return with false?
    There is currently no fully automated way to do that. How should that "automatically moled" method behave? What should it return? Pex needs a concrete return value to proceed.

    However, you can manually plug in such a mole. And instead of you picking a hard-coded return value, you can let it return a test parameter (possibly obtained via PexChoose). It's even possible to state that all methods of a class should be moled, and that their behavior should be to let Pex choose a return value. But this is something that needs to be configured, as part of the parameterized unit test from which the exploration starts. If so far you have been using "implicit" parameterized unit tests (by just right-clicking into your product code to invoke Pex), then this may be the time to move to explicit parameterized unit tests, which reside in a test project and are annotated with [PexMethod].

    > If not then this feels more like integration testing rather than unit testing.
    Pex doesn't automatically introduce good abstraction boundaries that make unit testing possible. The developer still has to do this by hand (either by introducing interfaces/non-sealed classes, or Moles).

    > For example, when I unit test CheckInput by hand (referencing my example in my first post now), I put in a mole...
    Exactly! And Moles were designed to work well with Pex.

    > So, is there some way to say "don't make any real calls to a specific method"?
    Once you have a parameterized unit test to start from, you can tweak how your code-under-test gets invoked, and you can write the code in there to attach a mole to the offending method at the beginning.

    Nikolai Tillmann - Tell us how you use Pex
    Friday, February 12, 2010 5:58 PM
  • > Could not pex look at the way the return value is used and infer what a mole should return?
    Yes, Pex can.

    And I had just that in mind when I wrote the following paragraph earlier. When the result value is another test input, Pex will try different values to exercise all possible behaviors.
    > However, you can manually plug in such a mole. And instead of you picking a hard-coded return value, you can let it return a test parameter (possibly obtained via PexChoose). It's even possible to state that all methods of a class should be moled, and that their behavior should be to let Pex choose a return value. But this is something that needs to be configured, as part of the parameterized unit test from which the exploration starts.

    Btw: You can tell Pex to treat all return values of all methods of a type SomeClass as test inputs by writing
        MSomeClass.Behavior = PexChooseBehavedBehavior.Instance; 
    

    (Assuming that you generated the mole type MSomeClass for SomeClass.)

    In the end, it really boils down to the question where you would like to draw the abstraction boundary, and that is something you have to do.

    If this does what you want, maybe you can give us a hint where we can improve our documentation?


    Nikolai Tillmann - Tell us how you use Pex
    Friday, February 12, 2010 9:30 PM

All replies

  • When Pex executes your code, all calls may be executed.

    However,
    1) when you code "runs too long" (which is configurable), Pex may terminate the code early.
    2) even though Pex may execute all your code, Pex may not be able to "understand" all of it. Pex only understands .NET code that gets instrumented.

    [edit] Let me clarify how this relates to Moles: When you attach a mole to a method before the code-under-test gets invoked, then this mole will take the role of the method method.

     
    Nikolai Tillmann - Tell us how you use Pex
    Friday, February 12, 2010 5:19 PM
  • I am confused.  :(

    You say "all calls may be executed".  How deep does it go?  Almost all code does something if you go far enough down the call list.  That is the whole point of mocking and unit testing.  (To run the code without db and file system calls.)

    If I have code that follows this tree:

    AcceptUserUpdate(User user)
        -> calls bool ParseUserUpdate(User user)
                      -> calls PerformUserUpdate(User user)
                                   -> SaveUserToDatabase(User user)

    If I run Pex on AcceptUserUpdate, will the explorations be calling SaveUserToDatabase?

    Is there any way to say "don't make that call"?  Stop at ParseUserUpdates and just mole out a return with true and a return with false?

    If not then this feels more like integration testing rather than unit testing.   (Please correct me if I am wrong.)

    For example, when I unit test CheckInput by hand (referencing my example in my first post now), I put in a mole that intercepts the static call to MyClass.MyStaticMethod.  That way that call does nothing.  That is because it is not part of what I want to test.  I am just looking to test my current method and the logic in it.  I am not interested in what that class method may or may not do.

    If that method happened to return a result then I would want to run different scenarios to explore what I do with the result of that method, but the actual method call does not interest me at this point.

    That said, I can see it make sense to make a call to a method in the same class as that is part of my "Unit" that I am testing.

    So, is there some way to say "don't make any real calls to a specific method"?

    Thanks for the help!
    Friday, February 12, 2010 5:37 PM
  • > If I run Pex on AcceptUserUpdate, will the explorations be calling SaveUserToDatabase?
    Yes. (If the bounds are set high enough and Pex doesn't stop execution early.)

    > Is there any way to say "don't make that call"?  Stop at ParseUserUpdates and just mole out a return with true and a return with false?
    There is currently no fully automated way to do that. How should that "automatically moled" method behave? What should it return? Pex needs a concrete return value to proceed.

    However, you can manually plug in such a mole. And instead of you picking a hard-coded return value, you can let it return a test parameter (possibly obtained via PexChoose). It's even possible to state that all methods of a class should be moled, and that their behavior should be to let Pex choose a return value. But this is something that needs to be configured, as part of the parameterized unit test from which the exploration starts. If so far you have been using "implicit" parameterized unit tests (by just right-clicking into your product code to invoke Pex), then this may be the time to move to explicit parameterized unit tests, which reside in a test project and are annotated with [PexMethod].

    > If not then this feels more like integration testing rather than unit testing.
    Pex doesn't automatically introduce good abstraction boundaries that make unit testing possible. The developer still has to do this by hand (either by introducing interfaces/non-sealed classes, or Moles).

    > For example, when I unit test CheckInput by hand (referencing my example in my first post now), I put in a mole...
    Exactly! And Moles were designed to work well with Pex.

    > So, is there some way to say "don't make any real calls to a specific method"?
    Once you have a parameterized unit test to start from, you can tweak how your code-under-test gets invoked, and you can write the code in there to attach a mole to the offending method at the beginning.

    Nikolai Tillmann - Tell us how you use Pex
    Friday, February 12, 2010 5:58 PM
  • OK, the confusion is lifting a bit.

    I can see that for unit testing code that eventually writes to a database (or makes a call to save or load from an external system) using Pex automatically is not the way to go.  I must admit that this seems like a significant limitation for my code.  Most of my code eventually calls a Web Service or loads or saves something.

    I plan to look into how to write my own PUT tests and see how that works out.

    Obviously I am new to Pex and how it works, but this part of your response confused me:

    >How should that "automatically moled" method behave? What should it return? Pex needs a concrete return value to proceed.

    Could not pex look at the way the return value is used and infer what a mole should return?

    For example:

    User myUser = DBObject.GetUserFromDatabase(userID);
    if (myUser.IsAdminUser)
    {  
        if (myUser.IsLoggedIn)
        // Do somthing admin like here
    }
    else
    {
        throw ApplicationAccessException("User is not an admin user");
    }

    From my point of view there is no need to call GetUserFromDatabase.  From exploring the method it is clear that the method only cares about the user returned having the IsAdminUser flag set (or not)  and if it is set then it cares if the IsLoggedIn flag is set.  So Pex could run this a few times.  Once returning a moled User with IsAdminUser not set and twice with it set (once with IsLoggedIn set and once with it not set). 

    Clearly this is a simple example, but I think it could scale out to more complex scenarios.

    Again, I admit that I am new to Pex and what I am thinking of is high level and may not work at a "real" implementation level.  I mention it because this is what I thought Pex did from the descriptions I read and saw.
    Friday, February 12, 2010 6:20 PM
  • > Could not pex look at the way the return value is used and infer what a mole should return?
    Yes, Pex can.

    And I had just that in mind when I wrote the following paragraph earlier. When the result value is another test input, Pex will try different values to exercise all possible behaviors.
    > However, you can manually plug in such a mole. And instead of you picking a hard-coded return value, you can let it return a test parameter (possibly obtained via PexChoose). It's even possible to state that all methods of a class should be moled, and that their behavior should be to let Pex choose a return value. But this is something that needs to be configured, as part of the parameterized unit test from which the exploration starts.

    Btw: You can tell Pex to treat all return values of all methods of a type SomeClass as test inputs by writing
        MSomeClass.Behavior = PexChooseBehavedBehavior.Instance; 
    

    (Assuming that you generated the mole type MSomeClass for SomeClass.)

    In the end, it really boils down to the question where you would like to draw the abstraction boundary, and that is something you have to do.

    If this does what you want, maybe you can give us a hint where we can improve our documentation?


    Nikolai Tillmann - Tell us how you use Pex
    Friday, February 12, 2010 9:30 PM
  • I think that this will be a common enough case that it should be in a tutorial (maybe even at the end of the main one with reference to it from the note about not running code that "could launch real rockets!").

    Either way, I would love to see step by step instructions that show how to include a Moled object into pex so that it is used BEFORE explorations (this is critical because the idea is that it would be used to avoid launching the rockets).

    Just to be clear, the Mole tutorial is great.  It shows really clearly how to make a mole to use in a normal unit test.  But the step that I have not seen is how to to tell Pex to use a Moled object before any Pex Explorations (or anything else Pex like) has been run.

    That would be great (and would make Pex super useful to me).

    And by the way, thanks for the great forums support!
    Tuesday, February 16, 2010 5:15 PM
  • OK, so I think this example that I am looking for can be found in the Pex Sharepoint example .

    I am not done with it so I will post any issues I have (prop on a separate thread).  But it seems to go through a more "real world" scenario.

    Wednesday, February 17, 2010 5:29 AM