locked
Pex run is not matching a normal run

    Question

  • I am trying to figure out Pex still.  I have a PUT that is throwing an exception.  I tried creating a normal Unit test using moles that is similar to see if it was Pex (or how I had setup Pex) and the normal test works fine. 

    I am going to post them both in hopes that someone can tell me what I am doing wrong (or at least steer me in the right direction).

     

    First the method under test:

            public List<vIndividualCustomer> GetCustomersByCity(string city)
            {
                var customerList = from customer in _context.vIndividualCustomers
                             where customer.City == city
                             select customer;
                return customerList .ToList();
            }

    This uses Linq to SQL to connect to the Sales schema of AdventureWorks.

    Now the "Normal" Moled unit test:

            [TestMethod]
            [HostType("Moles")]
            public void TestGetCustomersByCity()
            {
                // Arrange
                var ctx = new MAdventureWorksSalesDataContext();

                CustomersDAL dal = new CustomersDAL(ctx);

                var customerLinqList = new List<vIndividualCustomer>
                   {   new vIndividualCustomer {CustomerID = 1, City = "London"},
                       new vIndividualCustomer {CustomerID = 2, City = "London"},
                       new vIndividualCustomer {CustomerID = 3, City = "San Francisco"},
                       new vIndividualCustomer {CustomerID = 4, City = "San Francisco"}};

                MTable<vIndividualCustomer> customerTable = new MTable<vIndividualCustomer>();
                customerTable.Bind(customerLinqList.AsQueryable());
                ctx.vIndividualCustomersGet = () => customerTable;
               
                // Act
                var result = dal.GetCustomersByCity(null);

                // Assert
                Assert.IsTrue(result.Count == 0);

            }

    When I examine "Result View" from customerList (in the method undertest) it says "Enumeration yielded no results " (what I would expect by passing in null). 

    Now, when I Pex GetCustomersByCity my PUT looks like this:

            [PexMethod]
            public List<vIndividualCustomer> GetCustomersByCity([PexAssumeUnderTest]CustomersDAL target, string city)
            {
                List<vIndividualCustomer> result = target.GetCustomersByCity(city);
                return result;
            }

    Nice and simple.  When I ran it complained about being able to make a CustomerDAL.  So I made a class factory that look like this:

            [PexFactoryMethod(typeof(CustomersDAL))]
            public static CustomersDAL Create()
            {
                var ctx = new MAdventureWorksSalesDataContext();

                var customerLinqList = new List<vIndividualCustomer>
                                           {
                                               new vIndividualCustomer {CustomerID = 1, City = "London"},
                                               new vIndividualCustomer {CustomerID = 2, City = "San Francisco"},
                                           };

                MTable<vIndividualCustomer> customerTable = new MTable<vIndividualCustomer>();
                customerTable.Bind(customerLinqList.AsQueryable());

                ctx.vIndividualCustomersGet = () => customerTable;

                CustomersDAL customersDAL = new CustomersDAL(ctx);
                return customersDAL;

            }

    I then ran explorations and the only value it tried to pass in was "null".  (I found it a bit disappointing that it would only try one value, but oh well.) 

    This run throws an ArgumentException.  When I debug the run and examine Results View of customerList it says:

            {"type parameter is not a delegate "}    System.SystemException {System.ArgumentException}

    So, to sum up.  It says "Enumeration yielded no results" with a normal moled unit test and threw an exception with message of "type parameter is not a delegate" with the Pex run.

    Why the difference?  Where have I gone wrong?

    Tuesday, March 23, 2010 6:17 PM

All replies

  • Pex uses an instrumentation that is much more involved than Moles. Therefore, a lot of things can happen. There is always a stack trace associated with the errors in Pex. Could you cut and paste in the mail thread? For better stack traces, go to Tools -> Options -> Pex -> General and set Diagnostic = true.

    In order for Pex to support Linq, make sure you add a reference to Microsoft.Pex.Linq.dll and add the [assembly: PexLinqPackage] attribute to the test project. Our Linq support wors for .NET 2.0/3.5 but we seem to have a regression for .NET4.0. If you are running 4.0, that would explain why Pex cannot generate interresting test cases - we don't instrument Linq correctly.

    Why does Pex struggle with Linq anyway: in a nutshell, Linq uses DynamicMethod's to generate code. DynamicMethod methods are not reported to the profiler when they are jitted. Because our profiler cannot inject callbacks in the DynamicMethod, Pex cannot track the data flow through the Linq query. We have a workaround that intercepts the internals of the Linq to Object compiler and forces it to use Reflection.Emit instead.

     


    Jonathan "Peli" de Halleux - Give us your input about Pex!
    Wednesday, March 24, 2010 3:31 AM
    Owner
  • I am using Visual Studio 2008.  I do have Visual Studio 2010 installed (with .NET 4.0) but my target framework is 3.5 (can't even use 4.0 in 2008).

    I had the reference to Microsoft.Pex.Linq.dll already in my project (it must be added automatically).  I added [assembly: PexLinqPackage] to my AssemblyInfo.cs file.  I deleted my created tests (using pex) did a clean on the project and the solution and then a rebuild all on the solution.  I still only got the one value input (null).

     

    Here is the text from the stack trace window:

    [TestMethod]
    [PexGeneratedBy(typeof(PexCustomersDALTest))]
    [PexRaisedException(typeof(ArgumentException))]
    [HostType("Moles")]
    public void GetCustomersByCityThrowsArgumentException951()
    {
        CustomersDAL customersDAL;
        List<vIndividualCustomer> list;
        customersDAL = CustomersDALFactory.Create();
        list = this.GetCustomersByCity(customersDAL, (string)null);
    }
    
    --- Exception details
    
    System.ArgumentException: type parameter is not a delegate      at !!0 __Substitutions.System.Linq.Expressions.ExpressionCompiler.Compile(System.Object receiver, System.Linq.Expressions.Expression`1<!!0> lambda)
          at System.Collections.Generic.IEnumerator`1<!0> System.Linq.EnumerableQuery`1.GetEnumerator()
          at System.Collections.Generic.IEnumerator`1<!0> System.Linq.EnumerableQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator()
          at System.Void System.Collections.Generic.List`1.$stub$.ctor(System.Collections.Generic.IEnumerable`1<!0> )
          at System.Collections.Generic.List`1<!!0> System.Linq.Enumerable.ToList(System.Collections.Generic.IEnumerable`1<!!0> source)
        C:\src\ESP-Tester\ITTV\Episode6-Moles\MolesDemoPresent\Services\DAL\CustomersDAL.cs(29): at System.Collections.Generic.List`1<Services.DAL.vIndividualCustomer> Services.DAL.CustomersDAL.GetCustomersByCity(System.String city)
        C:\src\ESP-Tester\ITTV\Episode6-Moles\MolesDemoPresent\Services.Tests\DAL\PexCustomersDALTest.cs(27): at System.Collections.Generic.List`1<Services.DAL.vIndividualCustomer> Services.DAL.PexCustomersDALTest.GetCustomersByCity(Services.DAL.CustomersDAL target, System.String city)
          at Microsoft.ExtendedReflection.Interpretation.Term Microsoft.ExtendedReflection.__DynamicMethods.invoke:Services.DAL.PexCustomersDALTest.GetCustomersByCity(Microsoft.ExtendedReflection.DataAccess.ITermConstructor`1<Microsoft.ExtendedReflection.Interpretation.Term> , Microsoft.ExtendedReflection.DataAccess.ITermDestructor`1<Microsoft.ExtendedReflection.Interpretation.Term> , Microsoft.ExtendedReflection.Interpretation.Term[] )
          at System.Void Microsoft.Pex.Engine.PathExecution.PathExecutorBase.InternalRun(Microsoft.ExtendedReflection.DataAccess.ITermDestructor`1<Microsoft.ExtendedReflection.Interpretation.Term> destructor, Microsoft.Pex.Engine.PathExecution.PathSegmentPreparer segmentPreparer, Microsoft.Pex.Engine.PathExecution.PathSegmentReporter segmentReporter, System.Collections.Generic.IEnumerable`1<Microsoft.Pex.Engine.PathExecution.PathSegment> segments)
          at System.Boolean Microsoft.Pex.Engine.PathExecution.PathExecutorBase.TryRun(Microsoft.Pex.Engine.PathExecution.PexPathExecutionResult& runResult, Microsoft.ExtendedReflection.Interpretation.ISubstitution& finalSubstitution, Microsoft.ExtendedReflection.Interpretation.States.IState& finalState)
          at System.Boolean Microsoft.Pex.Engine.Drivers.PexPathDriver.InternalRun(System.Object concreteFixtureObject)
          at System.Boolean Microsoft.Pex.Engine.Drivers.PexPathDriver.Run(System.Object concreteFixtureObject)
          at System.Void Microsoft.Pex.Engine.Drivers.PexExplorationDriver.InternalExplore()
          at System.Void Microsoft.Pex.Engine.Drivers.PexExplorationDriverBase.CheckedExploreInternal()
          at System.Void Microsoft.Pex.Engine.Drivers.PexExplorationDriverBase.Execute()
          at System.Void Microsoft.Pex.Engine.Drivers.PexExplorationDriverBase.Explore()
          at System.Void Microsoft.Pex.Engine.TestFrameworks.PexTestFrameworkDriver.ExecuteExploration(Microsoft.Pex.Engine.PexExplorationBase exploration)
          at System.Void Microsoft.Pex.Engine.TestFrameworks.PexTestFrameworkDriver.ExecuteFixture(Microsoft.Pex.Engine.PexFixture fixture)
          at System.Boolean Microsoft.Pex.Engine.TestFrameworks.PexTestFrameworkDriver.ExecuteAssembly(Microsoft.ExtendedReflection.Metadata.AssemblyEx assemblyEx)
          at System.Void Microsoft.Pex.Engine.Drivers.PexDriver.ExecuteAssembly()
          at System.Void Microsoft.Pex.Engine.Drivers.PexDriver.ExecuteAll()
          at System.Int32 Microsoft.Pex.Engine.Drivers.PexDriver.ExecuteExplorationProtectedInSeparateStack()
          at System.Void Microsoft.Pex.Engine.Drivers.PexDriver+<>c__DisplayClass7.<ExecuteExplorationProtected>b__6(System.Object arg)
          at System.Void System.Threading.ThreadHelper.ThreadStart_Context(System.Object state)
          at System.Void System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state)
          at System.Void System.Threading.ThreadHelper.ThreadStart(System.Object obj)
    

    Wednesday, March 24, 2010 4:04 PM
  • After looking at this a bit more I found that there is a warning on the run:

    --- Description
    invalid solution minimization: some provided objects were not used in the solution

    There were also 5 things called "Observed Mismatches".  They had some fairly long call stacks on each of them.  To prevent this post from going too long I have saved the text to each of them as a Gist.  Here are the links if it is useful:

     • enumerable_rewriter.args_match
     • enumerable_rewriter.fixup_quoted_args
     • expression.get_method_based_binary_operator.txt
     • expression.validate_argument_types
     • expression.validate_lambda_args

    And finally there were 51 "Uninstrumented Methods".  This was really really long so I posted the text for it on a Gist as well.  If you would like to take a look at them here they are:

     • Uninstrumented Methods

    All of the Uninstrumented Methods looked to be stuff that was not my code.  (Mostly code that was generated by Linq to SQL.)

    Have I missed a step somewhere that is causing all these issues?

    Note: My solution is just a test solution to try and help me figure out Pex.  If it would help for me to send it, I would be happy to do so.

    Wednesday, March 24, 2010 4:31 PM
  • If you can send a self contained soluation to pexbug ad microsoft dot com, we will try to look at it.
    Jonathan "Peli" de Halleux - Give us your input about Pex!
    Thursday, March 25, 2010 12:20 AM
    Owner