locked
Issue with lambda expression Compile() and [HostType("Pex")].

    Question

  • Here's an interesting one. The following simplified unit test in MSTest gets an exception if the [HostType("Pex")] attribute is present:
    [TestMethod]
    [HostType("Pex")]
    public void testname3()
    {
        int x = 5;
        Expression<Func<int>> expression = () => x;
        var func = expression.Compile();
        int y = func();
    }

    If I remove the attribute, it runs fine. If I debug through, it will jump from the Compile line to the end } and then the test will fail with the following exception:
    Test method testname3 threw exception:  System.ArgumentNullException: Value cannot be null.
    Parameter name: field.

    System.Reflection.Emit.DynamicILGenerator.Emit(OpCode opcode, FieldInfo field)
    System.Linq.Expressions.ExpressionCompiler.GenerateGlobalAccess(ILGenerator gen, Int32 iGlobal, Type type, StackType ask)
    System.Linq.Expressions.ExpressionCompiler.GenerateConstant(ILGenerator gen, Type type, Object value, StackType ask)
    System.Linq.Expressions.ExpressionCompiler.GenerateConstant(ILGenerator gen, ConstantExpression c, StackType ask)
    System.Linq.Expressions.ExpressionCompiler.Generate(ILGenerator gen, Expression node, StackType ask)
    System.Linq.Expressions.ExpressionCompiler.GenerateMemberAccess(ILGenerator gen, Expression expression, MemberInfo member, StackType ask)
    System.Linq.Expressions.ExpressionCompiler.GenerateMemberAccess(ILGenerator gen, MemberExpression m, StackType ask)
    System.Linq.Expressions.ExpressionCompiler.Generate(ILGenerator gen, Expression node, StackType ask)
    System.Linq.Expressions.ExpressionCompiler.GenerateLambda(LambdaExpression lambda)
    System.Linq.Expressions.ExpressionCompiler.CompileDynamicLambda(LambdaExpression lambda)
    System.Linq.Expressions.ExpressionCompiler.Compile(LambdaExpression lambda)
    System.Linq.Expressions.LambdaExpression.Compile()
    Compile()
    PlayWithMoles.testname3() in PlayWithMoles.cs: line 42

    If I change the expression to () => 5; then it works fine. I'm actually only using the Compile in my unit tests so it isn't a big deal but I would bet this will blow up and cause many weird questions. I am using Compile for my own Assert statements that I stole the idea from MBUnit. :-) If you care, the project is on CodePlex at http://jslmstesthelper.codeplex.com/. My Assert statements look like this:
        AssertEx.That(() => x == 5);
    Which are much easier to read once you get over the lambda expressions. The other neat thing is that the assert failures can look like "x was not equal to 5".

    In any case, (I never thought I would say this but,) I love your Moles and your Stubs! Keep up the incredible work!

    Saturday, September 19, 2009 9:06 PM

Answers

  • This issue will be fixed in the next version of Pex (0.17). Unfortunately there is no workaround for the current version.
    Jonathan "Peli" de Halleux
    Tuesday, September 22, 2009 5:38 PM
    Owner

All replies

  • Did you create your project through the project wizard? It might have added a special attribute to the project. Look for:

        [assembly:PexLinqPackage]

    This attribute loads a special rewritter for the Expression compiler and it might well be the reason for the crash.

    Regarding the new AssertEx.That(...) method, why not simply have a AssertEx.That(bool) method so your example for read

         AssertEx.That(x == 5);

    If you only care about readability, this is much better than using the lambda. I suppose that the reason for using ExpressionTrees is really to be able to generate nice errors message. I guess you have some visitor that takes an expression tree and does pattern matching to create a english sentence (have you tought about showing the actual value of x?).

    I can see the automated error message being useful but it comes with an ugly price. Instance of 3 MSIL instruction to evaluate the equality, you have millions of instructions being executed through the Expression compiler code path. If you plan to use Pex in the future, you should know that this overhead comes as a burden on the Pex whitebox analysis (Pex analyses every executed MSIL instruction).

    So could you have the best of both worlds: lean and mean code + nice error messages? Yes this is possible. Instead of doing the compilation at runtime, do it at compilation time. For example, you could use CCI (http://ccimetadata.codeplex.com ) to rewrite the compiled test assembly. CCI could be used to visit the AST, create a string and embed it in the assembly. When running the test, in case of an error, you would simply have to load the string.

    > In any case, (I never thought I would say this but,) I love your Moles and your Stubs! Keep up the incredible work!

    Thanks. We've got a lof of good stuff in that space coming up. Stay tuned.


    Jonathan "Peli" de Halleux
    Sunday, September 20, 2009 4:29 AM
    Owner
  • I was just playing with Moles so this was an existing project that used TypeMock before. I was replacing the TypeMock code with Moles. In any case I didn't have the [assembly: PexLinqPackage] attribute. I added it and now I get the following exception:
     Test method testname3 threw exception:  System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. --->  System.ArgumentNullException: Value cannot be null.
     Parameter name: con.

     System.Reflection.Emit.ModuleBuilder.GetFieldTokenNoLock(FieldInfo field)
     System.Reflection.Emit.ModuleBuilder.GetFieldToken(FieldInfo field)
     System.Reflection.Emit.ILGenerator.Emit(OpCode opcode, FieldInfo field)
     System.Linq.Expressions.ExpressionCompiler.GenerateGlobalAccess(ILGenerator gen, Int32 iGlobal, Type type, StackType ask)
     System.Linq.Expressions.ExpressionCompiler.GenerateConstant(ILGenerator gen, Type type, Object value, StackType ask)
     System.Linq.Expressions.ExpressionCompiler.GenerateConstant(ILGenerator gen, ConstantExpression c, StackType ask)
     System.Linq.Expressions.ExpressionCompiler.Generate(ILGenerator gen, Expression node, StackType ask)
     System.Linq.Expressions.ExpressionCompiler.GenerateArgs(ILGenerator gen, ParameterInfo[] pis, ReadOnlyCollection`1 args)
     System.Linq.Expressions.ExpressionCompiler.GenerateMethodCall(ILGenerator gen, MethodInfo mi, ReadOnlyCollection`1 args, Type objectType)
     System.Linq.Expressions.ExpressionCompiler.GenerateMethodCall(ILGenerator gen, MethodCallExpression mc, StackType ask)
     System.Linq.Expressions.ExpressionCompiler.Generate(ILGenerator gen, Expression node, StackType ask)
     System.Linq.Expressions.ExpressionCompiler.GenerateConvert(ILGenerator gen, UnaryExpression u)
     System.Linq.Expressions.ExpressionCompiler.Generate(ILGenerator gen, Expression node, StackType ask)
     System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
     System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
     System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
     System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
     System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
     __Substitutions.System.Linq.Expressions.ExpressionCompiler.InternalGenerateLambda(Object receiver, LambdaExpression lambda)
     __Substitutions.System.Linq.Expressions.ExpressionCompiler.CompileDynamicLambda(Object receiver, LambdaExpression lambda)
     __Substitutions.System.Linq.Expressions.ExpressionCompiler.Compile(Object receiver, LambdaExpression lambda)
     System.Linq.Expressions.LambdaExpression.Compile()
     Compile()
     testname3() in PlayWithMoles.cs: line 21

    As I mentioned, this isn't fatal for me because the projects I'm testing are in VS 2005 so no lambdas. I'm using VS 2008 for all my unit testing since thats when MSTest became part of VS Professional.

    About the lambda assertions. I understand I can replace my AssertEx.That(() => x == 5) statement with Assert.IsTrue(x == 5). The main reason for using the lambdas is to be able to parse the expression tree and create a good failure message. For example, AssertEx.That(() => Console.ReadLine() == null) would tell me that "ReadLine method returned <xxx> instead of the expected null" instead of saying the value "<xxx> is not null". I find the extra hint at what's going on very helpful.

    I know very well that I'm replacing a couple IL instructions with thousand or millions. My first computer only had 4K of memory so I'm aware of the evil I do. But like all evil, it feels so good when I do it! In normal unit tests this makes almost no visible time different. If I were to do something like compare two bitmaps pixel by pixel I would drop back to the regular Assert.AreEqual(pixel1, pixel2). I'm also aware of how bad this is for Pex. I'm working on legacy code that wasn't built for unit testing so it's a challenge to write unit tests in the first place. I'm trying to us Pex against the new functionality I write.

    I've worked with both Cecil (Mono open source project) and CCI for rewriting assemblies. The problem with both is that they don't handle updating the .pdb files. (I would love to find out I'm wrong in either of these cases.) That means I can't single step through my unit tests. This is a deal breaker for me. My hope is that .Net 5.0 has enough of the "Compiler as a Service" in it that I can do something like what you are talking about. I've seriously considered converting my unit tests to the Boo language because it lets you rewrite the AST (Abstract Syntax Tree) as its being compiled. I don't think this is part of .Net 5.0 Compiler as a Service feature.

    Thanks for the quick response. BTW, I think its cool having such "direct" communications with you!

    Sunday, September 20, 2009 3:21 PM
  • I'm pretty sure CCI handles pdbs correctly. It is used by the Code Contracts.NET rewritter which handles pdbs :). Have you raised the issue on the CCI forums?
    Jonathan "Peli" de Halleux
    Monday, September 21, 2009 1:35 AM
    Owner
  • The project I was going to use this on was basically the same as Moles except I was going to rewrite the referenced assemblies to insert callbacks at the start. I was using Cecil and had it working pretty well except the pdb would get wacked 1/2 the time. Probably because I didn't know what I was doing. :-)

    In any case, I was going to get back to playing with the project this weekend but you released Moles so I didn't have to. I can easily change my unit tests so I don't have this problem. We are putting our house on the market Monday so I really don't have time to play anyway!

    Maybe I will get lucky and the stuff we would be using IL rewriting for could be done in the compiler in .Net 5.0. Having Code Contracts ship with VS 2010 is a good sign that it's needed or at least useful. Of course if it does make it into .Net 5.0, I'm sure many people will do evil with it.

    Thanks for all your help!
    Monday, September 21, 2009 2:45 AM
  • > IL rewriting for could be done in the compiler in .Net 5.0.

    Code Contracts.NET uses CCI, which you can download today from http://ccimetadata.codeplex.com . It does not use the 'Compiler As Service' component.
    Jonathan "Peli" de Halleux
    Monday, September 21, 2009 3:52 AM
    Owner
  • I understand CCI (used by Code Contracts) and .Net 5.0 "Compiler as a Service" are different things. I was pointing out that Code Contracts would probably move to "Compiler as a Service" if it allowed AST (Abstract Syntax Tree) modification like Boo's implementation. Currently the Boo language will let you modify the AST while the code is compiling. If Code Contracts would ONLY have to support Boo then they would have used Boo's AST instead of IL rewriting. I haven't heard much about "Compiler as a Service" but I did hear that the Boo like AST rewriting might be part of .Net 5.0. If I recall it was a very weak might.
    Monday, September 21, 2009 2:28 PM
  • The Code Contracts questions should really go into the Contrats forum :).


    Jonathan "Peli" de Halleux
    Monday, September 21, 2009 3:30 PM
    Owner
  • Hi,

    Could you copy the instrumentation instruction you are using for your project? I cannot reproduce the issue.

    Thanks,
    Peli
    Jonathan "Peli" de Halleux
    Tuesday, September 22, 2009 5:22 AM
    Owner
  • I made a nice zip file but couldn't see a way to attach it...

    I started from zero and created a new Console project. Created a new empty public method. Right-clicked on the method and chose "Create Unit Tests...". (The method isn't important, just showing I created everything "legal".) Modified the resulting test class as follows:
    namespace TestPlayWithMoles
    {
        using System;
        using System.Linq.Expressions;
        using Microsoft.VisualStudio.TestTools.UnitTesting;
        
        [TestClass()]
        public class ProgramTest
        {
            public TestContext TestContext { get; set; }
    
            [TestMethod]
            [HostType("Pex")]
            public void testname3()
            {
                int x = 5;
                Expression<Func<int>> expression = () => x;
                var func = expression.Compile();
                int y = func();
            }
        }
    }
    Added the two following attributes to the AssemblyInfo.cs file:
    [assembly: PexLinqPackage()]
    [assembly: PexInstrumentAssembly(typeof(Program))]
    Single stepping on the expression.Compile() line cause the focus to move to the last } of the method. Another single step finished the test run and get the error we saw above. Adding/removing the PexLinqPackage attribute still causes the test to fail in the same fashion.

    Other potentially important stuff. I'm running Windows 7 Ultimate 64-bit RTM from MSDN (installed new, not upgraded), Quad Core Intel, 8GB memory. I tried the following with no differences:
        * run single test or all tests.
        * compile in x86 mode and 64-bit mode.
        * debug vs release mode.
    Tuesday, September 22, 2009 2:34 PM
  • Ok, I could reproduce the issue. Investigating.
    Jonathan "Peli" de Halleux
    Tuesday, September 22, 2009 3:16 PM
    Owner
  • This issue will be fixed in the next version of Pex (0.17). Unfortunately there is no workaround for the current version.
    Jonathan "Peli" de Halleux
    Tuesday, September 22, 2009 5:38 PM
    Owner
  • Excellent! Next version is fine. I can easily work around the problem for now. Thanks for your help.
    Tuesday, September 22, 2009 7:05 PM
  • Don't forget to post your feedback at http://social.msdn.microsoft.com/Forums/en/pex/thread/27c703ea-3928-41ef-b767-638c39966b99 . Cheers, Peli
    Jonathan "Peli" de Halleux
    Tuesday, September 22, 2009 8:22 PM
    Owner