locked
Using Pex to test XmlDiff RRS feed

  • Question

  • Hello,

    I tried running PEX against XmlDiff, which is part of the XmlNotepad source code. It has trouble generating tests for many of the subclasses of DiffgramOperation. For example, PEX would generate a PexMethod for the "WriteTo" method, but generate no actual tests when one selects "Run Pex Explorations".  I have tried to isolate the issues in a sample project, but was unable to reproduce the issues.  The XmlDiff library has several defects, which we only see when we diff large XML documents and I was hoping PEX can help me find some of those issues.

    This may be a good test project to test the capabilities of PEX. 

    Thanks,

    Werner

     

    Tuesday, July 13, 2010 2:23 AM

Answers

  • I tried to reproduce your findings. I created a new C# project (in VS2008), pasted your code into it, used Pex 0.92 to create a test project. At first, I could reproduce that Pex doesn't cover much, as XmlWriter is abstract, and Pex cannot create an instance. Then I used your first modified version of WriteTo, which takes a Class1 parameter, but creates the XmlWriter explicitly. Now Pex produces all desired test cases, including the InvalidOperationException.

    As Peli said, I suspect that you have done some kind of configuration, either instrumentation-related, or possibly accepting/modifying a factory method, that stops Pex from covering more.

    If you still cannot solve the issue, can you send us a self-contained, zipped, repro project to pexbug@microsoft.com?

    Thanks.


    Nikolai Tillmann
    Monday, July 19, 2010 7:59 PM

All replies

  • The problem is that XmlWriter is abstract and Pex probably simply does not know how to create such instance. In general, if you use the XML classes, the complexity will simply explose and defeat Pex.

    In order to test efficiently any method relying on XmlWriter, you would need to inherit from XmlWriter and for method of XmlWriter, overload with a method that gives Pex the choice to return the value. This should work conceptually, but this is definitely work.


    Jonathan "Peli" de Halleux - Try Pex online at www.pexforfun.com!
    Tuesday, July 13, 2010 3:06 AM
  • Thanks for the quick answer.  I played around further and even manually creating an XmlWriter seems to cause issues too. Consider the following class:

     

    public class Class1
    {
     private readonly int _x;
     private readonly int _y;
     private readonly int _z;
    
     public Class1(int x, int y, int z)
     {
      _x = x;
      _y = y;
      _z = z;
     }
    
     internal void WriteTo(XmlWriter writer)
     {
      if (writer == null)
      {
       throw new ArgumentNullException("writer");
      }
      writer.WriteStartElement("abc");
      writer.WriteAttributeString("x", _x.ToString());
    
      if (_y > 0)
      {
       writer.WriteAttributeString("y", _y.ToString()); 
      }
    
      if (_y > 12 && _z > 0)
      {
       throw new InvalidOperationException("Oops");
      }
      writer.WriteAttributeString("z", _z.ToString());
      writer.WriteEndElement();
     }
    }

     

    When creating parameterized tests using PEX and then run explorations, the resulting tests only covers when _x, _y and _z is 0 (zero).  As a result, a test for the InvalidOperationException is never raised. However, if I add another method to Class1 which looks like this:

     

    internal void Execute()
    {
     Console.WriteLine(_x);
     if (_y > 0)
     {
     Console.WriteLine(_y);
     }
    
     if (_y > 12 && _z > 0)
     {
     throw new InvalidOperationException("Oops");
     }
     Console.WriteLine(_z);
    }

     

    Then PEX generates enough inputs for the PexMethod to cover all the logic, making me believe that PEX may actually be generate the necessary tests to get 100% coverage.  So its rather clear that the logic isn't a problem for PEX. I changed the PexMethod for WriteTo to look as follows, with the hope that if I manually create the XmlWriter that PEX may be tricked into generating the tests to cover all the paths:

     

    [PexMethod]
    internal void WriteTo([PexAssumeUnderTest] Class1 target)
    {
     var builder = new StringBuilder();
     var writer = XmlWriter.Create(builder);
     target.WriteTo(writer);
    }

     

    PEX still generated only code for one permutation, namely when _x, _y and _z is 0 (zero).  For some reason, I would assume (perhaps incorrectly) that if I create an instance of XmlWriter, like the method above, that PEX will would be able to generate the same coverage as the Execute method. The only way to get PEX to generate all the permutations, it seems, is to create a PexMethod as follows:

     

    [PexMethod]
    internal string WriteTo(int x,
     int y,
     int z)
    {
     Class1 target = new Class1(x,y,z);
     StringBuilder builder = new StringBuilder();
     using (XmlWriter writer = XmlWriter.Create(builder))
     {
      target.WriteTo(writer);
     }
     return builder.ToString();
    }

     

    This seems strange. Is this the expected behavior?  Am I missing something?

    Thanks,

    Werner 

    Tuesday, July 13, 2010 3:21 AM
  • Does Pex give you a 'object creation' issue message? Maybe the object factory used to create class1 is not good enough - maybe it actually adds a call the WriteTo which would explain why Pex is not successfully creating the instance of class1 such that it hits the corner cases.
    Jonathan "Peli" de Halleux - Try Pex online at www.pexforfun.com!
    Tuesday, July 13, 2010 4:52 AM
  • No object creation message is generated for either XmlWriter or Class1.  Do keep in mind for the "Execute" case, PEX actually generates four test methods and creates Class1 with the right values.  The PexMethod for Execute looks as follows:

    [PexMethod]
    internal void Execute([PexAssumeUnderTest] Class1 target)
    {
     target.Execute();
    }
    If I add 

    [PexMethod]
    internal void WriteTo([PexAssumeUnderTest] Class1 target)
    {
     var builder = new StringBuilder();
     var writer = XmlWriter.Create(builder);
     target.WriteTo(writer);
    }
    

    then PEX only generates one test method.  It looks like a bug, but I can't be sure.

    Werner

     

    Tuesday, July 13, 2010 5:01 AM
  • When I tried your sample, I had to specify to instrument System.Xml to make things work. It seems that Pex got confused if System.Xml was not instrumented.

    Did you get 'uninstrumented method' messages?


    Jonathan "Peli" de Halleux - Try Pex online at www.pexforfun.com!
    Tuesday, July 13, 2010 11:24 PM
  • I did not see any such messages.
    Wednesday, July 14, 2010 11:36 PM
  • That is weird. Maybe you have suppressed those uninstrumented method message in the past? (suppression attrbiutes should be in your 'PexAssemblyInfo.cs' file if any).
    Jonathan "Peli" de Halleux - Try Pex online at www.pexforfun.com!
    Wednesday, July 14, 2010 11:50 PM
  • I tried to reproduce your findings. I created a new C# project (in VS2008), pasted your code into it, used Pex 0.92 to create a test project. At first, I could reproduce that Pex doesn't cover much, as XmlWriter is abstract, and Pex cannot create an instance. Then I used your first modified version of WriteTo, which takes a Class1 parameter, but creates the XmlWriter explicitly. Now Pex produces all desired test cases, including the InvalidOperationException.

    As Peli said, I suspect that you have done some kind of configuration, either instrumentation-related, or possibly accepting/modifying a factory method, that stops Pex from covering more.

    If you still cannot solve the issue, can you send us a self-contained, zipped, repro project to pexbug@microsoft.com?

    Thanks.


    Nikolai Tillmann
    Monday, July 19, 2010 7:59 PM