Pex uses a factory for only a subset of my tests

Question

I have a problem with the Pex explorations on one of the methods in my test-class (the code is included below).

The PUT CreateNewCert only gets null-values passed to it for the nodeID-parameter. I have a factory for NodeID and the other tests gets values from that factory, but not this one. Pex instead generates several tests for the method with nodeID==null. Is there a way to solve this (e.g. force Pex to use a factory for a certain type)?

When I run the test manually and provide the input myself it executes well so I don't think there's an obvious problem with the code (except for the fact that Pex has a hard time solving its constraints). The class is more of an intregration test than a unit test (something which I'm working on to fix) but shouldn't Pex be able to generate a non-null value of NodeID either way (especially considering that it's doing that for the other tests)?

Thank you in advance! 

The test-class:

    [TestClass]
    [PexClass(typeof(BasicAuthService))]
    [PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException), AcceptExceptionSubtypes = true)]
    [PexAllowedExceptionFromTypeUnderTest(typeof(InvalidOperationException))]
    public partial class BasicAuthServiceTest
    {
        private static HttpSimulator httpSimulator;
        private static NodeTable nodeTable;
        private static string testUserEmail;

        [ClassInitialize]
        public static void Init()
        {
            //Init the http-simulator.
            httpSimulator = new HttpSimulator("/", @"C:\SoftwareProjects\Java\Degoo\RestServer\RestWeb").SimulateRequest();
            testUserEmail = "[e-mail]";
            HttpContext.Current.User = new GenericPrincipal(new GenericIdentity(testUserEmail), null);
            nodeTable = new NodeTable();
        }

        [PexMethod(MaxBranches = Int32.MaxValue)]
        [TestMethod]
        public Node CreateNewNode([PexAssumeUnderTest]BasicAuthService target, string nodeName)
        {
            Node result = target.CreateNewNode(nodeName);

            var storedNode = nodeTable.GetNode(result.Id);
            Assert.AreEqual(result, storedNode);
            return result;
        }

        [PexMethod(MaxBranches = Int32.MaxValue, TestEmissionFilter = PexTestEmissionFilter.All, MaxWorkingSet = Int32.MaxValue)]
        public byte[] CreateNewCert([PexAssumeUnderTest]BasicAuthService target, NodeID nodeID)
        {
            PexObserve.ValueForViewing("Raw Condition", PexSymbolicValue.GetRawPathConditionString());
            byte[] result = target.CreateNewCert(nodeID);
            Assert.IsTrue(result.Length > 0);
            var storedCert = nodeTable.GetNodeCertificate(nodeID);
            Assert.AreEqual(result, storedCert);
            return result;
        }

        [PexMethod(MaxBranches = Int32.MaxValue, TestEmissionFilter = PexTestEmissionFilter.All, MaxWorkingSet = Int32.MaxValue)]
        public void UpdateDecryptionKey(
            [PexAssumeUnderTest]BasicAuthService target,
            NodeID nodeID,
            byte[] decryptionKey
        )
        {
            target.UpdateDecryptionKey(nodeID, decryptionKey);
            var storedDecryptionKey = target.GetDecryptionKey(nodeID);
            Assert.AreEqual(decryptionKey, storedDecryptionKey);
        }
        [PexMethod(MaxBranches = Int32.MaxValue)]
        public void UserIDGet([PexAssumeUnderTest]BasicAuthService target)
        {
            Guid result = target.UserID;
            var testUser = Membership.GetUser(testUserEmail);
            Assert.AreEqual(result, testUser.ProviderUserKey);
        }

        [PexMethod(MaxBranches = Int32.MaxValue)]
        public byte[] GetDecryptionKey([PexAssumeUnderTest]BasicAuthService target, NodeID nodeID)
        {
            byte[] result = target.GetDecryptionKey(nodeID);
            return result;
        }
    }

The factory-method:

 [PexFactoryMethod(typeof(NodeID))]
 public static NodeID Create(Guid input)
 {
     //ToNodeID() is implemented by an extension-method.
     return input.ToNodeID();
 }

Some of the code covered by CreateNewCert:

public byte[] CreateNewCert(NodeID nodeID)
        {
            if (nodeID == null)
            {
                throw new ArgumentException("nodeID");
            }
            var nodeTable = new NodeTable();
            nodeTable.AuthenticateNode(nodeID, UserID);
            
            var certCreator = new CertificateCreator();
            return certCreator.CreateNodeCertificate(nodeID, UserID);
        }

    public class CertificateCreator
    {
        public byte[] CreateNodeCertificate(NodeID nodeID, Guid userID)
        {
            var nodeTable = new NodeTable();

            Pkcs12Store caStore = CaStore;

            var caPrivate = caStore.GetKey("MyKeyName").Key;
            var caCert = caStore.GetCertificate("MyKeyName").Certificate;

            //Generate a new public/private key-pair. We always create a new key-pair for each new certificate.
           var nodeCertKeys = generateRSAKeyPair();

            //Generate the new certificate for the node.
            var nodeCertSigned = signNodeCert(nodeID, caCert, nodeCertKeys.Public, caPrivate);

            //Store the new certificate and the private key.
            var nodeStore = buildPkcs12Store();

            var certEntry = new X509CertificateEntry(nodeCertSigned);
            var caCertEntry = new X509CertificateEntry(caCert);
            var keyEntry = new AsymmetricKeyEntry(nodeCertKeys.Private);
            nodeStore.SetKeyEntry("key", keyEntry, new[] { certEntry, caCertEntry });

            byte[] nodeStoreBytes = GetNodeStoreBytes(nodeStore);

            nodeTable.UpdateNodeCertififcate(nodeID, nodeStoreBytes);

            return nodeStoreBytes;
        }

        private byte[] GetNodeStoreBytes(Pkcs12Store nodeStore)
        {
            var nodeStoreStream = new MemoryStream();
            //We use a hard-coded password here because if we would use the node's password we would need to ask for
            // it everytime the node needs to open the cert.
            nodeStore.Save(nodeStoreStream, Settings.Default.NodeStorePassword.ToCharArray(), new SecureRandom());
            return nodeStoreStream.ToArray();
        }

        private X509Certificate signNodeCert(NodeID nodeID, X509Certificate caCert, AsymmetricKeyParameter nodePublicKey, AsymmetricKeyParameter caPrivateKey)
        {
            var certificateGenerator = new X509V3CertificateGenerator();
            BigInteger serialNumber = createNewSerialNumber(nodeID);

            certificateGenerator.SetSerialNumber(serialNumber);
            certificateGenerator.SetIssuerDN(PrincipalUtilities.GetSubjectX509Principal(caCert));
            certificateGenerator.SetNotBefore(DateTime.UtcNow.AddSeconds(-50));
            certificateGenerator.SetNotAfter(DateTime.UtcNow.AddYears(1));
            certificateGenerator.SetSubjectDN(new X509Name("CN=" + nodeID.ToGuid()));
            certificateGenerator.SetPublicKey(nodePublicKey);
            certificateGenerator.SetSignatureAlgorithm("SHA256WithRSAEncryption");

            return certificateGenerator.Generate(caPrivateKey);
        }

        private Pkcs12Store buildPkcs12Store()
        {
            return new Pkcs12StoreBuilder()
                .SetKeyAlgorithm(PkcsObjectIdentifiers.PbeWithShaAnd3KeyTripleDesCbc)
                .SetCertAlgorithm(PkcsObjectIdentifiers.PbewithShaAnd40BitRC2Cbc)
                .Build();
        }

        private Pkcs12Store caStore;
        private Pkcs12Store CaStore
        {
            get
            {
                if (caStore == null)
                {
                    //Load our CA (issuer) certificate.
                    caStore = buildPkcs12Store();

                   var caStoreFileStream = new FileStream(Settings.Default.CaKeyStorePath, FileMode.Open);
                    caStore.Load(caStoreFileStream, Settings.Default.CaKeyStorePassword.ToCharArray());
                    caStoreFileStream.Close();
                    return caStore;
                }
                return caStore;
            }
        }

        private BigInteger createNewSerialNumber(NodeID nodeId)
        {
            var secureRandom = new SecureRandom();
            secureRandom.SetSeed(nodeId.Id.ToByteArray());
            var serialNumberBytes = new byte[32];
            secureRandom.NextBytes(serialNumberBytes);
            return new BigInteger(serialNumberBytes).Abs();
        }

        private static AsymmetricCipherKeyPair generateRSAKeyPair()
        {
            IAsymmetricCipherKeyPairGenerator kpGen = GeneratorUtilities.GetKeyPairGenerator("RSA");
            kpGen.Init(
                new RsaKeyGenerationParameters(
                    BigInteger.ValueOf(0x10001),
                    new SecureRandom(),
                    2048,
                    25));

            return kpGen.GenerateKeyPair();
        }  
    }

Answer 1

What can help to diagnose these issues is to inline the call to the nodeid factory in a new PUT and call that PUT. You will see if Pex is able to generate tests instance of that nodeid.

When Pex explores an object factory, it only considers successfull path so it silently cuts away any input that fails the factory. By embedding the factory in the PUT, you will have an opportunity to understand if it works.

---

Jonathan "Peli" de Halleux - Give us your input about Pex!

Answer 2

When I run it with the factory inlined I get the following error:

--- Description
assumption violation: PexAssumeFailedException
 

byte[] bs;
BasicAuthService basicAuthService = new BasicAuthService();
bs = this.CreateNewCert(basicAuthService);

 
[TestMethod]
[PexGeneratedBy(typeof(BasicAuthServiceTest))]
[Ignore]
[PexDescription("the test state was: assumption violation")]
public void CreateNewCert01()
{
    byte[] bs;
    BasicAuthService basicAuthService = new BasicAuthService();
    bs = this.CreateNewCert(basicAuthService);
}

--- Exception details
 
Microsoft.Pex.Framework.Exceptions.PexAssumeFailedException: BasicAuthServiceTest.CreateNewCert(BasicAuthService) at segment index 0 needs object of type BasicAuthService (some object#0); extended sequence by ExSig#1      at System.Void Microsoft.Pex.Engine.PathExecution.PathExecutorBase.PrepareSegment(Microsoft.Pex.Engine.Solutions.SolutionConcretizer concretizer, System.Int32 segmentIndex, System.Boolean isLastSegment, Microsoft.Pex.Engine.MethodInvocations.PexMethodInvocationRequest request, Microsoft.Pex.Engine.MethodInvocations.PexMethodInvocation invocation)
      at System.Void Microsoft.Pex.Engine.PathExecution.PathExecutorBase+<>c__DisplayClass17.<TryRun>b__13(System.Int32 segmentIndex)
      at System.Void Microsoft.Pex.Engine.PathExecution.PathExecutorBase.InternalRun(Microsoft.ExtendedReflection.DataAccess.ITermDestructor`1<Microsoft.ExtendedReflection.Interpretation.Term> destructor, Microsoft.Pex.Engine.PathExecution.PathSegmentPreparer segmentPreparer, Microsoft.Pex.Engine.PathExecution.PathSegmentReporter segmentReporter, System.Collections.Generic.IEnumerable`1<Microsoft.Pex.Engine.PathExecution.PathSegment> segments)
      at System.Boolean Microsoft.Pex.Engine.PathExecution.PathExecutorBase.TryRun(Microsoft.Pex.Engine.PathExecution.PexPathExecutionResult& runResult, Microsoft.ExtendedReflection.Interpretation.ISubstitution& finalSubstitution, Microsoft.ExtendedReflection.Interpretation.States.IState& finalState) 
 
 
My test-method looks like this:

[PexMethod(MaxBranches = Int32.MaxValue, TestEmissionFilter = PexTestEmissionFilter.All, MaxWorkingSet = Int32.MaxValue)]
        public byte[] CreateNewCert([PexAssumeUnderTest]BasicAuthService target)
        {
            NodeID nodeID = NodeIDFactory.Create(default(Guid));
            byte[] result = target.CreateNewCert(nodeID);
            Assert.IsTrue(result.Length > 0);
            var storedCert = nodeTable.GetNodeCertificate(nodeID);
            Assert.AreEqual(result, storedCert);
            return result;
        }

An interesting thing is that factory for BasicAuthService isn't used in the generated method. For other PUTs this factory is used. E.g.

[TestMethod]
[PexGeneratedBy(typeof(BasicAuthServiceTest))]
[ExpectedException(typeof(ArgumentException))]
public void UpdateDecryptionKey04()
{
    BasicAuthService basicAuthService;
    NodeID nodeID;
    basicAuthService = BasicAuthServiceFactory.Create();
    nodeID = NodeIDFactory.Create(default(Guid));
    this.UpdateDecryptionKey(basicAuthService, nodeID, (byte[])null);
}

So it seems like it doesn't use factories at all for CreateNewCert. I will now inline BasicAuthService as well and get back to you with results of that.

 

Answer 3

With BasicAuthService inlined as well, I run out of memory. In the method I use Bouncy-Castle to generate the certificate. I suspect that the combination of Bouncy-Castle's memory usage (generating RSA-keys is a bit expensive) and the fact that quite a lot of classes are involved in the test is too much for Pex to handle. As you see, I've set MaxWorkingSet to Int32.MaxValue. I'm running 64bit with 12gb ram so if it's possible I'd gladly increase it if possible. Is there anything else I can do to decrease the memory usage (besides mocking some of the classes).

--- Description

out-of-memory while executing exploration; not all clean-up action might have been performed.
 
--- Exception details
 
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.      at !!0 Microsoft.ExtendedReflection.Interpretation.Impl.InternalizingTermManager.add(!!0 t)
      at Microsoft.ExtendedReflection.Interpretation.Term+Type Microsoft.ExtendedReflection.Interpretation.Impl.InternalizingTermManager.<.ctor>b__1(Microsoft.ExtendedReflection.Metadata.TypeEx type, Microsoft.ExtendedReflection.Interpretation.Impl.InternalizingTermManager me)
      at !1 Microsoft.ExtendedReflection.Collections.SafeInternalizingDictionary`3.GetOrCreateValue(!0 key, !2 context)
      at !1 Microsoft.ExtendedReflection.Collections.SafeInternalizingDictionary`3.SynchronizedGetOrCreateValue(!0 key, !2 context)
      at Microsoft.ExtendedReflection.Interpretation.Term Microsoft.ExtendedReflection.Interpretation.Impl.InternalizingTermManager.Type(Microsoft.ExtendedReflection.Metadata.TypeEx type)
      at Microsoft.ExtendedReflection.Interpretation.Term Microsoft.ExtendedReflection.Interpretation.States.Impl.State.CreateNewObject(System.Object value, Microsoft.ExtendedReflection.Metadata.TypeEx type)
      at Microsoft.ExtendedReflection.Interpretation.Term Microsoft.ExtendedReflection.Interpretation.States.Impl.State+Constructor.Microsoft.ExtendedReflection.DataAccess.ITermConstructor<Microsoft.ExtendedReflection.Interpretation.Term>.Object(System.Object value)
      at System.Boolean Microsoft.Pex.Engine.PathExecution.PathExecutorBase.tryCharacterizeException(System.Boolean exceptionIsAssumptionViolation, System.Exception e, Microsoft.ExtendedReflection.Interpretation.States.IState state, Microsoft.Pex.Engine.PathExecution.PexPathExecutionResultKind& kind, System.Exception& exception, Microsoft.ExtendedReflection.Interpretation.Term& exceptionTerm)
      at System.Boolean Microsoft.Pex.Engine.PathExecution.PathExecutorBase.TryRun(Microsoft.Pex.Engine.PathExecution.PexPathExecutionResult& runResult, Microsoft.ExtendedReflection.Interpretation.ISubstitution& finalSubstitution, Microsoft.ExtendedReflection.Interpretation.States.IState& finalState)
      at System.Boolean Microsoft.Pex.Engine.Drivers.PexPathDriver.InternalRun(System.Object concreteFixtureObject)
      at System.Boolean Microsoft.Pex.Engine.Drivers.PexPathDriver.Run(System.Object concreteFixtureObject)
      at System.Void Microsoft.Pex.Engine.Drivers.PexExplorationDriver.InternalExplore()
      at System.Void Microsoft.Pex.Engine.Drivers.PexExplorationDriverBase.CheckedExploreInternal()
      at System.Void Microsoft.Pex.Engine.Drivers.PexExplorationDriverBase.Execute()
      at System.Void Microsoft.Pex.Engine.Drivers.PexExplorationDriverBase.Explore()
 
 

Answer 4

We do not provide the 64-bit flavor for Pex so you are still limited by 32-bit process memory. You can set the er_dumpheap_on_oom environment variable to 1 so that Pex takes a heap snapshot when an OOM occurs. That snapshot is helpful for us to diagnose memory consumption issues.

We can follow up on this issue on [e-mail] .

---

Jonathan "Peli" de Halleux - Give us your input about Pex!