What are the best practices to unit test expected code contract failures in debug and release builds.

Question

I was wondering if there are some best practices for unit testing scenarios where It is expected that a code contract failure is to be expected.

Environment:
We have a .NET component that can be considered a kind of library. Public API entry points will throw ArgumentExceptions when bad arguments are passed.

To validate that the library behaves in this way, we have unit tests that pass bad arguments and expect that these exceptions are thrown. These unit tests are executed on the debug and release build of the library.

I read that the BCL team uses code contracts, but decided not to rely on the rewriter. As our library is only delivered to other internal teams, we have some more freedom (it oke for example to not throw ArgumentExceptions anymore).

At the moment we are discussing if we should use Contract.Requires or Contract.Requires<E>. When we use Contract.Requires the checks will be removed in the release build and our checking unit tests will start to fail. We could extend our checking unit tests to detect at runtime that we are testing the release build and skip the unit test. After brainstorming this issue, the question became however: is it still worthwhile to create unit tests for these conditions? Are code contracts alone not enough and are these unit tests a thing of the past? Or should a library component play it on the safe side and use Contract.Requires<E> for all its public entry points?

Answer 1

Why would you remove Contract.Requires from your release build?  I like calling routines that have contracts enabled, If you have taken all the trouble to include these preconditions, why would you remove them in your release builds and deny your consumers (internal or not) the pleasure of the benefits they provide?

---

Best Regards, David K Allen Minneapolis, Minnesota, USA

Answer 2

I would say that if your release build does not contain the requires contract validation, then you should not have unit tests that check for it. If on the other hand you ship your release with requires validation, then the unit tests will work in all builds.

This should be independent of which kind of exception gets thrown by the validation.

---

Cheers, -MaF (Manuel Fahndrich)

Answer 3

Our plan was to ship both the debug build and the release build and the contract libraries to our consumers. The release build would be build with the ReleaseRequires setting and the debug build with the Full Checking Level. Building with the ReleaseRequires Checking Level would in essence remove the Contract.Requires checking at runtime.

But based on your feedback and that from Manuel I get the impression that the standard workflow would be to use the debug version during development of the library itself and to ship only the release version (+ contract library) to our consumers. The release version would then contain in essence the same runtime argument checking as we already did before code contract existed (checking for bad arguments at runtime and throwing ArgumentXXXExceptions).

Answer 4

The requirements of your customers would drive your design choices. Without more information, I'm hesitant to offer reocmmendations.  Have you had a dialog with your customers about this? Or someone who can reasonably serve as a proxy for them? If this is a new product, having such a conversation with a pilot cusotmer would be priceless to avoid incorrect assumptions.   It sounds as if you are having to make some guesses about your requirements. And misunderstood requirements can result in a lot of waste.

However, imagining myself as a consumer of a library, in general,  I certain favor SOME form of precondtions. I want to know whether I've called your modules correctly or not.  I do highly recommend you read the entire Contracts Manual that is installed with Code Contracts. Here is a snippet from Chapter 5 "Usage Guidelines" that may be useful.

"The easiest use of the contract tools is if you decide that you don't need to perform argument validation at runtime in release builds (Usage 1). In that case, you use the contract tools during development, but not on the shipped bits. Remember, you can ship a contract reference assembly along with your release bits so clients can get runtime checking of your parameter validations on their debug builds via call-site requires checking."

Some questions you need to answer to decide on an approach include:
- Who are your customers?
- Are they OK with the Code Contracts libraries being installed? Or will they dislike this?
- Do they expect argument exceptions to be in a certain form or are the Code Contracts exceptions ok?

I know you were asking for "best practices" but I'm not sure there is such a thing. The goodness of a practice varies with the context and requirements. But I hope these ideas are helpful to steer you in useful ways of approaching this.

---

Best Regards, David K Allen Minneapolis, Minnesota, USA