Why does this test pass instead of say "Detected nondeterminism outside the control of CHESS"?

Question

Why does this test pass instead of say "Detected nondeterminism outside the control of CHESS"?

Answer 1

Hmm... you are just looking for trouble, aren't you?  :) This gets into the issue of what "nondeterminism" means in the context of CHESS. CHESS is stateless (except for keeping track of the schedule explored so far), so when it wants to explore a new schedule, it reexecutes the TestMethod from the beginning up until the point where it wants to try a new decision (schedule a different thread, etc). Now, during this "replay" mode, if CHESS finds that it cannot replay the schedule, it has "detected nondeterminism" (because some branch in the program that CHESS has no control over took a different direction, for example). This means that the only way CHESS detects non-determinism is when it is unable to replay a schedule. (Now, CHESS doesn't give up in this case; It tries to recover and replays the schedule several times; most sources of such nondeterminism, like lazy initialization, can be dealt with by CHESS).

In your example, the code will either potentially deadlock (in the case where the then branch executes) or pass.  CHESS probably doesn't detect any nondeterminism because the condition goes the same way almost all the time, only switching when time of day transitions from AM to PM. To really make things non-deterministic, try using a random number generator. But still, recall, that (1) CHESS will only detect non-determinism because it is unable to replay a previous schedule; (2) CHESS will try hard to recover from non-determinism before reporting any message.

Best,

-- Tom

Answer 2

I tried a random number generator and now's milliseconds mod two but I didn't receive "Detected nondeterminism outside the control of CHESS":





Could you give an example of a test that produces "Detected nondeterminism outside the control of CHESS"?

Answer 3

Well, like I said before, CHESS has a pretty robust mechanism for dealing with non-determinism. Why, may I ask, are you so concerned with producing this error message?

-- Tom

Answer 4

Because I would like to know how CHESS responds to nondeterminism-containing code.

Answer 5

Got it. As I said before, CHESS has an algorithm for dealing with nondeterminism. This algorithm is invoked when CHESS finds that it is unable to replay a schedule. The algorithm basically tries to replay again several times. The result of the algorithm is that CHESS often can tolerate the non-determinism, but in doing so, it may not end up exploring all schedules (resulting in a lost bug). In your example, you should move the non-determinism inside the body of a thread, for example, controlling whether or not a lock is done. You won't see the error message you desire. What you will probably see is different numbers of schedules explored each time in the test result page. Also, if CHESS finds a deadlock, try pasting the repro text in and running again. Look at the number of schedules. You may see that CHESS was not able to find the deadlock on the first schedule because of non-determinism. Try this and let me know how it goes. Best,

-- Tom

Answer 6

I moved nondeterminism close to the locks and CHESS said Passed to that about 50% of the time, Inconclusive about 50% of the time, and Failed about 1% of the time:



I then pasted Failed's repro text on the top of the test method and CHESS said Failed to that about 90% of the time, Passed about 5% of the time, and Inconclusive about 5% of the time:



Is this good behavior for CHESS to return three different results for the same code?

I was able to receive a warning that includes the word "nondeterminism" by running mchess from the command prompt:

Answer 7

> Is this good behavior for CHESS to return three different results for the same code?

Hey, you asked for it, I delivered! This is the hallmark of non-determinism, multiple possible results. CHESS does not control all sources of non-determinism, as your experience above demonstrates. I agree that in the Visual Studio experience, we could give a lot better messaging about this (like in mchess, have a warning that we detected nondeterminism, so you know that you may get a different result if you run a second time). For now, you will get the most consistent results from running mchess (which always reports when it detects non-determinism). We do not have m(any) cycles to spend on making the Visual Studio integration better, being consumed with many other tasks. Best,

-- Tom

Answer 8

Thanks Tom for your and your team's hard work.