Unable to work out how to generate the set of tests I want RRS feed

  • Question

  • Hi, 

    I'm just starting to have a play around with Spec Explorer, and I'm a bit confused on how to slice the model, and generate the tests I'm interested in.

    To take the sample Accumulator example to illustrate what I'm trying to do - it defines a DoubleAddScenario, and we generate tests that are of the form Add, Add, ReadAndReset

    I want to generalise this, to an arbitrary number of Adds, then a ReadAndReset. So, modifying the DoubleAddScenario to :-

    (Add*; ReadAndReset)*

    I read this as meaning any number of Adds, followed by a ReadAndReset, and all this any number of times. The re-exploration of DoubleAddScenario gives me a general state diagram which seems to agree with this. 

    The re-exploration of SlicedAccumulatorModelProgram becomes a lot more complex, but again seems to give the expected more specific state diagram, taking into account the specified allowable values for x.

    However, when re-exploring the AccumulatorTestSuite, I end up with just a bunch of separate, single step diagrams, all adding 2, and then ending with a [state bound hit].

    I'm confused as to why this is happening, but am guessing that I'm not restricting something enough ? What state bound am I hitting ?

    What I'm expecting to be able to generate is a set of tests of increasing length, something along the lines of :-

    Add(1), ReadAndReset()
    Add(2), ReadAndReset()
    Add(1), Add(1), ReadAndReset()
    Add(1), Add(2), ReadAndReset()
    Add(2), Add(1), ReadAndReset()
    Add(2), Add(2), ReadAndReset()
    Add(1), Add(1), Add(1), ReadAndReset()
    Add(1), Add(1), Add(2), ReadAndReset()

    ..up to the limit of StepBound (or PathDepthBound ?) that is set in the Main config (both of these are 128 anyway which seems way above the limits of what I am actually generating...)

    Is what I am expecting to be able to do reasonable ? Or have I perhaps misunderstood some basic concepts ? As mentioned, it seems that the exploration of the scenario seems ok, but something goes wrong between the scenario and the test suite.

    I guess I could set up different machines, with different numbers of Adds, eg DoubleAddScenario, TripleAddScenario, etc, but this seems a lot of work. 

    Thanks in advance for any advice ! :)

    Sunday, November 13, 2011 2:04 AM


  • Hi Guraaku,

    I have tried your machine. Actually the exploration of "SlicedAccmulatorSlicedModelProgram" will hit bound also. That is because (Add*; ReadAndReset)* will result in infinite state machine. So what you used later (BoundedRepetition) is something that I suggest. :)



    Friday, November 25, 2011 9:51 AM

All replies

  • Hi Guraaku,

    * Since your model is infinite you have very soon reached all four bounds in a test suite:
        switch StateBound = 99999;
        switch StepBound = 99999;
        switch PathDepthBound = 99999;
        switch StepsPerStateBound = 99999;
    * Having the same bounds for model and test suite is a problem. Model grows - test suite grows much faster.
      The reason is, that traversal (test suite generation) produces a lot more steps, than the original model has.

    Here are some ideas how to change this:

    Introduce an accepting state (needed for path removal below), since your examples always end in the resetted state:
    public static bool Accepting
             return (accumulator == 0);
    * The test cases/examples you have shown fit better to this scenario :  (Add*; ReadAndReset) - having the reset only once.
    * Lift all bounds in the switches (see above - alternative: make an own config for the test suite with lifted bounds.).
    * Instead introduce a bound to cope with infinity in the sliced model, and erase all bounds you hit with path removal:

    machine SlicedAccumulatorModelProgram() : Main
        construct accepting paths where NonDeterministicPathRemoval = "RemoveFullPath" for
        construct bounded exploration where PathDepth = 10 for
        DoubleAddScenario || AccumulatorModelProgram

    You can also try "LongTests" instead of "ShortTests".

    Hope you find a good result with these ideas.

    Monday, November 14, 2011 2:28 PM
  • Hi bububa,

    Thanks for your suggestions :)

    I've done what you suggested, and played around with various combinations, but unfortunately I couldn't get it to work. It doesn't seem to be actually removing the paths which lead to non accepting states. So, it still keeps generating tests with errors.

    However, I did try one small modification to the original, and it seems to work ok !

    All I changed was to put defined limits on the number of repetitions of Add, rather than using *
    So, instead of Add*, I used Add{1,3}.

    This then generated tests with the number of Adds being between 1 and 3. Furthermore, no bad tests were produced.

    For now, this is ok - all I really wanted to achieve with the * was to say, give me any number of these. Now I can say, give me any number between this and this, which in practice, is what I'll need to do anyway.

    So, I'm not really sure what's going on - I was assuming that there was a fairly close correspondence between the SlicedAccumlatorModel machine and the AccumulatorTestSuite machine. After all, the AccumulatorTestSuite machine simply says generate the tests for SlicedAccumulatorModel. I thought it could just follow the paths in the exploration, and create tests for each path.

    I suspect I'll run into the same kind of issue though, of needing to cut down possible paths, when the number of paths becomes too big (very soon !) So I think the things you mentioned will come in handy soon.

    Also, I was a bit reluctant to put the AcceptingState annotation into my model, because in reality, in using the system, you can stop in any state. So, I was wanting to keep the model "pure" if you like, and let the cord file determine the format of the tests. This does seem to work ...so far. By defining the machine as Add{1,3}; ReadAndReset, it does generate tests for me of that structure, without having to define accumulator = 0 as an accepting state, to get the test to end in a ReadAndReset.

    Anyway, I will keep experimenting ! No doubt I'll be asking more questions along the way. I'm pretty enthusiastic about the whole idea of MBT, and hoping I can get some good results :)

    Thanks again for your help :)

    Tuesday, November 15, 2011 9:16 AM
  • Hi Guraaku,

    the root cause for the non-accepting ends is,
    that all the scenarios above force ending in a resetted state.

    Here is an idea how to change this too:

    * Taking a brand new static sample - project
    * Replacing the DoubleAddScenario with this : (Add; ReadAndReset*){0,}. Notice this
    allows to stop in any state now and it still reproduces the first problem with hitting bounds.

    Now, if I add:

    machine Sliced2() : Main 
        construct accepting paths where NonDeterministicPathRemoval = "RemoveFullPath" for
        DoubleAddScenario || AccumulatorModelProgram

    config Main2 : Main
        switch StateBound = none;
        switch StepBound = none;
        switch PathDepthBound = none;
        switch StepsPerStateBound = none;
    machine TestSuite2() : Main2
        construct test cases where strategy = "ShortTests" for Sliced2()

    Everything is beautiful green and fine within Sliced2 and TestSuite2.

    An alternative to changing your scenario is to add the "construct accepting paths" also in front of the "construct test cases", if this is your intention.

    Hope this can finally help.


    • Edited by bububa Saturday, November 19, 2011 7:42 AM
    Friday, November 18, 2011 1:56 PM
  • Hi Guraaku,

    I have tried your machine. Actually the exploration of "SlicedAccmulatorSlicedModelProgram" will hit bound also. That is because (Add*; ReadAndReset)* will result in infinite state machine. So what you used later (BoundedRepetition) is something that I suggest. :)



    Friday, November 25, 2011 9:51 AM