locked
Requirements Coverage RRS feed

  • Question

  • Hi,

    Being able to embed requirements into the model and visualize them during test case generation is a nice feature.  I started using this feature.

    I got a couple of questions:

    1. Is there any way to know how many requirements are not covered for the given machine? For example, for the attached view I do not quite know are there any requirements that are not covered by my tests for the given machine. Is there a possibility to know this?

    2. Another question is related to the state space: In order to generate requirements-based tests, I am using "construct requirement coverage where strategy = "selective" for MyMachine

    The problem is that the state/step size of MyMachine can be very big if I increase the no. of parameters to actions. Is there any way to construct a requirements-based machine and tests without first constructing the state space of MyMachine? The reason is that the requirements-based machine appears to be so much smaller than the base machine MyMachine.

    Any comments?

    Thanks,

    Dharma 


    • Edited by Dharma11 Wednesday, November 7, 2012 4:11 PM
    Wednesday, November 7, 2012 4:07 PM

All replies

  • Hi Dharma,

    1.) Here is a way how you can reach this using the tools found in the Spec Explorer sample "RequirementReport" :
    It needs a little bit programming, but this customization is expected, because every project might have different needs (see http://social.msdn.microsoft.com/Forums/en-US/specexplorer/thread/9e8f84ca-23f4-45a9-8eb5-26e121aca230):

    The example gives you already a table with all requirements in your current machine (step 1).So basically you write now an additional customized post processing method. This additional code searches all your cs-files for requirements (step 2 - and maybe store also cord-file requirements in this table, if there are tags too).
    Now you can easily compare the two tables and find all missing requirements (step 3).
    Luckily this method also works if your model is made from many components and/or you have many machines:  you simply merge finally the requirements from all components and all machines and redo step 3.

    2.) This performance-issue is the big disadvantage of "requirement coverage". According to the developers this problem is very deep (backward exploration). This is also the reason why I'm using symbolic exploration and path coverage instead: It heavily reduces the machine in the first place. If I want to reduce it further or add some things symbolic exploration can't do today I steer it with scenarios - this always worked very elegantly in my project and I think it can solve your problem you mentioned in another thread. You find some examples in this forum or give a new example, if you like to have a demonstration.

    Thursday, November 8, 2012 8:16 AM
  • Hi,

    thanks for your reply. Regarding item 2, I did watch videos on Channel 9. They are very helpful, but not sure there was any discussion on symbolic exploration. It is perhaps the case that I missed some parts. Do you know the part numbers of the videos that deal with symbolic exploration?

    Thanks,

    Dharma

    Tuesday, November 13, 2012 3:59 PM
  • Hi Dharma,

    On Channel 9 symbolic exploration is mentioned only very briefly in the discussion after the lecture (final session 4 - I think).
    Unfortunately I don't know a simple explanation/example from the developers.
    Symbolic exploration ís an advanced method. It is a good practice to encapsulate most of it into library functions for your project. You will not need to use it directly, after you have identified these important library functions for you.

    Moreover it has several different application areas. Two relevant threads with examples for you might be:

    http://social.msdn.microsoft.com/Forums/en-US/specexplorer/thread/9665bd8f-1d65-4ffc-a1c2-b65c2051a85d
    http://social.msdn.microsoft.com/Forums/en-US/specexplorer/thread/3169093e-64f5-4dae-ad8c-55dd6295942f/#9cbe39ac-506b-46eb-9ccd-8edc900f4216

    Wednesday, November 14, 2012 1:28 PM
  • Hi,

    I tried using symbolic exploration. For example, here is my model code for the subscribe method. I used "CombinationStrategy.None" in the first line of my model code. Unfortunately, I do not see any difference in the explored state machine even if I remove that CombinationStrategy line.

    If I remove all lines from CanSubscribe (which is my guard) until the end of the "if" statement, then I see a graph with a symbolic value to the subject parameter. But, of course, I cannot delete those important lines! 

    How will you use symbolic exploration?

    subscribe+ will result in traces such as 1) subscribe("xy");subscribe("uvw"), 2) subscribe("uvw");subscribe("xy").  I want to treat both sequences as equivalent. How will you define abstractions of sequencing that just differs due to parameters?

     [Rule]
            void Subscribe(string subject, bool result)
            {
                Combination.Strategy(CombinationStrategy.None);
                bool guardStatus = subscribersMgmt.CanSubscribe(this, subject, patternMatcher);
                if (guardStatus)
                {
                    subscribersMgmt.AddSubscription(this, subject);
                }
    
                Condition.IsTrue(result == guardStatus);
            }
    machine SubscribeScenario() : Main where ForExploration = true
    {
       let SoftwareBusAdapterImpl conn
    
       in
    
       new conn.SoftwareBusAdapterImpl;
       (
         conn.Create;
         (
            conn.Connect;
            conn.Subscribe+;
            conn.Disconnect;
         );
         conn.Destroy;
       )
    }
    
    machine SubscribeProgram() : Main where ForExploration = true
    {
      bind Subscribe({"xy", "uvw"}, _)
      
      in
       
      (SubscribeScenario;exit) || ModelProgram
    }




    • Edited by Dharma11 Friday, November 16, 2012 3:07 AM
    Friday, November 16, 2012 3:01 AM
  • Hi Dharma,

    One of the goals of symbolic exploration mentioned in the lecture is
    that we let Spec Explorer choose the arguments for your rule according to the goal to cover requirements.

    So the idea is to drop the line "bind  Subscribe({"xy", "uvw"}, _)" and replace it with something else.
    If you really want to have these same arguments, this replacement could be some code like this (added to your rule):

    if (subject == "xy" | subject == "uvw")
        Requirement.Capture("#1");

    The idea in this modeling is, that you've choosen the two different arguments, because you assume that different code is executed inside the system under test. We "model" this wit the if-clause.
    This technique should allow to reduce the number of combinations later and to track them with requirement-tags.

    So first I would like to ask, if this idea of modeling is making any sense in your project?

    Friday, November 16, 2012 2:16 PM
  • Hi,

    Thanks. I did try without the bind statement but then the spec explorer shows an error state. I notice this issue whenever there is a string argument for which I do not feed data.

    I am not in favor of embedding "xy" and "uvw" inside my rule/model code because this limit the usefulness of the rule in the context of working with other rules.

    Please let me know how you will use Symbolic exploration for the example machine I attached earlier.

    Thanks,

    Dharma

    Friday, November 16, 2012 2:22 PM
  • Hi Dharma,

    At least we see that Spec Explorer can:
    * explore your rule subscribe(a1) with a variable string a1 (symbolic).
    * Find out that for a1 = "xy" a branch of the if-clause is reached.
    * Finally expands a1 for reaching all branches one time.
    * Result: generate only the call subscribe("xy")

    In your example it could:
    * explore subscribe(a1);subscribe(a2) with two variable strings a1,a2.
    * Find out that for a1 = "xy" (or "uvw") a branch of the if-clause is reached.
    * Find also out that for any a2 the same branch is reached.
    * Finally expands a1,a2 for reaching all branches one time.
    * Result: generate only the call subscribe("xy");subscribe("uvw"), since you have only one path and since subscribe("uvw");subscribe("xy"); results in the exact same code-paths it is omitted.

    Does this help in your example?
    Friday, November 16, 2012 9:05 PM
  • Hi,

    My subscribe function takes a string regular expression pattern in fact. There are a lot of patterns I have to test for.Thus, I am not sure the above description fits my context. 

    For example, the first line here is good valid subject as an argument and the second line is invalid ones.

    {"xy", "x.y.z", "test.*.>", "test.*.*.>", "*",
       "**", "", "test.>.x", "test>", "test.a*.b", "test..a", "test.a.", ".a.test", "test b", "x/a", "x\\a"}

    I'm more concerned about the other thread where I mentioned not being able to express symmetry and constrain the state space due to parameters. Did you see my example involving subscribe and unsubscribe? Even with two arguments, the number of states grow a lot?
      

    Saturday, November 17, 2012 1:04 AM
  • Hi Dharma,

    ok I don't want to talk you into symbolic exploration.
    I see your point - I have no solution.
    If Nico has also already looked at it, a last chance might be conformiq's state coverage ...

    Here are some ideas - maybe they can still help somehow:
    * Indeed we can eliminate with symbolic exploration only a few transitions from your example.
      Example: S6 to S11, because it only differs in the arguments to transition S6 to S10.
      Symbolic Exploration could help, if you have an infinite number of arguments but a small number of  combinations of actions to call.
    * Your example has a lot of combinations of action calls : interleaving 5 actions. So the usual idea is to use a stronger scenario.
    * If you can see the covered requirements you quite smartly add a scenario, if some requirements are not covered, but in your case I see no way to switch over from state coverage to requirements.
    * You don't have to use something like:
      if (subject == "xy" | subject == "uvw") you can also use something like
      if (HasReservedCharacter(subject) | HasInCorrectWildCardUsage(subject) | HasInCorrectDelimiterUsage(subject)) etc.
      with functions checking the argument "subject". Spec Explorer can find examples for the arguments for you on its own.
    * If you want to drop the bind-statement you can use "CombinationStrategy.Coverage", else your exploration is infinite.
    * "CombinationStrategy.None" should indeed keep "subject" symbolic. What you have seen is a known issue for string-arguments.

    Monday, November 19, 2012 8:25 AM
  • Hi,

    Here are some ideas - maybe they can still help somehow:
    * Indeed we can eliminate with symbolic exploration only a few transitions from your example.
      Example: S6 to S11, because it only differs in the arguments to transition S6 to S10

    Is it possible in the current version of Spec Explorer to define constraints to avoid symmetry? I thought Nico mentioned that we can only avoid such symmetry by using the requirements-coverage construct because Spec Explorer is step-based and not state-based, if I get the message of Nico correct.

    I have a related issue. Is there a way to avoid certain inter-leavings. For example, consider this below machine. It is for two subscribers and one publisher. It does not really matter which subscriber reads the message.  How will we treat (subConn.GetNextMsg(_)* ||| subConn2.GetNextMsg(_)* ) as the same as  (subConn2.GetNextMsg(_)* ||| subConn.GetNextMsg(_)*)

    machine SubTwoPubOneScenario() : Main where ForExploration = false
    {
       let SoftwareBusAdapterImpl subConn,
           SoftwareBusAdapterImpl subConn2, 
           SoftwareBusAdapterImpl pubConn 
    
       in
    
       ConnSetup(subConn);
       ConnSetup(subConn2);
       ConnSetup(pubConn);
    
       subConn.Subscribe;
       subConn2.Subscribe; 
    
    
       (pubConn.Publish* ||| subConn.GetNextMsg(_)* ||| subConn2.GetNextMsg(_)*);
    
       ConnTearDown(subConn);
       ConnTearDown(subConn2);
       ConnTearDown(pubConn);
    
    }

     Thanks.

    Dharma


    • Edited by Dharma11 Monday, November 26, 2012 8:31 PM
    Monday, November 26, 2012 8:22 PM