CaseInsensitive Problem in the May CTP release

Question

I've been updating my DSL from the first release to the May CTP release.
I managed to update everything except case sensitivity.
I've tried the attribute "@{CaseInsensitive}" and "@{CaseInsensitive[]}" but it is not accepted by intellipad although it still accepts the CaseSensitive one (although not working).
I get the following exception in Intelipad:

DSL.mg : error :System.ArgumentNullException: Value cannot be null.

Parameter name: source

   at System.Linq.Enumerable.Contains[TSource](IEnumerable`1 source, TSource value, IEqualityComparer`1 comparer)

   at System.Linq.Enumerable.Contains[TSource](IEnumerable`1 source, TSource value)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(Char c, String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(Char c, String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(Char c, String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(Char c, String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(Char c, String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(Char c, String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(Char c, String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(Char c, String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.Nfa.SimulateLiteral(String literal, Int32 index, Boolean isFinal, Int32 tokenCode, NfaState current, Boolean allTransitionsExact, Boolean caseInsensitive)

   at Microsoft.M.Grammar.LexerGenerator.LexerGenerator.ToDfa(Boolean caseSensitive, List`1 protoTokens)

   at Microsoft.M.Grammar.LexerGenerator.LexerGenerator.MakeLexer(List`1 protoTokens, String initialTokenGroupName, Boolean caseSensitive, ProtoTokenActionData protoTokenActionData)

   at Microsoft.M.Grammar.LexerGenerator.LexerGenerator.Create(List`1 tokenTypes, String initialTokenGroupName, Boolean caseSensitive, Dictionary`2& tokenCodes, Dictionary`2& tokenOverlaps, ProtoTokenActionData& protoTokenActionData, List`1& protoTokens)

   at Microsoft.M.Grammar.ParserGenerator.ParserGenerator.GenerateParserTables(CompilerContext context, SyntaxType start, SyntaxType[] interleaveTypes, Boolean caseSensitive, BinaryWriter parserTableWriter, BinaryWriter lexerTableWriter, Object[]& productions, GrammarTerm[]& terms, Dictionary`2& tokenMap, Int32[]& productionTermMap, SyntaxType[]& syntaxes)

   at Microsoft.M.Grammar.LanguageDeclaration.GenerateParserData(CompilerContext context, SyntaxType defaultSyntaxType, SyntaxType[] interleaveTypes, Boolean caseSensitive)

   at Microsoft.M.Grammar.LanguageDeclaration.GenerateDynamicParserData(CompilerContext context)

   at Microsoft.M.Grammar.LanguageDeclaration.CreateDynamicParserFactory(CompilerContext context)

   at Microsoft.M.Compiler.CreateParserFactories(CompilerOptions options, CompilationResults results)

   at Microsoft.M.Compiler.SemanticAnalyzeGrammar(CompilerOptions options, CompilationResults results)

   at Microsoft.M.Compiler.Compile(CompilerOptions options)

   at Microsoft.M.LanguageServices.DslModeUpdater.Update(ITextSnapshot snapshot, CompilerOptions options)

Answer 1

Could you please post the grammar (or a subset which results in the error)? I just tried the following grammar and it does not exhibit this issue.

module test
{
    @{CaseInsensitive}
    language MyTest
    {
        syntax Main = "Hello World";
    }
}

Answer 2

You're right.
I may have something that is breaking it. I have created a new mg file with the definition like yours and it didn't gave me any error.
I have a definition that breaks when caseInsensitive is set. Here is my language:

module myApp {

  import Language;



  @{CaseInsensitive}

  language GUI {

    syntax Main = stepToken 

                            s:statement+ 

                  endToken

                  => Main{valuesof(s)};

    

    syntax statement =

        s:clickStatement => s |

        s:rightClickStatement => s;



    syntax clickStatement = 

        clickToken db:DialogButtons inToken dialogToken =>

            ClickOnDialog { Component => db }|

        clickToken al:component => ClickOn{ Alias => al } |

        clickToken el:String "@" al:component => 

            ClickAtElement { Alias => al, Element => el };

            

    syntax rightClickStatement = 

        rightClickToken al:component => RightClickOn {Alias => al} |

        rightClickToken el:String "@" al:component => 

            RightClickAtElement { Alias => al, Element => el };



    syntax component = a:Alias => a;



    token Alias = ("A".."Z" | "a".."z" | "_" | "0".."9")+;



    token DialogButtons = (

                "OK" |

                "OKCancel" | 

                "Abort" | 

                "Retry" |

                "Ignore" |

                "Yes" |

                "No"

                );



    @{Classification["Keyword"]}

    token clickToken = "click on";



    @{Classification["Keyword"]}

    token rightClickToken = "rightclick on";



    @{Classification["Keyword"]}

    token inToken = "in";



  }

}  

When I have both clickStatement and RightClickStatement defined I get an exception. If i comment one of the them it's ok. But this exception only happens when CaseInsensitive is set. Otherwise both definitions work with no problems

Thanks,
José Tavares

Answer 3

Well after posting this I tried one more thing and it is now working with both statements. What I did was create an onToken and take away the "on" of the clickToken and rightClickToken. Then in the definition of the clickStatement and rightClickStatement I added the onToken after the clickToken and rightClickToken.
Now everything is working fine. In the first release of oslo this definition was working. Any reasons why it isn't working in this CTP?

José Tavares

Answer 4

That does seem odd. I'm having difficulty reproing your issue though because the latest grammar in this thread has some missing terms (stepToken, endToken, dialogToken and String.) If you want to dive into this futher, please post a full grammar that repros the error. I'm happy to take a look. I'm particularly interested in the grammar that works if you remove CaseInsensitive or comment clickStatement or RightClickStatement.

Answer 5

Here is the rest of the definition:

token String = Language.Grammar.TextLiteral;

@{Classification["Keyword"]}
token stepToken = "Step";

@{Classification["Keyword"]}
token endToken = "end";

@{Classification["Keyword"]}
token dialogToken = "dialog";

One thing I've noticed is that, in that definition, when I have a token that is a string with two words case insensitivity does not work (I think it conflicts with the Alias definition). You can reproduce this by commenting one of the statements (clickStatement or rightClickStatement) in the syntax statement definition, and writing a script like this one:

Step
   click on OKCancel in dialog //"click on" cannot be case insensitive but dialog and in can
end

Thanks,
José Tavares

Answer 6

Ok I see the issue now. You have indeed found a bug. When there is a capital letter in the "click on" token, it is incorrectly getting recoginzed as an Alias due to some confusion with the interleave. (Note that I added a whitespace interleave and a String token to make this compile.) The good news is that it does not repro with our current bits due to some new features. The bad news is that those bits aren't out yet!

I think you've already found some workarounds, but here are two things that I tried with your grammar to make "case on" really be case insensitive:
1) Split "click on" into two tokens.
2) Change the keyword to "clickon" without the space.

Here is the full grammar with option 1:

module myApp {
  import Language;

  @{CaseInsensitive}
  language GUI {
    syntax Main = stepToken 
                            s:statement+ 
                  endToken
                  => Main{valuesof(s)};
    
    syntax statement =
        s:clickStatement => s |
        s:rightClickStatement => s;

    syntax clickStatement = 
        clickToken onToken db:DialogButtons inToken dialogToken =>
            ClickOnDialog { Component => db }|
        clickToken onToken al:component => ClickOn{ Alias => al } |
        clickToken onToken el:String "@" al:component => 
            ClickAtElement { Alias => al, Element => el };
            
    syntax rightClickStatement = 
        rightClickToken onToken al:component => RightClickOn {Alias => al} |
        rightClickToken onToken el:String "@" al:component => 
            RightClickAtElement { Alias => al, Element => el };

    syntax component = a:Alias => a;

    token Alias = ("A".."Z" | "a".."z" | "_" | "0".."9")+;

    token DialogButtons = (
                "OK" |
                "OKCancel" | 
                "Abort" | 
                "Retry" |
                "Ignore" |
                "Yes" |
                "No"
                );

    @{Classification["Keyword"]}
    token clickToken = "click";
    
    @{Classification["Keyword"]}
    token onToken = "on";

    @{Classification["Keyword"]}
    token rightClickToken = "rightclick";

    @{Classification["Keyword"]}
    token inToken = "in";

    @{Classification["Keyword"]}
    token stepToken = "Step";

    @{Classification["Keyword"]}
    token endToken = "end";

    @{Classification["Keyword"]}
    token dialogToken = "dialog";

    token String = ('A'..'Z' | 'a'..'z')+;
    interleave Whitespace = ' ' | '\r' | '\n';
  }
}  

I hope this helps!
Ben