Token types from internal rules end up in parser

[andreasabel]

In a grammar like

Parsable.  Exp ::= Integer ;
Internal.  Exp ::= String;

The definition of String should not end up in the lexer, and no parser for String should be generated.

Here is an example where parsing actually fails, similar to #204:

Main.              Prg ::= Ident     ;
internal IId.      Prg ::= Id        ;

token Id letter+ ;  -- This overlaps with Ident, but should not confuse the parser.

The generated (Haskell) parser does not accept input foo but should, since this is an Ident and Id is only used internally. (Dead code should not affect the semantics.)

[fonfalleh]

Hi,
I've been playing around writing a parser for Lilypond and I've been having an issue that I think relates to this issue.
In particular, the following should be equivalent, but I could only parse the latter: c_\ff, c _\ff.
I believe parsing fails because c_ is a valid Ident, however I am not using any Idents in my bnfc rules.

I managed to work around this by removing removing the references to identifier character, or $i in (Haskell backend) the generated Lex.x file.

Will this be solved by this issue, or could it be a separate issue?
Kind regards,
Jacob

[andreasabel]

Hello @fonfalleh, thanks for he report.

Atm BNFC does not correctly support grammars that do not have a notion of identifier. This is because keywords are first parsed (more precisely: lexed) as identifiers, and then looked up in a keyword dictionary. For instance, keyword "if" matches also the regex for an Ident and is lexed as that and then turned into a keyword token.

A fix would be to include keywords in the generated lexer definition as their separate rules. However, this blows up the generated lexer significantly. (This might even exceed the limits on the automaton size in e.g. in ocamllex.) Thus, in practice, one lexes keywords as identifiers.

I think allowing the redefinition of Ident (#197) would solve your problem.

[fonfalleh]

Thanks!
I make use of token for variable declarations and references, but they can only contain letters, so I didn't use Ident for that in particular.

Does this mean that everything that can be lexed must match an Ident? Not sure if I follow.

[andreasabel]

Does this mean that everything that can be lexed must match an Ident? Not sure if I follow.

No, but BNFC puts a regex for Ident in the lexer definition, whether you use it or not. This means that e.g. c_\ff will be lexed as Ident("c_") \ff rather than c _\ff because of the longest match rule (the match "c_" is longer than the match "c" and thus takes precedence). This explains the problems you are experiencing with your generated lexer.

Your problem would be fixed if Ident was not included in the lexer definition, but then the lexer has handle keywords differently.