Regression in 1.4.1

[unconed]

I have a grammar which worked fine with @lezer/lr 1.4.0 producing incorrect trees in 1.4.1 (same generated grammar).

Reduced test case is here: https://github.com/unconed/lezer-bug-2024-07-24/

I have a rule which may be empty:

AttributeList { Attribute* }

This seems to sometimes throw it off, inserting the AttributeList in the wrong place in the parse tree.

Compare:
https://github.com/unconed/lezer-bug-2024-07-24/blob/master/1.4.0.txt
https://github.com/unconed/lezer-bug-2024-07-24/blob/master/1.4.1.txt

Instead of:

  LocalDeclaration                                          
    GlobalConstantDeclaration                               
      AttributeList      

it emits:

  AttributeList                                             
  LocalDeclaration                                          
    GlobalConstantDeclaration       

The test case is a reduced version of my WGSL grammar with the following shortened input:

// Comment

// Comment Comment
const ACES_INPUT = ;

// Comment Comment Comment
const ACES_OUTPUT = ; 

The strangest part is that changing the contents of the comments will affect the bug. If you remove the last word Comment in either comment 2 or 3, it parses correctly. But single-line comments are parsed with a custom token that scans for \n\r\v\f, it shouldn't care about the contents at all.

[marijnh]

Thanks for reducing that to a test case. Attached patch should help.