Lexer reports false negative at end of input

[TheOnlyTails]

The lexer I wrote in #679 is now working better, but unfortunately it still gives this weird error:

2024-10-18T15:34:21.948752Z  INFO nymph_rust: tokens=Some([(Impl, 0..4), (Whitespace(" "), 4..5), (Identifier("Default"), 5..12), (Whitespace(" "), 12..13), (For, 13..16), (Whitespace(" "), 16..17), (IntType, 17..20), (Whitespace(" "), 20..21), (Braces([Whitespace("\n  "), Func, Whitespace(" "), Identifier("default"), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), DecimalInt(0), Whitespace("\n")]), 21..46), (Whitespace("\n\n"), 46..48), (Impl, 48..52), (Whitespace(" "), 52..53), (Identifier("Default"), 53..60), (Whitespace(" "), 60..61), (For, 61..64), (Whitespace(" "), 64..65), (FloatType, 65..70), (Whitespace(" "), 70..71), (Braces([Whitespace("\n  "), Func, Whitespace(" "), Identifier("default"), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), FloatInt(0.0), Whitespace("\n")]), 71..97), (Whitespace("\n\n"), 97..99), (Impl, 99..103), (Whitespace(" "), 103..104), (Identifier("Default"), 104..111), (Whitespace(" "), 111..112), (For, 112..115), (Whitespace(" "), 115..116), (BooleanType, 116..123), (Whitespace(" "), 123..124), (Braces([Whitespace("\n  "), Func, Whitespace(" "), Identifier("default"), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), False, Whitespace("\n")]), 124..153), (Whitespace("\n\n"), 153..155), (Impl, 155..159), (Whitespace(" "), 159..160), (Identifier("Default"), 160..167), (Whitespace(" "), 167..168), (For, 168..171), (Whitespace(" "), 171..172), (CharType, 172..176), (Whitespace(" "), 176..177), (Braces([Whitespace("\n  "), Func, Whitespace(" "), Identifier("default"), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), CharUnicode(0), Whitespace("\n")]), 177..206), (Whitespace("\n\n"), 206..208), (Impl, 208..212), (Whitespace(" "), 212..213), (Identifier("Default"), 213..220), (Whitespace(" "), 220..221), (For, 221..224), (Whitespace(" "), 224..225), (StringType, 225..231), (Whitespace(" "), 231..232), (Whitespace("\n  "), 233..236), (Func, 236..240), (Whitespace(" "), 240..241), (Identifier("default"), 241..248), (Parens([]), 248..250), (Whitespace(" "), 250..251), (Arrow, 251..253), (Whitespace(" "), 253..254), (Whitespace("\n"), 256..257), (Whitespace("\n\n"), 258..260), (Impl, 260..264), (Whitespace(" "), 264..265), (Identifier("Default"), 265..272), (Whitespace(" "), 272..273), (For, 273..276), (Whitespace(" "), 276..277), (Tuple([]), 277..280), (Whitespace(" "), 280..281), (Braces([Whitespace("\n  "), Func, Whitespace(" "), Identifier("default"), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), Tuple([]), Whitespace("\n")]), 281..308), (Whitespace("\n\n"), 308..310), (Impl, 310..314), (Lt, 314..315), (Identifier("T"), 315..316), (Gt, 316..317), (Whitespace(" "), 317..318), (Identifier("Default"), 318..325), (Whitespace(" "), 325..326), (For, 326..329), (Whitespace(" "), 329..330), (List([Identifier("T")]), 330..334), (Whitespace(" "), 334..335), (Braces([Whitespace("\n  "), Func, Whitespace(" "), Identifier("default"), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), List([]), Whitespace("\n")]), 335..362), (Whitespace("\n\n"), 362..364), (Impl, 364..368), (Lt, 368..369), (Identifier("K"), 369..370), (Comma, 370..371), (Whitespace(" "), 371..372), (Identifier("V"), 372..373), (Gt, 373..374), (Whitespace(" "), 374..375), (Identifier("Default"), 375..382), (Whitespace(" "), 382..383), (For, 383..386), (Whitespace(" "), 386..387), (Map([Identifier("K"), Colon, Whitespace(" "), Identifier("V")]), 387..394), (Whitespace(" "), 394..395), (Braces([Whitespace("\n  "), Func, Whitespace(" "), Identifier("default"), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), Map([]), Whitespace("\n")]), 395..422), (Whitespace("\n\n"), 422..424), (Impl, 424..428), (Lt, 428..429), (Identifier("T"), 429..430), (Gt, 430..431), (Whitespace(" "), 431..432), (Identifier("Default"), 432..439), (Whitespace(" "), 439..440), (For, 440..443), (Whitespace(" "), 443..444), (Identifier("Option"), 444..450), (Lt, 450..451), (Identifier("T"), 451..452), (Gt, 452..453), (Whitespace(" "), 453..454), (Braces([Whitespace("\n  "), Func, Whitespace(" "), Identifier("default"), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), Identifier("Option"), Dot, Identifier("None"), Whitespace("\n")]), 454..489), (Whitespace("\n\n"), 489..491), (Impl, 491..495), (Lt, 495..496), (Identifier("T"), 496..497), (Gt, 497..498), (Whitespace(" "), 498..499), (Identifier("Default"), 499..506), (Whitespace(" "), 506..507), (For, 507..510), (Whitespace(" "), 510..511), (Identifier("Iterator"), 511..519), (Lt, 519..520), (Identifier("T"), 520..521), (Gt, 521..522), (Whitespace(" "), 522..523), (Braces([Whitespace("\n  "), Func, Whitespace(" "), Identifier("default"), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), Identifier("Iterator"), DoubleColon, Braces([Whitespace(" "), Identifier("next"), Colon, Whitespace(" "), Parens([]), Whitespace(" "), Arrow, Whitespace(" "), Identifier("Option"), Dot, Identifier("None"), Whitespace(" ")]), Whitespace("\n")]), 523..584)])
Error: found end of input expected ''', '"', '(', '[', '{', '#', '-', '.', '?', '@', ',', ':', '_', '|', '+', '!', '*', '/', '%', '&', '^', '<', '>', '=', or '\'
    ╭─[../stdlib/src/default.nym:39:2]
    │
 39 │ }
    │  │ 
    │  ╰─ found end of input expected ''', '"', '(', '[', '{', '#', '-', '.', '?', '@', ',', ':', '_', '|', '+', '!', '*', '/', '%', '&', '^', '<', '>', '=', or '\'
────╯
Error: found end of input expected ''', '"', '(', '[', '{', '#', '-', '.', '?', '@', ',', ':', '_', '|', '+', '!', '*', '/', '%', '&', '^', '<', '>', '=', or '\'
    ╭─[../stdlib/src/default.nym:39:2]
    │
 39 │ }
    │  │ 
    │  ╰─ found end of input expected ''', '"', '(', '[', '{', '#', '-', '.', '?', '@', ',', ':', '_', '|', '+', '!', '*', '/', '%', '&', '^', '<', '>', '=', or '\'
────╯
Error: found end of input expected ''', '"', '(', '[', '{', '#', '-', '.', '?', '@', ',', ':', '_', '|', '+', '!', '*', '/', '%', '&', '^', '<', '>', or '='
    ╭─[../stdlib/src/default.nym:19:1]
    │
 19 │ }
    │ │ 
    │ ╰─ found end of input expected ''', '"', '(', '[', '{', '#', '-', '.', '?', '@', ',', ':', '_', '|', '+', '!', '*', '/', '%', '&', '^', '<', '>', or '='
────╯

It lists several unexpected EOIs, one of whom is is reported twice, and the other in them middle of the file (where the input doesn't end at all). What am I doing wrong here?

Lexer code: https://gist.github.com/TheOnlyTails/c37a5d6eddf6dccfb0ed6fc5238ca084
Input: https://gist.github.com/TheOnlyTails/248d6212832d9d3183f89c9bd65f79cf

[zesterer]

Ah, .then_ignore(end()) isn't required in chumsky 1.0. Chumsky parsers now expect end of input by default, and require .lazy() to opt out of this behaviour.

[TheOnlyTails]

It still acts the same way even without it, I only added it to make sure that wasn't the problem.

[Zij-IT]

Can you provide the definition of token so that we can run the code with the input?

[TheOnlyTails]

Yes, here you go: https://gist.github.com/TheOnlyTails/e081b1f459e86ddce28f88e24e31e476

[Zij-IT]

It doesn't falsely report anything ;) You have a bug in your string parsing logic.

        // You probably meant to write ^\\" here instead
        none_of(r"[^\\']").map(StringToken::Char),

Copy paste bit you in the butt :D

[TheOnlyTails]

ah, stupid mistakes like these always manage to sneak in! thank you very much!