fix(core): indexing problem in Regexish work

[elijah-potter]

Issues

This PR solves a bug introduced by #669. Pinging @hippietrail for visibility.

Description

The Regexish PR neglected to include indexing in its unit test suite. As a result, it could emit tokens with indices that lay outside the document, which caused character Span fetches to fail.

This PR fixes the off-by-one error and improves test coverage to match.

Checklist

• I have performed a self-review of my own code
• I have added tests to cover my changes

[elijah-potter]

This is somewhat critical, so I will be merging this relatively quickly.

[hippietrail]

This is somewhat critical, so I will be merging this relatively quickly.

Thanks for pinging me! I think I've been seeing evidence of this today. I wonder why I didn't see it while I was working on it. Was something missing from my tests?

From looking at your changes it was right in the place with Some() and destructuring a struct that I had real trouble grocking. I decided it must be right after getting LLMs to explain it and to check my code and by testing it. Of course we know LLMs have their limitations but usually my skepticism is enough to spot when they're misleading me.

I still don't really understand those bits of code well by the way if somebody can explain them better.

[elijah-potter]

@hippietrail, the real issue was not the assert!(matches!. It was the fact that you didn't check the output index. This is a common problem with asking LLMs to generate test code: they make stuff that's similar to what's already there rather than improving coverage in a way that matters.

The bigger issue comes from the fact that I didn't review your code as carefully as I should have. That's on me.

[hippietrail]

It's when we zoom out a tiny bit further that I had trouble grokking the syntax:

        assert!(matches!(
        assert_eq!(
            lex_token(&source),
            Some(FoundToken {
                token: TokenKind::Regexish,
                ..
                next_index: 3
            })
        ));
        );

There's some kind of struct destructuring along with optional desctructuring or something and that .. which I mimicked from other code without really understanding it where it would've apparently been next_index: source.len()

I understand the logic where the size of the token needs to be added while doing the lexing. I've done lexing in a few languages over the years.

[elijah-potter]

Oh I understand. matches! is a pattern-matching macro. It follows Rust's match syntax.

Good resources:

• https://doc.rust-lang.org/std/macro.matches.html
• https://doc.rust-lang.org/book/ch19-03-pattern-syntax.html