Lexer cleanup: Split literal lexing to separate file and simplify Cursor.

[Julian-Wollersberger]

The functions that lexed string and number literals were moved to a new file literals.rs, to make lib.rs smaller and more organized. I made them freestanding functions instead of methods on Cursor to have better separation of concerns: The Cursor should only be responsible for iterating char-by-char, and not for the lexer logic.

The first three commits only move code around and change the function calls from self.foo() to foo(cursor). The fourth renames the widely-used .first(), .second() and eat_while() methods to .peek(), .peek_second() and .bump_while(). The fifth commit inlines some helper functions. They were so small that it was harder to keep them in my head than to understand what the code does when inlined.

[rust-highfive]

r? @varkor

(rust-highfive has picked a reviewer for you, use r? to override)

[petrochenkov]

cc @matklad

[petrochenkov]

Style nit: could you avoid multi-line imports?

[petrochenkov]

• No opinion on the move from Cursor methods to free functions, I don't really understand what is a Cursor's job and what is not. I'm interested in @matklad's opinion on this.
• Same about renaming first/second to peek/peek_second.

Otherwise LGTM.

[matklad]

No strong feelings here!

I personally agree that using funcitons for "grammar rules" and methods for inspecting tokens is cleaner, but this is also somewhat unusual. If we do this, someone might "cleanup" them back to methods in a couple of years. See also this and this.

No opinion on helper's renaming.

Keeping all rules in a single file was intentional: ~1k lines is not that big, especially with method bodies folded in the editor, and grammars are inherently not compositional (two different rules might intersect with each other).

Helpers for grammar categories, like raw_ident or whitespace were intentional, to have "one grammar production = one function" correspondence

first_token/advance_token split was intentional, to make it easier to see the public API of the crate at a glance, and to have the signature advance_token match more closely to signatures of other "production" functions (should've named it token though).

But, as I've said, I have no strong opinion here!

[petrochenkov]

If we do this, someone might "cleanup" them back

Yeah, my impression about this PR is that is certainly a refactoring, but it's not obvious that it's a cleanup rather than changing the code to match a personal style of the author.

[varkor]

I feel I have less of a stake in the organisation of these files, and so if there isn't a strong consensus, it would be worth letting someone who is affected more directly by these changes make the final call. Perhaps if a refactoring isn't clearly an improvement, it's better not to make those changes, since at the very least it causes additional churn.

r? @matklad

[matklad]

It looks like there's a rough consensus that the benefits here do not out-weigh churn, so I am going to close this. I'd also like to add that the high order bit about code-cleanness here is that lexer doesn't depend on any other parts of the compiler and has a straightforward data-based interface. So it really doesn't matter much how is it structured internally.

Thanks for the pull request regardless @Julian-Wollersberger!