Improve tokenize error handling

[pablogsal]

There has been quite a lot of instances were poor error handling in the tokenizer leads to crashes or errors being overwritten. We should not rely on continuous patches to every single individual issue but we should improve the situation with more robust infrastructure.

Linked PRs

• gh-104169: Ensure the tokenizer doesn't overwrite previous errors #104170
• gh-104169: Refactor tokenizer into lexer and wrappers #110684
• gh-104169: Fix test_peg_generator after tokenizer refactoring #110727

[pablogsal]

CC: @lysnikolaou

[hugovk]

Ready to close? Thanks!