gh-105820: Fix tok_mode expression buffer in file & readline tokenizer

[lysnikolaou]

• Issue: patchcheck.py raises SystemError on main #105820
• Issue: F-string debug mode does not print correctly when in last line of file #105831

[pablogsal]

@lysnikolaou Hummm, can you quickly cover why moving this chunk solves the problem? I think I am missing some key insight here

[lysnikolaou]

Sure. There's two different cases to consider here:

1. When we're calling the tokenizer from the tokenize module, it uses the readline tokenizer, which you added in 9216e69. In that branch, update_fstring_expr wasn't being called at all from tok_underflow_readline, which resulted in the last_expr_size not being updated. That's why we were passing a negative count to PyUnicode_DecodeUTF8.

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ cat tmp/t.py
print(f'''{
3
=}''')%                                                                                                                                                                                                                                                                                                                                                                                                         

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ ./python.exe -m tokenize tmp/t.py
unexpected error: Negative size passed to PyUnicode_New
Traceback (most recent call last):
  File "/Users/lysnikolaou/repos/python/cpython/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lysnikolaou/repos/python/cpython/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line 547, in <module>
    main()
  File "/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line 489, in main
    tokens = list(tokenize(f.readline))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line 447, in tokenize
    yield from _generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True)
  File "/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line 537, in _generate_tokens_from_c_tokenizer
    for info in it:
SystemError: Negative size passed to PyUnicode_New

2. There was a second bug with an f-string being tokenized from a file, only when the closing brace of the f-string expression part is in the last line of the file and there's no newline at the end. In that branch we use tok->cur to calculate the size of the buffer before adding an implicit newline and then tok->start after adding the newline. That resulted in the end buffer being one character too short.

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 took 15s 
❯ cat tmp/t.py
print(f'''{
3
=}''')%                                                                                                                                                                                                                                                                                                                                                                                                         

cpython on  main via C v14.0.3-clang via 🐍 pyenv 3.11.3 
❯ ./python.exe tmp/t.py            

3
3

[lysnikolaou]

Do you think I should split this PR in two and open a second issue for the second case above?

[pablogsal]

Do you think I should split this PR in two and open a second issue for the second case above?

Nah, I think is fine, but we may want to add two NEWS entries

[pablogsal]

LGTM

Thanks for the explanation @lysnikolaou. Fantastic work as always! 💪

[pablogsal]

@lysnikolaou If you want add the second NEWS entry and then land it.

[miss-islington]

Thanks @lysnikolaou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[bedevere-bot]

GH-105832 is a backport of this pull request to the 3.12 branch.