Tokenizer produces different output on Windows on py312 for ends of files

[AlexWaygood]

Bug report

If you copy and paste the following code into a repro.py file and run python -m tokenize on it on a Windows machine, the output is different on 3.12/3.13 compared to what it was on 3.11 (the file ends with a single newline):

foo = 'bar'
spam = 'eggs'

On Python 3.11, on Windows, the output is this:

> python -m tokenize cpython/repro.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'foo'
1,4-1,5:            OP             '='
1,6-1,11:           STRING         "'bar'"
1,11-1,13:          NEWLINE        '\r\n'
2,0-2,4:            NAME           'spam'
2,5-2,6:            OP             '='
2,7-2,13:           STRING         "'eggs'"
2,13-2,15:          NEWLINE        '\r\n'
3,0-3,0:            ENDMARKER      ''

On Python 3.13 (@ 6e62eb2) on Windows, however, the output is this:

> python -m tokenize repro.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'foo'
1,4-1,5:            OP             '='
1,6-1,11:           STRING         "'bar'"
1,11-1,12:          NEWLINE        '\n'
2,0-2,4:            NAME           'spam'
2,5-2,6:            OP             '='
2,7-2,13:           STRING         "'eggs'"
2,13-2,14:          NEWLINE        '\n'
3,0-3,1:            NL             '\n'
4,0-4,0:            ENDMARKER      ''

There appear to be two changes here:

• All the NEWLINE tokens now have \n values, whereas on Python 3.11 they all had \r\n values
• There is an additional NL token at the end, immediately before the ENDMARKER token.

As discussed in PyCQA/pycodestyle#1142, this appears to be Windows-specific, and may be the cause of a large number of spurious W391 errors from the pycodestyle linting tool. (W391 dictates that there should be one, and only one, newline at the end of a file.) The pycodestyle tool is included in flake8, meaning that test failures in pycodestyle can cause test failures for other flake8 plugins. (All tests for the flake8-pyi plugin, for example, currently fail on Python 3.13 on Windows.)

Your environment

Python 3.13.0a0 (heads/main:6e62eb2e70, May 27 2023, 14:00:13) [MSC v.1932 64 bit (AMD64)] on win32

Linked PRs

• gh-105017: Fix including additional NL token when using CRLF #105022
• [3.12] gh-105017: Fix including additional NL token when using CRLF (GH-105022) #105023
• gh-105017: Include CRLF lines in strings and column numbers #105030
• [3.12] gh-105017: Include CRLF lines in strings and column numbers (GH-105030) #105041

[pablogsal]

CC: @mgmacias95

[AlexWaygood]

#105022 fixes the issue of the additional NL token at the end of the file on Windows, but there are still differences on Windows between what the tokenizer produced on 3.11 and what it produces on #105022.

On 3.11 on Windows:

>python -m tokenize cpython/repro.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'foo'
1,4-1,5:            OP             '='
1,6-1,11:           STRING         "'bar'"
1,11-1,13:          NEWLINE        '\r\n'
2,0-2,4:            NAME           'spam'
2,5-2,6:            OP             '='
2,7-2,13:           STRING         "'eggs'"
2,13-2,15:          NEWLINE        '\r\n'
3,0-3,0:            ENDMARKER      ''

Using #105022:

> python -m tokenize repro.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,3:            NAME           'foo'
1,4-1,5:            OP             '='
1,6-1,11:           STRING         "'bar'"
1,11-1,12:          NEWLINE        '\n'
2,0-2,4:            NAME           'spam'
2,5-2,6:            OP             '='
2,7-2,13:           STRING         "'eggs'"
2,13-2,14:          NEWLINE        '\n'
3,0-3,0:            ENDMARKER      ''

Note that the same tokens are emitted now as were emitted on 3.11, but the end-column coordinate is off by one for each NEWLINE token.

(I'm still seeing loads of flake8-pyi test failures even using #105022, and I'm presuming this is the cause?)