Skip to content

Commit 08036e5

Browse files
committed
Make some minor aditions to the tokenize changes
1 parent 93a9286 commit 08036e5

File tree

1 file changed

+14
-4
lines changed

1 file changed

+14
-4
lines changed

Doc/whatsnew/3.12.rst

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -614,6 +614,8 @@ tokenize
614614

615615
* The :mod:`tokenize` module includes the changes introduced in :pep:`701`. (
616616
Contributed by Marta Gómez Macías and Pablo Galindo in :gh:`102856`.)
617+
See :ref:`whatsnew312-porting-to-python312` for more information on the
618+
changes to the :mod:`tokenize` module.
617619

618620
types
619621
-----
@@ -1207,6 +1209,8 @@ Removed
12071209
Iceape, Firebird, and Firefox versions 35 and below (:gh:`102871`).
12081210

12091211

1212+
.. _whatsnew312-porting-to-python312:
1213+
12101214
Porting to Python 3.12
12111215
======================
12121216

@@ -1301,10 +1305,16 @@ Changes in the Python API
13011305
1,13-1,17: FSTRING_MIDDLE ' end'
13021306
1,17-1,18: FSTRING_END '"'
13031307

1304-
Aditionally, final ``DEDENT`` tokens are now emitted within the bounds of the
1305-
input. This means that for a file containing 3 lines, the old version of the
1306-
tokenizer returned a ``DEDENT`` token in line 4 whilst the new version returns
1307-
the token in line 3.
1308+
Aditionally, there may be some minor behavioral changes as a consecuence of the
1309+
changes required to support :pep:`701`. Some of these changes include:
1310+
1311+
* Some final ``DEDENT`` tokens are now emitted within the bounds of the
1312+
input. This means that for a file containing 3 lines, the old version of the
1313+
tokenizer returned a ``DEDENT`` token in line 4 whilst the new version returns
1314+
the token in line 3.
1315+
1316+
* The ``type`` attribute of the tokens emitted when tokenizing some invalid Python
1317+
characters such as ``!`` has changed from ``ERRORTOKEN`` to ``OP``.
13081318

13091319
Build Changes
13101320
=============

0 commit comments

Comments
 (0)