@@ -614,6 +614,8 @@ tokenize
614
614
615
615
* The :mod: `tokenize ` module includes the changes introduced in :pep: `701 `. (
616
616
Contributed by Marta Gómez Macías and Pablo Galindo in :gh: `102856 `.)
617
+ See :ref: `whatsnew312-porting-to-python312 ` for more information on the
618
+ changes to the :mod: `tokenize ` module.
617
619
618
620
types
619
621
-----
@@ -1207,6 +1209,8 @@ Removed
1207
1209
Iceape, Firebird, and Firefox versions 35 and below (:gh: `102871 `).
1208
1210
1209
1211
1212
+ .. _whatsnew312-porting-to-python312 :
1213
+
1210
1214
Porting to Python 3.12
1211
1215
======================
1212
1216
@@ -1301,10 +1305,16 @@ Changes in the Python API
1301
1305
1,13-1,17: FSTRING_MIDDLE ' end'
1302
1306
1,17-1,18: FSTRING_END '"'
1303
1307
1304
- Aditionally, final ``DEDENT `` tokens are now emitted within the bounds of the
1305
- input. This means that for a file containing 3 lines, the old version of the
1306
- tokenizer returned a ``DEDENT `` token in line 4 whilst the new version returns
1307
- the token in line 3.
1308
+ Aditionally, there may be some minor behavioral changes as a consecuence of the
1309
+ changes required to support :pep: `701 `. Some of these changes include:
1310
+
1311
+ * Some final ``DEDENT `` tokens are now emitted within the bounds of the
1312
+ input. This means that for a file containing 3 lines, the old version of the
1313
+ tokenizer returned a ``DEDENT `` token in line 4 whilst the new version returns
1314
+ the token in line 3.
1315
+
1316
+ * The ``type `` attribute of the tokens emitted when tokenizing some invalid Python
1317
+ characters such as ``! `` has changed from ``ERRORTOKEN `` to ``OP ``.
1308
1318
1309
1319
Build Changes
1310
1320
=============
0 commit comments