Emit diagnostics for new syntax as per the target Python version

[dhruvmanila]

Our parser doesn't take into account the Python version aka target-version setting while parsing the source code. This means that we would allow having a match statement when the target Python version is 3.9 or lower. We want to signal this to the user.

One solution is to provide a diagnostic after the parsing is done. This diagnostic would indicate to the user that there's a syntax usage which isn't valid for the current target version. This is what's being done in Rome and Pyright¹. We would still continue linting the file and emit other diagnostics as well.

Following is a non-exhaustive list of syntax changes with the minimum required Python version:

3.13

• Type defaults for type parameters (PEP 696) ([syntax-errors] Type parameter defaults before Python 3.13 #16447)

3.12

• Formalized grammar for f-strings (PEP 701) ([syntax-errors] PEP 701 f-strings before Python 3.12 #16543)
• type statements ([syntax-errors] type statements before Python 3.12 #16478)
• type parameter lists ([syntax-errors] Type parameter lists before Python 3.12 #16479)

3.11

• except* syntax ([syntax-errors] except* before Python 3.11 #16446)
• Star expressions are allowed in index operations via PEP 646 ([syntax-errors] Star expression in index before Python 3.11 #16544)
• Star expressions are allowed in annotations for *args via PEP 646 ([syntax-errors] Star annotations before Python 3.11 #16545)

3.10

• match statement (Start detecting version-related syntax errors in the parser #16090)

3.9

• Relaxing grammar restrictions on decorators (PEP 614) ([syntax-errors] Named expressions in decorators before Python 3.9 #16386)
• Parenthesized context managers (formally made part of the language spec in Python 3.10, but usable -- and commonly used -- in Python >=3.9) ([syntax-errors] Parenthesized context managers before Python 3.9 #16523)
• Starred unpacking expressions are allowed in for statements on Python 3.9+ (officially made part of the language spec in Python 3.11) (cpython#90881) ([syntax-errors] Tuple unpacking in for statement iterator clause before Python 3.9 #16558)
• Assignment expressions can now be used unparenthesized in sequence indexes (but not slices) (What’s New In Python 3.10 ([syntax-errors] Unparenthesized assignment expressions in sets and indexes #16404))
  • These also worked on 3.9 when testing manually

3.8

• Assignment expressions (PEP 572) (Detect assignment expressions before Python 3.8 #16383)
• Positional-only parameters (PEP 570) ([syntax-errors] Positional-only parameters before Python 3.8 #16481)
• Parenthesized keyword-argument calls (e.g. f((a)=1)) are no longer allowed on Python 3.8+ (cpython#78822) ([syntax-errors] Parenthesized keyword argument names after Python 3.8 #16482)
• Iterable unpacking in return and yield statements no longer needs to be parenthesized on Pythonn 3.8+ (cpython#76298) ([syntax-errors] Tuple unpacking in return and yield before Python 3.8 #16485)

Footnotes

1. Pyright: Type alias statement requires Python 3.12 or newer ↩

[dhruvmanila]

The Match and Type alias statement can be implemented in the AST Statement analyzer within the match arms of the respective statement and checking the target-version setting.

I would suggest to add this as a new rule under the RUF category and all of the above would be in the same rule code.

[td-anne]

There might be other new syntax worth addressing:

• 3.11 except *
• 3.11 Variadic generics
• 3.8 Walrus operator (if there's interest in supporting pre-3.8)
• 3.10 Type unions with | (?)
• 3.10 Parenthesized with statements (with (a_gen() as a, b_gen() as b):
• 3.12 type parameter syntax (def max[T](list[T]) -> T:)
• 3.12-only f-strings (?)
• 3.9 dict unions with | (?)
• 3.9 lower-case list[]/dict[] in type annotations
• 3.9 PEP 617 extended generator syntax

Some of these might be closer to functional changes than syntax and hard to detect. But perhaps the easy ones could be folded in? There are advantages to python 3.12 that might make people want to develop with it even though their code is meant to run on python 3.9 (say).

[s-cork]

• 3.8 Walrus operator (if there's interest in supporting pre-3.8)

Just to say we have a use case for targeting 3.7. Thanks for the great work on this!

[achimnol]

So far, most new syntaxes were not "auto-applied" during formatting, but PEP-701 f-string placeholder updates are auto-applied to all codes. This prevents backporting those codes to older Python versions seamlessly as Ruff does not report syntax errors in older Python versions. Looking forward to see this issue to be resolved. 👀

[zanieb]

We'd appreciate contributions here if someone is interested in working on the project. We can do this relatively incrementally, but there is some initial complexity to determine how diagnostics are propagated.

@dhruvmanila / @charliermarsh will provide some additional details.