Re-enable the T and C linter prefix selectors by charliermarsh · Pull Request #3452 · astral-sh/ruff

charliermarsh · 2023-03-11T20:38:59Z

Summary

In one of the prior refactors around lint rule prefixes, we removed the T and C selectors.

To take T as an example, the issue is that we have two "linters" that both use the T prefix -- flake8-debugger, whose rules are (e.g.) T100, and flake8-print, whose rules are (e.g.) T201. In the refactored model, flake8-debugger was represented as having a prefix of T10, and a rule under that prefix with code 0, which combines to an overall selector of T100.

Under this scheme, if the user provides the T prefix, we don't know where to map that code. Prior to this PR, it was redirected to T10, so if a user provided --select T, that aliased to --select T10, which enabled rule T100, thus ignoring the flake8-print rules entirely. I believe this was unintentional.

The same problem exists with the C prefix, where we have both the McCabe complexity and flake8-comprehensions rules. The McCabe rules no longer run under --select C, because it was redirected to C40.

This PR special-cases those two prefixes, introducing custom selectors for T and C to re-enable the expected user behavior.

Closes #3052.

github-actions · 2023-03-11T20:48:15Z

✅ ecosystem check detected no changes.

charliermarsh · 2023-03-11T20:50:46Z

        // The following are here because we don't yet have the many-to-one mapping enabled.
        ("SIM111", "SIM110"),
        // The following are deprecated.
-        ("C", "C4"),


No longer need to redirect these at all, since they have their own custom selectors.

charliermarsh · 2023-03-11T20:53:27Z

There are likely further refactors we can do here, but I'd like to get this out in the next release as it's probably surprising to users that certain codes are no longer enforced despite no change in their selectors (e.g., select = ["C"] no longer runs the complexity checks).

Extend the structureless-cycle → `Never` resolution to `place_by_id` so that value/place cycles with no base case — e.g. a cyclic import `A = A`, or a loop-carried local defined only in terms of itself — resolve to `Never` instead of exposing the bare `Divergent` marker, matching the attribute-cycle behavior. Factored the resolution into `Type::resolve_structureless_cycle_to_never` and reused it from `wrap_structural_recursive`. Type-alias value cycles are deliberately excluded (they go through `Type::cycle_normalized` directly, not the place path): resolving a self-referential alias to `Never` would reintroduce the generic recursive-alias type-argument proliferation (#3452). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…generic recursive-alias overflow) `PEP695TypeAliasType::value_type` applied the alias's own specialization (`apply_function_specialization`) to the raw body *before* folding self-references to `Divergent`. For a generic recursive alias whose self-reference uses a deeper specialization than the alias's own parameters — e.g. type A[T] = Callable[[A[T]], A[T | A[T]]] type A[**P] = Callable[[], A[Concatenate[int, P]]] the specialization walk descended into the nested `A[…]` reference, recomputed `value_type` for an ever-deeper specialization, and never reached a fixed point — a stack overflow / non-convergence (ty issue #3452). The same path also overflowed on calling a parser-recovered recursive alias (`type A = | A` then `x()`, ty issue #3196). Fix: fold self-references to `Divergent(binder_id)` on the *raw* body first (the fold already matches by definition, so any specialization of the alias folds), then apply the alias's own specialization to the now-finite, self-reference-free body. The recursive argument is conservatively simplified to the alias's own parameters at the recursion point (`A[int | A[int]]` shown as `A[int]`), which is sound and stops the proliferation. When the type argument does not grow through the recursion (`type F[T] = list[F[T]]`), distinct specializations stay distinct (`F[int]` vs `F[str]`). Regression tests added for #3196 (call path) and #3452 (generic/ParamSpec recursive callable aliases) in pep695_type_aliases.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Extend the structureless-cycle → `Never` resolution to `place_by_id` so that value/place cycles with no base case — e.g. a cyclic import `A = A`, or a loop-carried local defined only in terms of itself — resolve to `Never` instead of exposing the bare `Divergent` marker, matching the attribute-cycle behavior. Factored the resolution into `Type::resolve_structureless_cycle_to_never` and reused it from `wrap_structural_recursive`. Type-alias value cycles are deliberately excluded (they go through `Type::cycle_normalized` directly, not the place path): resolving a self-referential alias to `Never` would reintroduce the generic recursive-alias type-argument proliferation (#3452). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…generic recursive-alias overflow) `PEP695TypeAliasType::value_type` applied the alias's own specialization (`apply_function_specialization`) to the raw body *before* folding self-references to `Divergent`. For a generic recursive alias whose self-reference uses a deeper specialization than the alias's own parameters — e.g. type A[T] = Callable[[A[T]], A[T | A[T]]] type A[**P] = Callable[[], A[Concatenate[int, P]]] the specialization walk descended into the nested `A[…]` reference, recomputed `value_type` for an ever-deeper specialization, and never reached a fixed point — a stack overflow / non-convergence (ty issue #3452). The same path also overflowed on calling a parser-recovered recursive alias (`type A = | A` then `x()`, ty issue #3196). Fix: fold self-references to `Divergent(binder_id)` on the *raw* body first (the fold already matches by definition, so any specialization of the alias folds), then apply the alias's own specialization to the now-finite, self-reference-free body. The recursive argument is conservatively simplified to the alias's own parameters at the recursion point (`A[int | A[int]]` shown as `A[int]`), which is sound and stops the proliferation. When the type argument does not grow through the recursion (`type F[T] = list[F[T]]`), distinct specializations stay distinct (`F[int]` vs `F[str]`). Regression tests added for #3196 (call path) and #3452 (generic/ParamSpec recursive callable aliases) in pep695_type_aliases.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Extend the structureless-cycle → `Never` resolution to `place_by_id` so that value/place cycles with no base case — e.g. a cyclic import `A = A`, or a loop-carried local defined only in terms of itself — resolve to `Never` instead of exposing the bare `Divergent` marker, matching the attribute-cycle behavior. Factored the resolution into `Type::resolve_structureless_cycle_to_never` and reused it from `wrap_structural_recursive`. Type-alias value cycles are deliberately excluded (they go through `Type::cycle_normalized` directly, not the place path): resolving a self-referential alias to `Never` would reintroduce the generic recursive-alias type-argument proliferation (#3452). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

## Summary Fixes ~~astral-sh/ty#3452~~ I intended to fix it, but the fixes were separated into #26881, #26882 and #26898. This PR provides a foundation for fixes, but there are observable improvements on this PR alone. For example: ```py from ty_extensions import static_assert from ty_extensions._internal import is_subtype_of type Left[T] = tuple[Left[list[T]]] type Right[T] = tuple[Right[list[T]]] # TODO: Left[int] should be equivalent to (subtype of) Right[int] static_assert(not is_subtype_of(Left[int], Right[int])) ``` This will result in stack overflow in the current main. A recursive type like `type StableRecursiveList[T] = T | list[StableRecursiveList[T]]` can be checked without any problem even in the current main. This is because the specialization that appears in the recursive type on the right side is the same as the left side, so the recursion guard using simple type equality works. The problem here is that among recursive aliases, the specialization on the right-hand side grows with each expansion. Current recursion guards cannot notice such type alias reentrancy. Therefore, when considering generic recursive type aliases, two levels of protection must be applied, distinguishing between equivalence based on type definition identity and full type equivalence, including specialization. Applying a recursive guard that only considers the latter will fail to detect cases of growing specialization (this is exactly the problem exposed by MRE). On the other hand, if we guard by considering only the former, specialization will not be considered, so we will treat `GrowingList[int]` and `GrowingList[str]` as the same thing. With this PR, the `CycleDetector` will now return a `Cycle` state in addition to `Ready` and `Pending` as a result of a visit. If the type aliases, including specializations, are equivalent to types already seen during the visit, the detector will still return the fallback value as `Ready`. If the type aliases are the same but the specializations are different, return `Cycle` and ask the upstream relation checker to decide. In the case of `TypeRelationChecker`, when this is received, it executes `recursive_type_pair_fallback` to complete the determination. --- So, what should `recursive_type_pair_fallback` do? If we call `check_type_pair` during this process, we will enter recursion again, so we need to make it a finite process. As I thought about it, I realized that this problem was undecidable. There is no general subtyping algorithm for growing recursive type aliases. This is because they will have expressive power equal to or greater than that of context-free grammars [^1]. In other words, determining the subtype of two such recursive aliases is the same problem as determining the equivalence and inclusion of two CFGs that are known to be undecidable. ```py from typing import Literal, final from ty_extensions import static_assert from ty_extensions._internal import is_subtype_of @Final class End: pass type AAnd[Rest] = tuple[Literal["a"], Rest] type BAnd[Rest] = tuple[Literal["b"], Rest] # S -> ε | aSb # {a^n b^n | n >= 0} type S[Rest] = Rest | AAnd[S[BAnd[Rest]]] # T -> ε | aTb | aaTbb # {a^n b^n | n >= 0} type T[Rest] = Rest | AAnd[T[BAnd[Rest]]] | AAnd[AAnd[T[BAnd[BAnd[Rest]]]]] # S and T produce exactly the same language, but the type checker cannot tell that. static_assert(not is_subtype_of(S[End], T[End])) ``` Therefore, we have to give up on this kind of recursive alias typing at some point (FYI, mypy makes growing recursive aliases like this illegal in the first place; pyright allows them, but seems to just have a recursion depth limit). In this PR, it simply returns a conservative solution immediately when a growing pattern of recursive type alias is detected. In reality, we can extend the support a little more, but we'll leave that as future work and focus on fixes first. --- #26881 strengthens the recursion guard in `TypeTransformer` under this PR change. This directly fixes #3452. #26882 strengthens the recursion guard in `UnionBuilder` under this PR change. This will properly stop the expansion of recursive union aliases that would result in stack overflow in the current main. #26898 adds `RecursionGuard` that wraps `TypeCollector`. [^1]: To be more specific, the type argument of type alias can be considered as the stack memory of a pushdown automaton. If the generic type alias itself appears on the right side with a different specialization than the left side, it corresponds to being able to push additional information onto the stack along with the state transition. If it has only trivial specializations like the left side, it cannot be used as a stack, and its abilities are equivalent to a finite automaton. ## Test Plan new mdtest cases --------- Co-authored-by: Micha Reiser <micha@reiser.io>

Re-enable the T and C linter prefix selectors

7330c66

charliermarsh added the bug Something isn't working label Mar 11, 2023

charliermarsh requested review from MichaReiser and konstin March 11, 2023 20:50

charliermarsh commented Mar 11, 2023

View reviewed changes

charliermarsh mentioned this pull request Mar 11, 2023

C901 (max-complexity) not reported since 0.0.247 #3052

Closed

Kilo59 mentioned this pull request Mar 13, 2023

Bug - Cannot ignore TCH004 via config file #3472

Closed

MichaReiser approved these changes Mar 13, 2023

View reviewed changes

konstin approved these changes Mar 13, 2023

View reviewed changes

charliermarsh merged commit 227679b into main Mar 13, 2023

charliermarsh deleted the charlie/t branch March 13, 2023 12:20

bastimeyer mentioned this pull request Mar 13, 2023

0.0.255: C4 selector prefix emits warning and maps to C40 only (subset of flake8-comprehensions) #3486

Closed

chenrui333 mentioned this pull request Mar 14, 2023

ruff 0.0.255 Homebrew/homebrew-core#125563

Closed

mikelane mentioned this pull request Mar 14, 2023

Bump ruff from 0.0.254 to 0.0.255 in /{{ cookiecutter.__project_name }} mikelane/cookiecutter-python-library#1

Merged

mtshiba mentioned this pull request Jul 16, 2026

[ty] Prevent stack overflows in recursive type relation checks #26503

Merged

coderabbitai Bot mentioned this pull request Jul 31, 2026

Add tests, coverage, line-count hooks, and pip-audit kgrizz-git/dislocker-ui#2

Merged

3 tasks

Navigation Menu

Uh oh!

Re-enable the T and C linter prefix selectors - #3452

Re-enable the T and C linter prefix selectors#3452
charliermarsh merged 1 commit into
mainfrom
charlie/t

charliermarsh commented Mar 11, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 11, 2023

Uh oh!

charliermarsh Mar 11, 2023

Uh oh!

charliermarsh commented Mar 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

charliermarsh commented Mar 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

github-actions Bot commented Mar 11, 2023

Uh oh!

charliermarsh Mar 11, 2023

Choose a reason for hiding this comment

Uh oh!

charliermarsh commented Mar 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

charliermarsh commented Mar 11, 2023 •

edited

Loading