[ty] Prevent stack overflows in recursive type relation checks

[mtshiba]

Summary

Fixes astral-sh/ty#3452
I intended to fix it, but the fixes were separated into #26881, #26882 and #26898. This PR provides a foundation for fixes, but there are observable improvements on this PR alone. For example:

from ty_extensions import static_assert
from ty_extensions._internal import is_subtype_of

type Left[T] = tuple[Left[list[T]]]
type Right[T] = tuple[Right[list[T]]]

# TODO: Left[int] should be equivalent to (subtype of) Right[int]
static_assert(not is_subtype_of(Left[int], Right[int]))

This will result in stack overflow in the current main.

A recursive type like type StableRecursiveList[T] = T | list[StableRecursiveList[T]] can be checked without any problem even in the current main. This is because the specialization that appears in the recursive type on the right side is the same as the left side, so the recursion guard using simple type equality works.

The problem here is that among recursive aliases, the specialization on the right-hand side grows with each expansion. Current recursion guards cannot notice such type alias reentrancy.

Therefore, when considering generic recursive type aliases, two levels of protection must be applied, distinguishing between equivalence based on type definition identity and full type equivalence, including specialization. Applying a recursive guard that only considers the latter will fail to detect cases of growing specialization (this is exactly the problem exposed by MRE). On the other hand, if we guard by considering only the former, specialization will not be considered, so we will treat GrowingList[int] and GrowingList[str] as the same thing.

With this PR, the CycleDetector will now return a Cycle state in addition to Ready and Pending as a result of a visit. If the type aliases, including specializations, are equivalent to types already seen during the visit, the detector will still return the fallback value as Ready. If the type aliases are the same but the specializations are different, return Cycle and ask the upstream relation checker to decide. In the case of TypeRelationChecker, when this is received, it executes recursive_type_pair_fallback to complete the determination.

---

So, what should recursive_type_pair_fallback do? If we call check_type_pair during this process, we will enter recursion again, so we need to make it a finite process. As I thought about it, I realized that this problem was undecidable. There is no general subtyping algorithm for growing recursive type aliases. This is because they will have expressive power equal to or greater than that of context-free grammars ¹. In other words, determining the subtype of two such recursive aliases is the same problem as determining the equivalence and inclusion of two CFGs that are known to be undecidable.

from typing import Literal, final
from ty_extensions import static_assert
from ty_extensions._internal import is_subtype_of

@final
class End:
    pass

type AAnd[Rest] = tuple[Literal["a"], Rest]
type BAnd[Rest] = tuple[Literal["b"], Rest]

# S -> ε | aSb
# {a^n b^n | n >= 0}
type S[Rest] = Rest | AAnd[S[BAnd[Rest]]]
# T -> ε | aTb | aaTbb
# {a^n b^n | n >= 0}
type T[Rest] = Rest | AAnd[T[BAnd[Rest]]] | AAnd[AAnd[T[BAnd[BAnd[Rest]]]]]

# S and T produce exactly the same language, but the type checker cannot tell that.
static_assert(not is_subtype_of(S[End], T[End]))

Therefore, we have to give up on this kind of recursive alias typing at some point (FYI, mypy makes growing recursive aliases like this illegal in the first place; pyright allows them, but seems to just have a recursion depth limit).
In this PR, it simply returns a conservative solution immediately when a growing pattern of recursive type alias is detected. In reality, we can extend the support a little more, but we'll leave that as future work and focus on fixes first.

---

#26881 strengthens the recursion guard in TypeTransformer under this PR change. This directly fixes #3452.
#26882 strengthens the recursion guard in UnionBuilder under this PR change. This will properly stop the expansion of recursive union aliases that would result in stack overflow in the current main.
#26898 adds RecursionGuard that wraps TypeCollector.

Test Plan

new mdtest cases

Footnotes

1. To be more specific, the type argument of type alias can be considered as the stack memory of a pushdown automaton. If the generic type alias itself appears on the right side with a different specialization than the left side, it corresponds to being able to push additional information onto the stack along with the state transition. If it has only trivial specializations like the left side, it cannot be used as a stack, and its abilities are equivalent to a finite automaton. ↩

[codspeed-hq]

Merging this PR will not alter performance

✅ 95 untouched benchmarks
⏩ 64 skipped benchmarks¹

---

_{Comparing type-identity-recursion-guard (86ddae3) with main (8bbdb8e)}

Footnotes

1. 64 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[charliermarsh]

Does this aim to fix any of the other issues linked from #24683?

[mtshiba]

Does this aim to fix any of the other issues linked from #24683?

astral-sh/ty#3195 and astral-sh/ty#3196 cannot be fixed with this PR.
Further investigation is underway, but in my opinion the root causes are different (unlike #26503, neither astral-sh/ty#3195 nor astral-sh/ty#3196 can result from common coding style, so perhaps we haven't taken them into account properly).

[astral-sh-bot]

Typing conformance results

No changes detected ✅

Current numbers

The percentage of diagnostics emitted that were expected errors held steady at 96.83%. The percentage of expected errors that received a diagnostic held steady at 91.74%. The number of fully passing files held steady at 99/133.

[astral-sh-bot]

Memory usage report

Summary

Project	Old	New	Diff	Outcome
sphinx	165.58MB	165.61MB	+0.02% (29.56kB)	⏫
flake8	28.68MB	28.68MB	-	✅
prefect	447.03MB	447.03MB	-	✅
trio	69.83MB	69.83MB	-	✅

Significant changes

Click to expand detailed breakdown

sphinx

Name	Old	New	Diff	Outcome
TypeAliasType<'db>::references_alias_::interned_arguments	0.00B	5.34kB	+5.34kB (new)	⏫
TypeAliasType<'db>::references_alias_	0.00B	4.53kB	+4.53kB (new)	⏫
infer_expression_types_impl	15.17MB	15.18MB	+0.03% (4.22kB)	⏫
infer_scope_types_impl	8.01MB	8.01MB	+0.05% (3.95kB)	⏫
Type<'db>::cached_materialization_	234.65kB	238.45kB	+1.62% (3.80kB)	⏫
infer_definition_types	13.68MB	13.69MB	+0.02% (3.50kB)	⏫
FunctionType<'db>::signature_	1.74MB	1.74MB	+0.07% (1.24kB)	⏫
is_redundant_with_impl	670.39kB	671.08kB	+0.10% (704.00B)	⏫
all_narrowing_constraints_for_expression	2.16MB	2.16MB	+0.03% (632.00B)	⏫
infer_unpack_types	303.24kB	303.69kB	+0.15% (464.00B)	⏫
infer_statement_types_impl	500.20kB	500.62kB	+0.09% (440.00B)	⏫
when_constraint_set_assignable_to_owned_impl	899.13kB	899.32kB	+0.02% (192.00B)	⏫
StaticClassLiteral<'db>::try_mro_	1.82MB	1.82MB	+0.01% (144.00B)	⏫
loop_header_reachability	230.98kB	231.11kB	+0.05% (128.00B)	⏫
analyze_non_terminal_call	686.02kB	686.13kB	+0.02% (120.00B)	⏫
...			5 more

[astral-sh-bot]

ecosystem-analyzer results

No diagnostic changes detected ✅

Full report with detailed diff (timing results)

[carljm]

Please consider comments (including Micha's comments) and either address or explicitly defer them, but I don't see hard blockers here and I think the core approach makes sense. Thank you!

[carljm]

Currently this leads to regression on cases of converging recursion that we correctly handle in main today.

main branch (and all other type checkers) handle this, but this PR fails:

type L[T] = tuple[T] | tuple[T, L[int]]
type R[T] = tuple[T] | tuple[T, R[int]]

def f(left: L[str], right: R[str]):
    right = left  # incorrectly rejected in this PR
    left = right  # incorrectly rejected in this PR

I guess we now consider any recursion with differing specialization as evidence of recursion and give up immediately.

But I guess ecosystem report suggests such patterns may not be common.

[mtshiba]

In this case, when we find an item with the same identity, instead of immediately returning Cycle, we can return Pending once to determine whether it is a growing pattern or just a stable recurive alias.

But I realized that there are cases where the recursive alias has to be expanded with different specializations many times before reaching a stable point. For example:

type Left[A, B, C] = tuple[A, Left[B, C, None]]
type Right[A, B, C] = tuple[A, Right[B, C, None]]

# Left[int, int, int] = tuple[int, Left[int, int, None]] = tuple[int, tuple[int, Left[int, None, None]]] = tuple[int, tuple[int, tuple[int, Left[None, None, None]]]]
# Left[None, None, None] (= tuple[None, Left[None, None, None]]) is recursive but stable, so it can be completely determined
static_assert(is_subtype_of(Left[int, int, int], Right[int, int, int]))

By increasing the number of type variables, we can defer any number of steps until we reach the stable point. Practically speaking, we should set an expansion limit.

[carljm]

This means that recursion via class-backed protocols or typed-dicts is invisible to is_recursive. So this (and the equivalent TypedDict example) still stack overflow on this PR:

from __future__ import annotations
from typing import Protocol
from ty_extensions import static_assert
from ty_extensions._internal import is_subtype_of

class LP[T](Protocol):
    child: "LA[list[T]]"
class RP[T](Protocol):
    child: "RA[list[T]]"

type LA[T] = LP[T]
type RA[T] = RP[T]

static_assert(not is_subtype_of(LA[int], RA[int]))

If this is an intentional scope cut, do you have a plan to fix it as a follow up?

[mtshiba]

Yes, I opened #26990

[carljm]

Is this supposed to enumerate all possibly-recursion-producing types? Protocol and TypedDict seem like notable omissions.

[mtshiba]

Yes, Protocol, TypedDict should also be supported here.
It would be easily covered by generalizing this PR mechanism. I will create a follow-up immediately after merging this PR.

[mtshiba]

I'll merge this PR. As for the unresolved issues, I have solutions in mind, but they will require additional review, so I will create follow-ups soon.

[charliermarsh]

Congratulations on the merge!

[mtshiba]

Congratulations on the merge!

Thanks, but there are still 5 draft PRs on top of this PR 😅

[charliermarsh]

One step at a time :)