[ty] no more diverging query cycles in type expressions

[carljm]

Summary

Use Type::Divergent to short-circuit diverging types in type expressions. This avoids panicking in a wide variety of cases of recursive type expressions.

Avoids many panics (but not yet all -- I'll be tracking down the rest) from astral-sh/ty#256 by falling back to Divergent. For many of these recursive type aliases, we'd like to support them properly (i.e. really understand the recursive nature of the type, not just fall back to Divergent) but that will be future work.

This switches Type::has_divergent_type from using any_over_type to a custom set of visit methods, because any_over_type visits more than we need to visit, and exercises some lazy attributes of type, causing significantly more work. This change means this diff doesn't regress perf; it even reclaims some of the perf regression from #20333.

Test Plan

Added mdtest for recursive type alias that panics on main.

Verified that we can now type-check packaging (and projects depending on it) without panic; this will allow moving a number of mypy-primer projects from bad.txt to good.txt in a subsequent PR.

[github-actions]

Diagnostic diff on typing conformance tests

No changes detected when running ty on typing conformance tests ✅

[github-actions]

mypy_primer results

No ecosystem changes detected ✅
No memory usage changes detected ✅

[codspeed-hq]

CodSpeed WallTime Performance Report

Merging #20359 will improve performances by 5.89%

_{Comparing cjm/divergent-generic (814c42d) with main (c3f2187)}

Summary

⚡ 1 improvement
✅ 7 untouched

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	small[tanjun]	1.7 s	1.6 s	+5.89%

[carljm]

@mtshiba I can't request you as reviewer, but would be happy for you to take a look at this PR.

[AlexWaygood]

For many of these recursive type aliases, we'd like to support them properly (i.e. really understand the recursive nature of the type, not just fall back to Divergent) but that will be future work.

Will it still be easy to identify type aliases in user code that we don't properly support, if they no longer cause us to panic? The panics are obviously really bad UX, but the nice thing about them is it makes it very easy for us to spot where we have bugs

[carljm]

For many of these recursive type aliases, we'd like to support them properly (i.e. really understand the recursive nature of the type, not just fall back to Divergent) but that will be future work.

Will it still be easy to identify type aliases in user code that we don't properly support, if they no longer cause us to panic? The panics are obviously really bad UX, but the nice thing about them is it makes it very easy for us to spot where we have bugs

I agree this is a potential downside, but a) I think it's outweighed by the value of eliminating the cycles ASAP, and b) I'm not that worried about it. We already know the core recursive-alias cases that we need to support, and if there are a few edge-case bugs hiding, if they matter to anyone we'll discover them.

It's also quite easy to temporarily remove the Type::Divergent fallback, if we want to do that and run it on ecosystem projects as a bug-finding tool.

[AlexWaygood]

Sounds good!

[sharkdp]

Thank you very much!

Very much looking forward to fewer panics related to these.

[sharkdp]

Have you thought about implementing has_divergent_type using a custom TypeVisitor (similar to what any_over_type does), that would overwrite selected methods like visit_bound_type_var_type in order to skip visiting the parts that this needs/wants to skip?

[carljm]

Thanks for the prompt. I tried this but was slightly unhappy with how many types the visitor needed to acquire detailed knowledge of the internals of in order to correctly implement visiting types without visiting lazily-inferred type attributes. So I ended up creating a general visitor flag for this, so that each type's walk_ function can implement this distinction internally.

[marcelroed]

Here's an unhandled cycle using the latest commit:

from typing import Generic, TypeVar

AT = TypeVar("AT", bound="A")
BT = TypeVar("BT", bound="B")

class A(Generic[BT]):
    ...

class B(Generic[AT]):
    ...

This is from an ML modeling library where type A (model configs) need to be able to construct type B (model instances) and type B (model instances) need to be able to return type A (model configs). Feel free to delete/hide this comment if it should be a part of a separate issue instead.

[carljm]

@marcelroed Yes, thanks -- I'm aware this PR doesn't fix all the cases in astral-sh/ty#256. Your case is already mentioned in astral-sh/ty#256 (comment), so it will be addressed before we close that issue.