[ty] Create separate `FunctionLiteral` and `FunctionType` types #18360

dcreager · 2025-05-29T01:27:48Z

This updates our representation of functions to more closely match our representation of classes.

The new OverloadLiteral and FunctionLiteral classes represent a function definition in the AST. If a function is generic, this is unspecialized. FunctionType has been updated to represent a function type, which is specialized if the function is generic. (These names are chosen to match ClassLiteral and ClassType on the class side.)

This PR does not add a separate Type variant for FunctionLiteral. Maybe we should? Possibly as a follow-on PR?

Part of astral-sh/ty#462

* main: Add `offset` method to `ruff_python_trivia::Cursor` (#18371) ty_ide: improve completions by using scopes ruff_python_parser: add `Tokens::before` method [ty] Split `Type::KnownInstance` into two type variants (#18350) Bump 0.11.12 (#18369)

github-actions · 2025-05-29T17:45:31Z

`mypy_primer` results

No ecosystem changes detected ✅

crates/ty_python_semantic/src/types/function.rs

crates/ty_python_semantic/src/types/infer.rs

Co-authored-by: Micha Reiser <micha@reiser.io>

dcreager · 2025-05-30T15:27:34Z

@dhruvmanila's comment also makes me realize that these types are not well structured for specialized generic functions. (And what this is replacing wasn't either!) It is the individual overloads of a function that are generic or not, but we're currently record a list of specializations for the function as a whole! This is correct for the inherited_generic_context, which comes from the containing class and does apply to every overload. But it's not correct for specializations of individual overloads.

I'm going to put this back to draft while I have a think about this.

dcreager · 2025-05-30T19:44:08Z

@dhruvmanila's comment also makes me realize that these types are well structured for specialized generic functions. (And what this is replacing wasn't either!) It is the individual overloads of a function that are generic or not, but we're currently record a list of specializations for the function as a whole! This is correct for the inherited_generic_context, which comes from the containing class and does apply to every overload. But it's not correct for specializations of individual overloads.

We discussed this in Discord, and I think that the current representation is okay.

I think there are still open questions about what it would even mean to explicitly specialize an overloaded function. Until those questions are addressed, we don't need to worry about it, since we don't persist specializations of generic functions. We infer specializations of individual overloads as part of calling one, but we use those specializations immediately to infer the return type, and then throw it away. The only way that specialized functions show up persistently is via a containing generic class (e.g. C[int].method), and those are handled correctly by the current representation.

* main: [ty] support callability of bound/constrained typevars (#18389) [ty] Minor tweaks to "list all members" docs and tests (#18388) [ty] Fix broken property tests for disjointness (#18384) [ty] List available members for a given type (#18251) [`airflow`] Add unsafe fix for module moved cases (`AIR312`) (#18363) Add a `SourceFile` to `OldDiagnostic` (#18356) Update salsa past generational id change (#18362) [`airflow`] Add unsafe fix for module moved cases (`AIR311`) (#18366) [`airflow`] Add unsafe fix for module moved cases (`AIR301`) (#18367) [ty] Improve tests for `site-packages` discovery (#18374) [ty] _typeshed.Self is not a special form (#18377) [ty] Callable types are disjoint from non-callable `@final` nominal instance types (#18368) [ty] Add diagnosis for function with no return statement but with return type annotation (#18359) [`airflow`] Add unsafe fix module moved cases (`AIR302`) (#18093) Rename `ruff_linter::Diagnostic` to `OldDiagnostic` (#18355) [`refurb`] Add coverage of `set` and `frozenset` calls (`FURB171`) (#18035)

carljm

Pro refactoring: zero tests touched!

carljm · 2025-05-30T21:31:17Z

crates/ty_python_semantic/src/types/display.rs

+
+        // TODO: when generic function types are supported, we should add
+        // the generic type parameters to the signature, i.e.
+        // show `def foo[T](x: T) -> T`.


This TODO is confusing me. The "when generic function types are supported" part of it looks kind of obsolete -- haven't they been supported for a while now?

But it does look like we still don't show the generic context here, so maybe the overall TODO is still relevant? Is there something blocking us from resolving this? (If not, that doesn't mean it needs to be resolved in this PR.)

Nothing blocking other than deciding whether it's what we want to do. I've reworded the TODO comment for now

crates/ty_python_semantic/src/types/function.rs

carljm · 2025-05-30T21:37:30Z

crates/ty_python_semantic/src/types/function.rs

+//! - TODO: Some functions don't correspond to a function definition in the AST, and are instead
+//!   synthesized as we mimic the behavior of the Python interpreter. Even though they are
+//!   synthesized, and are “implemented” as Rust code, they are still functions from the POV of the
+//!   rest of the type system.


What is the actual TODO here? Is it to stop representing these as Type::Callable with the is_function_like flag that @sharkdp added, and instead include them in our function representation?

Yes, this comment #18242 (comment)

It's also somewhat combined with the point above about "known" functions, and the idea we've floated about pulling some of the special-case logic out from call/bind.rs and infer.rs into hook fields of the function representation.

I can reword or remove this if it's too muddled.

carljm · 2025-05-30T21:54:30Z

crates/ty_python_semantic/src/types/function.rs

+    /// The inherited generic context, if this function is a class method being used to infer the
+    /// specialization of its generic class. If any of the method's overloads are themselves


I find this comment slightly vague/confusing. This field is part of the identity of a FunctionLiteral type, so it kind of doesn't make sense to say "if... being used to..." -- this isn't an ephemeral thing that only shows up when the function is used in a certain way. Can we be more explicit/clear here about precisely under what circumstances we will have an inherited_generic_context on a function literal?

I'll answer the question here to help figure out what change to make to the comment.

For something like

class C[T]: def __init__(self, x: T) -> None: ...

when we encounter __init__ during type inference, we infer a FunctionLiteral for the definition. But we don't check right then whether the method is a generic class constructor, and so (like all other function definitions), this field is None:

ruff/crates/ty_python_semantic/src/types/infer.rs

Line 2038 in f379eb6

let inherited_generic_context = None;

During member lookup (which is performed as part of class construction), we special-case __init__ and __new__ to update this field to hold the generic context of the containing class:

ruff/crates/ty_python_semantic/src/types/class.rs

Lines 1285 to 1288 in f379eb6

"__new__" | "__init__",

) => Type::FunctionLiteral(

function.with_inherited_generic_context(db, generic_context),

),

That means we end up creating a second FunctionLiteral for the method, which is what we pass on to the call binding logic, to let us infer a specialization of a class.

So I think it actually is an ephemeral thing that only shows up during a constructor call.

Maybe it's enough to reword it as "when this function is a class method [etc]"?

Added a summary of ☝️ to the field comment

carljm · 2025-05-30T22:12:16Z

crates/ty_python_semantic/src/types/function.rs

+    /// Typed internally-visible signature for this function.
+    ///
+    /// This represents the annotations on the function itself, unmodified by decorators and
+    /// overloads.
+    ///
+    /// These are the parameter and return types that should be used for type checking the body of
+    /// the function.
+    ///
+    /// Don't call this when checking any other file; only when type-checking the function body
+    /// scope.


I think this doc comment has kind of migrated along with various refactors since I think I originally wrote it -- but looking at the current state of the code, I am not convinced it's fully accurate -- I don't think we actually use this at all for type checking the body of the function.

That's a good point, the signature is only used when analyzing call sites. (We do use the parameter and return type annotations when type-checking the body, but we don't get those from the function's Signature)

Removed

carljm · 2025-05-30T22:13:02Z

crates/ty_python_semantic/src/types/function.rs

+    /// This is the signature as seen by external callers, possibly modified by decorators and/or
+    /// overloaded.


This doc comment I think also deserves the same warning about not calling it cross-file. (I guess the fact that it's private should help here.) It's important that all cross-file access goes through the Salsa-tracked FunctionType::signature.

Added. I also added a similar comment to OverloadLiteral::signature above, which is crate-public, since it is called by the display() implementation, and when rendering diagnostics (which I think are both okay places to add the cross-file dependency?)

crates/ty_python_semantic/src/types/function.rs

…aration * origin/main: [ty] Treat lambda functions as instances of types.FunctionType (#18431) [ty] Fix false positives for legacy `ParamSpec`s inside `Callable` type expressions (#18426) [ty] Improve diagnostics if the user attempts to import a stdlib module that does not exist on their configured Python version (#18403) Update taiki-e/install-action action to v2.52.4 (#18420) Update docker/build-push-action action to v6.18.0 (#18422) [ty] Fix server hang after shutdown request (#18414) Update Rust crate libcst to v1.8.0 (#18424) Update Rust crate clap to v4.5.39 (#18419) Update cargo-bins/cargo-binstall action to v1.12.6 (#18416) Update dependency mdformat-mkdocs to v4.3.0 (#18421) Update pre-commit dependencies (#18418) Update dependency ruff to v0.11.12 (#18417) [ty] Ensure `Literal` types are considered assignable to anything their `Instance` supertypes are assignable to (#18351) [ty] Promote projects to good that now no longer hang (#18370) Sync vendored typeshed stubs (#18407) [ty] Fix multithreading related hangs and panics (#18238) Support relative `--ty-path` in ty-benchmark (#18385) [ty] Update docs for Python version inference (#18397) [ty] Infer the Python version from the environment if feasible (#18057) Implement template strings (#17851)

…aration * origin/main: [ty] Infer `list[T]` when unpacking non-tuple type (#18438) [ty] Meta-type of type variables should be type[..] (#18439) [`pyupgrade`] Make fix unsafe if it deletes comments (`UP050`) (#18390) [`pyupgrade`] Make fix unsafe if it deletes comments (`UP004`) (#18393) [ty] Support using legacy typing aliases for generic classes in type annotations (#18404) Use ty's completions in playground (#18425) Update editor setup docs about Neovim and Vim (#18324) Update NPM Development dependencies (#18423) Infer `list[T]` for starred target in unpacking (#18401) [`refurb`] Mark `FURB180` fix unsafe when class has bases (#18149) [`fastapi`] Avoid false positive for class dependencies (`FAST003`) (#18271)

* main: [ty] Add tests for empty list/tuple unpacking (astral-sh#18451) [ty] Argument type expansion for overload call evaluation (astral-sh#18382) [ty] Minor cleanup for `site-packages` discovery logic (astral-sh#18446) [ty] Add generic inference for dataclasses (astral-sh#18443) [ty] dataclasses: Allow using dataclasses.dataclass as a function. (astral-sh#18440) [ty] Create separate `FunctionLiteral` and `FunctionType` types (astral-sh#18360) [ty] Infer `list[T]` when unpacking non-tuple type (astral-sh#18438) [ty] Meta-type of type variables should be type[..] (astral-sh#18439) [`pyupgrade`] Make fix unsafe if it deletes comments (`UP050`) (astral-sh#18390) [`pyupgrade`] Make fix unsafe if it deletes comments (`UP004`) (astral-sh#18393) [ty] Support using legacy typing aliases for generic classes in type annotations (astral-sh#18404) Use ty's completions in playground (astral-sh#18425) Update editor setup docs about Neovim and Vim (astral-sh#18324) Update NPM Development dependencies (astral-sh#18423)

dcreager added 2 commits May 28, 2025 19:46

Move some stuff into a new function submodule

73679a1

Separate function literal and type types

022a1ae

dcreager added internal An internal refactor or improvement ty Multi-file analysis & type inference labels May 29, 2025

dcreager added 5 commits May 29, 2025 10:18

add missing normalized check

5317310

move constructor to top

7276957

cache overload resolution

852bb73

Don't consider gradual signatures when checking function subtyping

df24254

update generated files

6e71b21

dcreager marked this pull request as ready for review May 29, 2025 19:32

dcreager requested review from AlexWaygood, MichaReiser, carljm and sharkdp as code owners May 29, 2025 19:32

MichaReiser reviewed May 30, 2025

View reviewed changes

crates/ty_python_semantic/src/types/function.rs Outdated Show resolved Hide resolved

crates/ty_python_semantic/src/types/function.rs Outdated Show resolved Hide resolved

dhruvmanila approved these changes May 30, 2025

View reviewed changes

crates/ty_python_semantic/src/types/function.rs Show resolved Hide resolved

crates/ty_python_semantic/src/types/infer.rs Show resolved Hide resolved

dcreager and others added 2 commits May 30, 2025 10:14

Update crates/ty_python_semantic/src/types/function.rs

6b64965

Co-authored-by: Micha Reiser <micha@reiser.io>

use shifts in bitflags

4606b89

dcreager marked this pull request as draft May 30, 2025 15:27

dcreager added 5 commits May 30, 2025 16:03

Move the tests too

b1f5e2c

Only check overloaded functions

11dbcd3

current_overload → last_definition

86eba96

Add a big comment

9bb1c1e

dcreager marked this pull request as ready for review May 30, 2025 20:37

carljm approved these changes May 30, 2025

View reviewed changes

dhruvmanila reviewed Jun 2, 2025

View reviewed changes

crates/ty_python_semantic/src/types/function.rs Outdated Show resolved Hide resolved

dcreager added 8 commits June 2, 2025 13:25

update todo

53c5ef4

overloads can be independently generic

058006a

remove old comment

3efcc36

inline update_last_definition

0dbb452

add cross-file warnings

d3a559f

Expand description of inherited_generic_context

af74a2f

dcreager merged commit 2c3b3d3 into main Jun 3, 2025
35 checks passed

dcreager deleted the dcreager/function-separation branch June 3, 2025 14:59

		/// The inherited generic context, if this function is a class method being used to infer the
		/// specialization of its generic class. If any of the method's overloads are themselves

	"__new__" \| "__init__",
	) => Type::FunctionLiteral(
	function.with_inherited_generic_context(db, generic_context),
	),

		/// This is the signature as seen by external callers, possibly modified by decorators and/or
		/// overloaded.

[ty] Create separate FunctionLiteral and FunctionType types #18360

[ty] Create separate FunctionLiteral and FunctionType types #18360

Uh oh!

Conversation

dcreager commented May 29, 2025

Uh oh!

github-actions bot commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dcreager commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dcreager commented May 30, 2025

Uh oh!

carljm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[ty] Create separate `FunctionLiteral` and `FunctionType` types #18360

[ty] Create separate `FunctionLiteral` and `FunctionType` types #18360

github-actions bot commented May 29, 2025 •

edited

Loading

`mypy_primer` results

dcreager commented May 30, 2025 •

edited

Loading