PEP 695: Lazy evaluation, concrete scoping semantics, other changes

pep-0695.rst

@@ -225,10 +225,10 @@ requirement that parameter names within a function signature must be unique.
     def func1[T, **T](): ... # Syntax Error
 
 
-Class type parameter names are not mangled if they begin with a double
-underscore. Mangling would not make sense because type parameters, unlike other
-class-scoped variables, cannot be accessed through the class dictionary, and
-the notion of a "private" type parameter doesn't make sense.
+Class type parameter names are mangled if they begin with a double
+underscore, to avoid complicating the name lookup mechanism for names used
+within the class. However, the ``__name__`` attribute of the type parameter
+will hold the non-mangled name.
 
 
 Upper Bound Specification
@@ -302,6 +302,14 @@ the existing rules enforced by type checkers for a ``TypeVar`` constructor call.
     class ClassG[T: (list[S], str)]: ...  # Type checker error: generic type
 
 
+Runtime Representation of Bounds and Constraints
+------------------------------------------------
+
+The upper bounds and constraints of ``TypeVar`` objects are accessible at
+runtime through the ``__bound__`` and ``__constraints__`` attributes.
+For ``TypeVar`` objects defined through the new syntax, these attributes
+become lazily evaluated, as discussed under :ref:`695-lazy-evaluation` below.
+
 
 Generic Type Alias
 ------------------
@@ -365,13 +373,13 @@ At runtime, a ``type`` statement will generate an instance of
 include:
 
 * ``__name__`` is a str representing the name of the type alias
-* ``__parameters__`` is a tuple of ``TypeVar``, ``TypeVarTuple``, or
+* ``__type_params__`` is a tuple of ``TypeVar``, ``TypeVarTuple``, or
   ``ParamSpec`` objects that parameterize the type alias if it is generic
 * ``__value__`` is the evaluated value of the type alias
 
-The ``__value__`` attribute initially has a value of ``None`` while the type
-alias expression is evaluated. It is then updated after a successful evaluation.
-This allows for self-referential type aliases.
+All of these attributes are read-only.
+
+The value of the type alias is evaluated lazily (see :ref:`695-lazy-evaluation` below).
 
 
 Type Parameter Scopes
@@ -382,9 +390,9 @@ includes the type parameters. Type parameters can be accessed by name
 within inner scopes. As with other symbols in Python, an inner scope can
 define its own symbol that overrides an outer-scope symbol of the same name.
 
-Type parameters declared earlier in a type parameter list are visible to
-type parameters declared later in the list. This allows later type parameters
-to use earlier type parameters within their definition. While there is currently
+Type parameters are visible to other
+type parameters declared elsewhere in the list. This allows type parameters
+to use other type parameters within their definition. While there is currently
 no use for this capability, it preserves the ability in the future to support
 upper bound expressions or type argument defaults that depend on earlier
 type parameters.
@@ -401,11 +409,9 @@ defined in an outer scope.
     # eliminate this limitation.
     class ClassA[S, T: Sequence[S]]: ...
 
-    # The following generates a compiler error or runtime exception because T
-    # is referenced before it is defined. This occurs even though T is defined
-    # in the outer scope.
-    T = 0
-    class ClassB[S: Sequence[T], T]: ...  # Compiler error: T is not defined
+    # The following generates no compiler error, because the bound for ``S``
+    # is lazily evaluated. However, type checkers should generate an error.
+    class ClassB[S: Sequence[T], T]: ...
 
 
 A type parameter declared as part of a generic class is valid within the
@@ -475,7 +481,7 @@ Type parameter symbols defined in outer scopes cannot be bound with
 The lexical scope introduced by the new type parameter syntax is unlike
 traditional scopes introduced by a ``def`` or ``class`` statement. A type
 parameter scope acts more like a temporary "overlay" to the containing scope.
-It does not capture variables from outer scopes, and the only symbols contained
+The only new symbols contained
 within its symbol table are the type parameters defined using the new syntax.
 References to all other symbols are treated as though they were found within
 the containing scope. This allows base class lists (in class definitions) and
@@ -570,11 +576,14 @@ When the new type parameter syntax is used for a generic class, assignment
 expressions are not allowed within the argument list for the class definition.
 Likewise, with functions that use the new type parameter syntax, assignment
 expressions are not allowed within parameter or return type annotations, nor
-are they allowed within the expression that defines a type alias.
+are they allowed within the expression that defines a type alias, or within
+the bounds and constraints of a ``TypeVar``. Similarly, ``yield``, ``yield from``,
+and ``await`` expressions are disallowed in these contexts.
 
 This restriction is necessary because expressions evaluated within the
 new lexical scope should not introduce symbols within that scope other than
-the defined type parameters.
+the defined type parameters, and should not affect whether the enclosing function
+is a generator or coroutine.
 
 ::
 
@@ -590,15 +599,10 @@ the defined type parameters.
 Accessing Type Parameters at Runtime
 ------------------------------------
 
-A new read-only attribute called ``__type_variables__`` is available on class,
-function, and type alias objects. This attribute is a tuple of the active
-type variables that are visible within the scope of that class, function,
-or type alias. This attribute is needed for runtime evaluation of stringified
-(forward referenced) type annotations that include references to type
-parameters. Functions like ``typing.get_type_hints`` can use this attribute
-to populate the ``locals`` dictionary with values for type parameters that
-are in scope when calling ``eval`` to evaluate the stringified expression.
-The tuple contains ``TypeVar`` instances.
+A new read-only attribute called ``__type_params__`` is available on generic classes,
+functions, and type aliases. This attribute is a tuple of the
+type parameters that parameterize the class, function, or alias.
+The tuple contains ``TypeVar``, ``ParamSpec``, and ``TypeVarTuple`` instances.
 
 Type parameters declared using the new syntax will not appear within the
 dictionary returned by ``globals()`` or ``locals()``.
@@ -799,7 +803,7 @@ This PEP introduces a new AST node type called ``TypeAlias``.
 
 ::
 
-    TypeAlias(identifier name, typeparam* typeparams, expr value)
+    TypeAlias(expr name, typeparam* typeparams, expr value)
 
 It also adds an AST node type that represents a type parameter.
 
@@ -809,24 +813,80 @@ It also adds an AST node type that represents a type parameter.
         | ParamSpec(identifier name)
         | TypeVarTuple(identifier name)
 
+Bounds and constraints are represented identically in the AST. In the implementation,
+any expression that is a ``Tuple`` AST node is treated as a constraint, and any other
+expression is treated as a bound.
+
 It also modifies existing AST node types ``FunctionDef``, ``AsyncFunctionDef``
 and ``ClassDef`` to include an additional optional attribute called
-``typeparam*`` that includes a list of type parameters associated with the
+``typeparams`` that includes a list of type parameters associated with the
 function or class.
 
+.. _695-lazy-evaluation:
+
+Lazy Evaluation
+---------------
+
+This PEP introduces three new contexts where expressions may occur that represent
+static types: ``TypeVar`` bounds, ``TypeVar`` constraints, and the value of type
+aliases. These expressions may contain references to names
+that are not yet defined. For example, type aliases may be recursive, or even mutually
+recursive, and type variable bounds may refer back to the current class. If these
+expressions were evaluated eagerly, users would need to enclose such expressions in
+quotes to prevent runtime errors. :pep:`563` and :pep:`649` detail the problems with
+this situation for type annotations.
+
+To prevent a similar situation with the new syntax proposed in this PEP, we propose
+to use lazy evaluation for these expressions, similar to the approach in :pep:`649`.
+Specifically, each expression will be saved in a code object, and the code object
+is evaluated only when the corresponding attribute is accessed (``TypeVar.__bound__``,
+``TypeVar.__constraints__``, or ``TypeAlias.__value__``). After the value is
+successfully evaluated, the value is saved and later calls will return the same value
+without re-evaluating the code object.
+
+If :pep:`649` is implemented, additional evaluation mechanisms should be added to
+mirror the options that PEP provides for annotations. In the current version of the
+PEP, that might include adding an ``__evaluate_bound__`` method to ``TypeVar`` taking
+a ``format`` parameter with the same meaning as in PEP 649's ``__annotate__`` method
+(and a similar ``__evaluate_constraints__`` method, as well as an ``__evaluate_value__``
+method on ``TypeAliasType``).
+However, until PEP 649 is accepted and implemented, only the default evaluation format
+(PEP 649's "VALUE" format) will be supported.
+
+As a consequence of lazy evaluation, the value observed for an attribute may
+depend on the time the attribute is accessed.
+
+::
+
+   X = int
+
+   class Foo[T: X, U: X]:
+       t, u = T, U
+
+   print(Foo.t.__bound__)  # prints "int"
+   X = str
+   print(Foo.u.__bound__)  # prints "str"
+
+Similar examples affecting type annotations can be constructed using the
+semantics of PEP 563 or PEP 649.
+
+A naive implementation of lazy evaluation would handle class namespaces
+incorrectly, because functions within a class do not normally have access to
+the enclosing class namespace. The implementation will retain a reference to
+the class namespace so that class-scoped names are resolved correctly.
+
 
 Library Changes
 ---------------
 
 Several classes in the ``typing`` module that are currently implemented in
-Python must be reimplemented in C. This includes: ``TypeVar``,
-``TypeVarTuple``, ``ParamSpec``, ``Generic``, and ``Union``. The new class
-``TypeAliasType`` (described above) also must be implemented in C. The
+Python must be partially implemented in C. This includes ``TypeVar``,
+``TypeVarTuple``, ``ParamSpec``, and ``Generic``, and the new class
+``TypeAliasType`` (described above). The implementation may delegate to the
+Python version of ``typing.py`` for some behaviors that interact heavily with
+the rest of the module. The
 documented behaviors of these classes should not change.
 
-The ``typing.get_type_hints`` must be updated to use the new
-``__type_variables__`` attribute.
-
 
 Reference Implementation
 ========================
@@ -917,17 +977,6 @@ Furthermore, this approach is not compatible with techniques used for
 evaluating quoted (forward referenced) type annotations.
 
 
-Lambda Lifting
---------------
-When considering implementation options, we considered introducing a new
-scope and executing the ``class``, ``def``, or ``type`` statement within
-a lambda -- a technique that is sometimes referred to as "lambda lifting".
-We ultimately rejected this idea because it did not work well for statements
-within a class body (because class-scoped symbols cannot be accessed by
-inner scopes). It also introduced many odd behaviors for scopes that were
-further nested within the lambda.
-
-
 Appendix A: Survey of Type Parameter Syntax
 ===========================================