Annotation scopes containing nested scopes

[15r10nk]

Bug report

Bug description:

The following code causes a SystemError during compilation.

class name_2[*name_5, name_3: int]:
    (name_3 := name_4)

    class name_4[name_5: name_5]((name_4 for name_5 in name_0 if name_3), name_2 if name_3 else name_0):
        pass

output (Python 3.12.0rc2+):

SystemError: compiler_lookup_arg(name='name_3') with reftype=3 failed in <generic parameters of name_4>; freevars of code <genexpr>: ('name_3',)

removing the named-expression makes it worse and causes a crash

class name_2[*name_5, name_3: int]:

    class name_4[name_5: name_5]((name_4 for name_5 in name_0 if name_3), name_2 if name_3 else name_0):
        pass

output (Python 3.12.0rc2+):

Traceback (most recent call last):
  File "/home/frank/projects/pysource-codegen/bug2.py", line 2, in <module>
    class name_2[*name_5, name_3: int]:
  File "/home/frank/projects/pysource-codegen/bug2.py", line 2, in <generic parameters of name_2>
    class name_2[*name_5, name_3: int]:
  File "/home/frank/projects/pysource-codegen/bug2.py", line 4, in name_2
    class name_4[name_5: name_5]((name_4 for name_5 in name_0 if name_3), name_2 if name_3 else name_0):
  File "/home/frank/projects/pysource-codegen/bug2.py", line 4, in <generic parameters of name_4>
    class name_4[name_5: name_5]((name_4 for name_5 in name_0 if name_3), name_2 if name_3 else name_0):
                                                       ^^^^^^
NameError: name 'name_0' is not defined
Modules/gcmodule.c:113: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f4c77059a30
object refcount : 1
object type     : 0x560ea4b6cfe0
object type name: dict
object repr     : {'__module__': '__main__', '__qualname__': 'name_2', '__type_params__': (name_5, name_3)}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x0000560ea4cc9ba0)

Current thread 0x00007f4c7756c280 (most recent call first):
  Garbage-collecting
  <no Python frame>

side note:
The reason for the strange looking language constructs and variable names is that I generated this code. I found it while working on my pysource-codegen library.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-109118: Fix runtime crash when NameError happens in PEP 695 function #109123
• [3.12] gh-109118: Fix runtime crash when NameError happens in PEP 695 function (GH-109123) #109173
• gh-109118: Disallow nested scopes within PEP 695 scopes within classes #109196
• [3.12] gh-109118: Disallow nested scopes within PEP 695 scopes within classes (GH-109196) #109297
• gh-109118: Allow lambdas in annotation scopes in classes #118019
• gh-109118: Make comprehensions work within annotation scopes, but without inlining #118160

[JelleZijlstra]

Thanks, this is very helpful!

Minimized version:

class name_2[name_3]:
    name_3 = name_4
    class name_4[name_5]((name_3 for name_5 in name_0), name_3):
        pass

The runtime crash when removing the name_3 = name_4 line is a bug in the interpreter that I will send a PR for. The compiler crash (SystemError) is a different bug that I haven't yet tracked down.

[JelleZijlstra]

The first thing to figure out with the compiler crash is how the code should behave, as class scopes are weird. We have a genexp within the type parameters of a generic class nested inside another generic class.

Here's the test I wrote, demonstrating how this works if the inner class is not generic:

+    def test_gen_exp_in_nested_class(self):
+        class C[T]:
+            T = "class"
+            class Inner(make_base(T for _ in (1,)), make_base(T)):
+                pass
+        T, = C.__type_params__
+        base1, base2 = C.Inner.__bases__
+        self.assertEqual(list(base1.__arg__), [T])
+        self.assertEqual(base2.__arg__, "class")
+
+    def test_gen_exp_in_nested_generic_class(self):
+        class C[T]:
+            T = "class"
+            class Inner[U](make_base(T for _ in (1,)), make_base(T)):
+                pass
+        T, = C.__type_params__
+        base1, base2, _ = C.Inner.__bases__
+        self.assertEqual(list(base1.__arg__), [T])
+        self.assertEqual(base2.__arg__, "class")
+
+
+def make_base(arg):
+    class Base:
+        __arg__ = arg
+    return Base
+

In short, the genexp should not see the definition in the outer class body, only the one in the generic parameters of the outer class. However, the other bases should see the definition in the class body. This is how it works now if the inner class is not generic, and this is correct.

If the inner class is generic, the behavior should be analogous, but evidently there's something wrong in the bookkeeping.