SystemError for pep695 type parameter with the same name as the inner class

[15r10nk]

Bug report

Bug description:

The following script shows the SystemError:

class name_1[name_4]:

    class name_4[name_2](name_4):
        name_4

output (Python 3.12.0rc2+):

SystemError: compiler_lookup_arg(name='name_4') with reftype=3 failed in <generic parameters of name_4>; freevars of code name_4: ('.type_params', 'name_4')

minimizing it further leads some different crashes:
The propably most interresting one is this, which does not look special at all.

class name_1[name_3]:
    class name_4[name_2](name_5):
        pass

output (Python 3.12.0rc2+):

Traceback (most recent call last):
  File "/home/frank/projects/pysource-codegen/bug3.py", line 1, in <module>
    class name_1[name_3]:
  File "/home/frank/projects/pysource-codegen/bug3.py", line 1, in <generic parameters of name_1>
    class name_1[name_3]:
  File "/home/frank/projects/pysource-codegen/bug3.py", line 2, in name_1
    class name_4[name_2](name_5):
  File "/home/frank/projects/pysource-codegen/bug3.py", line 2, in <generic parameters of name_4>
    class name_4[name_2](name_5):
                         ^^^^^^
NameError: name 'name_5' is not defined. Did you mean: 'name_2'?
Modules/gcmodule.c:113: gc_decref: Assertion "gc_get_refs(g) > 0" failed: refcount is too small
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7f2c57a12930
object refcount : 1
object type     : 0x557ad9ef1fe0
object type name: dict
object repr     : {'__module__': '__main__', '__qualname__': 'name_1', '__type_params__': (name_3,)}

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x0000557ada04eba0)

Current thread 0x00007f2c57f21280 (most recent call first):
  Garbage-collecting
  <no Python frame>

@JelleZijlstra I think this is another one for you. Feel free to create a second issue if you think that they are unrelated.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-109219: propagate free vars through type param scopes #109377
• gh-109219: propagate used free vars through type param scopes [v2] #109389
• [3.12] gh-109219: propagate free vars through type param scopes (GH-109377) #109410

[JelleZijlstra]

Thanks. The second one should be fixed by #109123. I'll have to look into what is causing the SystemError on your first example.

[JelleZijlstra]

I spent some time last night trying to get this to work without success. I'll try more later today, but here are my thoughts so far on this example:

class name_1[name_4]:

    class name_4[name_2](name_4):
        name_4

The name_4 in the body of the inner class should refer to the type param of the outer class, because the bodies of inner classes can't see names defined in outer classes, but they can see names defined in outer function-like scopes.

The name_4 in the bases of the inner class should be a NameError. If a name is bound anywhere in a class body, then we look only at the class scope and the global scope, not any intervening scopes. I discuss this in https://jellezijlstra.github.io/pep695 in the section that starts "A few more edge cases came up later." In this case, the name is bound in the class body, but there is no binding yet when the bases of the inner class are evaluated, so next we look at the globals, where there is also no name_4, so we raise a NameError.

cc @carljm, would appreciate if you could see whether you agree with this diagnosis, and if you can see a simple way to implement the desired behavior in the compiler.

[carljm]

Your analysis looks correct to me! I'll take a look at implementation...

[JelleZijlstra]

I suppose the problem is that inside the annotation scope for the inner class, name_4 needs to be present with two different meanings: a GLOBAL_IMPLICIT for the use in the annotation scope itself, and a free variable, so that it can be put in the cell for the class scope. I'm not sure the compiler has a way to do that in general, though there are some cases involving class scopes that do work like this.

[carljm]

A similar example that still reproduces the bug, but simplifies and perhaps clarifies a few things (replace the outer annotation scope with a regular function scope, and use a normal re-definition of the typevar in the class scope instead of circularly naming the inner class the same as the typevar; this example should run successfully rather than raise a NameError):

def f():
    T = str
    class C:
        T = int
        class D[U](T):
            T
    assert int in C.D.__bases__

[carljm]

In my latter example, the T = str in the outer f() function (equivalent to the presence of name_4 in outer class type params in the original example) is key to reproducing the bug. It is this definition of T (not the redefinition in the class) which causes the innermost T (in the body of the inner class) to have scope FREE rather than GLOBAL_IMPLICIT (as it does if you remove the line T = str from my example.)

I haven't quite sorted out yet what should be happening here; I'll come back to it tomorrow.

[carljm]

I'm using this test case to debug, and focusing on the discrepancy between the behavior with and without the [U] type param on the inner class, since the presence of that type param should not change the behavior of the name T. This test case fails with the SystemError as written, and passes if the [U] is removed:

    def test_complex_nested_classes(self):
        code = """
            def f():
                T = str
                class C:
                    T = int
                    class D[U](T):
                        x = T
                return C
        """
        C = run_code(code)["f"]()
        self.assertIn(int, C.D.__bases__)
        self.assertIs(C.D.x, str)