Extension module is already cached error with mypyc on Python 3.13

[hauntsaninja]

@cdce8p already bisected this upstream to python/cpython#118532

To Reproduce

# Set up using something like:
# python3.13 -m pip install setuptools
# python3.13 -m pip install -e .

printf '
from non_native import pure_python_const
class CanRemoveThisClass:
    pass
' > native.py

printf '
import native  # circular import
pure_python_const = 10
' > non_native.py

rm -rf build
rm *.so
python3.13 -m mypyc native.py
python3.13 -c 'import native'

Using a debug build of Python, you get:

Assertion failed: (value == NULL), function _extensions_cache_set, file import.c, line 1325.
zsh: abort      python3.13 -c 'import native

Using a non-debug build, you get:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import native
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 921, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 819, in module_from_spec
  File "<frozen importlib._bootstrap>", line 782, in _init_module_attrs
SystemError: extension module 'native' is already cached

mypyc transpiles Python code to an extension module that uses the C API. In the above, we compile native.py into an extension module and form an import cycle with a normal pure Python module

The C code mypyc generates is visible in the build directory

[hauntsaninja]

cc @ericsnowcurrently in case you happen to have time and kindness to take a look and offer any advice :-)

[hauntsaninja]

Here's a repro that doesn't involve mypyc. I'll read more about multi-phase init tomorrow, maybe imports in single init is not allowed or something. mypyc does globals, which I think multi-phase init doesn't like, would be a big change

printf '
#define PY_SSIZE_T_CLEAN
#include <Python.h>

static struct PyModuleDef nativemodule = {
    PyModuleDef_HEAD_INIT,
    .m_name = "native",
};

PyObject* module = NULL;

PyMODINIT_FUNC PyInit_native(void) {
    if (module) {
        Py_INCREF(module);
        return module;
    }
    module = PyModule_Create(&nativemodule);
    assert(module);

    Py_XDECREF(PyImport_ImportModule("non_native"));
    PySys_WriteStdout("hello from native\\n");

    return module;
}
' > native.c

printf 'import native  # circular import' > non_native.py

printf '
from setuptools import setup, Extension

setup(name="native", ext_modules=[Extension("native", sources=["native.c"])])
' > setup.py

python setup.py build_ext --inplace

python -c 'import native'

[JukkaL]

My understanding is that multi-phase init would solve the issue. I also encountered this when running tests on the release candidates, but I haven't had time to look into it in detail yet (and probably won't have time in the next week or two at least, unfortunately). Several mypyc tests fail because of this issue.

We've been relying on circular imports working without multi-phase init for years now, even if they haven't been officially supported by the Python C API.

Just to be clear, this also blocks compiling mypy on 3.13 release candidates, since mypy contains (many) import cycles.

[hauntsaninja]

I opened python/cpython#123880 upstream

[hauntsaninja]

This was fixed upstream, we should be all set for 3.13.0

[hauntsaninja]

Would be nice to make a release, I opened #17815