[BUG] distutils cannot be imported on Python 3.12 (without distutils) if pip was imported

[vstinner]

setuptools version

65.5.0

Python version

3.12 (dev, without distutils)

OS

Linux

Additional environment information

No response

Description

I just removed the distutils packages from Python 3.12: commit. setuptools works as expected, it provides its copy of distutils. Good.

But if pip is imported first, import setuptools or import distutils fails on import setuptools, since _distutils_hack has a special path when pip is installed which requires to get the stdlib distutils... which no longer exists.

Extract of _distutils_hack/__init__.py:

    def spec_for_pip(self):
        """
        Ensure stdlib distutils when running under pip.
        See pypa/pip#8761 for rationale.
        """

See pypa/pip#8761 for the rationale.

Well, this special case should be revisited for Python >= 3.12.0 alpha 2.

Expected behavior

Importing setuptools on Python 3.12 after pip was imported should just work.

How to Reproduce

$ python3.12
Python 3.12.0a1+ (heads/main:f09da28768, Nov  4 2022, 17:46:22) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pip
>>> import setuptools

Output

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/py312/lib/python3.12/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/py312/lib/python3.12/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/opt/py312/lib/python3.12/site-packages/_distutils_hack/__init__.py", line 77, in do_override
    ensure_local_distutils()
  File "/opt/py312/lib/python3.12/site-packages/_distutils_hack/__init__.py", line 60, in ensure_local_distutils
    importlib.import_module('distutils')
  File "/opt/py312/lib/python3.12/importlib/__init__.py", line 124, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'distutils'

[AMDmi3]

It doesn't seem to be related to pip, setuptools just do not support python 3.12. When running setuptools' setup.py install with python 3.12.0a3, as it is done when installing py-setuptools package on FreeBSD:

...
byte-compiling /work/usr/ports/devel/py-setuptools/work-py312/stage/usr/local/lib/python3.12/site-packages/_distutils_hack/override.py to override.cpython-312.pyc
writing byte-compilation script '/tmp/tmph5sr5lui.py'
/usr/local/bin/python3.12 /tmp/tmph5sr5lui.py
Traceback (most recent call last):
  File "/tmp/tmph5sr5lui.py", line 1, in <module>
    from distutils.util import byte_compile
ModuleNotFoundError: No module named 'distutils'
error: command '/usr/local/bin/python3.12' failed with exit code 1

[OTheDev]

I get this even without explicitly importing pip (i.e. just importing setuptools). Any further information on this?

[sathyz]

Hi,
I'm trying to take up this PR. Would it be fair to move distutils modules from cpython to setuptools to handle this?

[abravalheri]

Hi @sathyz, thank you very much for working on this. If you need to change anything of distutils please submit a PR to https://github.com/pypa/distutils.

But on a first glance I think we could tackle this issue by modifying https://github.com/pypa/setuptools/blob/main/_distutils_hack/__init__.py... What if the finder is always enabled for Python >= 3.12?

[sathyz]

Hi @abravalheri
I'm able to get this working by installing pypa/distutils and with the following change,

@@ -61,7 +61,8 @@ def ensure_local_distutils():

     # check that submodules load as expected
     core = importlib.import_module('distutils.core')
-    assert '_distutils' in core.__file__, core.__file__
+    if sys.version_info < (3, 12):
+        assert '_distutils' in core.__file__, core.__file__
     assert 'setuptools._distutils.log' not in sys.modules

I tried installing a package that was breaking earlier and is now working fine. One of the problem that I see is, pypa/distutils is not in pypi.org that I couldn't do pip install distutils, until this is available in PyPi repository we wouldn't be able to have it as dependency, am I correct? Any thoughts?