AssertionError no _distutils when running get-pip with Setuptools and USE_DISTUTILS=local

[jaraco]

Somehow related: Since setuptools v60 at least on Debian running get-pip.py over an existing pip + setuptools fails, when the Debian python3-distutils package is installed, e.g. as dependency of the Python development headers package:

/usr/local/lib/python3.9/dist-packages/_distutils_hack/__init__.py:24: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
  warnings.warn(
/usr/local/lib/python3.9/dist-packages/_distutils_hack/__init__.py:36: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/tmp/DietPi-Software/./get-pip.py", line 27081, in <module>
    main()
  File "/tmp/DietPi-Software/./get-pip.py", line 139, in main
    bootstrap(tmpdir=tmpdir)
  File "/tmp/DietPi-Software/./get-pip.py", line 120, in bootstrap
    args = determine_pip_install_arguments()
  File "/tmp/DietPi-Software/./get-pip.py", line 65, in determine_pip_install_arguments
    import setuptools  # noqa
  File "/usr/local/lib/python3.9/dist-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/usr/local/lib/python3.9/dist-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/usr/local/lib/python3.9/dist-packages/_distutils_hack/__init__.py", line 73, in do_override
    ensure_local_distutils()
  File "/usr/local/lib/python3.9/dist-packages/_distutils_hack/__init__.py", line 61, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /usr/lib/python3.9/distutils/core.py

This _distutils_hack comes from get-pip.py. Looks like removing any use and detangling distutils from pip in favour of setuptools would solve both issues.

Originally posted by @MichaIng in pypa/pip#10742 (comment)

[jaraco]

Setuptools 60 intentionally introduced its local copy of distutils as default. As an escape hatch, you should be able to set the environment variable SETUPTOOLS_USE_DISTUTILS=stdlib while running get-pip to bypass that behavior.

I know the distutils hack has an exclusion for when running under pip. That was, however, to account for pip's use of distutils. In this case, get-pip is specifically importing setuptools. Importing setuptools causes the more aggressive form of requiring distutils, where if the local copy of distutils isn't found, it fails with the reported error.

So the question is - what is it about this environment that triggers this behavior? In particular, how is it that this code doesn't ensure that any existing distutils is unloaded, then loaded from Setuptools?

Can someone create a set of docker steps that will replicate the issue?

[jaraco]

I was able to replicate the error with this Dockerfile:

FROM ubuntu:focal
RUN apt update
RUN apt upgrade -y
RUN apt install -y python3-pip python3-distutils wget
RUN wget https://bootstrap.pypa.io/get-pip.py
RUN python3 -m pip install -U setuptools
CMD python3 get-pip.py

[jaraco]

I did a bit of debugging and it seems that the spec_for_pip had gotten invoked, causing the DistutilsMetaFinder to disable the distutils hook:

draft $ docker run -it @$(docker build -q .) python3 -m pdb get-pip.py
> /get-pip.py(23)<module>()
-> import sys
(Pdb) b _distutils_hack/__init__.py:57
Breakpoint 1 at /usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py:57
(Pdb) c
/usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py:24: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
  warnings.warn(
/usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py:36: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
> /usr/local/lib/python3.8/dist-packages/_distutils_hack/__init__.py(57)ensure_local_distutils()
-> importlib.import_module('distutils')
(Pdb) 'distutils' in sys.modules
False
(Pdb) sys.meta_path[0]
<_distutils_hack.DistutilsMetaFinder object at 0x7fad398a9370>
(Pdb) mf = sys.meta_path[0]
(Pdb) mf.pip_imported_during_build()
False
(Pdb) mf.spec_for_distutils()
(Pdb) mf.spec_for_distutils
<function DistutilsMetaFinder.spec_for_pip.<locals>.<lambda> at 0x7fad39199550>

The hack makes the assumption that if pip is imported and it wasn't imported as part of a setup.py invocation, then it's a pip invocation and needs to avoid making the local version of distutils available.

But setuptools makes the assertion that the hack must be available (if SETUPTOOLS_USE_DISTUTILS=local).

The question is - what is it about get-pip that (a) imports pip and then (b) imports setuptools?

[jaraco]

I see it's these two lines where import pip and then import setuptools happens.

I no longer think this issue is specific to Debian or distutils.

[jaraco]

Indeed, the same issue occurs on my mac. All it takes is an invocation of get-pip on an existing install of Setuptools>=60 (or Setuptools>=50 with SETUPTOOLS_USE_DISTUTILS='local').

draft $ python -m venv venv
draft $ venv/bin/pip install -U setuptools
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (58.1.0)
Collecting setuptools
  Using cached setuptools-60.2.0-py3-none-any.whl (953 kB)
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 58.1.0
    Uninstalling setuptools-58.1.0:
      Successfully uninstalled setuptools-58.1.0
Successfully installed setuptools-60.2.0
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/Users/jaraco/draft/venv/bin/python -m pip install --upgrade pip' command.
draft $ venv/bin/python get-pip.py
/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py:24: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
  warnings.warn(
/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py:36: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/Users/jaraco/draft/get-pip.py", line 27081, in <module>
    main()
  File "/Users/jaraco/draft/get-pip.py", line 139, in main
    bootstrap(tmpdir=tmpdir)
  File "/Users/jaraco/draft/get-pip.py", line 120, in bootstrap
    args = determine_pip_install_arguments()
  File "/Users/jaraco/draft/get-pip.py", line 65, in determine_pip_install_arguments
    import setuptools  # noqa
  File "/Users/jaraco/draft/venv/lib/python3.10/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 73, in do_override
    ensure_local_distutils()
  File "/Users/jaraco/draft/venv/lib/python3.10/site-packages/_distutils_hack/__init__.py", line 61, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/distutils/core.py
draft $

[jaraco]

In the aforementioned commit, I've devised another exception that bypasses the issue (and solves the problem at least for me in this environment). I'm a little concerned that the approach is unsound, because it's essentially bypassing the other exception that was created for pip due to pypa/pip#8761.

I suspect a better approach would be for get-pip not to "import setuptools" to determine if it's present, but merely to "find" it or check metadata for it.

[jaraco]

On further consideration, the conflict with the other exception may not be a real concern, because that exception was to support the pip upgrading setuptools case. Since get-pip is specifically not upgrading Setuptools, the concern may be moot. So my temptation is to apply the drafted fix while get-pip explores the reported issue.

[SeanMooney]

it looks like the release job failed https://github.com/pypa/setuptools/actions/runs/1650359780

so while the tag is available on github its not published to pypi currently

[jaraco]

Thanks for letting me know. I've identified and fixed the issue and a new release should be going out.

[jaraco]

In https://github.com/pypa/pip/issues/10775#issuecomment-1009375819, I summarize why I think the fix for this issue was inadequate:

...get-pip's import of setuptools happens before get-pip's import of pip, but the hook for the hack is on the import of pip.

I believe instead, the distutils hack should (a) detect get-pip, (b) detect "import setuptools", and in that case, satisfy "import setuptools" without actually importing anything and bypassing the distutils replacement.

[jaraco]

In the 2993-redux branch, I've re-visited the approach, and I believe this branch also addresses the issue reported in pypa/pip#3022:

draft $ cat Dockerfile
FROM ubuntu:focal
RUN apt update
RUN apt upgrade
RUN apt install -y python3-dev curl git
RUN curl -O https://bootstrap.pypa.io/get-pip.py
RUN python3 get-pip.py
RUN python3 -m pip install git+https://github.com/pypa/setuptools@bugfix/2993-redux
RUN python3 get-pip.py
CMD python3 -m pip -V
draft $ docker run -it @$(docker build -q .)
pip 21.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)

[grv87]

@jaraco, have you finished your work on this issue? Please, take a look at #3439