Isolated build environment is corrupted on Homebrew Python 3.10 without active virtual environment

[RazerM]

Description

I experienced the following error in a CI environment

  Getting requirements to build wheel: started
  Running command Getting requirements to build wheel
  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
      main()
    File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/usr/local/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
      return self._get_build_requires(config_settings, requirements=['wheel'])
    File "/usr/local/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
      self.run_setup()
    File "/usr/local/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
      exec(code, locals())
    File "<string>", line 2, in <module>
  ModuleNotFoundError: No module named 'skbuild'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

Since I know from the pip install -v log above this truncated output that skbuild was definitely picked up from pyproject.toml and installed, it's clear that the assertion error prevents the skbuild package from being found.

Enabling PYTHONVERBOSE=1:

  Adding directory: '/usr/local/lib/python3.10/site-packages'
  Processing .pth file: '/usr/local/lib/python3.10/site-packages/distutils-precedence.pth'
  Traceback (most recent call last):
    File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site.py", line 549, in execsitecustomize
      import sitecustomize
    File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
    File "<frozen importlib._bootstrap_external>", line 883, in exec_module
    File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    File "/private/var/folders/cb/yqyhmsb136xcxm64h9vrswcw0000gn/T/pip-build-env-f7sm6np4/site/sitecustomize.py", line 22, in <module>
      assert not path in sys.path
  AssertionError

I see that it comes from

pip/src/pip/_internal/build_env.py

Line 130 in ff207cf

assert not path in sys.path

This is not my project so I don't know if this is something wrong with the package itself, the CI environment where this happens, or something else.

Changing the CI job to use a venv gets rid of the issue (which is of course a good idea anyway), but it would be good to understand why this assertion is in place and what I should do to fix it.

I think the message should at least include path and sys.path, which I could open a PR for, but I would also like some context from someone familiar with the implementation about what one should do in this case.

Expected behavior

The AssertionError would include actionable information

pip version

22.3

Python version

3.10.8

OS

macOS 10.14.6

How to Reproduce

I don't yet know why this happens, and since this issue about improving the message I think it's ok.

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[RazerM]

FWIW /usr/local/lib/python3.10/site-packages is the path which is also the last item in sys.path

[uranusjr]

How did you install the Python 3.10.8? The assert checks that a path is not in sys.path (because the environment is supposed to be isolated) but something in the installation is breaking the promise. There’s likely some configuration issues.

[RazerM]

brew

[dnicolodi]

I'm hitting the same issue. This happens only with Python 3.9 and 3.10 installed via Homebrew. It works on earlier Python versions. I'm debugging the issue but the first actionable thing could be to make the assertion error more verbose and detailed: understanding that something is going wrong with how pip setups the build venv took me many hours of head scratching.

Mostly for reference to other maybe bumping into the same problem, the symptoms look like this:

$ python3.10 -m pip install -vv .[test]
Using pip 22.3.1 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)
Non-user install because site-packages writeable
Created temporary directory: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-tracker-lvays4wb
Initialized build tracking at /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-tracker-lvays4wb
Created build tracker: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-tracker-lvays4wb
Entered build tracker: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-tracker-lvays4wb
Created temporary directory: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-install-e_f9sj5i
Created temporary directory: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-ephem-wheel-cache-m8k1pvpm
Processing /Users/runner/work/meson-python/meson-python
  Added file:///Users/runner/work/meson-python/meson-python to build tracker '/private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-tracker-lvays4wb'
  Running command pip subprocess to install build dependencies
  Created temporary directory: /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-zpp5cr1c
  Installing build dependencies: started
  Using pip 22.3.1 from /usr/local/lib/python3.10/site-packages/pip (python 3.10)
  Ignoring typing-extensions: markers 'python_version < "3.8"' don't match your environment
  Collecting meson>=0.63.3
    Downloading meson-0.64.0-py3-none-any.whl (895 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 895.2/895.2 kB 21.6 MB/s eta 0:00:00
  Collecting ninja
    Downloading ninja-1.11.1-py2.py3-none-macosx_10_9_universal2.macosx_10_9_x86_64.macosx_11_0_arm64.macosx_11_0_universal2.whl (270 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 270.7/270.7 kB 22.7 MB/s eta 0:00:00
  Collecting pyproject-metadata>=0.5.0
    Downloading pyproject_metadata-0.6.1-py3-none-any.whl (7.4 kB)
  Collecting tomli>=1.0.0
    Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
  Collecting packaging>=19.0
    Downloading packaging-21.3-py3-none-any.whl (40 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 kB 5.9 MB/s eta 0:00:00
  Collecting pyparsing!=3.0.5,>=2.0.2
    Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 13.7 MB/s eta 0:00:00
  Installing collected packages: ninja, tomli, pyparsing, meson, packaging, pyproject-metadata
    Creating /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-zpp5cr1c/overlay/bin
    changing mode of /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-zpp5cr1c/overlay/bin/ninja to 755
    changing mode of /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-build-env-zpp5cr1c/overlay/bin/meson to 755
  Successfully installed meson-0.64.0 ninja-1.11.1 packaging-21.3 pyparsing-3.0.9 pyproject-metadata-0.6.1 tomli-2.0.1
  Running command Getting requirements to build wheel
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:
  Getting requirements to build wheel: finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  [ImportError because of missing build dependency folllows]

where the interesting part is:

  Error in sitecustomize; set PYTHONVERBOSE for traceback:
  AssertionError:

Setting PYTHONVERBOSE=1 produces the expected torrent of debug messages, but it does not add much to the AssertionError.

[uranusjr]

You might want to reach out to maintainers of Homebrew’s Python formula to get a better idea about the issue. It only happens to their Python installation so it’s likely something wrong with how they build or configurate it.

[dnicolodi]

The sitecustomize.py generated by pip for its isolated environment looks like this with Homebrew's Python 3.10 when not using a virtual environment:

import os, site, sys

# First, drop system-sites related paths.
original_sys_path = sys.path[:]
known_paths = set()
for path in {'/usr/local/lib/python3.10/site-packages'}:
    site.addsitedir(path, known_paths=known_paths)
system_paths = set(
    os.path.normcase(path)
    for path in sys.path[len(original_sys_path):]
)
original_sys_path = [
    path for path in original_sys_path
    if os.path.normcase(path) not in system_paths
]
sys.path = original_sys_path

# Second, add lib directories.
# ensuring .pth file are processed.
for path in ['/usr/local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/site-packages']:
    assert not path in sys.path
    site.addsitedir(path)

The assertion in there obviously fails.

[uranusjr]

Quoting myself above:

The assert checks that a path is not in sys.path (because the environment is supposed to be isolated) but something in the installation is breaking the promise.

The assertion is doing what it is supposed to do, those entries being in there is the problem.

[dnicolodi]

If I read it correctly, the first part of the generated sitecustomize.py removes the system paths from sys.path. The second path then checks that its job correctly. Apparently something fails. I am not yet saying that this is a bug in pip. Since getting to the generated sitecustomize.py is not trivial I wanted to record it here to aid other in understanding what is going on.

[dnicolodi]

Oh, and by the way, adding a thing twice to a list while checking that it is not already there, rarely succeeds.

The last three lines of that sitecustomize.py do the equivalent of

foo = []
for path in ['/usr/local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/site-packages']:
    assert not path in foo
    foo.append(path)

Deduplicating the list solves the problem.

[uranusjr]

The logic works everywhere except Homebrew’s Python installation, some I’m inclined to say the responsibility to correct this likely lies in Homebrew. pip can modify the heuristic to better support whatever Homebrew is doing, if we can have insight first on what Homebrew is doing with its Python installation (that nobody else is), why it’s doing it, whether it’s a correct thing (if it’s not, maybe fix that instead), and how pip can help without compromising what we (both sides) are trying to achieve.