Isolated build fails with generic StopIteration when pip is not provisioned in venv

[d-glenz]

Thank you for providing feedback on Python packaging!

To help us help you, please fill out as much of the following as you can. If a question is not relevant, feel free to skip it.

1. What is your operating system and version?

CentOS 7.6.1810

2. What is your Python version?

Python 3.6.5

3. What version of pip do you have?

pip-21.0.1

4. If following an online tutorial or guide, please provide a link to the page or section giving you trouble:

https://packaging.python.org/tutorials/packaging-projects/

5. Could you describe your issue in as much detail as possible?

Created a sample-project according to the tutorial.
Set up a virtual environment, upgraded pip to the latest and installed build-0.3.1.post1
Ran python -m build --sdist --wheel to create both a wheel and an sdist, and then encountered this:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/dominikglenz/.virtualenvs/tmp-5bea59413b8ffc04/lib/python3.6/site-packages/build/__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/home/dominikglenz/.virtualenvs/tmp-5bea59413b8ffc04/lib/python3.6/site-packages/build/__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "/home/dominikglenz/.virtualenvs/tmp-5bea59413b8ffc04/lib/python3.6/site-packages/build/__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "/home/dominikglenz/.virtualenvs/tmp-5bea59413b8ffc04/lib/python3.6/site-packages/build/__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "/home/dominikglenz/.virtualenvs/tmp-5bea59413b8ffc04/lib/python3.6/site-packages/build/env.py", line 91, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/home/dominikglenz/.virtualenvs/tmp-5bea59413b8ffc04/lib/python3.6/site-packages/build/env.py", line 203, in _create_isolated_env_venv
    pip_distribution = next(iter(metadata.distributions(name='pip', path=[purelib])))
StopIteration

It looks like build did not add pip to the /tmp/build-env-*/ site-packages.

[layday]

Transferred to the build tracker for greater visibility.

[layday]

This could have one of a number of causes:

• the purelib path is incorrect
• importlib_metadata is not able to find any meta path finders which implement find_distributions
• pip was genuinely not installed (least likely)

[d-glenz]

Looked into this a bit more and found that after venv.EnvBuilder(with_pip=True).create(path), so after the isolated build environment is created, path contains:

├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── python
│   └── python3
├── include
├── lib
│   └── python3.6
│       └── site-packages
├── lib64
│   └── python3.6
│       └── site-packages
└── pyvenv.cfg

Notice how bin does not contain either python3.6 or pip, and site-packages in both lib/python3.6 and lib64/python3.6 are empty.

[FFY00]

Looks like CentOS patches the venv module then 😕

You can install build with the virtualenv extra, build[virtualenv], which should fix this. We should make virrtualenv a hard requirement on CentOS though.

[uranusjr]

I don’t think there’s a good way to add hard dependencies based on Linux distribution variants.

I think this is less about patching venv, but that on most Linux distributions, pip is a separate package from the base Python installation, and ensurepip does not work without the user installing pip from the system package first. There has been long-running tension between Linux distribution maintainers and Python packaging tools; distributions really don’t want to put pip into the base Python package (for valid reasons), but most Python packaging tools want to be able to boostrap pip (also for valid reasons).

I think the best build can do here is to detect pip is correctly populated by venv, and error out (hinting the virtualenv extra) if venv is broken.

[layday]

Should ensurepip not error then if it can't install pip if pip is not installed system-wide? I was under the impression that it did on Debian.

[uranusjr]

That depends on how CentOS packages and patches things. IIRC Debian just outright removed ensurepip without noticing it’s used by venv, which (fortunately in a sense?) made the error very visible.

[freddrake]

I see this happening when running build under tox. The underlying Python is a clean installation from sources, not a distro-provided bastardization.

Requiring build[virtualenv] instead of build makes it work, but I only found out about the virtualenv extra from reading this issue.

[adamjstewart]

I just hit this issue as well. I'm a developer for the Spack package manager. Spack installs each package into a separate installation prefix, so we rely on things like PYTHONPATH for packages to find their dependencies. Even if I install build[virtualenv], that doesn't seem to solve the problem.

P.S. If pip is a run-time dependency, shouldn't it be listed in setup.cfg under install_requires?

P.P.S. This wouldn't be an issue if build vendored its dependencies like pip/setuptools/flit/poetry/etc.

[FFY00]

pip is not a dependency, we use the standard library venv module to build an environment with pip installed, which it does using the standard library ensurepip module. The issue arises from Python distributors patching the Python installation and straight out breaking ensurepip, they do this generally because they do not like Python vendoring a pip wheel for ensurepip to use, so they make it rely on the external pip.

Please fill a bug report with your distributor if their package for pypa/build does not depend on whatever is needed to make ensurepip usable on that environment. If you are using pypa/build from a virtual environment, you can still fill a bug report with your distributor to ask them to stop this behavior, but they will probably just gonna tell you that the user should be aware that ensurepip only works with certain packages installed (they usually make them optional dependencies) 🤷

[adamjstewart]

Spack builds Python with the --without-ensurepip flag, is that the cause of the issue? Even with this flag enabled, I'm still able to import ensurepip. My understanding is that this flag causes Python not to build pip, although ensurepip is still available if you want to build pip later.

Unfortunately I don't have a distributor to open an issue with, as I'm the distributor. I'm maintaining Spack and Spack's Python packages, so if it's possible to fix this on our side, I'm probably the one to do it. Can you tell me exactly how ensurepip needs to work for this library?

[FFY00]

I believe --without-ensurepip is equivalent to --with-ensurepip=no which IIRC prevents ensurepip from being run on install.

If you do not patch the Python installation in any way, this should work OOTB. CPython bundles the pip and setuptools wheels required for ensurepip, https://github.com/python/cpython/tree/main/Lib/ensurepip/_bundled. I'd say that the first step would be checking if they are there.

[adamjstewart]

Yes, they are there:

$ ls lib/python3.8/ensurepip/_bundled/
pip-21.1.1-py3-none-any.whl        setuptools-56.0.0-py3-none-any.whl

[FFY00]

Then try to use venv to setup a virtual environment with pip (it does by default) and check if that works. If it does, then you are likely running into other problem, and we will have to debug that.

[adamjstewart]

Can you give me some venv commands that you expect to work? Never used it before, as Spack has its own virtual environments.

[FFY00]

$ python -m venv test
$ test/bin/pip --version

You can delete test after.

[adamjstewart]

Looks like the venv doesn't have pip installed.

[adamjstewart]

FWIW, removing --without-ensurepip in my Python build solved the issue for me. So it sounds like this library requires the ensurepip package to be installed and working.

[adamjstewart]

Dug a bit deeper into this thanks to the help of @wdconinc. When you build Python, you have 3 options:

• --with-ensurepip=upgrade (default)
• --with-ensurepip=install
• --without-ensurepip

It looks like --with-ensurepip=upgrade is required for build to work properly. Both --with-ensurepip=install and --without-ensurepip cause the issue reported above. Unfortunately, --with-ensurepip=upgrade will try to replace the pip installed on the system, which users may not have admin privileges to upgrade.

I'm still unsure why --with-ensurepip=install doesn't work, and I'm also unsure why --without-ensurepip doesn't work. I tried the above suggestions with build[virtualenv] but that didn't help either. Any additional insight into this issue would be much appreciated.

[layday]

Are you manipulating the PYTHONPATH in Spack?

[layday]

I assume what's happening is pip does not install itself in the venv because it believes it's already installed. When you pass --with-ensurepip=upgrade, the version from ensurepip is higher than the pip from PYTHONPATH, and pip installs itself in the venv. Presumably the installation is performed with the -U option.

[adamjstewart]

Are you manipulating the PYTHONPATH in Spack?

Yes, we are manipulating PYTHONPATH in Spack. Spack installs all Python libraries to separate installation prefixes, so PYTHONPATH is required to locate these. This lets us do cool things like install multiple versions of numpy with different BLAS/LAPACK libraries and let the user choose which to use.

I assume what's happening is pip does not install itself in the venv because it believes it's already installed

Hmm, so I don't have any pip installed on my system. Both --with-ensurepip=upgrade and --with-ensurepip=install result in pip being installed alongside Python and available in the default site-packages directory (not in PYTHONPATH). It would make sense that "upgrade" would cause a newer version of pip to be installed in the venv. But wouldn't that mean that "upgrade" doesn't work either if the installed version of pip is already the newest version?

[layday]

So in the environment that you execute build there's no pip? Are you sure?

[layday]

I don't have Spack, but here's a simple repro using Nix, which manipulates the Python path in the same way:

$ nix-shell -p python39.pkgs.pip --command fish
$ python -m pip install build --target foo
Collecting build
  Using cached build-0.7.0-py3-none-any.whl (16 kB)
Collecting pep517>=0.9.1
  Using cached pep517-0.12.0-py2.py3-none-any.whl (19 kB)
Collecting tomli>=1.0.0
  Using cached tomli-2.0.0-py3-none-any.whl (12 kB)
Collecting packaging>=19.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.7-py3-none-any.whl (98 kB)
Installing collected packages: tomli, pyparsing, pep517, packaging, build
Successfully installed build-0.7.0 packaging-21.3 pep517-0.12.0 pyparsing-3.0.7 tomli-2.0.0
$ touch setup.py
$ PYTHONPATH="foo:$PYTHONPATH" python -m build
* Creating venv isolated environment...

Traceback (most recent call last):
  File "/[...]/foo/build/__main__.py", line 372, in main
    built = build_call(
  File "/[...]/foo/build/__main__.py", line 229, in build_package_via_sdist
    sdist = _build(isolation, builder, outdir, 'sdist', config_settings, skip_dependency_check)
  File "/[...]/foo/build/__main__.py", line 140, in _build
    return _build_in_isolated_env(builder, outdir, distribution, config_settings)
  File "/[...]/foo/build/__main__.py", line 104, in _build_in_isolated_env
    with _IsolatedEnvBuilder() as env:
  File "/[...]/foo/build/env.py", line 104, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/[...]/foo/build/env.py", line 261, in _create_isolated_env_venv
    pip_distribution = next(iter(metadata.distributions(name='pip', path=[purelib])))  # type: ignore[no-untyped-call]
StopIteration

ERROR
$ echo $PYTHONPATH
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/site-packages /nix/store/zs9mh2h826rgh2w39bv8gb1xc2nr77lk-python3.9-pip-21.3.1/lib/python3.9/site-packages

[layday]

Ultimately, this needs to be fixed in Python. If ensurepip should ignore entries on the PYTHONPATH, the environ needs to be manipulated in venv. Invoking Python with -I is not sufficient because ensurepip invokes pip in a subprocess of its own. Observe:

$ python -Ic 'import subprocess
  subprocess.run(["python", "-c", "import os; print(os.environ[\'PYTHONPATH\'])"])'
/nix/store/fkygdjhm2raz32xrzwnq8b89hkkdc9vv-python3-3.9.9/lib/python3.9/site-packages:/nix/store/zs9mh2h826rgh2w39bv8gb1xc2nr77lk-python3.9-pip-21.3.1/lib/python3.9/site-packages
$ PYTHONPATH= python -c 'import subprocess
  subprocess.run(["python", "-c", "import os; print(os.environ[\'PYTHONPATH\'])"])'

[layday]

In build, it will be mitigated by #406 (38099ad), should it be merged at some point.

[adamjstewart]

Thanks for the detailed diagnosis @layday. I think you're right, this is not an issue of how Python was built, but whether or not pip is present in PYTHONPATH. I tried building 3 combinations of --with-ensurepip={no,install,update} and all 3 caused the above issue when pip was in PYTHONPATH, but all 3 worked when pip was not in PYTHONPATH.

I'll see if I can extract a minimal patch from 38099ad and add it to our build recipe.

[layday]

Okay, I had assumed there was some interplay with installing pip with Python, but it makes things a lot easier if there isn't.

[adamjstewart]

For anyone else who runs into this problem, the following minimal patch solved it for me:

--- a/src/build/env.py	2021-09-16 16:20:01.000000000 -0500
+++ b/src/build/env.py	2022-01-23 15:08:26.000000000 -0600
@@ -254,6 +254,7 @@
     """
     import venv
 
+    os.environ.pop('PYTHONPATH', None)
     venv.EnvBuilder(with_pip=True, symlinks=_fs_supports_symlink()).create(path)
     executable, script_dir, purelib = _find_executable_and_scripts(path)
 

Hopefully #406 is merged soon. Thanks for your help @layday!

[salotz-sitx]

+1 on this. I am a Spack user and would like this to work :)

Interestingly in one of my spack environments python -m build seems to work and I don't know why, because they created in very similar ways.

[salotz-sitx]

@adamjstewart It looks like if you have python 3.9 in your spack env it works, but if you use 3.8 you get the OPs error. I tried specifically 3.9.9 and 3.8.12 btw.

[adamjstewart]

@salotz-sitx does spack/spack#28562 solve your issue? For me, it didn't matter which Python version I was using, it mattered if pip was in my PYTHONPATH.

[salotz-sitx]

I didn't try spack/spack#28562.
It looks like pip is on PYTHONPATH:

--> echo $PYTHONPATH
~/path/to/.spack-env/view/lib/python3.9/site-packages:/home/salotz/resources/spack/opt/spack/linux-ubuntu18.04-skylake/gcc-7.5.0/py-setuptools-58.2.0-46aor54ev6z5er27upmg4bzoc2gxnud2/lib/python3.9/site-packages 

and pip is in the first in the list: ~/path/to/.spack-env/view/lib/python3.9/site-packages

[jaraco]

I've run into this issue in jaraco/jaraco.packaging#8. After switching from pep517 to build for loading metadata from an unbuilt project, now all docs builds are failing in RTD. I don't know if RTD is using Spack or Nix or maybe just something with PYTHONPATH. Is the recommended workaround to include build[virtualenv] in the dependencies?