Fallback handling when downloading wheels for seeding may fail #2429
Description
Issue
When a virtualenv is being created and a bundled package that needs to be downloaded for seeding is being downloaded by pip at the very same moment, then pip may not output the exact location of the downloaded package. Fallback handling then incorrectly fails on being passed '' as the version to search for.
This issue was observed in highly parallel testing of code, where multiple environments were being created at the very same moment. Additional testing to isolate the issue showed that two virtualenv instances can at the same time instruct pip to download, for example, a newer setuptools, and both pip instances will start this download. pip itself is guarded against this situation and only saves the file in the first instance, and discards the downloaded file and merely reports success in the second. The output of the second pip instance does not contain the expected path for virtualenv to act on.
virtualenv is, in principle, guarded against this issue, by searching for the newest wheel for the package. If a bundled package was being downloaded, however, version translations along the path cause the version specifier (an empty string) to be rejected as invalid for finding the newest wheel.
Environment
Tested on an Ubuntu 20.04 VM with 4 cores.
appdirs 1.4.3
apt-xapian-index 0.49
blinker 1.4
certifi 2019.11.28
chardet 3.0.4
Click 7.0
colorama 0.4.3
command-not-found 0.3
cryptography 2.8
cupshelpers 1.0
dbus-python 1.2.16
distlib 0.3.0
distro 1.4.0
distro-info 0.23ubuntu1
entrypoints 0.3
filelock 3.0.12
gpg 1.13.1-unknown
httplib2 0.14.0
idna 2.8
importlib-metadata 1.5.0
keyring 18.0.1
language-selector 0.1
launchpadlib 1.10.13
lazr.restfulclient 0.14.2
lazr.uri 1.0.3
more-itertools 4.2.0
netifaces 0.10.4
oauthlib 3.1.0
olefile 0.46
packaging 20.3
pexpect 4.6.0
Pillow 7.0.0
pip 20.0.2
pluggy 0.13.0
py 1.8.1
pycairo 1.16.2
pycrypto 2.6.1
pycups 1.9.73
PyGObject 3.36.0
PyJWT 1.7.1
pymacaroons 0.13.0
PyNaCl 1.3.0
pyparsing 2.4.6
PyQt5 5.14.1
python-apt 2.0.0+ubuntu0.20.4.8
python-debian 0.1.36ubuntu1
PyYAML 5.3.1
reportlab 3.5.34
requests 2.22.0
requests-unixsocket 0.2.0
SecretStorage 2.3.1
setuptools 45.2.0
simplejson 3.16.0
sip 4.19.21
six 1.14.0
ssh-import-id 5.10
systemd-python 234
toml 0.10.0
tox 3.13.2
ubuntu-advantage-tools 27.10
ubuntu-drivers-common 0.0.0
ufw 0.36
unattended-upgrades 0.1
urllib3 1.25.8
usb-creator 0.3.7
virtualenv 20.0.17
wadllib 1.3.3
wheel 0.34.2
xkit 0.0.0
zipp 1.0.0Output of the virtual environment creation
Triggering this issue was done using a small script to set up several environments in parallel. It lists specific versions to recreate the environment where the issue was previously observed. Take care to run this script in an empty directory: it forcibly removes previous output.
#!/bin/bash
rm -rf venv* cache pip_cache
# Virtualenv 20.16.2
python3 -m virtualenv --download --pip 22.2.1 --wheel 0.37.1 --setuptools 63.4.0 --seeder pip venv_root
./venv_root/bin/pip install virtualenv==20.16.2
ARGS="--download --with-traceback --app-data ./cache --symlinks -vv --pip bundle --wheel bundle --setuptools bundle"
export PIP_CACHE_DIR=./pip_cache
./venv_root/bin/python3 -m virtualenv ${ARGS} venv1 &
./venv_root/bin/python3 -m virtualenv ${ARGS} venv2 &
./venv_root/bin/python3 -m virtualenv ${ARGS} venv3 &
./venv_root/bin/python3 -m virtualenv ${ARGS} venv4 &
./venv_root/bin/python3 -m virtualenv ${ARGS} venv5 &
./venv_root/bin/python3 -m virtualenv ${ARGS} venv6 &
./venv_root/bin/python3 -m virtualenv ${ARGS} venv7 &
./venv_root/bin/python3 -m virtualenv ${ARGS} venv8 &Adding output redirection, the following trace was obtained for one of the instances which failed:
457 setup logging to NOTSET [DEBUG report:35]
461 find interpreter for spec PythonSpec(path=/home/schaap/tst/issue/venv_root/bin/python3) [INFO builtin:56]
462 discover exe for PythonInfo(spec=CPython3.8.10.final.0-64, exe=/home/schaap/tst/issue/venv_root/bin/python3, platform=linux, version='3.8.10 (default, Jun 22 2022, 20:18:18) \n[GCC 9.4.0]', encoding_fs_io=utf-8-utf-8) in /usr [DEBUG py_info:407]
462 filesystem is case-sensitive [DEBUG info:24]
463 Attempting to acquire lock 140008185598160 on /home/schaap/tst/issue/cache/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [DEBUG _api:172]
463 Lock 140008185598160 not acquired on /home/schaap/tst/issue/cache/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock, waiting 0.05 seconds ... [DEBUG _api:186]
513 Attempting to acquire lock 140008185598160 on /home/schaap/tst/issue/cache/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [DEBUG _api:172]
513 Lock 140008185598160 acquired on /home/schaap/tst/issue/cache/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [DEBUG _api:176]
514 got python info of /usr/bin/python3.8 from /home/schaap/tst/issue/cache/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.json [DEBUG via_disk_folder:129]
532 Attempting to release lock 140008185598160 on /home/schaap/tst/issue/cache/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [DEBUG _api:209]
532 Lock 140008185598160 released on /home/schaap/tst/issue/cache/py_info/1/df0893f56f349688326838aaeea0de204df53a132722cbd565e54b24a8fec5f6.lock [DEBUG _api:212]
532 proposed PythonInfo(spec=CPython3.8.10.final.0-64, system=/usr/bin/python3.8, exe=/home/schaap/tst/issue/venv_root/bin/python3, platform=linux, version='3.8.10 (default, Jun 22 2022, 20:18:18) \n[GCC 9.4.0]', encoding_fs_io=utf-8-utf-8) [INFO builtin:63]
533 accepted PythonInfo(spec=CPython3.8.10.final.0-64, system=/usr/bin/python3.8, exe=/home/schaap/tst/issue/venv_root/bin/python3, platform=linux, version='3.8.10 (default, Jun 22 2022, 20:18:18) \n[GCC 9.4.0]', encoding_fs_io=utf-8-utf-8) [DEBUG builtin:65]
603 create virtual environment via CPython3Posix(dest=/home/schaap/tst/issue/venv1, clear=False, no_vcs_ignore=False, global=False) [INFO session:48]
603 create folder /home/schaap/tst/issue/venv1/bin [DEBUG _sync:9]
604 create folder /home/schaap/tst/issue/venv1/lib/python3.8/site-packages [DEBUG _sync:9]
604 write /home/schaap/tst/issue/venv1/pyvenv.cfg [DEBUG pyenv_cfg:30]
604 home = /usr [DEBUG pyenv_cfg:34]
604 implementation = CPython [DEBUG pyenv_cfg:34]
604 version_info = 3.8.10.final.0 [DEBUG pyenv_cfg:34]
604 virtualenv = 20.16.2 [DEBUG pyenv_cfg:34]
604 include-system-site-packages = false [DEBUG pyenv_cfg:34]
604 base-prefix = /usr [DEBUG pyenv_cfg:34]
604 base-exec-prefix = /usr [DEBUG pyenv_cfg:34]
604 base-executable = /usr/bin/python3.8 [DEBUG pyenv_cfg:34]
604 symlink /usr/bin/python3.8 to /home/schaap/tst/issue/venv1/bin/python [DEBUG _sync:28]
604 create virtualenv import hook file /home/schaap/tst/issue/venv1/lib/python3.8/site-packages/_virtualenv.pth [DEBUG api:89]
605 create /home/schaap/tst/issue/venv1/lib/python3.8/site-packages/_virtualenv.py [DEBUG api:92]
605 ============================== target debug ============================== [DEBUG session:50]
605 debug via /home/schaap/tst/issue/venv1/bin/python /home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/create/debug.py [DEBUG creator:197]
605 {
"sys": {
"executable": "/home/schaap/tst/issue/venv1/bin/python",
"_base_executable": "/home/schaap/tst/issue/venv1/bin/python",
"prefix": "/home/schaap/tst/issue/venv1",
"base_prefix": "/usr",
"real_prefix": null,
"exec_prefix": "/home/schaap/tst/issue/venv1",
"base_exec_prefix": "/usr",
"path": [
"/usr/lib/python38.zip",
"/usr/lib/python3.8",
"/usr/lib/python3.8/lib-dynload",
"/home/schaap/tst/issue/venv1/lib/python3.8/site-packages"
],
"meta_path": [
"<class '_virtualenv._Finder'>",
"<class '_frozen_importlib.BuiltinImporter'>",
"<class '_frozen_importlib.FrozenImporter'>",
"<class '_frozen_importlib_external.PathFinder'>"
],
"fs_encoding": "utf-8",
"io_encoding": "utf-8"
},
"version": "3.8.10 (default, Jun 22 2022, 20:18:18) \n[GCC 9.4.0]",
"makefile_filename": "/usr/lib/python3.8/config-3.8-x86_64-linux-gnu/Makefile",
"os": "<module 'os' from '/usr/lib/python3.8/os.py'>",
"site": "<module 'site' from '/usr/lib/python3.8/site.py'>",
"datetime": "<module 'datetime' from '/usr/lib/python3.8/datetime.py'>",
"math": "<module 'math' (built-in)>",
"json": "<module 'json' from '/usr/lib/python3.8/json/__init__.py'>"
} [DEBUG session:51]
697 add seed packages via FromAppData(download=True, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/schaap/tst/issue/cache) [INFO session:55]
729 download wheel setuptools 3.8 to /home/schaap/tst/issue/cache/wheel/house [DEBUG acquire:44]
732 download wheel pip 3.8 to /home/schaap/tst/issue/cache/wheel/house [DEBUG acquire:44]
738 download wheel wheel 3.8 to /home/schaap/tst/issue/cache/wheel/house [DEBUG acquire:44]
14006 downloaded wheel wheel-0.37.1-py2.py3-none-any.whl [DEBUG acquire:69]
14007 adding wheel-0.37.1-py2.py3-none-any.whl information to /home/schaap/tst/issue/cache/wheel/3.8/embed/3/wheel.json [DEBUG periodic_update:75]
14007 got embed update of distribution wheel from /home/schaap/tst/issue/cache/wheel/3.8/embed/3/wheel.json [DEBUG via_disk_folder:129]
14010 wheel-0.37.1-py2.py3-none-any.whl already present in /home/schaap/tst/issue/cache/wheel/3.8/embed/3/wheel.json [WARNING periodic_update:78]
15706 fail [ERROR via_app_data:95]
Traceback (most recent call last):
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 82, in _get
result = get_wheel(
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 27, in get_wheel
wheel = download_wheel(
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 68, in download_wheel
result = _find_downloaded_wheel(distribution, version_spec, for_py_version, to_folder, out)
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 82, in _find_downloaded_wheel
return find_compatible_in_house(distribution, version_spec, for_py_version, to_folder)
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 94, in find_compatible_in_house
raise ValueError(version_spec)
ValueError
15706 ValueError('') [ERROR via_app_data:109]
15848 fail [ERROR via_app_data:95]
Traceback (most recent call last):
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 82, in _get
result = get_wheel(
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 27, in get_wheel
wheel = download_wheel(
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 68, in download_wheel
result = _find_downloaded_wheel(distribution, version_spec, for_py_version, to_folder, out)
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 82, in _find_downloaded_wheel
return find_compatible_in_house(distribution, version_spec, for_py_version, to_folder)
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/wheels/acquire.py", line 94, in find_compatible_in_house
raise ValueError(version_spec)
ValueError
15849 ValueError('') [ERROR via_app_data:109]
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/__main__.py", line 69, in <module>
run_with_catch() # pragma: no cov
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/__main__.py", line 54, in run_with_catch
run(args, options, env)
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/__main__.py", line 16, in run
session = cli_run(args, options, env)
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/run/__init__.py", line 30, in cli_run
of_session.run()
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/run/session.py", line 43, in run
self._seed()
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/run/session.py", line 56, in _seed
self.seeder.run(self.creator)
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 40, in run
with self._get_seed_wheels(creator) as name_to_whl:
File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/home/schaap/tst/issue/venv_root/lib/python3.8/site-packages/virtualenv/seed/embed/via_app_data/via_app_data.py", line 125, in _get_seed_wheels
raise RuntimeError(f"seed failed due to failing to download wheels {', '.join(fail.keys())}")
RuntimeError: seed failed due to failing to download wheels pip, setuptools