pip>=20.2 needlessly re-installs dependencies available on the system in system-site-packages venv

[TimoRoth]

What did you want to do?

We are using a venv that explicitly does use system-site-packages, in order to allow users in a container, which does not allow write access to /, to install their own packages while still using all the (partially very complex to install/build) packages provided by the container image.
This used to work fine until pip>=20.2, and is still broken in latest pip dev build as of today.

Output

Singularity> python3 -m venv --system-site-packages test_venv
Singularity> source ./test_venv/bin/activate
(test_venv) Singularity> pip list | grep numpy
numpy              1.19.1
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/home/users/fmaussion/test_venv/bin/python3 -m pip install --upgrade pip' command.
(test_venv) Singularity> pip install numpy
Requirement already satisfied: numpy in /usr/local/pyenv/versions/3.7.8/lib/python3.7/site-packages (1.19.1)
WARNING: You are using pip version 20.1.1; however, version 20.2.1 is available.
You should consider upgrading via the '/home/users/fmaussion/test_venv/bin/python3 -m pip install --upgrade pip' command.
(test_venv) Singularity> pip install --upgrade pip setuptools
Collecting pip
  Using cached pip-20.2.1-py2.py3-none-any.whl (1.5 MB)
Collecting setuptools
  Using cached setuptools-49.2.1-py3-none-any.whl (789 kB)
Installing collected packages: pip, setuptools
  Attempting uninstall: pip
    Found existing installation: pip 20.1.1
    Uninstalling pip-20.1.1:
      Successfully uninstalled pip-20.1.1
  Attempting uninstall: setuptools
    Found existing installation: setuptools 47.1.0
    Uninstalling setuptools-47.1.0:
      Successfully uninstalled setuptools-47.1.0
Successfully installed pip-20.2.1 setuptools-49.2.1
(test_venv) Singularity> pip install numpy
Collecting numpy
  Using cached numpy-1.19.1-cp37-cp37m-manylinux2010_x86_64.whl (14.5 MB)
Installing collected packages: numpy
Successfully installed numpy-1.19.1

As you can see, before updating pip to 20.2, it correctly notices that numpy is already installed and does not need to do anything.
After the update, it goes and installs a copy of the same numpy version in the venv.
The same happens when installing a package that depends on numpy. Also the same goes for any other dependency that's already provided by the system site-packages.
"pip list" correctly lists numpy as installed in either case.

Additional information

The reason we are doing it this way is because building some of those dependencies, which are binary and annoying to deal with (most prominently gdal and fiona). The singularity container image has all of them pre-built, with optimizations specific for the host systems CPU.

For now I've instructed users to either use --no-deps when installing things, or to explicitly pin pip to <20.2.

[sbidoul]

I can reproduce. It might be a side effect of changes made to check_if_exists.

[sbidoul]

This is due to the local_only argument of search_distributions that is True by default when called from utils.misc.get_distribution().

[fdebrabander]

Can confirm, also have the same issue.

[sbidoul]

@fdebrabander the issue is fixed in #8702 and should be part of the next patch release. You can test it by installing that branch with pip install -U "pip @ git+https://github.com/pypa/pip@refs/pull/8702/head".

[EricCousineau-TRI]

Can also confirm; on Ubuntu 18.04 with python3-numpy installed on the system (numpy==1.13.3), I can see this happening:

$ mkdir -p ~/venv/sys
$ (
set -eu
cd ~/venv/sys
rm -rf ./*
python3 -m virtualenv -p python3 --system-site-packages .
source bin/activate
pip install pip==20.2.1
pip install numpy
)
...
Successfully installed numpy-1.19.

$ (                                                                                                                                                                                                 
set -eu                                                                                                                                                                                                        
cd ~/venv/sys
rm -rf ./*
python3 -m virtualenv -p python3 --system-site-packages .
source bin/activate
pip install pip==20.2
pip install numpy
)
...
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (1.13.3)

[EricCousineau-TRI]

FWIW Our investigation in #8726 was motivated by us running into this problem.

Something to note is that this new behavior does seem to be in line with what these docs say about --upgrade-strategy=only-if-needed:
https://github.com/pypa/pip/blob/f17c1d6d3bf4b4ee8b7b0e669c449b9cd86e94f6/docs/html/development/architecture/upgrade-options.rst#controlling-what-gets-installed

• only-if-needed - packages are only upgraded if they are named in the pip command or a requirement file (i.e, they are direct requirements), or an upgraded parent needs a later version of the dependency than is currently installed.
• to-satisfy-only (undocumented) - packages are not upgraded (not even direct requirements) unless the currently installed version fails to satisfy a requirement (either explicitly specified or a dependency).

[EricCousineau-TRI]

Eh, not sure about the direct correlation, tho.

20.2.1 was released on Aug. 4.
20.2 was released on July 28.

The commit that introduced these docs (5da3a7d) was made back in Feb.

EDIT: Didn't read upstream thread. Source was root-caused - sorry for the noise!