Wheel cache is not forward/backward compatible between 22.2.x and 22.3

[cboylan]

Description

Installing packages and having the resulting wheels go into a cache dir with pip 22.3 produces a cache that appears to be ignored by 22.2.2 forcing 22.2.2 to install from sdists. The same thing seems to happen if you create the wheel cache with 22.2.2 and then try to install packages with that cache using 22.3.

We noticed this because we've got some multistage docker image builds that will install all build requirements (like gcc) and create wheels for all deps in a throwaway image then copy only the wheels over to the final image and pip install what we need there using the wheel cache to avoid needing build requirements like gcc. This broke when the pip versions on either side of that got out of sync.

I don't know if this was an expected change (it hasn't been a problem in the past and I strongly suspect that we've gotten out of sync previously). This was also not called out in the release notes. Just want to make sure this gets reported even if ultimately this was expected.

Expected behavior

I expected that the cache would be forward compatible at least. It seems a little odd to force users to rebuild all new wheels when they update pip versions. Old pip recognizing newer pips wheel cache and using it is maybe a bit less expected (though would've been nice in our situation).

That said I'm not entirely sure there is a bug here. I'm mostly interested in calling the behavior change out to double check that this was expected. And to see if this sort of behavior change is worthy of a release note.

pip version

22.3

Python version

3.10

OS

Debian Bullseye, OpenSUSE Tumbleweed

How to Reproduce

A working cache with the same pip version across environments:

> python3 -m venv venv1
> venv1/bin/python3 --version
Python 3.10.7
> venv1/bin/pip --version
pip 22.2.2 from /home/clark/tmp/venv1/lib64/python3.10/site-packages/pip (python 3.10)
> venv1/bin/pip install build wheel
Collecting build
  Using cached build-0.8.0-py3-none-any.whl (17 kB) 
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB) 
Collecting packaging>=19.0
  Using cached packaging-21.3-py3-none-any.whl (40 kB) 
Collecting pep517>=0.9.1
  Using cached pep517-0.13.0-py3-none-any.whl (18 kB) 
Collecting tomli>=1.0.0
  Using cached tomli-2.0.1-py3-none-any.whl (12 kB) 
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB) 
Installing collected packages: wheel, tomli, pyparsing, pep517, packaging, build
Successfully installed build-0.8.0 packaging-21.3 pep517-0.13.0 pyparsing-3.0.9 tomli-2.0.1 wheel-0.37.1

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python3 -m pip install --upgrade pip 
> venv1/bin/pip install --cache-dir ./pip-testing/wheels netifaces
Collecting netifaces
  Downloading netifaces-0.11.0.tar.gz (30 kB) 
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: netifaces
  Building wheel for netifaces (setup.py) ... done
  Created wheel for netifaces: filename=netifaces-0.11.0-cp310-cp310-linux_x86_64.whl size=35306 sha256=0bc5bfaf024c86104588db92a0124cf6d181d45c1935bccd75ec0812b7327644
  Stored in directory: /home/clark/tmp/pip-testing/wheels/wheels/48/65/b3/4c4cc6038b81ff21cc9df69f2b6774f5f52e23d3c275ed15aa
Successfully built netifaces
Installing collected packages: netifaces
Successfully installed netifaces-0.11.0

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python3 -m pip install --upgrade pip 
> python3 -m venv venv2
> venv2/bin/python --version
Python 3.10.7
> venv2/bin/pip --version
pip 22.2.2 from /home/clark/tmp/venv2/lib64/python3.10/site-packages/pip (python 3.10)
> venv2/bin/pip install --cache-dir ./pip-testing/wheels netifaces
Collecting netifaces
  Using cached netifaces-0.11.0-cp310-cp310-linux_x86_64.whl
Installing collected packages: netifaces
Successfully installed netifaces-0.11.0

[notice] A new release of pip available: 22.2.2 -> 22.3
[notice] To update, run: python3 -m pip install --upgrade pip

And now using the same wheel cache but with a different virtualenv that has newer pip installed it fallsback to building from the sdist:

> python3 -m venv venv2
> venv2/bin/python --version
Python 3.10.7
> venv2/bin/pip --version
pip 22.3 from /home/clark/tmp/venv2/lib64/python3.10/site-packages/pip (python 3.10)
> venv2/bin/pip install --cache-dir ./pip-testing/wheels netifaces
Collecting netifaces
  Using cached netifaces-0.11.0.tar.gz (30 kB)
  Preparing metadata (setup.py) ... done
Installing collected packages: netifaces
  DEPRECATION: netifaces is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for netifaces ... done
Successfully installed netifaces-0.11.0

Note netifaces was chosen as an example here because it does not publish python3.10 wheels on pypi and its install is quick and simple. I don't expect this to be a package specific issue, but I suppose that is possible as well.

Output

Output was provided inline with the reproduction steps above.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

The format of the cache isn't public, and you shouldn't really be using the cache like this. I don't know if it was accidental or deliberate that the cache format changed - I would expect that we'd keep it stable when we can, simply because it makes it work better as a cache, but we don't guarantee that.

For the use case you describe, you probably shouldn't be doing this anyway. It seems like you'd be better using pip wheel to download/build the wheels you want, and then copying those wheels to the target and using --find-links so that pip install will find the wheels when installing on the target. This is likely to be a lot more robust than your current approach.

[fungi]

So, just to be clear, there's no expectation that a new version of pip will be able to use a wheel cache created by the previous version, and such behavior changes are neither considered a regression nor do they rise to the level of requiring a release note.

Should pip's documentation state that, and perhaps also recommend clearing your wheel cache as a part of each upgrade?

[pfmoore]

That's a lot stronger than what I said, which was that we don't make any particular promises, but we won't change the cache format without good reason.

Also, it's a cache, intended purely to improve performance by avoiding unnecessary rebuilds. If you want to guarantee pip won't try to build on the target, you should be using pip wheel and --find-links as I said.

[fungi]

Setting the reported use case where this was discovered aside entirely, I'm just wondering more generally if cache layout/format changes for pip are something which should be mentioned in release notes so that people upgrading know when they should blow away the cache because its contents have been rendered useless and now it's only full of cruft, or if it's simply good hygiene to do that at every pip upgrade on the basis that it may have changed.

[cboylan]

I did more digging into why this is happening and have learned some things. I think this may actually be a bug in bad03ef. The wheel caching behavior simply acted as a canary. In particular that commit updated the Link class's handling of the hashes received from the simple API. It added Link.from_json() which is called by parse_links() to set the valid hashes on the Link object.

But the same commit updated Link.hash and Link.hash_name to only consult self.link_hash which is not set by Link.from_json(). I'm not familiar enough with pip to say one way or another, but is it possible that this means the files downloaded from pypi are no longer having their index listed hashes double checked locally to ensure there aren't any bit flips (or worse)?

The reason this affects the wheel cache behavior is that the wheel caching mechanism hashes Link object information to determine where to read and write the cached wheels. In 22.2.2 this included hash information for the packages but in 22.3 it does not.

22.2.2 wheel cache path hash input:

{"interpreter_name":"cp","interpreter_version":"310","sha256":"043a79146eb2907edf439899f262b3dfe41717d34124298ed281139a8b93ca32","url":"https://files.pythonhosted.org/packages/a6/91/86a6eac449ddfae239e93ffc1918cf33fd9bab35c04d1e963b311e347a73/netifaces-0.11.0.tar.gz"}

22.3.0 wheel cache path hash input:

{"interpreter_name":"cp","interpreter_version":"310","url":"https://files.pythonhosted.org/packages/a6/91/86a6eac449ddfae239e93ffc1918cf33fd9bab35c04d1e963b311e347a73/netifaces-0.11.0.tar.gz"}

This patch against 22.3 seems to fix the wheel cache issues we have with the wheel cache, but I also think this ensures Link objects created by parse_links() using the simple json api have a valid link_hash set which I believe may be important. If the hash information from the simple json API should be set on the Link objects some other way then it might be a good idea to clean up Link.hashes to reduce confusion and update parse_links() to set the values correctly.

diff --git a/src/pip/_internal/models/link.py b/src/pip/_internal/models/link.py
index c792d128b..03cc34b42 100644
--- a/src/pip/_internal/models/link.py
+++ b/src/pip/_internal/models/link.py
@@ -248,6 +248,11 @@ class Link(KeyBasedCompareMixin):
         yanked_reason = file_data.get("yanked")
         dist_info_metadata = file_data.get("dist-info-metadata")
         hashes = file_data.get("hashes", {})
+        link_hash = None
+        for hash_name, hash_value in hashes.items():
+            if hash_name in _SUPPORTED_HASHES:
+                link_hash = LinkHash(name=hash_name, value=hash_value)
+                break
 
         # The Link.yanked_reason expects an empty string instead of a boolean.
         if yanked_reason and not isinstance(yanked_reason, str):
@@ -262,6 +267,7 @@ class Link(KeyBasedCompareMixin):
             requires_python=pyrequire,
             yanked_reason=yanked_reason,
             hashes=hashes,
+            link_hash=link_hash,
             dist_info_metadata=dist_info_metadata,
         )
 

Finally, while I agree that our image build process can probably be made more robust, I think we shouldn't fixate too strongly on that specific use case here. I will work on improving our image builds. But at the same time I also expect that if I build perfectly valid wheels with 22.2.2 then upgrade to 22.3 that pip will continue to effectively use its wheel cache and check the hashes of its downloads.

[sbidoul]

@cboylan I just finished bissecting and found the same commit. I've no time right now to investigate your proposed fix but there is definitely some bug to fix here, and your explanation makes a lot of sense at first sight.

[pradyunsg]

FWIW, I'll note that in #11143 -- the fact that it requires changing the cache format and the lack of a migration path for the same, was treated as a blocking concern.

if cache layout/format changes for pip are something which should be mentioned in release notes

I don't think we intentionally changed the cache format here. It's probably reasonable to treat this as a bug/regression IMO. :)

[pradyunsg]

A PR fixing this, with a test to validate the caching key would be welcome!