PACKAGES DO NOT MATCH error when a new wheel is added for an existing version

[nickwilliams-eventbrite]

Using Pipenv 2018.11.26...

Issue description (steps to reproduce)

• Upload a new version of a package to PyPi or a private wheelhouse, but upload just the source tarball (.tar.gz)
• Update a Pipfile to use this new version, then lock that file — the digest for the new source tarball gets placed in the lockfile
• pipenv install --deploy from the lockfile — everything works
• Upload a precompiled wheel for the same package and version to PyPi or the private wheelhouse containing the source tarball, but keep the source tarball (don't remove it)
• pipenv install --deploy from the lockfile ... that package fails to install due to the PACKAGES DO NOT MATCH error, even though the source tarball is still available and still matches

Expected behavior

When multiple install candidates are available, and some of them match the digests in the lockfile while others do not match the digests in the lockfile, Pipenv should discard the candidates that do not match the digests in the lockfile and install one of the candidates that do match.

Actual behavior

When multiple install candidates are available, and some of them match the digests in the lockfile while others do not match the digests in the lockfile, Pipenv discards all candidates and refuses to install the package.

Just to head off any quick replies of "just re-lock your lockfiles," this is not practicable for us. We have hundreds of projects using this package, and having them all broken for hours or days while we re-lock them all is not tenable. Obviously, if there were no matching candidates, we would have to re-lock. That's not in question. But Pipenv found some matching and some non-matching candidates—it should discard the non-matching candidates and install a matching one.

Our exact use case

In PyPi, mysqlclient has the source tarball for all versions and Windows wheels for some versions, but it does not have any Linux or macOS wheels. In order to speed up builds (by not recompiling the mysqlclient C code every install) and eliminate the need to have the MySQL client library header files installed everywhere, we decided to precompile a Linux wheel for mysqlclient compatible with our infrastructure and place that wheel in our private wheelhouse. As long as we re-lock a lockfile, it works and the pre-compiled wheel is installed, but before we do that, we get the following output:

Collecting mysqlclient==1.3.14 (from -r /tmp/pipenv-z17i8cot-requirements/pipenv-5jrmav9h-requirement.txt (line 1))
  2 location(s) to search for versions of mysqlclient:
  * https://our-wheelhouse/simple/mysqlclient/
  * https://pypi.python.org/simple/mysqlclient/
...
    Found link https://our-wheelhouse/simple/mysqlclient/mysqlclient-1.3.14-cp37-cp37m-linux_x86_64.whl#sha256=7936bcb57c01f3bb6e3b23a160cbb171fb9bdd8745435a3e998f929a92adc047 (from https://our-wheelhouse/simple/mysqlclient/), version: 1.3.14
...
    Found link https://files.pythonhosted.org/packages/f7/a2/1230ebbb4b91f42ad6b646e59eb8855559817ad5505d81c1ca2b5a216040/mysqlclient-1.3.14.tar.gz#sha256=3981ae9ce545901a36a8b7aed76ed02960a429f75dc53b7ad77fb2f9ab7cd56b (from https://pypi.org/simple/mysqlclient/), version: 1.3.14
...
  Using version 1.3.14 (newest of versions: 1.3.14)
  Created temporary directory: /tmp/pip-unpack-hnxccb3h
  Looking up "https://our-wheelhouse/simple/mysqlclient/mysqlclient-1.3.14-cp37-cp37m-linux_x86_64.whl" in the cache
  No cache entry available
  Starting new HTTPS connection (1): our-wheelhouse:443
  https://our-wheelhouse:443 "GET /simple/mysqlclient/mysqlclient-1.3.14-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 106449
  Downloading https://our-wheelhouse/simple/mysqlclient/mysqlclient-1.3.14-cp37-cp37m-linux_x86_64.whl (106kB)
  Downloading from URL https://our-wheelhouse/simple/mysqlclient/mysqlclient-1.3.14-cp37-cp37m-linux_x86_64.whl#sha256=7936bcb57c01f3bb6e3b23a160cbb171fb9bdd8745435a3e998f929a92adc047 (from https://our-wheelhouse/simple/mysqlclient/)
  Updating cache with response from "https://our-wheelhouse/simple/mysqlclient/mysqlclient-1.3.14-cp37-cp37m-linux_x86_64.whl"
  Caching due to etag
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-036br1_4'
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    mysqlclient==1.3.14 from https://our-wheelhouse/simple/mysqlclient/mysqlclient-1.3.14-cp37-cp37m-linux_x86_64.whl#sha256=7936bcb57c01f3bb6e3b23a160cbb171fb9bdd8745435a3e998f929a92adc047 (from -r /tmp/pipenv-z17i8cot-requirements/pipenv-5jrmav9h-requirement.txt (line 1)):
        Expected sha256 062d78953acb23066c0387a8f3bd0ecf946626f599145bb7fd201460e8f773e1
        Expected     or 3981ae9ce545901a36a8b7aed76ed02960a429f75dc53b7ad77fb2f9ab7cd56b
        Expected     or b3591a00c0366de71d65108627899710d9cfb00e575c4d211aa8de59b1f130c9
        Expected     or be7f8145aaf75ba96aa0a61b3d3a3ea7430aa8677ac9e26b305e450473e29567
             Got        7936bcb57c01f3bb6e3b23a160cbb171fb9bdd8745435a3e998f929a92adc047

Of note ... the pipenv install --deploy that gave me that ^^ output had a return code of 0. It "succeeded," even though mysqlclient did not install. I think this is because Pipenv tries again in "Installing initially failed dependencies," and the success/failure of that step is ignored, per #3537 (which is fixed but still has not been released). However, it could be a different issue, so I wanted to mention it.

---

# pipenv --support | xclip
Error: Can't open display: (null)

[frostming]

I guess I could reproduce it if I manually remove the hash from the lockfile and do pipenv install --deploy? But I can't. Tested on master branch(c3a9e8d)

Pipfile.lock: (Note I removed the hash of wheel.)

{
    "_meta": {
        "hash": {
            "sha256": "e6728243c3982bd95c23ad63dc24514046979d9cf07efbb6dc60150bdf182788"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.7"
        },
        "sources": [{
            "name": "pypi",
            "url": "https://pypi.org/simple",
            "verify_ssl": true
        }]
    },
    "default": {
        "click": {
            "hashes": [
                "sha256:5b94b49521f6456670fdb30cd82a4eca9412788a93fa6dd6df72c94d5a8ff2d7"
            ],
            "index": "pypi",
            "version": "==7.0"
        }
    },
    "develop": {}
}

[thehesiod]

just ran into this as well, we have a custom pypi server, uploaded an sdist awhile ago, then added some wheels for efficiency, and then all prod builds broke :( It would be nice if pipenv could hook into pip to control what is allowed based on the lock file.

[matteius]

@nickwilliams-eventbrite and @thehesiod I don't think this should be an issue in the current 2022.4.8 release because of the requirement to use index restricted packages for locking. Please advise if you think this is still a concern and we can revisit it.