Pipfile: Unable to install from url - gives error "THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!"

[thernstig]

Issue description

No Pipefile.lock exists to start with. Just a Pipfile with the following:

bsddb3 = {file = "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl", sys_platform = "== 'win32'", platform_machine = "== 'AMD64'"}

When trying to execute this the hashes in the newly generated Pipfile.lock do not match.

Expected result

I expect the package to be installed without any issues.

Actual result

~\Desktop\projects\test> pipenv install --verbose
Creating a virtualenv for this project...
Pipfile: C:\Users\Moo\Desktop\projects\test\Pipfile
Using C:\Users\Moo\AppData\Local\Programs\Python\Python36\python.exe (3.6.1) to create virtualenv...
Running virtualenv with interpreter C:\Users\Moo\AppData\Local\Programs\Python\Python36\python.exe
Using base prefix 'C:\\Users\\Moo\\AppData\\Local\\Programs\\Python\\Python36'
New python executable in C:\Users\Moo\Desktop\projects\test\.venv\Scripts\python.exe
Installing setuptools, pip, wheel...done.

Virtualenv location: C:\Users\Moo\Desktop\projects\test\.venv
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
using sources: [{'url': 'https://pypi.org/simple', 'verify_ssl': True, 'name': 'pypi'}]
Using pip: -i https://pypi.org/simple

                          ROUND 1
Current constraints:
  bsddb3==6.2.6 from https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3 (from -r C:\Users\Moo\AppData\Local\Temp\pipenv-0gzvry7n-requirements\pipenv-n0a9xltf-constraints.txt (line 2))

Finding the best candidates:
  found candidate bsddb3==6.2.6 (constraint was ==6.2.6)

Finding secondary dependencies:
  bsddb3==6.2.6             requires bsddb3==6.2.6

New dependencies found in this round:
  adding ['bsddb3', '==6.2.6', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  bsddb3==6.2.6 from https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3 (from -r C:\Users\Moo\AppData\Local\Temp\pipenv-0gzvry7n-requirements\pipenv-n0a9xltf-constraints.txt (line 2))

Finding the best candidates:
  found candidate bsddb3==6.2.6 (constraint was ==6.2.6)

Finding secondary dependencies:
  bsddb3==6.2.6             requires bsddb3==6.2.6
------------------------------------------------------------
Result of round 2: stable, done

Updated Pipfile.lock (cd1455)!
Installing dependencies from Pipfile.lock (cd1455)...
Installing "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3; sys_platform == 'win32' --hash=sha256:42d621f4037425afcb16b67d5600c4556271a071a9a7f7f2c2b1ba65bc582d05"
$ "C:\\Users\\Moo\\Desktop\\projects\\test\\.venv\\Scripts\\pip.exe" install   --verbose  --no-deps  -r "C:\\Users\\Moo\\AppData\\Local\\Temp\\pipenv-b5i1sysf-requirements\\pipenv-e103omt2-requirement.txt" --require-hashes -i https://pypi.org/simple --exists-action w
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Created temporary directory: C:\Users\Moo\AppData\Local\Temp\pip-ephem-wheel-cache-424tiglb
Created temporary directory: C:\Users\Moo\AppData\Local\Temp\pip-req-tracker-43l1eko7
Created requirements tracker 'C:\\Users\\Moo\\AppData\\Local\\Temp\\pip-req-tracker-43l1eko7'
Created temporary directory: C:\Users\Moo\AppData\Local\Temp\pip-install-t6vli0ca
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Collecting bsddb3==6.2.6 from https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3 (from -r C:\\Users\\Moo\\AppData\\Local\\Temp\\pipenv-b5i1sysf-requirements\\pipenv-e103omt2-requirement.txt (line 1))
  Created temporary directory: C:\Users\Moo\AppData\Local\Temp\pip-unpack-ts5i4iz_
  Looking up "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl" in the cache
  Current age based on date: 181
  Starting new HTTPS connection (1): download.lfd.uci.edu:443
  https://download.lfd.uci.edu:443 "GET /pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl HTTP/1.1" 304 0
  Using cached https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl
  Downloading from URL https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3
Cleaning up...
Removed build tracker 'C:\\Users\\Moo\\AppData\\Local\\Temp\\pip-req-tracker-43l1eko7'
THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    bsddb3==6.2.6 from https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3 (from -r C:\\Users\\Moo\\AppData\\Local\\Temp\\pipenv-b5i1sysf-requirements\\pipenv-e103omt2-requirement.txt (line 1)):
        Expected sha256 42d621f4037425afcb16b67d5600c4556271a071a9a7f7f2c2b1ba65bc582d05
             Got        0cc7071e8108d3306f6d25afc5c0bb61501fa524090915c2577d3ba139904faa

Exception information:
Traceback (most recent call last):
  File "c:\users\Moo\desktop\projects\test\.venv\lib\site-packages\pip\_internal\basecommand.py", line 141, in main
    status = self.run(options, args)
  File "c:\users\Moo\desktop\projects\test\.venv\lib\site-packages\pip\_internal\commands\install.py", line 299, in run
    resolver.resolve(requirement_set)
  File "c:\users\Moo\desktop\projects\test\.venv\lib\site-packages\pip\_internal\resolve.py", line 109, in resolve
    raise hash_errors
pip._internal.exceptions.HashErrors: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    bsddb3==6.2.6 from https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3 (from -r C:\\Users\\Moo\\AppData\\Local\\Temp\\pipenv-b5i1sysf-requirements\\pipenv-e103omt2-requirement.txt (line 1)):
        Expected sha256 42d621f4037425afcb16b67d5600c4556271a071a9a7f7f2c2b1ba65bc582d05
             Got        0cc7071e8108d3306f6d25afc5c0bb61501fa524090915c2577d3ba139904faa


An error occurred while installing https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3; sys_platform == 'win32'! Will try again.
  ================================ 1/1 - 00:00:02
Installing initially failed dependencies...
Installing "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3; sys_platform == 'win32' --hash=sha256:42d621f4037425afcb16b67d5600c4556271a071a9a7f7f2c2b1ba65bc582d05"
$ "C:\\Users\\Moo\\Desktop\\projects\\test\\.venv\\Scripts\\pip.exe" install   --verbose  --no-deps  -r "C:\\Users\\Moo\\AppData\\Local\\Temp\\pipenv-b5i1sysf-requirements\\pipenv-aim0vwlb-requirement.txt" --require-hashes -i https://pypi.org/simple --exists-action w
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Created temporary directory: C:\Users\Moo\AppData\Local\Temp\pip-ephem-wheel-cache-hdoctb1r
Created temporary directory: C:\Users\Moo\AppData\Local\Temp\pip-req-tracker-oqeza5wz
Created requirements tracker 'C:\\Users\\Moo\\AppData\\Local\\Temp\\pip-req-tracker-oqeza5wz'
Created temporary directory: C:\Users\Moo\AppData\Local\Temp\pip-install-g39tgel3
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Collecting bsddb3==6.2.6 from https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3
  Created temporary directory: C:\Users\Moo\AppData\Local\Temp\pip-unpack-0xgkm_8d
  Looking up "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl" in the cache
  Current age based on date: 2
  Starting new HTTPS connection (1): download.lfd.uci.edu:443
  https://download.lfd.uci.edu:443 "GET /pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl HTTP/1.1" 304 0
  Using cached https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl
  Downloading from URL https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3
Cleaning up...
Removed build tracker 'C:\\Users\\Moo\\AppData\\Local\\Temp\\pip-req-tracker-oqeza5wz'
Exception information:
Traceback (most recent call last):
  File "c:\users\Moo\desktop\projects\test\.venv\lib\site-packages\pip\_internal\basecommand.py", line 141, in main
    status = self.run(options, args)
  File "c:\users\Moo\desktop\projects\test\.venv\lib\site-packages\pip\_internal\commands\install.py", line 299, in run
    resolver.resolve(requirement_set)
  File "c:\users\Moo\desktop\projects\test\.venv\lib\site-packages\pip\_internal\resolve.py", line 109, in resolve
    raise hash_errors
pip._internal.exceptions.HashErrors: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    bsddb3==6.2.6 from https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3
        Expected sha256 42d621f4037425afcb16b67d5600c4556271a071a9a7f7f2c2b1ba65bc582d05
             Got        0cc7071e8108d3306f6d25afc5c0bb61501fa524090915c2577d3ba139904faa


THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    bsddb3==6.2.6 from https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl#egg=bsddb3 (from -r C:\\Users\\Moo\\AppData\\Local\\Temp\\pipenv-b5i1sysf-requirements\\pipenv-aim0vwlb-requirement.txt (line 1)):
        Expected sha256 42d621f4037425afcb16b67d5600c4556271a071a9a7f7f2c2b1ba65bc582d05
             Got        0cc7071e8108d3306f6d25afc5c0bb61501fa524090915c2577d3ba139904faa


     ================================ 0/1 - 00:00:02
~\Desktop\projects\test>

Steps to replicate

Create a Pipfile with the following content. Note that this needs to be run on Windows 64-bit (since the wheel pointed out is specific to that architecture).

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
bsddb3 = {file = "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl", sys_platform = "== 'win32'", platform_machine = "== 'AMD64'"}

[requires]
python_version = "3.6"

Then execute a pipenv install --verbose.

---

$ pipenv --support

Pipenv version: '2018.7.1'

Pipenv location: 'c:\\users\\Moo\\appdata\\local\\programs\\python\\python36\\lib\\site-packages\\pipenv'

Python location: 'c:\\users\\Moo\\appdata\\local\\programs\\python\\python36\\python.exe'

Other Python installations in PATH:

• 3.5: C:\Users\Moo\AppData\Local\Programs\Python\Python35\python.exe

• 3.6: C:\Users\Moo\AppData\Local\Programs\Python\Python36\python.exe

• 3.6.1: C:\Users\Moo\AppData\Local\Programs\Python\Python36\python.exe

• 3.5.4: C:\Users\Moo\AppData\Local\Programs\Python\Python35\python.exe

• 3.6.1: C:\Windows\py.exe

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.1',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.17134',
 'python_full_version': '3.6.1',
 'python_version': '3.6',
 'sys_platform': 'win32'}

System environment variables:

• ALLUSERSPROFILE
• APPDATA
• COMMONPROGRAMFILES
• COMMONPROGRAMFILES(X86)
• COMMONPROGRAMW6432
• COMPOSE_CONVERT_WINDOWS_PATHS
• COMPUTERNAME
• COMSPEC
• DOCKER_CERT_PATH
• DOCKER_HOST
• DOCKER_MACHINE_NAME
• DOCKER_TLS_VERIFY
• DOCKER_TOOLBOX_INSTALL_PATH
• DRIVERDATA
• FP_NO_HOST_CHECK
• HOME
• HOMEDRIVE
• HOMEPATH
• LOCALAPPDATA
• LOGONSERVER
• NO_PROXY
• NUMBER_OF_PROCESSORS
• ONEDRIVE
• OS
• PATH
• PATHEXT
• PIPENV_VENV_IN_PROJECT
• PROCESSOR_ARCHITECTURE
• PROCESSOR_IDENTIFIER
• PROCESSOR_LEVEL
• PROCESSOR_REVISION
• PROGRAMDATA
• PROGRAMFILES
• PROGRAMFILES(X86)
• PROGRAMW6432
• PSMODULEPATH
• PUBLIC
• SESSIONNAME
• SYSTEMDRIVE
• SYSTEMROOT
• TEMP
• TMP
• USERDOMAIN
• USERDOMAIN_ROAMINGPROFILE
• USERNAME
• USERPROFILE
• VBOX_MSI_INSTALL_PATH
• VS140COMNTOOLS
• VSCODE_CWD
• WINDIR
• TERM_PROGRAM
• TERM_PROGRAM_VERSION
• LANG
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH

Pipenv?specific environment variables:

• PIPENV_VENV_IN_PROJECT: true

Debug?specific environment variables:

• PATH: C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files (x86)\ctags58;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\PuTTY\;C:\Users\Moo\AppData\Local\Programs\Python\Python36\Scripts\;C:\Users\Moo\AppData\Local\Programs\Python\Python36\;C:\Users\Moo\AppData\Local\Programs\Python\Python35\Scripts\;C:\Users\Moo\AppData\Local\Programs\Python\Python35\;C:\Users\Moo\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Users\Moo\AppData\Local\Microsoft\WindowsApps;C:\Program Files\Microsoft VS Code\bin;C:\Program Files\Docker Toolbox;C:\Users\Moo\AppData\Local\Programs\Microsoft VS Code\bin;c:\users\Moo\appdata\local\programs\python\python36\lib\site-packages\pywin32_system32
• LANG: en_US.UTF-8

---

Contents of Pipfile ('C:\Users\Moo\Desktop\projects\test\Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
bsddb3 = {file = "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl", sys_platform = "== 'win32'", platform_machine = "== 'AMD64'"}

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('C:\Users\Moo\Desktop\projects\test\Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "a47b65d9674071fb07079ca13c8e7c4df34f8c1ebfd624846dfcc9316acd1455"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "bsddb3": {
            "file": "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl",
            "hashes": [
                "sha256:42d621f4037425afcb16b67d5600c4556271a071a9a7f7f2c2b1ba65bc582d05"
            ],
            "index": "pypi",
            "markers": "sys_platform == 'win32'",
            "version": "==6.2.6"
        }
    },
    "develop": {}
}

[techalchemy]

Huh. This is incredibly weird. I can definitely reproduce it, but I am not really sure why....

There is no wheel available on pypi, and the wheel cache has digest which matches what pip says.... I ran it myself by hand

This will need some investigation

[techalchemy]

/cc @uranusjr something we should see about with passa

[thernstig]

@techalchemy Any ideas why this happens? Maybe this should be closed as part of #3146?

[techalchemy]

Ah most likely so, will have to check!

[uranusjr]

Also related: pypa/pip#4344

[BramVanroy]

I'd like to add my ticket to this thread.

I have tested this out on Windows 10 with different URLs on Python 3.6 and 3.7 but I always get the same issue, namely THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE.. But I found that, for instance with torch, the genereated Pipfile.lock (which failed to install) would look like this:

{
    "_meta": {
        "hash": {
            "sha256": "87323994796c9e00db1071ff23c430265176a50a2770dcf739cf5641e107fcd5"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "torch": {
            "file": "https://download.pytorch.org/whl/cpu/torch-1.0.0-cp36-cp36m-win_amd64.whl",
            "hashes": [
                "sha256:012a9c7efce86c7a0ce78cd7945fe7c798049537fc3e85af9f14e8789d13c17f",
                "sha256:4aadc7124afc431ac6a227a05dc8eff417b6fd8f90fc6c44d514ddfca9a6b474",
                "sha256:53e12607830ccb1e5fc4076aafe19bdbbc380799793fbaad696714b72859bde6",
                "sha256:7e73a141bf817c0a914131dec51ea24a2f1946b96749b003af664230a9b95197",
                "sha256:cb92ac65fcc7685fa6c5920b24101182dcb706d841fc6154ada604f749b615e3",
                "sha256:cedbc382a0e992a169c73d2c469887c2e5ce0c6fa88b1dabe8f9021e1acb564f",
                "sha256:ded9e2e59c086127423c23e902e2bec42b3b443a0e458fae76c013f62a7e0748",
                "sha256:df005dff3e3f12911630e48e0e75d3594a424a317d785b49426c23d0810a4682",
                "sha256:f4196ce8ba17797f3c2d13c0d53cf461a8e32f6130a08e6e4ce917637afccdc6"
            ],
            "index": "pypi",
            "version": "==1.0.0"
        }
    },
    "develop": {}
}

I'm not sure why a pypi index is chosen for packages that are downloaded from a URL anyway. pipenv --lock does not solve anything. The only thing that worked for me was: removing the hashes (and maybe also the index) from Pipfile.lock, saving it, and the running pipenv sync which installs packages without a re-lock or verification if I'm not mistaken. Working lock file:

{
    "_meta": {
        "hash": {
            "sha256": "87323994796c9e00db1071ff23c430265176a50a2770dcf739cf5641e107fcd5"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "torch": {
            "file": "https://download.pytorch.org/whl/cpu/torch-1.0.0-cp36-cp36m-win_amd64.whl",
            "version": "==1.0.0"
        }
    },
    "develop": {}
}

As soon as you install another package, the same issue will arise though and you will have to go through the same process of removing the hashes and syncing.