Bug: 404 error while calculating hashes with 2023.8.28

[matejsp]

Issue description

when upgrading from pipenv 2023.7.3 to 2023.8.28. First there is very verbose logging for INFO:pipenv.patched.pip._internal.operations.prepare. In another bug.

For some repostories (like nexus) that used to return md5 instead of sha256, pipenv downloads the file and calculates the hash.
It seems that automatic calculation of hash in case md5 is used in nexus it uses incorrect url.

INFO:pipenv.patched.pip._internal.operations.prepare:Collecting wcmatch==8.3 (from semgrep==0.78.0->-r /var/folders/l6/nmk965l50gv_90_c4xh5x0_4ctpk94/T/pipenv-jgl0rhw4-requirements/pipenv-an17je3b-constraints.txt (line 19))
INFO:pipenv.patched.pip._internal.network.download:Using cached https://nexus.company.net/repository/pypi-blessed/packages/wcmatch/8.3/wcmatch-8.3-py3-none-any.whl (42 kB)

Downloading file wcmatch-8.3-py3-none-any.whl to obtain hash...
HTTP error 404 while getting https://u:p@nexus.company.net/packages/wcmatch/8.3/wcmatch-8.3-py3-none-any.whl#md5=8d2acdbf5586e2175b9f88d16e599ac8
Downloading file wcmatch-8.3.tar.gz to obtain hash...
HTTP error 404 while getting https://u:p@nexus.company.net/packages/wcmatch/8.3/wcmatch-8.3.tar.gz#md5=d3e45a9f6853ebbb4d64c60d279de2f8
Downloading file wcmatch-8.3.tar.gz to obtain hash...
Downloading file wcmatch-8.3-py3-none-any.whl to obtain hash...

When resolving downloading hashes part of url is missing. But it works when downloading original whl or tar.gz.

[[source]]
name = "nexus"
url = "https://u:p@nexus.company.net/repository/pypi-blessed/simple"
verify_ssl = true

Expected result

It should download from correct url and calculate the hash.

Actual result

It fetches from wrongly build url and gets 404 from the nexus.

Steps to replicate

It seems that this behaviour works for https://pypi.python.org/simple because /simple is at root.
However for https://u:p@nexus.company.net/repository/pypi-blessed/simple it does not.

Pipfile:
[[source]]
name = "nexus"
url = "https://u:p@nexus.company.net/repository/pypi-blessed/simple"
verify_ssl = true

---

[matejsp]

It seems that hashing seems quite broken in pipenv currently.
Another issue with missing hashes for different naming (case sensitivity, dot vs dash)

What we have in nexus:

and with upgraded pipenv all the hashes are removed.

Same story for pyyaml:

and removed hahes:

Or dots vs dashes:

MarkupSafe is another case for broken hashes.

[matteius]

@matejsp What makes you say it has to do with /simple/ -- from what I can see for non pypi sources, it is trying to find the hashes and package links from:

pkg_url = f"{source['url']}/{ireq.name}/"

Does https://u:p@nexus.company.net/repository/pypi-blessed/simple/{package_name} exist in your case?

I notice in your logs:
https://u:p@nexus.company.net/packages/wcmatch/8.3/wcmatch-8.3.tar.gz#md5=d3e45a9f6853ebbb4d64c60d279de2f8 but that seems to be missing the /repository/pypi-blessed/simple part of the path from your original source URL.

This is high on my list to get to the bottom of, your help is appreciated.

[matteius]

My current best guess is there is a relative href on an index that is like https://u:p@nexus.company.net/repository/pypi-blessed/simple/{package_name} and that its trying to get the package url from package_url = urljoin(source["url"], package_url)
since your source has a path like /repository/pypi-blessed/simple/ its getting truncated and replaced with the partial relative path on that index page. Does this sound accurate?

[matejsp]

Yes first url exists and where whl were always downloaded from, first line from is downloading (or using from cache):
https://nexus.company.net/repository/pypi-blessed/packages/wcmatch/8.3/wcmatch-8.3-py3-none-any.whl

versus (wrongly concated)

https://u:p@nexus.company.net/packages/wcmatch/8.3/wcmatch-8.3.tar.gz#md5=d3e45a9f6853ebbb4d64c60d279de2f8

Md5 is returned when downloading the package from nexus. old uploaded ones have md5 new one have sha256 (but no way to fix old ones).

[matteius]

@matejsp I installed a nexus server locally and explored the issue more, please try out this patch: #5898

[matejsp]

So I installed via (btw it would be nice to have it in readme how install itself):
pipenv install -e git+https://github.com/pypa/pipenv.git@hash-collection-refinements#egg=pipenv

And run it across our setup and it works!
It correctly identifies all the hashes (small differences in naming for wheels).
What I like is that this version also correctly resolves and does not lock:

Requires-Dist: exceptiongroup ; python_version < "3.11"

Good job, looking forward to a release with fixed logging and this.

[matteius]

2023.9.1 has been released