Pip 21 performs one unneeded fetch when -U is supplied #9516
Labels
type: bug
A confirmed bug or unintended behavior
Comments
|
The upgrade ( |
|
It seems that |
uranusjr
changed the title
|
i don't know about "one" for me if i try and update https://pypi.org/project/psycopg2/ it tries to download all the linux versions too one by one until it ends up on 2 functioning windows ports in a row. |
|
Can you paste the command you used and the output it generated? |
|
|
Probably 20.3.4 had the issue of the unneeded fetch too. Only reverting to 20.3.3 fixes the issue for me. |
|
it did not, but downgrading from 21 keeps the problem. you probably have it fixed now because you went one step further below |
|
Downgrading to 20.3.3 and then upgrading to 20.3.4 causes the error to come back. This means that 20.3.3 is able to ignore something, but does not restore it to a sane state. The fact that pip 21 causes pip 20.3.4 that otherwise would work fine to behave weirdly suggesta that pip 21 corrupts some piece of "system state". And where is system state stored? May I suggest that pip 21 is retired from being automatically proposed for an upgrade as in |
|
Or maybe 20.3.4 really has the problem too: in a clean (new) virtualenv where I only set pip.conf to not use the cache and I only did doing |
|
Pure guess, but could this change in 21.0 be relevant? "Remove support for legacy wheel cache entries that were created with pip versions older than 20.0 (#7502)". |
|
@callegar i was wrong. I was sure i had version 20.3.4 before updating to 21.0 as i do automatic daily updates and i was sure that the pip updates happened every 2 weeks, so i assumed "the mess" only started happening with version 21. But i see now that daily updates means i skipped version 20.3.4 as it happened only 3 hours before 21. My bad. |
|
I did some debugging. The cause to this is similar to #9432. pip would perform exactly one additional download (or cache access) when @Butterfly-Dragon pip is downloading multiple versions for backtracking, so you see more than one version being downloaded, to find a working combination of your dependencies. But only exactly one of the downloaded versions from each project is superfulous (the one matching the already-installed version). |
|
@uranusjr yeah, i got that... it... should resolve dependencies without having to download everything, even for incompatible operating systems. 😅 As you see it tires to install Linux versions under windows 10. I don't know if that might be due to me installing the linux support under windows (because reasons) but ... it should not work like that. |
|
Would it be possible if you send me your requirements.txt? I am working on a fix and it would be a good test case. My email is available in my GitHub profile if you’re not comfortable with posting it publicly. |
|
requirements.txt |
|
@uranusjr, with reference to your comment #9516 (comment), I would like to ask for a quick clarification. If I understand correctly, after 20.3.3 pip tries to get (either from download or from the cache) two versions of every package may be upgradable. One is the upgrade candidate and the other is the current version. My understanding from your message is that this is for enabling backtracking if the upgrade is not possible or if something goes wrong with it. Is this the case and what is planned to always happen after 20.3.3? Because 20.3.3 was not doing this, and was working reasonably well, I would like to ask what specific issue the change is trying to solve, because I am having the impression that there is a trade off with other issues. One of them is that whenever the cache cannot be used, this is basically doubling internet traffic, which is of some concern for those under metered connections (some ISPs, connections via UMTS/CDMA, some virtual private servers having networking caps, etc.) or under slow connections (since in these cases it is making package updates twice as slow). Because almost no other package manager does this (deb, rpm, conda, tlmgr), I wonder if this is really necessary. In principle, it should be only the package metadata, not the whole package that is needed to establish the possibility for an upgrade and the workable version combinations. Furthermore, given that you have the package already installed, wouldn't it be possible to backup that one, or move it to a safe location, rather than re-downloading it to account for the need of a backtrace? |
|
From what I can tell with the available information, the resolver needs to backtrack when it resolves the dependency set you give to it. This is caused by something that changed between 20.3.3 and 21.0, and may or may not be related to the issue being discussed here.
No, not always. If the already installed version is satisfactory, the resolver would not need to backtrack. But if not, backtrack will be needed and the additional download would happen. But I don’t know which is the case, which is why I asked for your requirements.txt. |
|
i can confirm that flushing the cache and on a new 3.9.1 python install... this still happens. Ate least 2 packages are always downloaded. |
|
I proposed a fix to the issue described in the top post in #9522. It would be nice if you could test whether it affects your issue as well @Butterfly-Dragon. |
|
it's currently blocked and i'm no programmer anymore, i was at the time of fortran 🤣👍💖 |
bors bot
referenced
this issue
in duckinator/emanate
Jan 31, 2021
215: Update pip to 21.0.1 r=duckinator a=pyup-bot This PR updates [pip](https://pypi.org/project/pip) from **20.3.3** to **21.0.1**. <details> <summary>Changelog</summary> ### 21.0.1 ``` =================== Bug Fixes --------- - commands: debug: Use packaging.version.parse to compare between versions. (`9461 <https://github.com/pypa/pip/issues/9461>`_) - New resolver: Download and prepare a distribution only at the last possible moment to avoid unnecessary network access when the same version is already installed locally. (`9516 <https://github.com/pypa/pip/issues/9516>`_) Vendored Libraries ------------------ - Upgrade packaging to 20.9 ``` ### 21.0 ``` ================= Deprecations and Removals ------------------------- - Drop support for Python 2. (`6148 <https://github.com/pypa/pip/issues/6148>`_) - Remove support for legacy wheel cache entries that were created with pip versions older than 20.0. (`7502 <https://github.com/pypa/pip/issues/7502>`_) - Remove support for VCS pseudo URLs editable requirements. It was emitting deprecation warning since version 20.0. (`7554 <https://github.com/pypa/pip/issues/7554>`_) - Modernise the codebase after Python 2. (`8802 <https://github.com/pypa/pip/issues/8802>`_) - Drop support for Python 3.5. (`9189 <https://github.com/pypa/pip/issues/9189>`_) - Remove the VCS export feature that was used only with editable VCS requirements and had correctness issues. (`9338 <https://github.com/pypa/pip/issues/9338>`_) Features -------- - Add ``--ignore-requires-python`` support to pip download. (`1884 <https://github.com/pypa/pip/issues/1884>`_) - New resolver: Error message shown when a wheel contains inconsistent metadata is made more helpful by including both values from the file name and internal metadata. (`9186 <https://github.com/pypa/pip/issues/9186>`_) Bug Fixes --------- - Fix a regression that made ``pip wheel`` do a VCS export instead of a VCS clone for editable requirements. This broke VCS requirements that need the VCS information to build correctly. (`9273 <https://github.com/pypa/pip/issues/9273>`_) - Fix ``pip download`` of editable VCS requirements that need VCS information to build correctly. (`9337 <https://github.com/pypa/pip/issues/9337>`_) Vendored Libraries ------------------ - Upgrade msgpack to 1.0.2. - Upgrade requests to 2.25.1. Improved Documentation ---------------------- - Render the unreleased pip version change notes on the news page in docs. (`9172 <https://github.com/pypa/pip/issues/9172>`_) - Fix broken email link in docs feedback banners. (`9343 <https://github.com/pypa/pip/issues/9343>`_) .. note You should *NOT* be adding new change log entries to this file, this file is managed by towncrier. You *may* edit previous change logs to fix problems like typo corrections or such. To add a new change log entry, please see https://pip.pypa.io/en/latest/development/contributing/#news-entries .. towncrier release notes start ``` ### 20.3.4 ``` =================== Features -------- - ``pip wheel`` now verifies the built wheel contains valid metadata, and can be installed by a subsequent ``pip install``. This can be disabled with ``--no-verify``. (`9206 <https://github.com/pypa/pip/issues/9206>`_) - Improve presentation of XMLRPC errors in pip search. (`9315 <https://github.com/pypa/pip/issues/9315>`_) Bug Fixes --------- - Fixed hanging VCS subprocess calls when the VCS outputs a large amount of data on stderr. Restored logging of VCS errors that was inadvertently removed in pip 20.2. (`8876 <https://github.com/pypa/pip/issues/8876>`_) - Fix error when an existing incompatibility is unable to be applied to a backtracked state. (`9180 <https://github.com/pypa/pip/issues/9180>`_) - New resolver: Discard a faulty distribution, instead of quitting outright. This implementation is taken from 20.2.2, with a fix that always makes the resolver iterate through candidates from indexes lazily, to avoid downloading candidates we do not need. (`9203 <https://github.com/pypa/pip/issues/9203>`_) - New resolver: Discard a source distribution if it fails to generate metadata, instead of quitting outright. This implementation is taken from 20.2.2, with a fix that always makes the resolver iterate through candidates from indexes lazily, to avoid downloading candidates we do not need. (`9246 <https://github.com/pypa/pip/issues/9246>`_) Vendored Libraries ------------------ - Upgrade resolvelib to 0.5.4. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pip - Changelog: https://pyup.io/changelogs/pip/ - Homepage: https://pip.pypa.io/ </details> Co-authored-by: pyup-bot <github-bot@pyup.io>
1 task
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Environment
Description
See the following example:
Why is
cvxoptbeing collected and downloaded if the requirement about it is already satisfied?Why is the past version of
cvxoptalso being downloaded?I have the suspicion that this may have something to do with the fact that I have the package cache disabled with
no-cache-dir = falseinpip.confand that the cache code has been refactored in this version.Expected behavior
If a requirement is satisfied, then
pipshould not download anything. If a requirement is not satisfied,pipshould download its code at the required version only.How to Reproduce
pipcache (probably needed)cvxopt)pip -install -UOutput
The text was updated successfully, but these errors were encountered: