Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metadata inconsistent error message should contain all metadata caused the inconsistency #9186

Closed
larsoner opened this issue Nov 30, 2020 · 14 comments · Fixed by #9469
Closed
Labels
C: error messages Improving error messages type: enhancement Improvements to functionality

Comments

@larsoner
Copy link

  • pip version: 20.3
  • Python version: 3.9 or 3.8 (only two I've tested)
  • OS: Linux or Windows (only two I've tested)

See numpy/numpy#17885 (comment) -- in 20.2.4 this works:

$ python3.9 -m pip install --progress-bar off --upgrade "pip!=20.3"
...
Successfully installed pip-20.2.4
$ python3.9 -m pip install --progress-bar off --upgrade --pre --only-binary ":all:" -i "https://pypi.anaconda.org/scipy-wheels-nightly/simple" numpy
...
Collecting numpy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2Bd0a457f/numpy-1.21.0.dev0%2B20201129035606_d0a457f-cp38-cp38-manylinux2010_x86_64.whl (15.4 MB)

Installing collected packages: numpy
  Attempting uninstall: numpy
    Found existing installation: numpy 1.21.0.dev0+d3cffdc
    Uninstalling numpy-1.21.0.dev0+d3cffdc:
      Successfully uninstalled numpy-1.21.0.dev0+d3cffdc
Successfully installed numpy-1.21.0.dev0+d0a457f

But with 20.4:

$ python3.9 -m pip install --progress-bar off --upgrade --pre --only-binary ":all:" -i "https://pypi.anaconda.org/scipy-wheels-nightly/simple" numpy
...
Collecting numpy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2Bd0a457f/numpy-1.21.0.dev0%2B20201129040310_d0a457f-cp39-cp39-manylinux2010_x86_64.whl (15.4 MB)

ERROR: Requested numpy from https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2Bd0a457f/numpy-1.21.0.dev0%2B20201129040310_d0a457f-cp39-cp39-manylinux2010_x86_64.whl has different version in metadata: '1.21.0.dev0+d0a457f'
@dstufft
Copy link
Member

dstufft commented Nov 30, 2020

I'm guessing a call to urlunquote needs added.

@brainwane brainwane added state: needs eyes Needs a maintainer/triager to take a closer look S: needs triage Issues/PRs that need to be triaged labels Nov 30, 2020
@uranusjr
Copy link
Member

Weird, the new resolver does not reimplement any HTTP logic, it’s all shared with the legacy resolver.

@pradyunsg pradyunsg added C: error messages Improving error messages type: enhancement Improvements to functionality and removed state: needs eyes Needs a maintainer/triager to take a closer look S: needs triage Issues/PRs that need to be triaged labels Nov 30, 2020
@pradyunsg pradyunsg added this to the 20.3.1 milestone Nov 30, 2020
@pradyunsg
Copy link
Member

pradyunsg commented Nov 30, 2020

diff --git a/src/pip/_internal/resolution/resolvelib/candidates.py b/src/pip/_internal/resolution/resolvelib/candidates.py
index 1fc2ff479..a50e888c6 100644
--- a/src/pip/_internal/resolution/resolvelib/candidates.py
+++ b/src/pip/_internal/resolution/resolvelib/candidates.py
@@ -210,6 +210,8 @@ def _check_metadata_consistency(self, dist):
         if self._name is not None and self._name != name:
             raise MetadataInconsistent(self._ireq, "name", dist.project_name)
         version = dist.parsed_version
+        print(version)
+        print(self._version)
         if self._version is not None and self._version != version:
             raise MetadataInconsistent(self._ireq, "version", dist.version)

Gives:

1.21.0.dev0+d0a457f
1.21.0.dev0+20201129033659.d0a457f

I don't think there's a functionality bug here. The versions are actually different. We should probably present them both in the error message though, to avoid confusion like this.

@larsoner
Copy link
Author

larsoner commented Nov 30, 2020

I don't think there's a functionality bug here. The versions are actually different. We should probably present them both in the error message though, to avoid confusion like this.

From this it's clear at least that in the pip internals somewhere they are different in a way that they must not have been (or that did not matter) in 20.2.4. Can you clarify where these two versions get parsed from the wheel/setuptools/etc. so that the NumPy wheels can be fixed at their end?

@charris
Copy link

charris commented Nov 30, 2020

The date of wheel creation is added to the wheel names so that pip will pick up the latest. The wheel generation itself is automated, are you suggesting that the date should be added when setup is run? This used to work, so the question we are asking is: bug or feature.

@pradyunsg
Copy link
Member

@larsoner Sorry, I was very much in "triage all the notifications" mode when I looked at this and responded here.

pip's new resolver is stricter across the board for a lot of things related to a package's metadata. One of these is being stricter about the version information that pip sees about a package -- ensuring that what's provided from the METADATA file in that distribution (or as generated from that, in case of sdists) matches what the distribution can/may provide from other mechanisms -- in this case, the wheel filename.

What's happening here is that the wheel file (https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2Bd0a457f/numpy-1.21.0.dev0%2B20201129040310_d0a457f-cp39-cp39-manylinux2010_x86_64.whl) says the version is 1.21.0.dev0+20201129040310_d0a457f but all the metadata within the file says that it is 1.21.0.dev0+d0a457f.

The resolver is noticing this, and aborting since inconsistent metadata => can't rely on it => can't use it => AAAAAA. 😉

@pfmoore
Copy link
Member

pfmoore commented Nov 30, 2020

Also, just to be clear, this behaviour was present in 20.2.4, however it was opt-in only, as it's part of the new resolver. So the change here is actually just that the new resolver is now default. You should be able to demonstrate this by using the --use-feature=2020-resolver flag on pip 20.2.4, and you should see the same behaviour as 20.3. Conversely, you can (for a short period) opt out of the new resolver using --use-deprecated=legacy-resolver to get the old behaviour back. But this option will be getting dropped next release, so you should definitely only view this as a stopgap solution.

@charris
Copy link

charris commented Dec 1, 2020

How does pip handle build tags? Do they need to be internally specified and equivalent to those found in the file name?

@pradyunsg
Copy link
Member

pradyunsg commented Dec 1, 2020

@charris What do you mean by build tags? If you're referring to wheel's build tags, pip does exactly what the wheel spec says, as far as I know. https://www.python.org/dev/peps/pep-0427/#file-name-convention

@uranusjr
Copy link
Member

uranusjr commented Dec 1, 2020

They can be present in the version string, as long as the one listed in METADATA matches the one in the wheel’s file name. So the most simplistic fix would be to modify the METADATA content (and its corresponding entry in RECORD) to match.

@pradyunsg
Copy link
Member

I just tried pip install --upgrade --pre --only-binary ":all:" -i "https://pypi.anaconda.org/scipy-wheels-nightly/simple" numpy on Python 3.8 (MacOS), with pip's master branch and it worked.

❯ pip install --upgrade --pre --only-binary ":all:" -i "https://pypi.anaconda.org/scipy-wheels-nightly/simple" numpy 
Looking in indexes: https://pypi.anaconda.org/scipy-wheels-nightly/simple
Collecting numpy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.21.0.dev0%2B438.gbf4ac1f81/numpy-1.21.0.dev0%2B438.gbf4ac1f81-cp38-cp38-macosx_10_9_x86_64.whl (16.1 MB)
     |████████████████████████████████| 16.1 MB 401 kB/s 
Installing collected packages: numpy
Successfully installed numpy-1.21.0.dev0+438.gbf4ac1f81
❯ pip --version                                                                                                     
pip 21.0.dev0 from /Users/pradyunsg/Projects/pip/src/pip (python 3.8)
❯ pwd                                 
/Users/pradyunsg/Projects/pip
❯ git show HEAD                              
commit 7b7469b8454be1c0f968957e93ae23264187ee41 (HEAD -> master, upstream/master, upstream/HEAD, origin/master, origin/HEAD)
Merge: aa089883a a79758c13
Author: Paul Moore <p.f.moore@gmail.com>
Date:   Sat Jan 16 17:55:28 2021 +0000

    Merge pull request #9460 from pfmoore/vendor_21_0
    
    Vendoring changes for release 21.0 (excluding setuptools)

I think this either got fixed in our code, or on Anaconda's end. If someone else could provide a new reproducer for this or confirm that this is no longer an issue, that'll be great. :)

@charris
Copy link

charris commented Jan 17, 2021

@pradyunsg NumPy changed its versioning in master to use versioneer, so that might have fixed some things along with pip 20.3.

@uranusjr
Copy link
Member

uranusjr commented Jan 17, 2021

The numpy wheels now have consistent versions in metadata and filename, so I think it’s the versioneer change that fixed it for numpy. I think we can close this now? I’ll change the title to log the error message improvement instead.

@uranusjr uranusjr changed the title 20.3 fails to resolve NumPy pre wheel name properly Metadata inconsistent error message should contain all metadata caused the inconsistency Jan 17, 2021
@larsoner
Copy link
Author

I see the edit about the improved title, sounds good to me and thanks for the quick information and fixes!

bors bot referenced this issue in duckinator/emanate Jan 31, 2021
215: Update pip to 21.0.1 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.3.3** to **21.0.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 21.0.1
   ```
   ===================

Bug Fixes
---------

- commands: debug: Use packaging.version.parse to compare between versions. (`9461 &lt;https://github.com/pypa/pip/issues/9461&gt;`_)
- New resolver: Download and prepare a distribution only at the last possible
  moment to avoid unnecessary network access when the same version is already
  installed locally. (`9516 &lt;https://github.com/pypa/pip/issues/9516&gt;`_)

Vendored Libraries
------------------

- Upgrade packaging to 20.9
   ```
   
  
  
   ### 21.0
   ```
   =================

Deprecations and Removals
-------------------------

- Drop support for Python 2. (`6148 &lt;https://github.com/pypa/pip/issues/6148&gt;`_)
- Remove support for legacy wheel cache entries that were created with pip
  versions older than 20.0. (`7502 &lt;https://github.com/pypa/pip/issues/7502&gt;`_)
- Remove support for VCS pseudo URLs editable requirements. It was emitting
  deprecation warning since version 20.0. (`7554 &lt;https://github.com/pypa/pip/issues/7554&gt;`_)
- Modernise the codebase after Python 2. (`8802 &lt;https://github.com/pypa/pip/issues/8802&gt;`_)
- Drop support for Python 3.5. (`9189 &lt;https://github.com/pypa/pip/issues/9189&gt;`_)
- Remove the VCS export feature that was used only with editable VCS
  requirements and had correctness issues. (`9338 &lt;https://github.com/pypa/pip/issues/9338&gt;`_)

Features
--------

- Add ``--ignore-requires-python`` support to pip download. (`1884 &lt;https://github.com/pypa/pip/issues/1884&gt;`_)
- New resolver: Error message shown when a wheel contains inconsistent metadata
  is made more helpful by including both values from the file name and internal
  metadata. (`9186 &lt;https://github.com/pypa/pip/issues/9186&gt;`_)

Bug Fixes
---------

- Fix a regression that made ``pip wheel`` do a VCS export instead of a VCS clone
  for editable requirements. This broke VCS requirements that need the VCS
  information to build correctly. (`9273 &lt;https://github.com/pypa/pip/issues/9273&gt;`_)
- Fix ``pip download`` of editable VCS requirements that need VCS information
  to build correctly. (`9337 &lt;https://github.com/pypa/pip/issues/9337&gt;`_)

Vendored Libraries
------------------

- Upgrade msgpack to 1.0.2.
- Upgrade requests to 2.25.1.

Improved Documentation
----------------------

- Render the unreleased pip version change notes on the news page in docs. (`9172 &lt;https://github.com/pypa/pip/issues/9172&gt;`_)
- Fix broken email link in docs feedback banners. (`9343 &lt;https://github.com/pypa/pip/issues/9343&gt;`_)


.. note

    You should *NOT* be adding new change log entries to this file, this
    file is managed by towncrier. You *may* edit previous change logs to
    fix problems like typo corrections or such.

    To add a new change log entry, please see
        https://pip.pypa.io/en/latest/development/contributing/#news-entries

.. towncrier release notes start
   ```
   
  
  
   ### 20.3.4
   ```
   ===================

Features
--------

- ``pip wheel`` now verifies the built wheel contains valid metadata, and can be
  installed by a subsequent ``pip install``. This can be disabled with
  ``--no-verify``. (`9206 &lt;https://github.com/pypa/pip/issues/9206&gt;`_)
- Improve presentation of XMLRPC errors in pip search. (`9315 &lt;https://github.com/pypa/pip/issues/9315&gt;`_)

Bug Fixes
---------

- Fixed hanging VCS subprocess calls when the VCS outputs a large amount of data
  on stderr. Restored logging of VCS errors that was inadvertently removed in pip
  20.2. (`8876 &lt;https://github.com/pypa/pip/issues/8876&gt;`_)
- Fix error when an existing incompatibility is unable to be applied to a backtracked state. (`9180 &lt;https://github.com/pypa/pip/issues/9180&gt;`_)
- New resolver: Discard a faulty distribution, instead of quitting outright.
  This implementation is taken from 20.2.2, with a fix that always makes the
  resolver iterate through candidates from indexes lazily, to avoid downloading
  candidates we do not need. (`9203 &lt;https://github.com/pypa/pip/issues/9203&gt;`_)
- New resolver: Discard a source distribution if it fails to generate metadata,
  instead of quitting outright. This implementation is taken from 20.2.2, with a
  fix that always makes the resolver iterate through candidates from indexes
  lazily, to avoid downloading candidates we do not need. (`9246 &lt;https://github.com/pypa/pip/issues/9246&gt;`_)

Vendored Libraries
------------------

- Upgrade resolvelib to 0.5.4.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 4, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
C: error messages Improving error messages type: enhancement Improvements to functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants