Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[2020-resolver] Resolver fails to install editable when self in dependencies #9143

Closed
jaraco opened this issue Nov 18, 2020 · 10 comments
Closed
Labels
type: bug A confirmed bug or unintended behavior

Comments

@jaraco
Copy link
Member

jaraco commented Nov 18, 2020

In jaraco/pytest-checkdocs#6, I discovered an apparent regression with the resolver. With the old resolver, installing a package in editable mode would satisfy dependencies. Wtih the new resolver, a separate installation pytest-checkdocs==2.1.1 gets installed even though pytest-checkdocs-2.1.2dev17+g3df4d42 would be installed (the local checkout).

~ $ git clone gh://jaraco/pytest-checkdocs
~ $ cd pytest-checkdocs
pytest-checkdocs master $ python -m venv env
pytest-checkdocs master $ env PIP_USE_FEATURE=2020-resolver env/bin/pip install -e '.[testing]'
Obtaining file:///Users/jaraco/code/main/pytest-checkdocs
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting jaraco.test>=3.2.0
  Using cached jaraco.test-3.2.0-py3-none-any.whl (5.9 kB)
Collecting pytest-checkdocs>=1.2.3
  Using cached pytest_checkdocs-2.1.1-py3-none-any.whl (4.1 kB)
Collecting docutils>=0.15
  Using cached docutils-0.16-py2.py3-none-any.whl (548 kB)
Processing /Users/jaraco/Library/Caches/pip/wheels/06/5c/1c/dcab820c7e72f8794a11e3d73aa6308c1fe264736b4fa0bd35/pytest_black-0.3.12-py3-none-any.whl
Collecting jaraco.context
  Using cached jaraco.context-3.0.0-py2.py3-none-any.whl (5.8 kB)
Collecting jaraco.apt
  Using cached jaraco.apt-2.0-py2.py3-none-any.whl (3.2 kB)
Collecting yg.lockfile
  Using cached yg.lockfile-2.3-py2.py3-none-any.whl (6.8 kB)
Collecting jaraco.functools
  Using cached jaraco.functools-3.0.1-py3-none-any.whl (6.7 kB)
Collecting zc.lockfile
  Using cached zc.lockfile-2.0-py2.py3-none-any.whl (9.7 kB)
Requirement already satisfied: setuptools in ./env/lib/python3.9/site-packages (from zc.lockfile->yg.lockfile->jaraco.context->jaraco.test>=3.2.0->pytest-checkdocs==2.1.2.dev17+g3df4d42) (49.2.1)
Collecting pytest-mypy
  Using cached pytest_mypy-0.8.0-py3-none-any.whl (6.6 kB)
Collecting mypy>=0.780
  Using cached mypy-0.790-py3-none-any.whl (2.4 MB)
Collecting mypy-extensions<0.5.0,>=0.4.3
  Using cached mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB)
Collecting typed-ast<1.5.0,>=1.4.0
  Using cached typed_ast-1.4.1-cp39-cp39-macosx_10_15_x86_64.whl (225 kB)
Collecting typing-extensions>=3.7.4
  Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting attrs>=19.0
  Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
Collecting filelock>=3.0
  Using cached filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting jaraco.collections
  Using cached jaraco.collections-3.0.0-py2.py3-none-any.whl (9.4 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting jaraco.classes
  Using cached jaraco.classes-3.1.0-py2.py3-none-any.whl (5.7 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting pytest-flake8
  Using cached pytest_flake8-1.0.6-py2.py3-none-any.whl (6.3 kB)
Collecting flake8>=3.5
  Using cached flake8-3.8.4-py2.py3-none-any.whl (72 kB)
Collecting pyflakes<2.3.0,>=2.2.0
  Using cached pyflakes-2.2.0-py2.py3-none-any.whl (66 kB)
Collecting pycodestyle<2.7.0,>=2.6.0a1
  Using cached pycodestyle-2.6.0-py2.py3-none-any.whl (41 kB)
Collecting mccabe<0.7.0,>=0.6.0
  Using cached mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
Processing /Users/jaraco/Library/Caches/pip/wheels/4e/57/9a/e704bdd859ee892dc46fff03fd499422dc9e99fd9bd5c446d3/black-20.8b1-py3-none-any.whl
Collecting click>=7.1.2
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting pathspec<1,>=0.6
  Using cached pathspec-0.8.1-py2.py3-none-any.whl (28 kB)
Collecting appdirs
  Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting regex>=2020.1.8
  Using cached regex-2020.11.13-cp39-cp39-macosx_10_9_x86_64.whl (284 kB)
Collecting jaraco.text
  Using cached jaraco.text-3.2.0-py2.py3-none-any.whl (8.1 kB)
Collecting tempora
  Using cached tempora-4.0.1-py3-none-any.whl (14 kB)
Collecting more-itertools
  Using cached more_itertools-8.6.0-py3-none-any.whl (45 kB)
Collecting pytest-cov
  Using cached pytest_cov-2.10.1-py2.py3-none-any.whl (19 kB)
Collecting coverage>=4.4
  Using cached coverage-5.3-cp39-cp39-macosx_10_13_x86_64.whl (205 kB)
Collecting pytest!=3.7.3,>=3.5
  Using cached pytest-6.1.2-py3-none-any.whl (272 kB)
Collecting py>=1.8.2
  Using cached py-1.9.0-py2.py3-none-any.whl (99 kB)
Collecting pluggy<1.0,>=0.12
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging
  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pytz
  Using cached pytz-2020.4-py2.py3-none-any.whl (509 kB)
Installing collected packages: more-itertools, pytz, jaraco.functools, zc.lockfile, tempora, six, pyparsing, yg.lockfile, typing-extensions, typed-ast, toml, regex, pyflakes, pycodestyle, py, pluggy, pathspec, packaging, mypy-extensions, mccabe, jaraco.text, jaraco.classes, jaraco.apt, iniconfig, click, attrs, appdirs, pytest, mypy, jaraco.context, jaraco.collections, flake8, filelock, docutils, coverage, black, pytest-mypy, pytest-flake8, pytest-cov, pytest-checkdocs, pytest-black, jaraco.test
Successfully installed appdirs-1.4.4 attrs-20.3.0 black-20.8b1 click-7.1.2 coverage-5.3 docutils-0.16 filelock-3.0.12 flake8-3.8.4 iniconfig-1.1.1 jaraco.apt-2.0 jaraco.classes-3.1.0 jaraco.collections-3.0.0 jaraco.context-3.0.0 jaraco.functools-3.0.1 jaraco.test-3.2.0 jaraco.text-3.2.0 mccabe-0.6.1 more-itertools-8.6.0 mypy-0.790 mypy-extensions-0.4.3 packaging-20.4 pathspec-0.8.1 pluggy-0.13.1 py-1.9.0 pycodestyle-2.6.0 pyflakes-2.2.0 pyparsing-2.4.7 pytest-6.1.2 pytest-black-0.3.12 pytest-checkdocs-2.1.1 pytest-cov-2.10.1 pytest-flake8-1.0.6 pytest-mypy-0.8.0 pytz-2020.4 regex-2020.11.13 six-1.15.0 tempora-4.0.1 toml-0.10.2 typed-ast-1.4.1 typing-extensions-3.7.4.3 yg.lockfile-2.3 zc.lockfile-2.0
WARNING: You are using pip version 20.2.3; however, version 20.2.4 is available.
You should consider upgrading via the '/Users/jaraco/code/main/pytest-checkdocs/env/bin/python -m pip install --upgrade pip' command.

Why would you declare a dependency on self? The declared dependency is part of a package template used across hundreds of packages. It would be possible to customize that (test) dependency (omit it) just for this package, and doing so does work around the issue, but it should also be the case that if . satisfies one of the dependencies, that dependency shouldn't be downloaded/installed, or at the very least, the editable copy should supersede the downloaded version.

Since this behavior worked as I would expect prior to the 2020-resolver, it would be nice to restore that behavior.

@uranusjr
Copy link
Member

uranusjr commented Nov 18, 2020

I believe this is a variant to #8711 (wrong issue, see below)

@brainwane brainwane added the type: bug A confirmed bug or unintended behavior label Nov 29, 2020
@brainwane
Copy link
Contributor

I just spoke about this issue with @pradyunsg. This regression is not strictly a duplicate of other issues, but is probably being solved or partially addressed by pull requests subsequent to Jason opening the issue.

It's a tricky problem! The user is installing a package. Within the dependency graph, they see the same package. But since pip is doing an editable install of a local directory, the resolver doesn't realize it's the same package, and does the wrong things.

This may be difficult to fix -- that's a little unknown right now -- and we probably have not gotten enough testing of editable environments, during the beta, compared to testing other setups/use cases. We can also assume that approximately everyone dealing with this problem will be a Python developer (with a little more level of skill, at least, than our less-expert users).

Since the 20.3 release #8936 is so delayed compared to our plans, I have made the call to defer this particular issue to post-release work. In the meantime, until we fix this (maybe in a point release between now and 21.0), the user may want to use --use-deprecated=legacy-resolver. Sorry.

@jaraco
Copy link
Member Author

jaraco commented Nov 29, 2020

Thanks for the update and the diligent work. No worries on the delay. I recognize that it's a difficult problem. Since there's an escape hatch, that's probably acceptable. Thanks for everything.

@nshgraph
Copy link

nshgraph commented Dec 1, 2020

This is affecting us also. Thanks for the workaround.

@uranusjr
Copy link
Member

uranusjr commented Dec 3, 2020

Merging the conversation with #8785 to avoid messages from scattering.

@jaraco
Copy link
Member Author

jaraco commented Jan 10, 2021

I encountered this issue again in python/importlib_metadata#273. In this case, importlib_metadata, the project under test, is an indirect dependency of the test dependencies, but still triggers the behavior. I was testing with pip 20.3.1.

jaraco added a commit to jaraco/zipp that referenced this issue Jan 30, 2021
clrpackages pushed a commit to clearlinux-pkgs/zipp that referenced this issue Mar 9, 2021
Jason R. Coombs (18):
      Use inline flags with local scope.
      Honor TOX_WORK_DIR if set. Workaround for tox-dev/tox#20.
      Collapse skeleton history from archive/2020-12
      Update skeleton description to describe the periodic collapse. Fixes #27.
      Enable automerge
      Automatically inject project name in docs heading.
      pre-commit autoupdate
      Rename 'Automated Tests' to simply 'tests'
      Add note about automatic merging of PRs and the requirements and limitations.
      Prefer pytest-enabler to jaraco.test
      Enable complexity limit. Fixes jaraco/skeleton#34.
      Add support for namespace packages. Closes jaraco/skeleton#40.
      Disable 2020-resolver. Workaround for pypa/pip#9143.
      Normalize indentation
      Exclude dist from discovered packages. Fixes jaraco/skeleton#46.
      It's no longer necessary to filter this warning and it's not a warning anymore.
      Bump minimum pytest
      Update changelog.

KOLANICH (1):
      Added an .editorconfig. Pull request jaraco/skeleton#43.

Sviatoslav Sydorenko (3):
      Replace pep517.build with build (#37)
      Use license_files instead of license_file in meta (#35)
      Use `extend-ignore` in flake8 config (#33)
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 4, 2021
@jaraco
Copy link
Member Author

jaraco commented Nov 8, 2021

In sphinx-doc/sphinx#9825 and tox-dev/tox#2272, I found that the workaround for this issue (pinning to legacy resolver) was causing other problems (legacy resolver was installing conflicting requirements).

In jaraco/pytest-checkdocs@ba62865, I attempted to restore the 2020 resolver, but now the tests fail with this error:

The conflict is caused by:
    pytest-checkdocs[testing] 0.1.dev1+gba62865 depends on pytest-checkdocs 0.1.dev1+gba62865 (from /home/runner/work/pytest-checkdocs/pytest-checkdocs)
    pytest-checkdocs[testing] 0.1.dev1+gba62865 depends on pytest-checkdocs>=2.4; extra == "testing"

Somehow it seems that the package is being assigned a version of 0.1.dev1 even though the version in metadata is 2.7.2.dev:

pytest-checkdocs main $ pip-run -q jaraco.develop -- -m jaraco.develop.print-meta -f Version
2.7.2.dev11+gba62865

Is this bug still a duplicate of #8785?

@jaraco jaraco reopened this Nov 8, 2021
@jaraco
Copy link
Member Author

jaraco commented Nov 8, 2021

Interestingly, I don't encounter the issue when I test locally (macOS), so maybe there's something about the GHA environment that has an older virtualenv/pip that doesn't include the fix. I see the failed tests get the latest virtualenv (20.10.0), so that's likely not the issue.

@jaraco
Copy link
Member Author

jaraco commented Nov 8, 2021

Okay. I see GHA checks out the repo with --depth=1, so that's probably the issue (not enough repo history to detect the last version tag). I'll experiment with some options to get a bit more history.

@jaraco
Copy link
Member Author

jaraco commented Nov 8, 2021

It looks like in jaraco/pytest-checkdocs#16, I have a working fix. Thanks for fixing the issue in pip!

@jaraco jaraco closed this as completed Nov 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

No branches or pull requests

4 participants