Editable dependency requirements sometimes override top-level ones #3296

greysteil · 2018-11-24T10:54:40Z

Issue description

I'm not 100% sure how this is happening, but when locking a Pipfile that includes an editable requirement, sometimes the requirements from that dependency are "preferred" (i.e., overwrite) the top-level requirements for dependencies.

An example is probably the best illustration:

Pipfile

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
requests = { editable = true, git = 'https://github.com/requests/requests.git', ref = 'v2.18.4' }
urllib3 = "==1.21.1"

Result from calling pyenv exec pipenv lock:

{
    "_meta": {
        "hash": {
            "sha256": "71c49547d80da72c513f8e15ba6c8c070007d930c22dff508902f491b7665107"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:339dc09518b07e2fa7eda5450740925974815557727d6bd35d319c1524a04a4c",
                "sha256:6d58c986d22b038c8c0df30d639f23a3e6d172a05c3583e766f4c0b785c0986a"
            ],
            "version": "==2018.10.15"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "idna": {
            "hashes": [
                "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f",
                "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4"
            ],
            "version": "==2.6"
        },
        "requests": {
            "editable": true,
            "git": "https://github.com/requests/requests.git",
            "ref": "a3d7cf3f27e74c28ef30f01e9f2e483570ab042e"
        },
        "urllib3": {
            "hashes": [
                "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
            ],
            "version": "==1.22"
        }
    },
    "develop": {}
}

Expected result

The version of urllib3 should be 1.21.1.

Actual result

The version of urllib3 is 1.22.

Steps to replicate

Included in description. Happens on 3ce1394.

$ pipenv --support

Pipenv version: '2018.11.15.dev0'

Pipenv location: '/Users/greysteil/code/dependabot-script/.bundle/bundler/gems/dependabot-core-df7316130ca8/helpers/python/src/pipenv/pipenv'

Python location: '/Users/greysteil/.pyenv/versions/3.6.6/bin/python'

Python installations found:

3.7.0: /Users/greysteil/.pyenv/versions/3.7.0/bin/python3
3.7.0: /Users/greysteil/.pyenv/versions/3.7.0/bin/python3.7m
3.6.7: /Users/greysteil/.pyenv/versions/3.6.7/bin/python3
3.6.7: /Users/greysteil/.pyenv/versions/3.6.7/bin/python3.6m
3.6.6: /Users/greysteil/.pyenv/versions/3.6.6/bin/python3
3.6.6: /Users/greysteil/.pyenv/versions/3.6.6/bin/python3.6m
3.6.5: /Users/greysteil/.pyenv/versions/3.6.5/bin/python3
3.6.5: /Users/greysteil/.pyenv/versions/3.6.5/bin/python3.6m
3.6.5: /usr/local/bin/python3
3.6.5: /usr/local/bin/python3.6m
3.6.2: /Users/greysteil/.pyenv/versions/3.6.2/bin/python3
3.6.2: /Users/greysteil/.pyenv/versions/3.6.2/bin/python3.6m
3.6.1: /Users/greysteil/.pyenv/versions/3.6.1/bin/python3
3.6.1: /Users/greysteil/.pyenv/versions/3.6.1/bin/python3.6m
3.5.0: /Users/greysteil/.pyenv/versions/3.5.0/bin/python3
3.5.0: /Users/greysteil/.pyenv/versions/3.5.0/bin/python3.5m
2.7.15: /Users/greysteil/.pyenv/versions/2.7.15/bin/python
2.7.14: /Users/greysteil/.pyenv/versions/2.7.14/bin/python
2.7.10: /usr/bin/python
2.7.10: /usr/bin/pythonw
2.7.10: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '17.7.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 17.7.0: Wed Oct 10 23:06:14 PDT '
                     '2018; root:xnu-4570.71.13~1/RELEASE_X86_64',
 'python_full_version': '3.6.6',
 'python_version': '3.6',
 'sys_platform': 'darwin'}

System environment variables:

TERM_PROGRAM
PYENV_ROOT
TERM
SHELL
CLICOLOR
TMPDIR
Apple_PubSub_Socket_Render
TERM_PROGRAM_VERSION
TERM_SESSION_ID
PYENV_VERSION
USER
SSH_AUTH_SOCK
PYENV_DIR
LSCOLORS
PATH
PWD
EDITOR
LANG
PYENV_HOOK_PATH
XPC_FLAGS
RBENV_SHELL
XPC_SERVICE_NAME
HOME
SHLVL
LOGNAME
__CF_USER_TEXT_ENCODING
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

PATH: /Users/greysteil/.pyenv/versions/3.6.6/bin:/usr/local/Cellar/pyenv/1.2.8/libexec:/Users/greysteil/.pyenv/plugins/pyenv-virtualenv/bin:/Users/greysteil/.pyenv/plugins/pyenv-update/bin:/Users/greysteil/.pyenv/plugins/pyenv-installer/bin:/Users/greysteil/.pyenv/plugins/pyenv-doctor/bin:/Users/greysteil/.poetry/bin:/Users/greysteil/.config/yarn/global/node_modules/.bin:/Users/greysteil/.poetry/bin::/Users/greysteil/go/bin::/usr/local/opt/erlang@20/bin:/Users/greysteil/.pyenv/bin:/Users/greysteil/.cargo/bin:/usr/local/heroku/bin:/Users/greysteil/.rbenv/shims:/usr/local/bin:./node_modules/.bin:.bundle/binstubs:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/go/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/local/git/bin:/usr/local/sbin
SHELL: /bin/bash
EDITOR: subl -w
LANG: en_GB.UTF-8
PWD: /Users/greysteil/code/python-test

Contents of Pipfile ('/Users/greysteil/code/python-test/Pipfile'):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true

[packages]
requests = { editable = true, git = 'https://github.com/requests/requests.git', ref = 'v2.18.4' }
urllib3 = "==1.21.1"

Contents of Pipfile.lock ('/Users/greysteil/code/python-test/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "71c49547d80da72c513f8e15ba6c8c070007d930c22dff508902f491b7665107"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:339dc09518b07e2fa7eda5450740925974815557727d6bd35d319c1524a04a4c",
                "sha256:6d58c986d22b038c8c0df30d639f23a3e6d172a05c3583e766f4c0b785c0986a"
            ],
            "version": "==2018.10.15"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "idna": {
            "hashes": [
                "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f",
                "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4"
            ],
            "version": "==2.6"
        },
        "requests": {
            "editable": true,
            "git": "https://github.com/requests/requests.git",
            "ref": "a3d7cf3f27e74c28ef30f01e9f2e483570ab042e"
        },
        "urllib3": {
            "hashes": [
                "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
            ],
            "version": "==1.22"
        }
    },
    "develop": {}
}

The text was updated successfully, but these errors were encountered:

techalchemy · 2018-11-24T16:21:58Z

It’s definitely happening, we don’t always trust the resolver on editavke dependencies so we don’t pass no-deps during install. These get installed last. Question is, is this a regression?

techalchemy · 2018-11-24T16:22:43Z

Basically if it is, I’d love to trust what we resolved.

greysteil · 2018-11-24T16:28:27Z

It's a regression from 2018.10.13 - I get the correct version of urllib3 when running that. It had already regressed in 2018.11.14, though.

- Stop preferring resolution of VCS dependencies in all cases - Resolve vcs dependencies together with non-vcs dependencies - Clarify blocking and no-deps logic - Add artifacts and tests - Add vendoring task for artifacts - Clean up release tasks - Fixes #3296 Signed-off-by: Dan Ryan <dan@danryan.co>

jpcope · 2019-12-12T19:34:11Z

We've had this happen to us still on release 2018.11.26

I wish I had an example I could share, but I do not because the repos involved are private and were accessed via a gitconfig url rewrite that adds a private github auth token ( if that helps identify where it is still a problem )

greysteil mentioned this issue Nov 24, 2018

Pipenv updates pinned packages in Pipfile.lock when there is an editable git dependency with unpinned sub-deps dependabot/dependabot-core#805

Closed

greysteil changed the title ~~Editable dependency requirements are sometimes preferred to top-level ones~~ Editable dependency requirements sometimes overwrite top-level ones Nov 24, 2018

greysteil changed the title ~~Editable dependency requirements sometimes overwrite top-level ones~~ Editable dependency requirements sometimes override top-level ones Nov 24, 2018

techalchemy mentioned this issue Nov 24, 2018

Don't give child deps of vcs deps auto-precedence #3297

Merged

techalchemy added Type: Bug 🐛 This issue is a bug. Category: Dependency Resolution Issue relates to dependency resolution. Type: Regression This issue is a regression of a previous behavior. labels Nov 24, 2018

techalchemy added this to the November Bugfix Release milestone Nov 24, 2018

techalchemy added the Type: Release Blocker Must be resolved before the next release can be cut. label Nov 25, 2018

techalchemy closed this as completed in #3297 Nov 25, 2018

josephcopenhaver mentioned this issue Dec 21, 2019

editable dependency causes wrong versions to be installed #4078

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Editable dependency requirements sometimes override top-level ones #3296

Editable dependency requirements sometimes override top-level ones #3296

greysteil commented Nov 24, 2018

techalchemy commented Nov 24, 2018

techalchemy commented Nov 24, 2018

greysteil commented Nov 24, 2018 •

edited

Loading

jpcope commented Dec 12, 2019

Editable dependency requirements sometimes override top-level ones #3296

Editable dependency requirements sometimes override top-level ones #3296

Comments

greysteil commented Nov 24, 2018

Issue description

Expected result

Actual result

Steps to replicate

techalchemy commented Nov 24, 2018

techalchemy commented Nov 24, 2018

greysteil commented Nov 24, 2018 • edited Loading

jpcope commented Dec 12, 2019

greysteil commented Nov 24, 2018 •

edited

Loading