Skip to content

Pipenv selects wrong PyPi repository for private packages with non canonical names #6056

@mangin

Description

@mangin

Issue description

Let's assume that we have a package 'some-cool-package-name'. Pip allows non-canonical names using underscores. In this particular case it would look like 'some_cool_package_name`

if the package is located in private PyPi index than pipenv would select default index during creating a lock. That would create problems with locking.

The reason of the problem

In this file we normalize the names of packages:
https://github.com/pypa/pipenv/blob/main/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py#L250

And we pass normalized name to finder:
https://github.com/pypa/pipenv/blob/main/pipenv/patched/pip/_internal/resolution/resolvelib/factory.py#L284

Here we get the repository of packages by package name:

index_urls = [self.index_lookup[project_name]]

Here you can see how we create mapping between package name and repository:

index_mapping[source["name"]] = source["url"]

But there we create mapping between package name and repository. (Pay attention that here we don't use normalized package name)

Expected result

Pipenv should be able to work with packages that have non-canonical names.

Actual result

Pipenv has a problem with locking non-canonical names

Steps to replicate

pipenv install -d 
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://test.pypi.org/simple"
verify_ssl = true
name = "testpypi"

[packages]
pipenv_test_private_package = {version = "*", index = "testpypi"}

Stack trace:

Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠸ Locking...False
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement pipenv_test_private_package (from versions: none)
[ResolutionFailure]:   File "~/.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 645, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "~/.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 612, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "~/.venv/lib/python3.10/site-packages/pipenv/resolver.py", line 592, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "~/.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 908, in resolve_deps
[ResolutionFailure]:       results, hashes, internal_resolver = actually_resolve_deps(
[ResolutionFailure]:   File "~/.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 681, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "~/.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 442, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for pipenv_test_private_package

Traceback (most recent call last):
  File "~/.venv/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "~/.venv/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "~/.venv/lib/python3.10/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "~/.venv/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "~/.venv/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "~/.venv/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "~/.venv/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "~/.venv/lib/python3.10/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "~/.venv/lib/python3.10/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "~/.venv/lib/python3.10/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "~/.venv/lib/python3.10/site-packages/pipenv/routines/install.py", line 164, in do_install
    do_init(
  File "~/.venv/lib/python3.10/site-packages/pipenv/routines/install.py", line 672, in do_init
    do_lock(
  File "~/.venv/lib/python3.10/site-packages/pipenv/routines/lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "~/.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 849, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
  File "~/.venv/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 718, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

$ pipenv --support

Pipenv version: '2023.11.15'

Pipenv location: '/.venv/lib/python3.10/site-packages/pipenv'

Python location: '/.venv/bin/python3.10'

OS Name: 'posix'

User pip version: '23.3.1'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.15.0-91-generic',
 'platform_system': 'Linux',
 'platform_version': '#101~20.04.1-Ubuntu SMP Thu Nov 16 14:22:28 UTC 2023',
 'python_full_version': '3.10.4',
 'python_version': '3.10',
 'sys_platform': 'linux'}

System environment variables:

  • SHELL
  • SESSION_MANAGER
  • QT_ACCESSIBILITY
  • PIPENV_VENV_IN_PROJECT
  • COLORTERM
  • XDG_CONFIG_DIRS
  • NVM_INC
  • XDG_MENU_PREFIX
  • GNOME_DESKTOP_SESSION_ID
  • LC_ADDRESS
  • GNOME_SHELL_SESSION_MODE
  • LC_NAME
  • SSH_AUTH_SOCK
  • ADFS_USER
  • XMODIFIERS
  • DESKTOP_SESSION
  • LC_MONETARY
  • SSH_AGENT_PID
  • PTC_ARTIFACTORY_TOKEN
  • EDITOR
  • GTK_MODULES
  • SYS_DST_APPROVER_STASH_TOKEN
  • PWD
  • AD_USERNAME
  • LOGNAME
  • XDG_SESSION_DESKTOP
  • XDG_SESSION_TYPE
  • GPG_AGENT_INFO
  • SYSTEM_USER
  • XAUTHORITY
  • GJS_DEBUG_TOPICS
  • WINDOWPATH
  • SNYK_TOKEN
  • HOME
  • USERNAME
  • IM_CONFIG_PHASE
  • LC_PAPER
  • LANG
  • LS_COLORS
  • XDG_CURRENT_DESKTOP
  • VTE_VERSION
  • ARTIFACTORY_PASSWORD
  • GNOME_TERMINAL_SCREEN
  • MARVIN_JENKINS_PASSWORD
  • INVOCATION_ID
  • MANAGERPID
  • GOROOT
  • SYS_DST_ROBOT_STASH_TOKEN
  • GJS_DEBUG_OUTPUT
  • NVM_DIR
  • LESSCLOSE
  • XDG_SESSION_CLASS
  • MARVIN_JENKINS_USER
  • TERM
  • LC_IDENTIFICATION
  • DST_ARTIFACTORY_TOKEN
  • LESSOPEN
  • USER
  • GNOME_TERMINAL_SERVICE
  • VISUAL
  • DISPLAY
  • SHLVL
  • NVM_CD_FLAGS
  • LC_TELEPHONE
  • QT_IM_MODULE
  • LC_MEASUREMENT
  • ARTIFACTORY_USERNAME
  • XDG_RUNTIME_DIR
  • ARTIFACTORY_TOKEN
  • LC_TIME
  • PTC_STASH_TOKEN
  • RAPID7_TOKEN
  • JOURNAL_STREAM
  • XDG_DATA_DIRS
  • PATH
  • GDMSESSION
  • DBUS_SESSION_BUS_ADDRESS
  • NVM_BIN
  • LC_NUMERIC
  • OLDPWD
  • GOPATH
  • _
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

  • PIPENV_VENV_IN_PROJECT: True

Debug–specific environment variables:

  • PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/UNKNOWN/dev/bin:/home/UNKNOWN/.local/bin/:/opt/python/3.7.9/bin/
  • SHELL: /bin/bash
  • EDITOR: vim
  • LANG: en_US.UTF-8
  • PWD: /

Contents of Pipfile ('/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://test.pypi.org/simple"
verify_ssl = true
name = "testpypi"

[packages]
pipenv_test_private_package = {version = "*", index = "testpypi"}

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type: RegressionThis issue is a regression of a previous behavior.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions