Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installing bad package name removes version specifiers from Pipfile causing lock failures #5865

Open
edwinjosegeorge opened this issue Aug 25, 2023 · 13 comments
Labels
Priority: High This item is high priority and should be resolved quickly. Type: Bug 🐛 This issue is a bug. validation

Comments

@edwinjosegeorge
Copy link

Issue description

Unable to lock using pipenv==2023.8.23 while pipenv==2023.8.21 can. The latest version reports mismatch in sub-dependency but upon inspection, the resolution is actually possible.

Expected result

Lock to succeed

Actual result

> pipenv lock
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠦ Locking...
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Cannot install -r /var/folders/pm/7grpkr192dq50xdn86kw8clh0000gp/T/pipenv-1ka_j20u-requirements/pipenv-xf2cewnc-constraints.txt (line 3) and pydantic[email] because these package versions have conflicting dependencies.
INFO:pipenv.patched.pip._internal.resolution.resolvelib.factory:
The conflict is caused by:
    The user requested pydantic[email]
    private-lib 0.3.19 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.18 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.17 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.16 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.15 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.14 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.13 depends on pydantic[email]<2.0.0 and >=1.10.4
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 647, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 614, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:       ^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 594, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 848, in resolve_deps
[ResolutionFailure]:       results, hashes, internal_resolver = actually_resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 621, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 447, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/cli/command.py", line 340, in lock
    do_lock(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 789, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 658, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

My Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "${PRIVATE_URL}"
verify_ssl = true
name = "myprivatelib"

[packages]
private-lib = {version = "*", index = "myprivatelib"}
pydantic = {extras = ["email"]}

[dev-packages]

[requires]
python_version = "3.11"

remove the old lock file

rm Pipfile.lock

Create new lock

pipenv lock

This command fails for pipenv==2023.8.23 but succeeds for pipenv=2023.8.21 as the later installs pydantic==1.10.12

@matteius
Copy link
Member

Is it because you are missing a version specifier on pydantic?

Nothing else really stands out to me about it: v2023.8.21...v2023.8.23

@edwinjosegeorge
Copy link
Author

Does the new version check for the "version" tag in pipfile? Somehow the previous version is able to achieve locking without this constraint. When I add version=* for pydantic, I am able to build the lock.

The error seems misleading as pipenv should be iterating the version of pydantic and not the private-repo

@matteius
Copy link
Member

I was under the assumption in the refactor away from requirementslib that if the pipfile entry is a string, its likely a version specifier (but learned it could be a vcs string) and then I thought if it was a dictionary there would always be a version key for a named requirement. I just checked out to 2023.7.23 which was before that refactor was merged, the basic install command does add a version key:

$ pipenv install pydantic["email"]
Creating a virtualenv for this project...
Pipfile: C:\Users\matte\Projects\pipenv-triage\pipenv-5865\Pipfile
Using default python from C:\Users\matte\AppData\Local\Programs\Python\Python311\python.exe (3.11.2) to create virtualenv...
[   =] Creating virtual environment...created virtual environment CPython3.11.2.final.0-64 in 3210ms
  creator CPython3Windows(dest=C:\c\users\matte\.virtualenvs\pipenv-5865-uw0NJOV8, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\matte\AppData\Local\pypa\virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.0.0, wheel==0.41.1
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: C:\c\Users\matte\.virtualenvs\pipenv-5865-uw0NJOV8
Creating a Pipfile for this project...
Installing pydantic[email]...
Resolving pydantic...
Adding pydantic to Pipfile's [packages] ...
Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Resolving dependencies...
Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (c85e765bf1ae1df8dd2ada36ab6adccb8c888756749bbf045a47708346853ede)!
Installing dependencies from Pipfile.lock (853ede)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5865
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pydantic = {version = "*", extras = ["email"]}

[dev-packages]

[requires]
python_version = "3.9"

While on the one hand we want to be backwards compatible, I think this is a case where it is better if we move forward with requiring the version specifier and not inferring you mean star specifier. I think then the enhancement would be based around Pipfile validation before entering into a lock phase. I think that was something you were interested in looking at one point @oz123 ?

@oz123
Copy link
Contributor

oz123 commented Aug 25, 2023

It seems like we should reinclude Cerberus in our vendor. That pipfile without version specificier should trigger an exception.

@matteius
Copy link
Member

we should reinclude Cerberus in our vendor

I'd be in favor of consider it vs a vanilla data-class solution that does some validations -- its such an old library that I think if we can avoid it we will be better off.

@oz123
Copy link
Contributor

oz123 commented Aug 26, 2023

Re-adding Cerberus is only temporary until we do a rewrite with dataclasses.
Also, Cerberus is already an optional dependancy of plette. And while it's old - it's stable and has a concise API. I'm kind of sorry I removed it prematurely from our vendor library.

@edwinjosegeorge
Copy link
Author

I have a couple of doubts here.... I am finding some odd behaviour

Installing a package that does not exist??

> pipenv install some_random_package          
Installing some_random_package...
Resolving some_random_package...
Added some-random-package to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock (7a3e87) out of date, updating to (ed4d91)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠇ Locking...
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement some-random-package (from versions: none)
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 647, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 614, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:       ^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 594, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 848, in resolve_deps
[ResolutionFailure]:       results, hashes, internal_resolver = actually_resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 621, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 447, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for some-random-package

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/install.py", line 297, in do_install
    raise e
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/install.py", line 281, in do_install
    do_init(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/install.py", line 648, in do_init
    do_lock(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 789, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 658, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

The command have failed as expected, but what does Added some-random-package to Pipfile's [packages] ...✔ Installation Succeeded means??

Automatically removes the version specifiers in pipfile

This is my pipfile initially

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pydantic = {extras = ["email"], version = "<2.0"}

[dev-packages]

[requires]
python_version = "3.11"

Lets say I made some typo and wrote the following

pipenv install --dev some_random_package

The output is same as the first one, installation have failed, BUT now pipfile is updated to following

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pydantic = {extras = ["email"] }

[dev-packages]

[requires]
python_version = "3.11"

pydantic have lost its version tag. In the first command i ran, the neither the pipfile, nor the lock file was updated

It was this change that caused me to face the issue I raised, and just adding the version in pydantic helped to lock new package that was being installed

@matteius
Copy link
Member

@edwinjosegeorge Are you saying trying to install a non existent package that fails is removing the version specifier for an unrelated Pipfile entry? 🤔

@edwinjosegeorge
Copy link
Author

Yep. There is no such package

  1. I do get a message "Installation Succeeded"
  2. It removes the version specifiers in pipfile ( when used with pipenv install --dev )

@matteius matteius added the Type: Bug 🐛 This issue is a bug. label Aug 28, 2023
@matteius matteius changed the title Unable to lock even if resolution is possible Installing bad package name removes version specifiers from Pipfile causing lock failures Aug 28, 2023
@matteius
Copy link
Member

That is really bizarre -- I just confirmed what you are saying -- the interesting thing is the version specifiers were left alone in dev group, but in the packages it was stripped from the dictionary.

@phette23
Copy link

I think any failed install causes this behavior, not just nonexistent packages, FYI. I'm running into the problem like this:

> pipenv --version
pipenv, version 2023.8.23
> pipenv install invenio-app-rdm[opensearch2,postgresql]==11.0.4 # succeeds, version in Pipfile
> pipenv install invenio-saml # fails with a build error on my MacOS
> grep invenio-app-rdm Pipfile
invenio-app-rdm = {extras = ["postgresql", "opensearch2"] }

@matteius matteius added the Priority: High This item is high priority and should be resolved quickly. label Sep 19, 2023
@matteius
Copy link
Member

I believe this has been fixed in 2024.3.0 or recent version, hard to confirm, though I haven't seen it happen.

@phette23
Copy link

I can't reproduce it anymore in my tests, trying

> pipenv --version
pipenv, version 2024.3.0
> pipenv install requests==2.32.2
> pipenv install cchardet==2.1.7 # this fails for me
> grep requests Pipfile
requests = "==2.32.2"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: High This item is high priority and should be resolved quickly. Type: Bug 🐛 This issue is a bug. validation
Projects
None yet
Development

No branches or pull requests

4 participants