Installing bad package name removes version specifiers from Pipfile causing lock failures

[edwinjosegeorge]

Issue description

Unable to lock using pipenv==2023.8.23 while pipenv==2023.8.21 can. The latest version reports mismatch in sub-dependency but upon inspection, the resolution is actually possible.

Expected result

Lock to succeed

Actual result

> pipenv lock
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠦ Locking...
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Cannot install -r /var/folders/pm/7grpkr192dq50xdn86kw8clh0000gp/T/pipenv-1ka_j20u-requirements/pipenv-xf2cewnc-constraints.txt (line 3) and pydantic[email] because these package versions have conflicting dependencies.
INFO:pipenv.patched.pip._internal.resolution.resolvelib.factory:
The conflict is caused by:
    The user requested pydantic[email]
    private-lib 0.3.19 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.18 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.17 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.16 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.15 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.14 depends on pydantic[email]<2.0.0 and >=1.10.4
    The user requested pydantic[email]
    private-lib 0.3.13 depends on pydantic[email]<2.0.0 and >=1.10.4
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 647, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 614, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:       ^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 594, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 848, in resolve_deps
[ResolutionFailure]:       results, hashes, internal_resolver = actually_resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 621, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 447, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/cli/command.py", line 340, in lock
    do_lock(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 789, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 658, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

My Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "${PRIVATE_URL}"
verify_ssl = true
name = "myprivatelib"

[packages]
private-lib = {version = "*", index = "myprivatelib"}
pydantic = {extras = ["email"]}

[dev-packages]

[requires]
python_version = "3.11"

remove the old lock file

rm Pipfile.lock

Create new lock

pipenv lock

This command fails for pipenv==2023.8.23 but succeeds for pipenv=2023.8.21 as the later installs pydantic==1.10.12

[matteius]

Is it because you are missing a version specifier on pydantic?

Nothing else really stands out to me about it: v2023.8.21...v2023.8.23

[edwinjosegeorge]

Does the new version check for the "version" tag in pipfile? Somehow the previous version is able to achieve locking without this constraint. When I add version=* for pydantic, I am able to build the lock.

The error seems misleading as pipenv should be iterating the version of pydantic and not the private-repo

[matteius]

I was under the assumption in the refactor away from requirementslib that if the pipfile entry is a string, its likely a version specifier (but learned it could be a vcs string) and then I thought if it was a dictionary there would always be a version key for a named requirement. I just checked out to 2023.7.23 which was before that refactor was merged, the basic install command does add a version key:

$ pipenv install pydantic["email"]
Creating a virtualenv for this project...
Pipfile: C:\Users\matte\Projects\pipenv-triage\pipenv-5865\Pipfile
Using default python from C:\Users\matte\AppData\Local\Programs\Python\Python311\python.exe (3.11.2) to create virtualenv...
[   =] Creating virtual environment...created virtual environment CPython3.11.2.final.0-64 in 3210ms
  creator CPython3Windows(dest=C:\c\users\matte\.virtualenvs\pipenv-5865-uw0NJOV8, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\matte\AppData\Local\pypa\virtualenv)
    added seed packages: pip==23.2.1, setuptools==68.0.0, wheel==0.41.1
  activators BashActivator,BatchActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

Successfully created virtual environment!
Virtualenv location: C:\c\Users\matte\.virtualenvs\pipenv-5865-uw0NJOV8
Creating a Pipfile for this project...
Installing pydantic[email]...
Resolving pydantic...
Adding pydantic to Pipfile's [packages] ...
Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Resolving dependencies...
Success!
Locking [dev-packages] dependencies...
Updated Pipfile.lock (c85e765bf1ae1df8dd2ada36ab6adccb8c888756749bbf045a47708346853ede)!
Installing dependencies from Pipfile.lock (853ede)...
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

matte@LAPTOP-N5VSGIBD MINGW64 ~/Projects/pipenv-triage/pipenv-5865
$ cat Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pydantic = {version = "*", extras = ["email"]}

[dev-packages]

[requires]
python_version = "3.9"

While on the one hand we want to be backwards compatible, I think this is a case where it is better if we move forward with requiring the version specifier and not inferring you mean star specifier. I think then the enhancement would be based around Pipfile validation before entering into a lock phase. I think that was something you were interested in looking at one point @oz123 ?

[oz123]

It seems like we should reinclude Cerberus in our vendor. That pipfile without version specificier should trigger an exception.

[matteius]

we should reinclude Cerberus in our vendor

I'd be in favor of consider it vs a vanilla data-class solution that does some validations -- its such an old library that I think if we can avoid it we will be better off.

[oz123]

Re-adding Cerberus is only temporary until we do a rewrite with dataclasses.
Also, Cerberus is already an optional dependancy of plette. And while it's old - it's stable and has a concise API. I'm kind of sorry I removed it prematurely from our vendor library.

[edwinjosegeorge]

I have a couple of doubts here.... I am finding some odd behaviour

Installing a package that does not exist??

> pipenv install some_random_package          
Installing some_random_package...
Resolving some_random_package...
Added some-random-package to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock (7a3e87) out of date, updating to (ed4d91)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠇ Locking...
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement some-random-package (from versions: none)
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 647, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 614, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:       ^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/resolver.py", line 594, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 848, in resolve_deps
[ResolutionFailure]:       results, hashes, internal_resolver = actually_resolve_deps(
[ResolutionFailure]:       ^^^^^^^^^^^^^^^^^^^^^^
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 621, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 447, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for some-random-package

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/install.py", line 297, in do_install
    raise e
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/install.py", line 281, in do_install
    do_init(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/install.py", line 648, in do_init
    do_lock(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/routines/lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 789, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2023.8.23/libexec/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 658, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

The command have failed as expected, but what does Added some-random-package to Pipfile's [packages] ...✔ Installation Succeeded means??

Automatically removes the version specifiers in pipfile

This is my pipfile initially

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pydantic = {extras = ["email"], version = "<2.0"}

[dev-packages]

[requires]
python_version = "3.11"

Lets say I made some typo and wrote the following

pipenv install --dev some_random_package

The output is same as the first one, installation have failed, BUT now pipfile is updated to following

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pydantic = {extras = ["email"] }

[dev-packages]

[requires]
python_version = "3.11"

pydantic have lost its version tag. In the first command i ran, the neither the pipfile, nor the lock file was updated

It was this change that caused me to face the issue I raised, and just adding the version in pydantic helped to lock new package that was being installed

[matteius]

@edwinjosegeorge Are you saying trying to install a non existent package that fails is removing the version specifier for an unrelated Pipfile entry? 🤔

[edwinjosegeorge]

Yep. There is no such package

1. I do get a message "Installation Succeeded"
2. It removes the version specifiers in pipfile ( when used with pipenv install --dev )

[matteius]

That is really bizarre -- I just confirmed what you are saying -- the interesting thing is the version specifiers were left alone in dev group, but in the packages it was stripped from the dictionary.

[phette23]

I think any failed install causes this behavior, not just nonexistent packages, FYI. I'm running into the problem like this:

> pipenv --version
pipenv, version 2023.8.23
> pipenv install invenio-app-rdm[opensearch2,postgresql]==11.0.4 # succeeds, version in Pipfile
> pipenv install invenio-saml # fails with a build error on my MacOS
> grep invenio-app-rdm Pipfile
invenio-app-rdm = {extras = ["postgresql", "opensearch2"] }