The flag does not respect package resolver results and leads to inconsistent lock files.

[matejsp]

Issue description

We have just updated pipenv and were negatively surprised that this feature is about to be removed.

The flag --keep-outdated has been deprecated for removal.The flag does not respect package resolver results and leads to inconsistent lock files. Please pin relevant requirements in your Pipfile and discontinue use of this flag.

I never saw any discussion regarding keeping keep-outdated flag but this feature is one of the most useful features in pipenv at least in very large projects such as monoliths. Please don't remove it. #5544 by @matteius

In our project we have around 110 depedencies and we rely on pipenv keep-updated so the developers update only their and not all dependencies. We specify only directly used dependencies and rely on lock file to have a list of transient dependencies.
We occasionally pin transient dependencies when we see incompatibilities.

Since we have the work split between different teams and dependencies are common in our monolith we have a problem if one team member updates all other dependencies and very often breaks the project. Some (binary wheels) are really problematic such as cryptography because it could fails just on some target systems.

Based on proposal that we should pin ALL transient dependencies if we want to retain control over dependencies making everything unreadable and hard to maintain:

• you cannot distinct between pinning a transient or first level dependency (at least in not current Pipfile format)
• you cannot do updates one by one without affecting all other dependencies
• you cannot split dev process of updating all dependencies a release with just updated dependencies
• pipenv graph is broken since it displays all pinned dependencies as first level dependencies.
• every developer when introducing new depedency or even upgrading will need to track down all the transient dependencies and pin them. if they don't other developers will broke their code by accidentally updating their dependencies.

Basically we rely on pipenv to do hard lifting for us and not manually track all dependencies. Manual work is error prone and would like to avoid. This is such important feature to us that we will migrate to another solution (like poetry).

The main reason why we even need such feature at all is because pipenv is lacking tooling for updating one by one dependency:
Like https://python-poetry.org/docs/cli/#update or npm or basically every other dep manager.

Expected result

Retain this feature or introduce new commands for updating controllably dependencies.

Actual result

The flag --keep-outdated has been deprecated for removal.The flag does not respect package resolver results and leads to inconsistent lock files. Please pin relevant requirements in your Pipfile and discontinue use of this flag.

Steps to replicate

use flag --keep-outdated

$ pipenv --support

Pipenv version: '2023.2.4'

Pipenv location: '/Users/myuser/.virtualenvs/bitstamp38/lib/python3.8/site-packages/pipenv'

Python location: '/Users/myuser/.virtualenvs/bitstamp38/bin/python3'

OS Name: 'posix'

User pip version: '22.3.1'

user Python installations found:

• 3.11.2: /usr/local/bin/python3
• 3.10.10: /Users/myuser/.pyenv/versions/3.10.10/bin/python3
• 3.10.7: /Users/myuser/.pyenv/versions/3.10.7/bin/python3
• 3.10.6: /Users/myuser/.pyenv/versions/3.10.6/bin/python3
• 3.9.16: /usr/local/bin/python3.9
• 3.9.13: /Users/myuser/.pyenv/versions/3.9.13/bin/python3
• 3.9.6: /usr/bin/python3
• 3.8.16: /Users/myuser/.virtualenvs/bitstamp38/bin/python3
• 3.8.16: /Users/myuser/.virtualenvs/bitstamp38/bin/python
• 3.8.16: /Users/myuser/.virtualenvs/bitstamp38/bin/python3
• 3.8.16: /Users/myuser/.virtualenvs/bitstamp38/bin/python
• 3.8.16: /usr/local/bin/python3.8
• 3.8.16: /Users/myuser/.pyenv/versions/3.8.16/bin/python3
• 3.8.13: /Users/myuser/.pyenv/versions/3.8.13/bin/python3
• 3.8.12: /Users/myuser/.pyenv/versions/bitstamp38/bin/python3
• 3.8.12: /Users/myuser/.pyenv/versions/3.8.12/bin/python3
• 3.6.8: /usr/local/bin/python3.6
• 3.6.8: /usr/local/bin/python3.6m

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.16',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '22.3.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 22.3.0: Thu Jan  5 20:53:49 PST '
                     '2023; root:xnu-8792.81.2~2/RELEASE_X86_64',
 'python_full_version': '3.8.16',
 'python_version': '3.8',
 'sys_platform': 'darwin'}

System environment variables:

• TERM_PROGRAM
• SHELL
• TERM
• TMPDIR
• TERM_PROGRAM_VERSION
• TERM_SESSION_ID
• USER
• SSH_AUTH_SOCK
• PATH
• LaunchInstanceID
• __CFBundleIdentifier
• PWD
• XPC_FLAGS
• XPC_SERVICE_NAME
• SHLVL
• HOME
• LOGNAME
• SECURITYSESSIONID
• OLDPWD
• ZSH
• PAGER
• LESS
• LSCOLORS
• _VIRTUALENVWRAPPER_API
• VIRTUALENVWRAPPER_SCRIPT
• VIRTUALENVWRAPPER_PYTHON
• NVM_DIR
• NVM_CD_FLAGS
• NVM_BIN
• NVM_INC
• VIRTUALENVWRAPPER_PROJECT_FILENAME
• VIRTUALENVWRAPPER_WORKON_CD
• WORKON_HOME
• VIRTUALENVWRAPPER_HOOK_DIR
• VIRTUAL_ENV
• PS1
• CD_VIRTUAL_ENV
• LANG
• LC_ALL
• LC_CTYPE
• _
• __CF_USER_TEXT_ENCODING
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PYTHONDONTWRITEBYTECODE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /Users/myuser/.virtualenvs/b38/bin:/Users/myuser/.rd/bin:/Users/myuser/.nvm/versions/node/v14.18.2/bin:/Users/myuser/apache-maven-3.8.2/bin:/usr/local/opt/mysql@5.7/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /Users/myuser/projects/b/pipenvs
• VIRTUAL_ENV: /Users/myuser/.virtualenvs/b38

---

[matteius]

In our project we have around 110 dependencies and we rely on pipenv keep-updated so the developers update only their and not all dependencies

How do you handle when the resolver creates an inconsistent or lock file that simply cannot be installed? The whole point of removing --keep-outdated is because it creates tons of support tickets to the pipenv maintainers about why their project lock file has issues, and it when numerous times it has relates back to the implementation of --keep-outdated, it has been determined there is no fix --keep-outdated because the idea behind it is flawed. It requires a complete lock lock file and takes that and adds any new things from a complete package resolution without any regard for what is compatible in the dependency chain.

Since we have the work split between different teams and dependencies are common in our monolith we have a problem if one team member updates all other dependencies and very often breaks the project. Some (binary wheels) are really problematic such as cryptography because it could fails just on some target systems.

Ok, pin the appropriate problematic packages and start using named package categories where it makes sense if some of these different teams don't actually need the whole set of 110 packages.

Based on proposal that we should pin ALL transient dependencies if we want to retain control over dependencies making everything unreadable and hard to maintain:

The proposal was never that you pin ALL transient dependencies, that is exactly what the Lock file is -- the proposal was that you pin ALL problematic dependencies such that you can trust when you lock that the specifiers you don't want to be affected won't be. It was less of a proposal -- this is how pipenv is intended to work and --keep-outdated was never documented because it was a hack meant to appease those that refuse to listen about how the resolver requirements and the intentions of the Pipfile.

Basically we rely on pipenv to do hard lifting for us and not manually track all dependencies. . Manual work is error prone and would like to avoid. This is such important feature to us that we will migrate to another solution (like poetry).

--keep-outdated is also quite error prone and I would like if you told that side of the story. Have you personally encountered any times that --keep-outdated didn't actually yield a healthy or valid result? I am not sure I understand the argument that manual work is error prone, or what the manual work is even that you are describing? Using the Pipfile more appropriately should be a goal for any organization; I would also be curious to hear about how poetry doesn't pull in transient dependencies and installs things one-by-one -- that doesn't really line up with my understanding of how that tool works.

I personally cannot support the --keep-outdated flag and I have never heard of a reasonable proposal of how to make it create consistent locking results.

[matteius]

Here is an example of the problem:

Let's try: First I install old Django, imagine this was some time ago:
pipenv install Django==2.*

Then now today I want to add django-model-utils:
pipenv install django-model-utils==4.2.0 --keep-outdated

So it "works" and generates a lockfile, but it is not valid. Django-model-utils 4.2.0 will not work with even Django 3.0 (support was dropped) so having it install 4.2.0 along side 2.2.28 is begging for problems.

The Lockfile:

matteius@matteius-VirtualBox:~/pipenv-triage/pipenv-4536$ cat Pipfile.lock 
{
    "_meta": {
        "hash": {
            "sha256": "44175eea762b2510933033b1a915a8cfeaa2edbe36be9d5407be47cc103fef52"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "django": {
            "hashes": [
                "sha256:0200b657afbf1bc08003845ddda053c7641b9b24951e52acd51f6abda33a7413",
                "sha256:365429d07c1336eb42ba15aa79f45e1c13a0b04d5c21569e7d596696418a6a45"
            ],
            "index": "pypi",
            "version": "==2.2.28"
        },
        "django-model-utils": {
            "hashes": [
                "sha256:a768a25c80514e0ad4e4a6f9c02c44498985f36c5dfdea47b5b1e8cf994beba6",
                "sha256:e7a95e102f9c9653427eadab980d5d59e1dea972913b9c9e01ac37f86bba0ddf"
            ],
            "index": "pypi",
            "version": "==4.2.0"
        },
        "pytz": {
            "hashes": [
                "sha256:1e760e2fe6a8163bc0b3d9a19c4f84342afa0a2affebfaa84b01b978a02ecaa7",
                "sha256:e68985985296d9a66a881eb3193b0906246245294a881e7c8afe623866ac6a5c"
            ],
            "version": "==2022.1"
        },
        "sqlparse": {
            "hashes": [
                "sha256:0c00730c74263a94e5a9919ade150dfc3b19c574389985446148402998287dae",
                "sha256:48719e356bb8b42991bdbb1e8b83223757b93789c00910a616a071910ca4a64d"
            ],
            "markers": "python_version >= '3.5'",
            "version": "==0.4.2"
        }
    },
    "develop": {}
}

[matejsp]

How do you handle when the resolver creates an inconsistent or lock file that simply cannot be installed? The whole point of removing --keep-outdated is because it creates tons of support tickets to the pipenv maintainers about why their project lock file has issues, and it when numerous times it has relates back to the implementation of --keep-outdated, it has been determined there is no fix --keep-outdated because the idea behind it is flawed. It requires a complete lock lock file and takes that and adds any new things from a complete package resolution without any regard for what is compatible in the dependency chain.

Yes it occasionally happens. That's the reason why we run pipenv lock & sync twice. Sometimes first fails and second one usually succeeds. And we also generate requirements and install into empty venv. So we can detect such issues. But it outweighs the downside of not being able to upgrade in small chunks at all. This is HUGE minus for handling packages especially in large project in production.

Ok, pin the appropriate problematic packages and start using named package categories where it makes sense if some of these different teams don't actually need the whole set of 110 packages.

One version that is not problematic today, usually is tomorrow. Every package breaks compatibility at one point. That's why we always check the release notes for each package when we upgrade it. That's why iterative upgrading is important to us. Categories don't work for us, since we have monolith and we need all packages installed at the same time. We also pick upgrading of dependencies in such way that it is easy to see if anything breaks in production (or testing). For example major version upgrade of core library is a planned effort and not something that just accidentally slips in.

The proposal was never that you pin ALL transient dependencies, that is exactly what the Lock file is -- the proposal was that you pin ALL problematic dependencies such that you can trust when you lock that the specifiers you don't want to be affected won't be. It was less of a proposal -- this is how pipenv is intended to work and --keep-outdated was never documented because it was a hack meant to appease those that refuse to listen about how the resolver requirements and the intentions of the Pipfile.

Defeats the purpose if you cannot not control what you want to upgrade. I understand it was a hack and poorly documented. But people were apparently using this feature, if they were complaining about it :) Solution to take the feature away without some alternative is a bad solution.

--keep-outdated is also quite error prone and I would like if you told that side of the story. Have you personally encountered any times that --keep-outdated didn't actually yield a healthy or valid result? I am not sure I understand the argument that manual work is error prone, or what the manual work is even that you are describing? Using the Pipfile more appropriately should be a goal for any organization; I would also be curious to hear about how poetry doesn't pull in transient dependencies and installs things one-by-one -- that doesn't really line up with my understanding of how that tool works.
I personally cannot support the --keep-outdated flag and I have never heard of a reasonable proposal of how to make it create consistent locking results.

The fact that it is error prone is not such a bit deal, because you have GIT, pull requests and CI. So if pipenv would produce incorrect results this would be caught very early in the process. No big deal really. Committing only parts of lock file seems more error prone to me.

Regarding your example:

• what happens if you have private repository and dont upload the latest Django. Will you also get inconsistent result if you pipenv install django-model-utils==4.2.0. If that would happen then you have a bigger problem. You should detect and complain.
• what happens if you try to install older version of django-model-utils that is not compatible with current latest Django? Will you get inconsistent result or complain or use latest version still compatible?

If you would add pipenv upgrade 1-package-name 2-package-name (and leave the rest intact) like this would remove the need for this. This is the way npm update works if you list packages. And if you upgrade library that creates incompatibility in lock file you get an error and package.json (or pipfile) is not updated remaining in the consistent state. I saw the same solution used in poetry (BUT haven't tested it, have plan for sometime in the future).

Leave it buggy, it's not perfect, but it works just fine at least for us :)

[matteius]

That's the reason why we run pipenv lock & sync twice. Sometimes first fails and second one usually succeeds

Can you explain this in more detail.

If you are running pipenv lock it is essentially locking based on your Pipfile constraints and that would update all dependencies that aren't pinned anyway. The same thing install does.

Defeats the purpose if you cannot not control what you want to upgrade. Every package breaks compatibility at one point.

That is debatable -- you can control it with the specifiers in the Pipfile. Every package has the potential to break compatibility, review the packages being updated as part of the PR that adds a new dependency and decide at that time if it needs to be pinned in the Pipfile. You are describing having a testing cycle, but the scope of testing you want to control using --keep-outdated is less predictable than maintaining a well defined Pipfile.

The fact that it is error prone is not such a bit deal, because you have GIT, pull requests and CI. So if pipenv would produce incorrect results this would be caught very early in the process.

The same could be said about what is recommended -- that you go through a PR process and review your lock file constraints.

Committing only parts of lock file seems more error prone to me.

That is precisely what --keep-outdated does though, it commits only part of the lock file resolution.

If you would add pipenv upgrade 1-package-name 2-package-name (and leave the rest intact) like this would remove the need for this.

There is an upgrade option though I have not used it much -- but when you upgrade a package and it has a dependency that needs to be upgraded too, you would expect both to be upgraded, right?

Side note: Have you ever explored the behavior of --selective-upgrade? To me its a more complicated version of --keep-outdated that may be safer, but I have not done enough analysis of that code path to know for sure.

--keep-outdated in the current form is definitely on the chopping block, but there is not a timeline on when it will be removed. Having it be deprecated allows me to close any issue that the user happens to be passing that flag, because for me my time is valuable and there is no reasonable way to support it.

[matteius]

FWIW poetry does let you install incompatible versions of things together and won't affect the lockfile. I don't know why you would want that behavior, as the package authors already specify that they are not compatible, and the pip resolver respects that. Pipenv uses the pip resolver and poetry does not. Therefore:

matteius@matteius-VirtualBox:~/pipenv-triage/outdated$ poetry add django-model-utils==4.2.0

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing django-model-utils (4.2.0)


matteius@matteius-VirtualBox:~/pipenv-triage/outdated$ poetry add Django==2.*
Creating virtualenv outdated-cSQLnpBf-py3.10 in /home/matteius/.cache/pypoetry/virtualenvs

Updating dependencies
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/97/d3/31dd2c3e48fc2060819f4acb0686248250a0f2326356306b38a42Resolving dependencies... Downloading https://files.pythonhosted.org/packages/2e/09/fbd3c46dce130958ee8e0090f910f1fe39e502cc5ba0aadca1e8aResolving dependencies... Downloading https://files.pythonhosted.org/packages/2e/09/fbd3c46dce130958ee8e0090f910f1fe39e502cc5ba0aadca1e8aResolving dependencies... (1.0s)

Writing lock file

Package operations: 3 installs, 0 updates, 0 removals

  • Installing pytz (2022.7.1)
  • Installing sqlparse (0.4.3)
  • Installing django (2.2.28)



matteius@matteius-VirtualBox:~/pipenv-triage/outdated$ cat poetry.lock 
# This file is automatically @generated by Poetry and should not be changed by hand.

[[package]]
name = "django"
version = "2.2.28"
description = "A high-level Python Web framework that encourages rapid development and clean, pragmatic design."
category = "main"
optional = false
python-versions = ">=3.5"
files = [
    {file = "Django-2.2.28-py3-none-any.whl", hash = "sha256:365429d07c1336eb42ba15aa79f45e1c13a0b04d5c21569e7d596696418a6a45"},
    {file = "Django-2.2.28.tar.gz", hash = "sha256:0200b657afbf1bc08003845ddda053c7641b9b24951e52acd51f6abda33a7413"},
]

[package.dependencies]
pytz = "*"
sqlparse = ">=0.2.2"

[package.extras]
argon2 = ["argon2-cffi (>=16.1.0)"]
bcrypt = ["bcrypt"]

[[package]]
name = "django-model-utils"
version = "4.2.0"
description = "Django model mixins and utilities"
category = "main"
optional = false
python-versions = "*"
files = [
    {file = "django-model-utils-4.2.0.tar.gz", hash = "sha256:e7a95e102f9c9653427eadab980d5d59e1dea972913b9c9e01ac37f86bba0ddf"},
    {file = "django_model_utils-4.2.0-py3-none-any.whl", hash = "sha256:a768a25c80514e0ad4e4a6f9c02c44498985f36c5dfdea47b5b1e8cf994beba6"},
]

[package.dependencies]
Django = ">=2.0.1"

[[package]]
name = "pytz"
version = "2022.7.1"
description = "World timezone definitions, modern and historical"
category = "main"
optional = false
python-versions = "*"
files = [
    {file = "pytz-2022.7.1-py2.py3-none-any.whl", hash = "sha256:78f4f37d8198e0627c5f1143240bb0206b8691d8d7ac6d78fee88b78733f8c4a"},
    {file = "pytz-2022.7.1.tar.gz", hash = "sha256:01a0681c4b9684a28304615eba55d1ab31ae00bf68ec157ec3708a8182dbbcd0"},
]

[[package]]
name = "sqlparse"
version = "0.4.3"
description = "A non-validating SQL parser."
category = "main"
optional = false
python-versions = ">=3.5"
files = [
    {file = "sqlparse-0.4.3-py3-none-any.whl", hash = "sha256:0323c0ec29cd52bceabc1b4d9d579e311f3e4961b98d174201d5622a23b85e34"},
    {file = "sqlparse-0.4.3.tar.gz", hash = "sha256:69ca804846bb114d2ec380e4360a8a340db83f0ccf3afceeb1404df028f57268"},
]

[metadata]
lock-version = "2.0"
python-versions = "^3.10"
content-hash = "b0508819df0416bff36c4a482ce7c3f0a9e7341bbbb9879bce3cbccaaeefb314"


matteius@matteius-VirtualBox:~/pipenv-triage/outdated$ poetry show
django             2.2.28   A high-level Python Web framework that encourages rapid development and clean, pragmatic design.
django-model-utils 4.2.0    Django model mixins and utilities
pytz               2022.7.1 World timezone definitions, modern and historical
sqlparse           0.4.3    A non-validating SQL parser.

[matejsp]

Can you explain this in more detail.

I believe it is related to #4351 and problems regarding updating of hashes. We update dependency in pipfile and run lock --keep-outdated and in first run it updates the hashes in lock file BUT it runs install with old hashes and in another run it "fixes" itself. There is also another issue but I presume it is related to --keep-outdated, that old hashes are left inside even if version is updated.

How we run it:

pipenv lock --dev --keep-outdated --pre --clear
pipenv sync --dev --pre

That is debatable -- you can control it with the specifiers in the Pipfile. Every package has the potential to break compatibility, review the packages being updated as part of the PR that adds a new dependency and decide at that time if it needs to be pinned in the Pipfile. You are describing having a testing cycle, but the scope of testing you want to control using --keep-outdated is less predictable than maintaining a well defined Pipfile.
We want to have granular control what and when it is updated with tooling support if possible.
The same could be said about what is recommended -- that you go through a PR process and review your lock file constraints.

How would that work in practice? You cannot partially accept PR. So you do CI, QA test with 5 lib changes. You send to production and it fails for some reason. You rollback PR. Then you want to selectively update one version at the time (or even using bisection). How would pipenv help without keep outdated? Manually fiddling with lock file? We had that case multiple times. Last time was grpc lib introduced perf degradation (still opened issue with new io lib) doubling the cpu usage in production under high load increasing our latencies. We easily tracked it to grpc and pin it in pipfile, since we only upgraded grpc dep.
Good luck tracking such issues with updates all over the place. Sometimes we don't update some specific lib because we don't have enough QA resources in current week or we need more info which functionality it affects. So we should pin it in pipfile and remove it one week later?

That is precisely what --keep-outdated does though, it commits only part of the lock file resolution.

Yes, manual is more still more errors prone and fiddle with lock files directly is bad.

There is an upgrade option though I have not used it much -- but when you upgrade a package and it has a dependency that needs to be upgraded too, you would expect both to be upgraded, right?

Not in one go. I would expected to get and error if I introduced incompatibilities (NIT: flag with --update-needed).

pipenv upgrade django-model-utils==4.2.0
 You cannot upgrade because django-model-utils requires Django >= 3.0
pipenv upgrade django-model-utils==4.2.0 Django==3.2 (we want LTS not latest for example)
 OK! lock updated. Good to go!

Side note: Have you ever explored the behavior of --selective-upgrade? To me its a more complicated version of --keep-outdated that may be safer, but I have not done enough analysis of that code path to know for sure.

Thank you I will try. I haven't explored it yet, but I will try, it sounds what could work for us if it works. But I need to see how to do selective-upgrade and update pipfile version or do install of new dep.

--keep-outdated in the current form is definitely on the chopping block, but there is not a timeline on when it will be removed. Having it be deprecated allows me to close any issue that the user happens to be passing that flag, because for me my time is valuable and there is no reasonable way to support it.

I understand. That's why I would like shed light on my use case and perhaps also from other people using this feature because nobody just happens to accidently pass this flag :) Perhaps fixing this issue: #4351 would solve a lot (especially if you document those alternatives).

FWIW poetry does let you install incompatible versions of things together and won't affect the lockfile. I don't know why you would want that behavior

I don't want that and I never did. I want to have upgrading under control without breaking everything as reliably as possible. If tooling sometimes produces wrong output you can easily create automated test for this (like installing into new venv). Not perfect but acceptable.

I don't want to sound pushy or negative or anything. I really do appreciate pipenv and your work. Just want to get our use case across (and from people using this "complex" feature).

[matejsp]

I have tried the solution that you proposed. I don't have time to investigate how poetry works. I still prefer pipenv for now.

I was using our own existing pipenv lock with pipfile.

Inside it was (and many many others):
requests = {extras = ["use_chardet_on_py3"], version = "==2.28.1"}
and chardet 4.0.0 as transitive depedency.

Option 1 [broken]:

pipenv update --selective-upgrade requests

• requests was NOT updated to latest version not in lock and not in pipfile
• hashes were added for some deps (that's fine)
• pbr and pyparsing depedencies were removed for some reason (maybe it detected that is is no longer needed)
• it updated chardet to latest version -> ok fine requests depends on it
• it updated around 8 other unrelated depedencies (like isort, colorama, packaging)

Option 2 [working as workaround for direct dependency if you revert pipfile]:

 pipenv install --selective-upgrade requests==2.28.2

• requests were changed in pipfile (but all formatting lost :( and also extras)
• updated requests in lock file
• no other depedencies changed (good!)
• if I run without selective-upgrade it updates also unrelated dependencies (good!)

Option 3 [broken]:

pipenv update --selective-upgrade chardet

• chardet is transient dependency
• pipenv error: Warning: chardet was not found in your Pipfile! Aborting.

Option 4 [as expected]:

pipenv install --selective-upgrade chardet

• it adds chardet = "*" to pipfile and nothing else happens since it is already in a lock file

Option 5 [working as workaround if you revert pipfile changes]:

pipenv install --selective-upgrade chardet==5.1.0

• it adds chardet = "5.1.0" to pipfile and reformats it
• updates only chardet in lock file
• you need to revert whole pipfile to leave chardet transitive depedency

Option 6 [broken]:

update manually pipfile to point to requests 2.28.2
pipenv install --selective-upgrade chardet==5.1.0

• versions are updated in lock file and pipfile for requests & chardet
• hashes are not updated not for requests and not for chardet
• lock file is broken and unable to recover

(.venv) ➜  py38 pipenv install --selective-upgrade chardet==5.1.0
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Installing chardet==5.1.0...
Resolving chardet==5.1.0...
Installing...
Adding chardet to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock (197860) out of date, updating to (606399)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (9271a329ce45de2e5e5a762b1fc04edd891ce37724a2c3df057cd08bef606399)!
Installing dependencies from Pipfile.lock (606399)...
An error occurred while installing requests[use_chardet_on_py3]==2.28.2 --hash=sha256:907470a00798ef67935597a924c8af60aeab20e9666d76ef7e1b7793cd34a3d5 --hash=sha256:7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983! Will try again.
Installing initially failed dependencies...
[pipenv.exceptions.InstallError]: Looking in indexes: https://nexus.xyz.net/repository/pypi/simple
[pipenv.exceptions.InstallError]: Collecting requests[use_chardet_on_py3]==2.28.2
[pipenv.exceptions.InstallError]:   Using cached https://nexus.xyz.net/repository/pypi/packages/requests/2.28.2/requests-2.28.2-py3-none-any.whl (62 kB)
[pipenv.exceptions.InstallError]: ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
[pipenv.exceptions.InstallError]:     requests[use_chardet_on_py3]==2.28.2 from https://nexus.xyz.net/repository/pypi/packages/requests/2.28.2/requests-2.28.2-py3-none-any.whl#sha256=dd4a51d9add94a243743d19fb7beb5f5063255359a02b45282e1812e46087f4d (from -r /var/folders/l6/nmk965l50gv_90_c4xh5x0_4ctpk94/T/pipenv-g0dkm4nc-requirements/pipenv-zmku8mrq-hashed-reqs.txt (line 1)):
[pipenv.exceptions.InstallError]:         Expected sha256 7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983
[pipenv.exceptions.InstallError]:         Expected     or 907470a00798ef67935597a924c8af60aeab20e9666d76ef7e1b7793cd34a3d5
[pipenv.exceptions.InstallError]:              Got        dd4a51d9add94a243743d19fb7beb5f5063255359a02b45282e1812e46087f4d
ERROR: Couldn't install package: [Requirement(_name='requests', vcs=None, req=NamedRequirement(name='requests', version='==2.28.2', req=Requirement.parse('requests[use_chardet_on_py3]==2.28.2'), extras=['use_chardet_on_py3'], editable=False, _parsed_line=<Line (editable=False, name=requests, path=None, uri=None, extras=('use_chardet_on_py3',), markers=None, vcs=None, specifier===2.28.2, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=requests[use_chardet_on_py3]==2.28.2)>), markers=None, _specifiers='==2.28.2', index='nxsbts', editable=False, hashes=frozenset({'sha256:907470a00798ef67935597a924c8af60aeab20e9666d76ef7e1b7793cd34a3d5', 'sha256:7c5599b102feddaa661c826c56ab4fee28bfd17f5abca1ebbe3e7f19d7c97983'}), extras=('use_chardet_on_py3',), abstract_dep=None, _line_instance=<Line (editable=False, name=requests, path=None, uri=None, extras=('use_chardet_on_py3',), markers=None, vcs=None, specifier===2.28.2, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=requests[use_chardet_on_py3]==2.28.2)>, _ireq=None)]
 Package installation failed...

[matejsp]

Tested another case that you mentioned:

[requires]
python_version = "3.8"

[packages]
Django = "==2.2.18"

test> PIPENV_IGNORE_VIRTUALENVS=1 pipenv lock
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!

added to pipfile:
django-model-utils = "==4.3.1"

test> PIPENV_IGNORE_VIRTUALENVS=1 pipenv lock
or
test> PIPENV_IGNORE_VIRTUALENVS=1 pipenv lock --keep-outdated

are getting the same result (like expected!):

✘ Locking Failed!
⠇ Locking...
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Cannot install -r /var/folders/l6/nmk965l50gv_90_c4xh5x0_4ctpk94/T/pipenvf0hll074requirements/pipenv-h_q7gcay-constraints.txt (line 3) and django==2.2.18 because these package versions have conflicting dependencies.
[ResolutionFailure]:   File "/Users/myuser/.virtualenvs/b38/lib/python3.8/site-packages/pipenv/resolver.py", line 811, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/Users/myuser/.virtualenvs/b38/lib/python3.8/site-packages/pipenv/resolver.py", line 759, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/Users/myuser/.virtualenvs/b38/lib/python3.8/site-packages/pipenv/resolver.py", line 738, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/Users/myuser/.virtualenvs/b38/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 1102, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/Users/myuser/.virtualenvs/b38/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 899, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/Users/myuser/.virtualenvs/b38/lib/python3.8/site-packages/pipenv/utils/resolver.py", line 687, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

It's the same in poetry. Both detect conflict.

➜  my-package poetry add django-model-utils==4.3.1
Updating dependencies
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/17/38/5381cfbee475ddf6c109e434516ac969163124dcaad2cfc258586c571515/django_model_utils-4.3.1-py3-none-any.whl (0.3s)

Because django-model-utils (4.3.1) depends on Django (>=3.2)
 and my-package depends on django (2.2.18), django-model-utils is forbidden.
So, because my-package depends on django-model-utils (4.3.1), version solving failed.

4.2.0 django-models-utils requires Django > 2.0.0. That why you didn't get the error in poetry and pipenv.

[matteius]

@matejsp Thanks for your detailed responses ... I am working through them slowly.

How would that work in practice? You cannot partially accept PR. So you do CI, QA test with 5 lib changes. You send to production and it fails for some reason. You rollback PR. Then you want to selectively update one version at the time (or even using bisection). How would pipenv help without keep outdated? Manually fiddling with lock file?

Say you roll back because its critical, or its not critical so you have a bit of time to fix it -- in either case, the path forward would be to isolate the offending upgrade and pin that in your Pipfile and re-lock pipenv lock and re-test.

Yes, manual is more still more errors prone and fiddle with lock files directly is bad.

I am not, and never have, recommended people fiddle with the lock files directly.

Sounds like there are numerous bugs in --selective-upgrade as well -- I am wondering though, as I've thought more about what an ideal behavior would be. Either --selective-upgrade or pipenv upgrade could consider resolving the package in question to figure out which packages are required (not necessarily the versions), then run a complete lock phase and pull the required packages (determined from the isolated resolve phase) and their allowed versions (determined from the complete lock phase) and just merge in those updates to the Lock file. This may be more deterministic and prevent inconsistent lock files (which is what I am trying to solve for here with removing --keep-outdated) -- it does mean that if upgrading requests brings in a new package or requires one you already have to be updated, that it would be changed in the lock file, but the idea is everything else would be left the same. Thoughts?

[matejsp]

Say you roll back because its critical, or its not critical so you have a bit of time to fix it -- in either case, the path forward would be to isolate the offending upgrade and pin that in your Pipfile and re-lock pipenv lock and re-test.

My issue was how to isolate the offending upgrade (since we couldn't reproduce it in QA). So for each dep & pin & deploy to production, rollback if perf degradation & repeat.. And at the end unpin all the deps that were not problematic.

We can both agree that your ideal way does not fit how we do it :) Did I mention our SecOps team that is doing a fast upgrade of only CVE impacted deps and not touching others because the hotfix release will go fast to prod without much testing. Like last two CVEs in Django this month.

Sounds like there are numerous bugs in --selective-upgrade as well -- I am wondering though, as I've thought more about what an ideal behavior would be. Either --selective-upgrade or pipenv upgrade could consider resolving the package in question to figure out which packages are required (not necessarily the versions), then run a complete lock phase and pull the required packages (determined from the isolated resolve phase) and their allowed versions (determined from the complete lock phase) and just merge in those updates to the Lock file. This may be more deterministic and prevent inconsistent lock files (which is what I am trying to solve for here with removing --keep-outdated) -- it does mean that if upgrading requests brings in a new package or requires one you already have to be updated, that it would be changed in the lock file, but the idea is everything else would be left the same. Thoughts?

That would be ok solution, but really can't speculate about corner cases. One thing that I don't understand is if you produce corrupted lock file for some reason, why isn't there a validation at the end? Better to show error than to silently produce corrupted lock file? In this case developer could update/pin more libraries to help pipenv resolver (or file a bug report).

Now to the fun part :D PoC :)
After I tried how it fares with django and django-model-utils I tested also our env with poetry (using https://pypi.org/project/pipenv-poetry-migrate/). It is a good case since it's big and ugly monolith :D

Differences:

• poetry lock --no-update: (works the same as --keep-outdated) Do not update locked versions, only refresh lock file.
• It is brutally fast. Pipenv takes 90 seconds each time to process lock file while poetry's first time was around 90 seconds, but now it is cached and we are under 10 seconds. bazinga
• Wierd behaviour in poetry that is pulling LONG obsolete dependencies (futures, backport-abc, singledispatch) while pipenv doesn't (Poetry not honouring (deprecated) dependencies with tornado 5.1.1 python-poetry/poetry#7534).
• Poetry lock file hashes are tied to a wheel name (so it is easy to see if any platform is missing)
• Poetry lock file has information about dependencies (easier to find issues)
• Pipenv lock file is json and poetry is toml (I still prefer json)
• Both support pyenv
• I haven't really do a proper testing

Example from lock file:

[[package]]
name = "sphinx"
version = "5.3.0"
description = "Python documentation generator"
category = "dev"
optional = false
python-versions = ">=3.6"
files = [
    {file = "Sphinx-5.3.0.tar.gz", hash = "sha256:51026de0a9ff9fc13c05d74913ad66047e104f56a129ff73e174eb5c3ee794b5"},
    {file = "sphinx-5.3.0-py3-none-any.whl", hash = "sha256:2724a6c7894cdbd21b011446d1b92f9b8201741fb826483fcdd3caae3af009c3"},
]

[package.dependencies]
alabaster = ">=0.7,<0.8"
babel = ">=2.9"
colorama = {version = ">=0.4.5", markers = "sys_platform == \"win32\""}
docutils = ">=0.14,<0.20"
imagesize = ">=1.3"
importlib-metadata = {version = ">=4.8", markers = "python_version < \"3.10\""}
Jinja2 = ">=3.0"
packaging = ">=21.0"
Pygments = ">=2.12"
requests = ">=2.5.0"
snowballstemmer = ">=2.0"
sphinxcontrib-applehelp = "*"
sphinxcontrib-devhelp = "*"
sphinxcontrib-htmlhelp = ">=2.0.0"
sphinxcontrib-jsmath = "*"
sphinxcontrib-qthelp = "*"
sphinxcontrib-serializinghtml = ">=1.1.5"

[matteius]

That would be ok solution, but really can't speculate about corner cases.

There would be fewer corner cases than --keep-outdated because keep outdated doesn't update a sub-dependency if its already present in your lockfile (but needs to be updated).

One thing that I don't understand is if you produce corrupted lock file for some reason, why isn't there a validation at the end? Better to show error than to silently produce corrupted lock file? In this case developer could update/pin more libraries to help pipenv resolver (or file a bug report).

There is validation if the lock phase cannot be resolved -- but I don't understand your comment. As far as I know, we don't produce a "corrupted lock file", its just that using --keep-outdated will not consider the case of:

resolving the package in question to figure out which packages are required (not necessarily the versions), then run a complete lock phase and pull the required packages (determined from the isolated resolve phase) and their allowed versions (determined from the complete lock phase) and just merge in those updates to the Lock file.

That is to say, the correct way to isolate a package upgrade is to consider all the required packages that it has and ensure they all meet specified version requirements, which is not what --keep-outdated does. The problem you are running into is complete resolver phase includes other package upgrades that are not specific to your upgrade in question. The biggest corner case I could see with this new approach is it maybe would leave behind old stale packages in the lock file because it wouldn't be taking the complete result of the new lock (say package A required package B, but now requires C--your lock file would likely have A, B and C, but at least the versions would not conflict against what the authors allowed).

Last note -- I do know that poetry uses its own resolver different from pip, so that can explain some of the differences you are seeing there.

[matejsp]

There is validation if the lock phase cannot be resolved -- but I don't understand your comment. As far as I know, we don't produce a "corrupted lock file", its just that using --keep-outdated will not consider the case of

Sorry from earlier comments I was under the impression that --keep-outdated sometimes produces inconsistent/corrupted lock files.

What we observed are the following cases:

• hashes were not updated to new version, but version was -> resulting in a corrupted lock (second run usually fixes this)
• new hashes were added but old ones for old version were not cleared -> soft corrupted since lock is still working (we manually delete them from lock file)
• no longer needed transient dependencies due to version upgrade were not removed -> soft corruption, we manually deleted them from lock file

Regarding A,B,C case ... try updating, if conflict, detect and report an error :) no need to do a magic here and auto resolve ... say C is incompatible and let the developer pin the C to correct version.

IMHO if it is hard to maintain the magic with keep-outdated ... make it simpler :D it will still cover 95% use cases. Alternative as --selective-upgrade are fine too if working :)

[dimbleby]

Just a drive-by comment on the claim that

poetry does let you install incompatible versions of things together

I see no incompatibility between django-model-utils 4.2.0 and django 2.*. The requirement from django-model-utils is for django >= 2.0.1

I know nothing about pipenv but so far as I can see this output is incorrect:

pipenv upgrade django-model-utils==4.2.0
 You cannot upgrade because django-model-utils requires Django >= 3.0