Pinned version in dev-packages ignored when used by main packages

[juanitosvq]

Hi, this may be expected behaviour but I wanted to double check. Couldn't find a clear answer in the docs or in an issue.

Issue description

When I pin a dependency in my dev-packages, I would expect pipenv to honour that and either:

• Install that version as part of the dev packages
• Or fail with a resolution failure if there is another package trying to install a different version from the one I specified.

Expected result

In my example below, if I pin packaging to v21.3 in my dev-packages, I would expect to either see that version or get resolution failure.

Actual result

In my example below, given that apispec requires any version of packaging, it is installing v22.0 in both packages and dev-packages. I wouldn't expect to see this version in dev-packages, instead I would expect to see v21.3 or resolution failure.

Steps to replicate

Provided pipenv --support output below.

Thanks in advance.

---

Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.

$ pipenv --support

Pipenv version: '2022.11.30'

Pipenv location: '/home/REDACTED/lib/python3.9/site-packages/pipenv'

Python location: '/home/REDACTED/bin/python'

OS Name: 'posix'

User pip version: '22.3'

user Python installations found:

• 3.9.16: /home/REDACTED/bin/python3.9
• 3.9.16: /home/REDACTED/bin/python3
• 3.9.16: /home/REDACTED/bin/python
• 3.9.16: /home/REDACTED/bin/python3.9
• 3.9.16: /home/REDACTED/bin/python3
• 3.9.16: /home/REDACTED/bin/python
• 3.9.16: /usr/bin/python3.9
• 3.9.16: /bin/python3.9
• 3.8.10: /usr/bin/python3
• 3.8.10: /usr/bin/python3.8
• 3.8.10: /bin/python3
• 3.8.10: /bin/python3.8
• 3.7.16: /usr/bin/python3.7m
• 3.7.16: /usr/bin/python3.7
• 3.7.16: /bin/python3.7m
• 3.7.16: /bin/python3.7
• 2.7.18: /usr/bin/python2.7
• 2.7.18: /usr/bin/python2
• 2.7.18: /bin/python2.7
• 2.7.18: /bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.16',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.15.0-56-generic',
 'platform_system': 'Linux',
 'platform_version': '#62~20.04.1-Ubuntu SMP Tue Nov 22 21:24:20 UTC 2022',
 'python_full_version': '3.9.16',
 'python_version': '3.9',
 'sys_platform': 'linux'}

System environment variables:

• SHELL
• SESSION_MANAGER
• QT_ACCESSIBILITY
• NVM_RC_VERSION
• SNAP_REVISION
• XDG_CONFIG_DIRS
• XDG_MENU_PREFIX
• GNOME_DESKTOP_SESSION_ID
• SNAP_REAL_HOME
• TERMINAL_EMULATOR
• SNAP_USER_COMMON
• LANGUAGE
• GNOME_SHELL_SESSION_MODE
• SSH_AUTH_SOCK
• TERM_SESSION_ID
• SNAP_INSTANCE_KEY
• XMODIFIERS
• DESKTOP_SESSION
• SSH_AGENT_PID
• BAMF_DESKTOP_FILE_HINT
• GTK_MODULES
• PWD
• XDG_SESSION_DESKTOP
• LOGNAME
• XDG_SESSION_TYPE
• GPG_AGENT_INFO
• XAUTHORITY
• DESKTOP_STARTUP_ID
• SNAP_CONTEXT
• GJS_DEBUG_TOPICS
• WINDOWPATH
• HOME
• USERNAME
• IM_CONFIG_PHASE
• LANG
• LS_COLORS
• XDG_CURRENT_DESKTOP
• VIRTUAL_ENV
• SNAP_ARCH
• SNAP_INSTANCE_NAME
• SNAP_USER_DATA
• INVOCATION_ID
• MANAGERPID
• SNAP_REEXEC
• GJS_DEBUG_OUTPUT
• NVM_DIR
• LESSCLOSE
• XDG_SESSION_CLASS
• TERM
• AWS_EIO_CD_PROFILE
• LESSOPEN
• USER
• SNAP
• SNAP_COMMON
• SNAP_VERSION
• DISPLAY
• SHLVL
• NVM_CD_FLAGS
• SNAP_LIBRARY_PATH
• SNAP_COOKIE
• QT_IM_MODULE
• SNAP_DATA
• XDG_RUNTIME_DIR
• PS1
• SNAP_NAME
• JOURNAL_STREAM
• XDG_DATA_DIRS
• PATH
• GDMSESSION
• DBUS_SESSION_BUS_ADDRESS
• GIO_LAUNCHED_DESKTOP_FILE_PID
• GIO_LAUNCHED_DESKTOP_FILE
• _
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PYTHONDONTWRITEBYTECODE
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/REDACTED/bin:/home/REDACTED/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
• SHELL: /bin/bash
• LANG: en_CA.UTF-8
• PWD: /home/REDACTED
• VIRTUAL_ENV: /home/REDACTED

---

Contents of Pipfile ('/home/REDACTED/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
apispec = "*"

[requires]
python_version = "3.9"

[pipenv]
allow_prereleases = true

[dev-packages]
packaging = "==21.3"

Contents of Pipfile.lock ('/home/REDACTED/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "46f6381ed3ae611fdfc008749d0e138dacb07e741b0b7888ecc136adbbfad371"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "apispec": {
            "hashes": [
                "sha256:d97f0ae9c65133185b9ed9c5be1a434eb85627dfa33c4c53cabda122256c1b67",
                "sha256:e76d80b739edef4be213092a6384ad7fd933ba7d64f6d5a0aff8d4da1bef7887"
            ],
            "index": "pypi",
            "version": "==6.0.2"
        },
        "packaging": {
            "hashes": [
                "sha256:2198ec20bd4c017b8f9717e00f0c8714076fc2fd93816750ab48e2c41de2cfd3",
                "sha256:957e2148ba0e1a3b282772e791ef1d8083648bc131c8ab0c1feba110ce1146c3"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==22.0"
        }
    },
    "develop": {
        "packaging": {
            "hashes": [
                "sha256:2198ec20bd4c017b8f9717e00f0c8714076fc2fd93816750ab48e2c41de2cfd3",
                "sha256:957e2148ba0e1a3b282772e791ef1d8083648bc131c8ab0c1feba110ce1146c3"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==22.0"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        }
    }
}

[matteius]

@juanitosvq Yeah this is expected behavior; there exists a compromise between the desire to not have something like django-debug-toolbar upgrade the version of Django in dev that is pinned in packages. During the addition of named package categories I decided to continue having the default or packages group continue to constrain (this is pip terminology, we use a constraint file for this) the other package groups.

For users that do not want this behavior I am considering a global pipenv option to not constrain from default, but right now the recommendation is to use named package categories. For your example you could have a prod group that included the apispec version you wanted, or not use default/packages group at all and nothing would be constrained this way.

I guess if there is enough interest in an option in the Pipfile that does not constrain when locking, we can do it. Then the support flow becomes --> default is constrained (issue reported) --> you switch to use no constraints (different issue reported) --> Compromise and keep things you want constrained in default packages group, and things you don't want constrained defined in other named categories.

The documentation could/should be improved, but there is a note about it here: https://pipenv.pypa.io/en/latest/basics/#specifying-package-categories

[juanitosvq]

Thanks for the prompt and detailed response, @matteius . I wasn't aware of the named package categories feature, I think that may help us sorting out the issue we are having, thanks!

[yukihiko-shinoda]

For now, the way I got succeed to get expected behavior with pipenv==2023.2.4 is:

Succeed example

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

- [packages]
+ [prod-packages]
apispec = "==6.0.2"

- [dev-packages]
+ [packages]
packaging = "==21.3"
colorama = "==0.4.6"

Then:

$ pipenv lock

Creating a virtualenv for this project...
Pipfile: /workspace/Pipfile
Using /usr/local/bin/python (3.11.1) to create virtualenv...
⠙ Creating virtual environment...created virtual environment CPython3.11.1.final.0-64 in 120ms
  creator CPython3Posix(dest=/workspace/.venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==22.3.1, setuptools==66.0.0, wheel==0.38.4
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /workspace/.venv
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Locking [dev-packages] dependencies...
Locking [prod-packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success!
Updated Pipfile.lock (11b641784365aea8803e3ee8c4be5a3a43e6691a28cd2b0b0bfeac06a83318f4)!

Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "11b641784365aea8803e3ee8c4be5a3a43e6691a28cd2b0b0bfeac06a83318f4"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "colorama": {
            "hashes": [
                "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44",
                "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"
            ],
            "index": "pypi",
            "version": "==0.4.6"
        },
        "packaging": {
            "hashes": [
                "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
                "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
            ],
            "index": "pypi",
            "version": "==21.3"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        }
    },
    "develop": {},
    "prod-packages": {
        "apispec": {
            "hashes": [
                "sha256:d97f0ae9c65133185b9ed9c5be1a434eb85627dfa33c4c53cabda122256c1b67",
                "sha256:e76d80b739edef4be213092a6384ad7fd933ba7d64f6d5a0aff8d4da1bef7887"
            ],
            "index": "pypi",
            "version": "==6.0.2"
        },
        "packaging": {
            "hashes": [
                "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
                "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
            ],
            "index": "pypi",
            "version": "==21.3"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        }
    }
}

To create development environment:

pipenv sync --categories='packages prod-packages'

Result:

$ pipenv run pip list
Package    Version
---------- -------
apispec    6.0.2
colorama   0.4.6
packaging  21.3
pip        22.3.1
pyparsing  3.0.9
setuptools 66.0.0
wheel      0.38.4

The command to deploy into production:

pipenv install --deploy --categories=prod-packages

Result:

$ pipenv run pip list
Package             Version
------------------- -------
Package    Version
---------- -------
apispec    6.0.2
packaging  21.3
pip        22.3.1
pyparsing  3.0.9
setuptools 66.0.0
wheel      0.38.4

Failed example

The following is failed:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

- [packages]
+ [prod-packages]
apispec = "==6.0.2"

[dev-packages]
packaging = "==21.3"
colorama = "==0.4.6"

The locked packaging versions were different between development and production:

Pipfile.lock

{
    "_meta": {
        "hash": {
            "sha256": "23f267f4e6a9557b26caadaf4a4574299298594ec6690a5d0b64fe39ef568023"
        },
        "pipfile-spec": 6,
        "requires": {},
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {
        "colorama": {
            "hashes": [
                "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44",
                "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"
            ],
            "index": "pypi",
            "version": "==0.4.6"
        },
        "packaging": {
            "hashes": [
                "sha256:dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb",
                "sha256:ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522"
            ],
            "index": "pypi",
            "version": "==21.3"
        },
        "pyparsing": {
            "hashes": [
                "sha256:2b020ecf7d21b687f219b71ecad3631f644a47f01403fa1d1036b0c6416d70fb",
                "sha256:5026bae9a10eeaefb61dab2f09052b9f4307d44aee4eda64b309723d8d206bbc"
            ],
            "markers": "python_full_version >= '3.6.8'",
            "version": "==3.0.9"
        }
    },
    "prod-packages": {
        "apispec": {
            "hashes": [
                "sha256:d97f0ae9c65133185b9ed9c5be1a434eb85627dfa33c4c53cabda122256c1b67",
                "sha256:e76d80b739edef4be213092a6384ad7fd933ba7d64f6d5a0aff8d4da1bef7887"
            ],
            "index": "pypi",
            "version": "==6.0.2"
        },
        "packaging": {
            "hashes": [
                "sha256:714ac14496c3e68c99c29b00845f7a2b85f3bb6f1078fd9f72fd20f0570002b2",
                "sha256:b6ad297f8907de0fa2fe1ccbd26fdaf387f5f47c7275fedf8cce89f99446cf97"
            ],
            "markers": "python_version >= '3.7'",
            "version": "==23.0"
        }
    }
}

[yukihiko-shinoda]

Can't it be thought that many people who use dependency management tools do so because they want to ensure same packages in test and production environments?
And aren't there so many users who don't know this specification change of behavior?

#5366

https://github.com/pypa/pipenv/pull/5366/files#diff-ef852c4ac364f946819f765a6bc26f04f1b0968f31fc512949a60fa2ab0685e8R1027

Behavior of lock function in representative dependency management tools

I checked followings:

• PHP Composer
• Ruby Bundler
• Python Poetry
• Python PDM
• Node Npm
• Node Yarn
• Golang Go Modules
• Java Maven
• Java Gradle

Almost dependency management tools lock by both default category and dev-package.
If dev-package category locks lower version than default category, locked version is adjusted to lower version.
Only Gradle prioritize default category than dev-package category, however it locks into only 1 version of each package which is common with all categories, and it also supports function to pin transitive dependencies.