[idea] Compile from install_requires to an extras_require

[JulienPalard]

What's the problem this feature will solve?

It'll allow to easily do with pip what some other packaging/installing/things tools are trying to solve: installing with pinned dependencies.

Currently I'm having a project (an application) that I deploy in a venv: I first build it, upload it to PyPI, and install it using pip install on my servers. As the app is alone in its venv, and as the pinned dependencies from pip-compile are the one used during testing¹, I'd like to use those pinned dependencies during the installation too.

1. And as I have no idea how it's possible to check for all dependencies combinations in a search for the "minimal acceptable working set", I think distro packagers are doing magic...

Describe the solution you'd like

I'd like a new option, maybe --to-extra EXTRA, so pip-compile --to-extra pinned . to emit the pinned dependencies in an extras_require.pinned section of setup.cfg instead of in a requirements.txt file.

(pip-compile . already works with a pyproject.toml and a setup.cfg, no need for a setup.py file)

It would allow both kind of installations:

• pip install . to install with "large", "loosely pinned", "normal" dependencies (some >= when it's needed, as it should be), and
• pip install .[pinned] to install with pinned dependencies.

Also it could be used as pip-compile --to-extra dev-pinned .[dev] to compile an extras_require.dev to an extras_require.dev-pinned.

For oeis for example, it would look like this:

[options]
py_modules = oeis
python_requires = >= 3.7
install_requires =
  matplotlib
  sympy

[options.entry_points]
console_scripts = oeis=oeis:main

[options.extras_require]
pinned=
  cycler==0.10.0
      # via matplotlib
  kiwisolver==1.3.2
      # via matplotlib
  matplotlib==3.4.3
      # via oeis (pyproject.toml)
  mpmath==1.2.1
      # via sympy
  numpy==1.21.2
      # via matplotlib
  pillow==8.3.2
      # via matplotlib
  pyparsing==2.4.7
      # via matplotlib
  python-dateutil==2.8.2
      # via matplotlib
  six==1.16.0
      # via
      #   cycler
      #   python-dateutil
  sympy==1.8
      # via oeis (pyproject.toml)

Alternative Solutions

An alternative solution would be to use file: in the setup.cfg file for the install_requires and extra_require, pointing to requirements.in and requirements.txt respectively, like pypa/setuptools#1951.

Another alternative solution would be not to let the user choose the section name (or suffix), to ensure consistency between users, I can think of recommended it make sense, but it's too long I think, so pinned looks good to me, explicit enough, it would simplify the command, instead of typing pip-compile --to-extra pinned . one could do pip-compile --to-extra ..

Bonus

As a bonus, the repo would have less files (I already don't have a requirements.in, I use my setup.cfg's install_requires for this): it'll allow not to have a requirement.txt as they're stored in setup.cfg. One could move like all requirements files to setup.cfg in fact! Less files! setup.cfg FTW!

[AndydeCleyre]

I don't know if any part of this will be useful or interesting, but FWIW... in my frontend to pip-tools, there are functions to achieve a similar result, but targeting the pyproject.toml instead of the setup.cfg, and sourcing data from req files. So it admittedly does not achieve the total consolidation you're after.

$ git clone git@github.com:JulienPalard/oeis
$ cd oeis
$ a8  # create/activate venv
$ pipac 'matplotlib>=2<4' 'sympy>=1<2'  # add lines to requirements.in then compile to requirements.txt
$ pipa -c pinned '-r requirements.txt'  # add line to pinned-requirements.in
$ pypc
$ git status --short
 M pyproject.toml
?? pinned-requirements.in
?? requirements.in
?? requirements.txt
$ git diff

diff --git a/pyproject.toml b/pyproject.toml
index 221883f..bc11878 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -6,3 +6,9 @@ build-backend = "setuptools.build_meta"
 markers = [
     "slow: marks tests as slow (deselect with '-m \"not slow\"')",
 ]
+
+[project]
+dependencies = ["matplotlib>=2<4", "sympy>=1<2"]
+
+[project.optional-dependencies]
+pinned = ["cycler==0.11.0", "fonttools==4.29.1", "kiwisolver==1.3.2", "matplotlib==3.5.1", "mpmath==1.2.1", "numpy==1.22.2", "packaging==21.3", "pillow==9.0.1", "pyparsing==3.0.7", "python-dateutil==2.8.2", "six==1.16.0", "sympy==1.9"]

It happens in pypc, which is a little shell wrapping a little python using tomlkit to inject.

[apljungquist]

Have you looked at doing something like pip install -c requirements.txt .? Or better yet, PIP_CONSTRAINT=requirements.txt pip install ..

The latter ensures that the version locks are respected during isolated build, if it occurs. But beware of pip-compile ignoring build-system.requires or extras when compiling from pyproject.toml.

[JulienPalard]

This is a bit different:

• In my solution the packager chooses the pinned dependencies, so users of the package can opt-in to use them.
• In your solution the system administrator chooses the pinned dependencies (maybe versioning them in their Ansible playbooks), to upload them to the server before the installation.

(I don't see a clean way for the packager to distribute the constraint file.)

[apljungquist]

I think I understand now. So then one could do something like pipx install oeis[pinned] to install the app with the dependencies it has been tested with? Interesting idea.

Do you explicitly want to not use a setup.py file to read the dependencies from file?

Do you want to support multiple python versions, operating systems, etc? I believe pip-tools excludes any dependencies that are not enabled for the current environment.

[JulienPalard]

Do you explicitly want to not use a setup.py file to read the dependencies from file?

Yes, I'm more on the side of PEP 621 pyproject.toml-only packaging today. At the time I wrote the issue I was using setup.cfg+pyproject.toml but no setup.py.

Do you want to support multiple python versions, operating systems, etc?

That's the hard point. Since I wrote this issue I encontered real cases were pinning a version of Python would make the project not installable with another Python version (and almost stopped pinning any dependencies since then).

For Python versions it may be solvable (#1326). For multiple OSes I don't know I have the same OS on all my laptops/desktops/servers ;)