Remove implicit setuptools dependency

[jaraco]

What's the problem this feature will solve?

Currently, pip has an implicit dependency on setuptools to build legacy packages (pre PEP 517). Removing setuptools from a build environment will cause builds of these packages or install of these packages from source to fail.

For example:

$ python -m venv env
$ env/bin/pip uninstall -y setuptools
$ env/bin/pip install --no-cache-dir termcolor
Collecting pytest-virtualenv
  Downloading https://files.pythonhosted.org/packages/21/7f/0ec1ab9b8fcd4a3a063b97143a97d1c093d01227353b682c9027d14b14c2/pytest_virtualenv-1.7.0-py2.py3-none-any.whl
Collecting virtualenv (from pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/05/f1/2e07e8ca50e047b9cc9ad56cf4291f4e041fa73207d000a095fe478abf84/virtualenv-16.7.9-py2.py3-none-any.whl (3.4MB)
     |████████████████████████████████| 3.4MB 2.0MB/s 
Collecting pytest (from pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/e6/94/29b5a09edc970a2f24742e6186dfe497cd6c778e60e54693215a36852613/pytest-5.3.3-py3-none-any.whl (235kB)
     |████████████████████████████████| 245kB 26.2MB/s 
Collecting pytest-fixture-config (from pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/83/6e/ebc4d3d5c51cc440bdb0adcd274ff2436dc6814ed1cc2cbc6d5386aaf85c/pytest_fixture_config-1.7.0-py2.py3-none-any.whl
Collecting pytest-shutil (from pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/26/b7/ef48a8f1f81ae4cd6f22992f6ffb7e9bf030d6e6654e2e626a05aaf5e880/pytest_shutil-1.7.0-py2.py3-none-any.whl
Collecting attrs>=17.4.0 (from pytest->pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl
Collecting py>=1.5.0 (from pytest->pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/99/8d/21e1767c009211a62a8e3067280bfce76e89c9f876180308515942304d2d/py-1.8.1-py2.py3-none-any.whl (83kB)
     |████████████████████████████████| 92kB 20.7MB/s 
Collecting more-itertools>=4.0.0 (from pytest->pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/bc/e2/3206a70758a21f9878fcf9478282bb68fbc66a5564718f9ed724c3f2bb52/more_itertools-8.1.0-py3-none-any.whl (41kB)
     |████████████████████████████████| 51kB 67.3MB/s 
Collecting pluggy<1.0,>=0.12 (from pytest->pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/a0/28/85c7aa31b80d150b772fbe4a229487bc6644da9ccb7e427dd8cc60cb8a62/pluggy-0.13.1-py2.py3-none-any.whl
Collecting wcwidth (from pytest->pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/58/b4/4850a0ccc6f567cc0ebe7060d20ffd4258b8210efadc259da62dc6ed9c65/wcwidth-0.1.8-py2.py3-none-any.whl
Collecting packaging (from pytest->pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/d8/5b/3098db49a61ccc8583ffead6aedc226f08ff56dc03106b6ec54451e27a30/packaging-20.0-py2.py3-none-any.whl
Collecting execnet (from pytest-shutil->pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/d3/2e/c63af07fa471e0a02d05793c7a56a9f7d274a8489442a5dc4fb3b2b3c705/execnet-1.7.1-py2.py3-none-any.whl
Collecting contextlib2 (from pytest-shutil->pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/85/60/370352f7ef6aa96c52fb001831622f50f923c1d575427d021b8ab3311236/contextlib2-0.6.0.post1-py2.py3-none-any.whl
Collecting termcolor (from pytest-shutil->pytest-virtualenv)
  Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
    ERROR: Command errored out with exit status 1:
     command: /Users/jaraco/code/public/pypa/setuptools/env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-nsnw5epl/termcolor/setup.py'"'"'; __file__='"'"'/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-nsnw5epl/termcolor/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-install-nsnw5epl/termcolor/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

This issue is often masked by the fact that pip will cache the wheel, so if the wheel was built with one environment, it might succeed in the next without setuptools due to the cache, leading to situations where the same code will run successfully in some CI environments (with cache support) but fail in others.

I encountered this issue in setuptools test suite itself, as I'm attempting to install pytest-virtualenv which depends transitively on termcolor, and would like to be able to install those test requirements without the need for an implicit setuptools.

Describe the solution you'd like

Why can't pip, for legacy projects, supply an implied 'pyproject.toml':

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta:__legacy__"

This would effectively build the project using those dependencies (if not already met), and would allow for old, unmaintained projects like termcolor to be installed in environments with only pip.

As an illustration:

draft $ pip freeze --all
argcomplete==1.10.0
Click==7.0
pip==19.3.1
pip-run==5.3
pipx==0.14.0.0
userpath==1.3.0
draft $ http -d https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
HTTP/1.1 200 OK
...
Downloading 3.82 kB to "termcolor-1.1.0.tar.gz"
Done. 3.82 kB in 0.00059s (6.36 MB/s)
draft $ tar xfz termcolor-1.1.0.tar.gz
draft $ cd termcolor-1.1.0/
termcolor-1.1.0 $ cat > pyproject.toml
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta:__legacy__"
termcolor-1.1.0 $ pip wheel .                                                                                                                                                  
Processing /Users/jaraco/draft/termcolor-1.1.0
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: termcolor
  Building wheel for termcolor (PEP 517) ... done
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=2ccf30d5778fdef8c1cac2d2c24697e8ffdaae5bbb99d01a39b1faaec7694251
  Stored in directory: /Users/jaraco/draft/termcolor-1.1.0
Successfully built termcolor

Alternative Solutions

The alternative is that every environment needs one of:

1. maintain this implicit dependency
2. avoid installing legacy packages
3. update all legacy packages to have wheels

[jaraco]

This feature could be rolled out incrementally, first as an opt-in feature, through a command-line arg / environment var, that would address the concern from my perspective and early-adopters could elect to beta-test the behavior. Then, it could be the default with an opt-out feature, and then ultimately it could be the default behavior.

Once this behavior is the default, it should substantially simplify the code, eliminating the current code path where setup.py is invoked with the setuptools wrapper.

[pfmoore]

Why can't pip, for legacy projects, supply an implied 'pyproject.toml'

That's essentially what the --use-pep517 flag will do. So in effect, you're asking for --use-pep517 to be the default. We made the decision at the time of implementing PEP 517 support to make it opt-in rather than opt-out, to minimise disruption. We could revisit that decision now that some time has passed, but I suspect the outcome will be the same. Any project that wants to opt into PEP 517 can do so by supplying a pyproject.toml, and any that don't are potentially impacted by the semantic differences between "legacy" mode and "PEP 517 with no pyproject.toml" mode:

1. If wheel isn't available in the build environment, legacy mode does a direct setup.py install, which is very different from building a wheel and installing that.
2. Build isolation - this would trigger build isolation, which is a definite semantic change.

... and quite possibly others.

I don't know what would constitute a reasonable transition plan for moving to --use-pep517 being the default, and then to having it be the only option. We'd need some way of judging how many projects would break as a result of such a change, and how hard they would be to fix, and I don't think anyone has a good way of getting that information (short of making the change and listening for the screams, which is not an option I'm particularly keen on - the packaging tools don't have a particularly good track record on backward compatibility, and as a result we should probably be cautious about the risks we take).

Note: Your proposal might not be identical to --use-pep517, but I see little point in expending effort (and risk) in changing legacy (direct execution of setup.py) install mode - that code is there solely to support projects that can't or won't move to the modern source format, and should therefore remain semantically unchanged, as such projects will be equally unable to react to semantic changes.

Note 2: If you prefer PEP 517 behaviour by default, you can set --use-pep517 in your config file. But in that case, you'd have to be prepared to deal with projects that don't build correctly in that mode.

[jaraco]

Oh, wow. I think I might have heard about --use-pep517, but I was unaware. Yeah, that's pretty much what I was looking for as to the opt-in feature. Setuptools may be able to use that to avoid issues bootstrapping its test dependencies. I'll explore more, but that does seem to be the ticket.

[jaraco]

Works like a charm:

~ $ pip freeze --all                                                                                                                                                           
argcomplete==1.10.0
Click==7.0
pip==20.0.1
pip-run==5.3
pipx==0.14.0.0
userpath==1.3.0
~ $ pip-run --no-cache-dir --use-pep517 termcolor -- -m pip freeze                                                                                                             
Collecting termcolor
  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: termcolor
  Building wheel for termcolor (PEP 517) ... done
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=42caf6b58ce04c99c6cac5e5a6127dc1f1ef2b3873965727d510f159e3e22469
  Stored in directory: /private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pip-ephem-wheel-cache-e2ty0ri4/wheels/a0/16/9c/5473df82468f958445479c59e784896fa24f4a5fc024b0f501
Successfully built termcolor
Installing collected packages: termcolor
Successfully installed termcolor-1.1.0
argcomplete==1.10.0
Click==7.0
pip-run==5.3
pipx==0.14.0.0
termcolor==1.1.0
userpath==1.3.0