Making setuptools the reference implementation of the distutils API

[ncoghlan]

For the last few years, we've stopped updating distutils to support new packaging interoperability standards, and have instead recommended that folks that want to rely on features in metadata 1.2+, or to generate wheel archives, switch to using setuptools. This has allowed the packaging metadata support to evolve independently of the exact version of Python that folks are using to publish their packages, and it's worked well enough that I'm willing to call the approach a success.

However, it has left us with a maintainability issue, which is that we still have a distutils API implementation shipping with CPython that we don't actually want people to be using, and the source of the problem isn't always obvious when folks using that implementation aren't getting the behaviour they expect or want.

pip has long injected the setuptools monkeypatches of distutils before running setup.py scripts, and the setuptools folks are considering switching to providing their own setuptools-maintained copy of distutils rather than continuing to monkeypatch the standard library version at import time: pypa/setuptools#417

I think the cleanest way to resolve this will be to actively seek to switch distutils over to the ensurepip model of maintenance: add an ensuredistutils module to the standard library that installs bundled versions of setuptools and wheel, just as ensurepip installs a bundled copy of pip.

Identified substeps:

• Agreement from at least me, @jaraco, and @dstufft that this is the direction we want to start moving
• Adopt distutils setuptools#417 (removes setuptools dependency on stdlib distutils)
• Make the setuptools documentation self-contained (rather than being a delta on top of the distutils documentation). This will mean also covering the injected distutils API in the setuptools documentation.
• Write a PEP for an ensuredistutils module targeting either Python 3.9 (feature freeze early 2020) or 3.10 (feature freeze early 2021)
• As part of that PEP, consider whether or not we might want to use the opportunity to make the installation of distutils entirely opt-in (given that pip 10+ will automatically install setuptools+wheel when asked to build a distribution from source that doesn't specify a different build system, and that any projects relying on pkg_resources at runtime should be explicitly declaring a dependency on setuptools anyway)
• Assuming the PEP is accepted, make the required changes to CPython and its documentation
• Assuming the PEP is accepted, update the triaging instructions in the CPython developer guide to advise that distutils bugs reports and enhancement requests should be closed as "third party", and the submitters redirected to the setuptools issue tracker.

[ncoghlan]

This was prompted in part by https://bugs.python.org/issue33069, where the fact distutils is still mostly metadata 1.1 focused means it doesn't set the maintainer fields in the metadata (since they were new in metadata 1.2).

[jaraco]

Overall, this proposal sounds good to me and is the right direction.

add an ensuredistutils module

Why not ensuresetuptools (as that's what's being ensured and consistent with what ensurepip does)? I would actually prefer ensure-setuptools, though there's some value in the runtogether form for consistency with ensurepip. I use this dash-separated form for runpy scripts that I expect users to invoke (has the added value of not being importable by accident, making it a pure runner).

This effort might run into subtle conflicts with the effort in pypa/setuptools#863, to make pkg_resources an independent package... or maybe not. Based on the outcome of pypa/setuptools#581, setuptools must vendor its dependencies, so pkg_resources will just be one of those vendored dependencies, but that will change:

and that any projects relying on pkg_resources at runtime should be explicitly declaring a dependency on setuptools anyway

Instead to require those packages to depend on pkg_resources.

It sounds like a good plan. Perhaps we can even secure some sponsorship to hire assistance with the implementation.

[dstufft]

I'm still digesting this idea, however to play a bit of a devil's advocate here, is ensure(distutils|setuptools) even required?

As we move into a post PEP 518 world we are eliminating the bootstrapping problem for tools like setuptools, where people no longer need to manually ensure that they are installed prior to attempting to install anything that uses them. Thus the eventual goal it seems to me should be that nothing needs to be preinstalled except for a package manager that understands and supports PEP 518. Right now that tool is pip, but it could be something else a decade from now.

[jaraco]

I agree. It's worth serious consideration introducing yet another ensure step (ideally, there'd be 0). I do feel that perhaps it's an anti-pattern to be avoided.

The issue I've encountered is one of bootstrapping. Thinking about a system where one wishes to build an environment from source without going to the network. One needs a mechanism to make those PEP 518 dependencies available. If one of those is setuptools, how does one make setuptools available, especially when setuptools depends on packaging (among others), which in turn depends on setuptools to build? Can pip (or another build tool), using the metadata and source tarballs of setuptools and its dependencies, build setuptools? It's not sufficient that pip can install from wheels, because these packaging tools (like Spack) wish for compliance and principles of open source, to always build from source.

The ensure-setuptools approach addresses that, somewhat tangentially, by abstracting the bootstrapping challenge. Packaging tools don't have to care about how the sausage is made as long as they get setuptools when they ask for it.

Moreover, an important goal we have is that no build tool should be elevated. I'd like to create a packaging environment where setuptools isn't a specially-blessed package, but could be theoretically (if not in practice) superseded by another build tool. Baking ensure-setuptools into the packaging model violates that principle.

In light of these considerations, I think I'm now -1 on the proposal for another ensure*. Perhaps it makes more sense to focus on finalizing and refining the PEP 518 enhancements and see if that does in fact achieve the promise of making setuptools a largely opt-in. If that's the case, then we can still do the other items in the list, but skip 4 entirely.

[pfmoore]

I agree with @dstufft - under PEP 518, we don't need any distutils implementation in the standard library, except as a support library used by setuptools. IMO, this fits in with the spirit of PEP 517 and 518, making setuptools just another contender in the build system ecosystem, with no special privilege,

This would still mean that setuptools would need to extend its documentation to include the core distutils API.

There would remain the question of what to do with the core distutils code. Doing nothing basically leaves things unchanged, so that's not much help. If pypa/setuptools#417 were implemented, core distutils could simply be dropped (although other tools may use it - I don't think flit does, but I didn't check thoroughly). Alternatively, as it's perfectly OK under PEP 518 for build tools to have dependencies, we could simply make distutils a 3rd party library and have setuptools (and any other build tools that use distutils functionality) depend on it. People wouldn't be able to mistakenly write projects that used raw distutils to build, as distutils itself has no wheel building support (so it doesn't conform to PEP 517/518).

[ncoghlan]

Given that distutils has been part of the standard library for a couple of decades, I do think we need a straightforward way for folks to say "Make this Python installation provide the distutils API", and it's highly desirable for that mechanism to work without internet access at install time.

The bundling could potentially be phased out (and I agree that "no pre-installed build tool" is a desirable longer term goal), but I'd want there to be at least one transitional release that omitted the distutils source tree, but included a setuptools wheel file for easy installation. (I also genuinely doubt we could get a PEP accepted if we omitted such a transitional release)

It wouldn't necessarily need to be its own bootstrapping module though - that's just the first approach that occurred to me. Another possibility might be to add a --with-distutils option to ensurepip that emits a DeprecationWarning saying to instead rely on pyproject.toml and/or explicit preinstallation of setuptools.

[pfmoore]

I was thinking in terms of a transitional release that issued a warning if users imported distutils. That warning could point users to the docs on setting up a pyproject.toml based project, explaining that users then don't need distutils installed to build the project.

I'm not actively against a means of making the distutils API be "on request", I'm just not sure it's needed. Normal deprecation cycles, combined with the fact that pip 10+ (which will be bundled in Python 3.8, even if it's too late for 3.7) will work without distutils being present, seem sufficient to me.

By the way, with pip 10 being released soon (approx 4 weeks from now, if things go to plan), what's the process for discussing what ends up in Python 3.7? Are we too late for 3.7? Should I raise an issue on bpo? Is the process for updating the bundled pip documented anywherem (I couldn't see anything in the devguide)? So many questions...

[ncoghlan]

Raise an issue for upgrading 3.7 to pip 10 before 3.7.0b4 (due April 20th) on bpo - while we're willing to upgrade pip in maintenance releases, switching from 9.x to 10.x as the default post-release would be overly disruptive compared to launching with it for beta 4. The lack of docs for how to do this would make a good devguide issue :)

[ncoghlan]

Back on the topic of a migration strategy, I agree a plain DeprecationWarning in distutils.core.setup and potentially some of the other public APIs should work - that would even trigger when using a version of setuptools that still monkeypatched distutils instead of injecting its own version of the modules.

[pganssle]

Possibly too detailed a question at this point, but would the actual implementation of this be to just import all the code from the existing distutils, or write a fresh implementation of the API? The problem with pulling in all the code directly from distutils is that the Python 2.7.x and master branches of distutils have already diverged somewhat (though I haven't checked to see if there are any obvious differences in the public API).

One possibility might be to cut a bugfix-only LTS version of setuptools supporting 2.7 and move the primary setuptools over to being 3-only before dropping the distutils dependency, but that does involve maintaining two branches.

[merwok]

I don’t know if we can honestly talk about such a thing as a «distutils API» that would exist separate from the distutils implementation. It seems to me that having distutils very stable and setuptools patching and extending it has worked acceptably in recent years, and I have an uneasy feeling about engineering a system to remove and install it.

(Not commenting on setuptools deciding to copy distutils instead of importing it, which sounds like a good idea, only on the removal + ensuredistutils proposal.)

ensurepip is a technical necessity that enables the success story of «Python comes with pip», and it's not so hard to explain it to people. I see less value in removing distutils (and breaking existing code and docs) now that it’s becoming less and less prominent thanks to newer packaging tutorials, PEP 518, flit, cookiecutter, etc.

I would like to see in what ways PEP 518 moves the ecosystem and best practices in the upcoming years, and then I think it would make sense to make plans for the evolution of setuptools and eventual removal of distutils.

[njsmith]

@merwok Right now, there are a lot of setup.py's out there that still do import distutils. When run under pip this is fine, because they transparently get switched to use setuptools instead. But a lot of people still run setup.py directly, and so they're using actual distutils rather than setuptools, and distutils is increasingly broken in the context of the modern packaging ecosystem, so this is bad. In the long run the solution is for everyone to switch to using pip instead of calling setup.py directly, but in the mean time it would be nice if import distutils could consistently give you setuptools, instead of only doing it most of the time. But to accomplish that, you need some sort of bundling.

By the way, has anyone done an audit of pip 10 to check that it's ready to replace direct setup.py calls in most cases? Does it actually have replacements for setup.py bdist_wheel and setup.py sdist? Should it? Is this the release where we start evangelizing "don't use setup.py directly"? (Probably this should get split off into its own issue, but it's closely linked to the overall transition plan here, and, I'm on my phone keyboard :-).)

[pfmoore]

@njsmith You're right that's probably a separate issue, but I don't think it's ever going to be a goal for pip to replace every possible invocation of setup.py. PEP 517 encapsulates the build system interface that pip needs, but build systems are entirely at liberty to have additional commands that only they support. Actually, there's no pip equivalent for setup.py sdist, and there's been debate in the past over whether that's outside pip's remit (I don't know what the current consensus on that is).

So I don't think we're looking to tell people to stop calling setup.py directly (even if we do want to stop them doing so for commands like install and bdist_wheel)....

[ncoghlan]

Regarding the question of "Why not leave the distutils code around indefinitely?", my main concern is with cases like https://bugs.python.org/issue33069, where we have to choose between leaving a subtle misbehaviour in distutils, or else upgrading it to newer metadata versions (duplicating work that has already been done elsewhere, either in distribute and then merged back in to setuptools, or directly in setuptools post-merger).

As things stand, building a package with plain distutils can appear to work in simple cases, but you'll be missing quite a few capabilities, and have a debugging session ahead of you to figure out you need the setuptools enhancements.

That all gets a lot simpler to understand if "import distutils" starts reporting a DeprecationWarning (and perhaps eventually ImportError) in the absence of the setuptools enhancements.

[dstufft]

Distutils could just stop overwriting author metadata with maintainer metadata. You’d then get the warning it doesn’t understand that keyword when you tried to use it. Folks are more likely to reach for setuptools for things distutils doesn’t support rather than for things where distutils has weird behavior.

…

Sent from my iPhone

On Mar 19, 2018, at 6:07 PM, Nick Coghlan ***@***.***> wrote: Regarding the question of "Why not leave the distutils code around indefinitely?", my main concern is with cases like https://bugs.python.org/issue33069, where we have to choose between leaving a subtle misbehaviour in distutils, or else upgrading it to newer metadata versions (duplicating work that has already been done elsewhere, either in distribute and then merged back in to setuptools, or directly in setuptools post-merger). As things stand, building a package with plain distutils can appear to work in simple cases, but you'll be missing quite a few capabilities, and have a debugging session ahead of you to figure out you need the setuptools enhancements. That all gets a lot simpler to understand if "import distutils" starts reporting a DeprecationWarning (and perhaps eventually ImportError) in the absence of the setuptools enhancements. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[njsmith]

Regarding the question of migrating maintainers away from invoking setup.py directly, see #129

[jaraco]

@doko42 - Can you share the list of Debian packages that currently depend on python-distutils?

[doko42]

I would not consider usage of distutils.version, .utils (stringtobool), .spawn (find_executable) and distutils.sysconfig really using distutils. These all look like references to old distutils code which should be replaced.

Currently remaining dependencies on distutils in Debian (after splitting out distutils from Python3):
$ apt-cache rdepends python3-distutils

python3-testtools
snapcraft
python3-virtualenv
python3-versiontools
python3-venv
python3-setuptools-scm
python3-pyroma
python3-pyassimp
python3-pip
python3-path
python3-mypy
python3-iptables
python3-fswrap
python3-editor
python3-distutils-extra
python3-capstone
python-bootstrapform-doc
onioncircuits
dirtbike
diffoscope
dell-recovery
python3-testtools
python3-setuptools
libglib2.0-dev-bin
gobject-introspection

[brainwane]

Since this is a backlogged task that we want to finish and that lots of other things depend on, I've added an item for it in the Fundable Packaging Improvements wiki page.

[jaraco]

FYI, pypa/setuptools#2143 has an implementation of step 2.