add PEP 517/8 build backend and requirements specification

[gaborbernat]

👍

[giampaolo]

Both setuptools and wheels are not mandatory so I don't think this is worth it.

[gaborbernat]

@giampaolo I think you're mistaken here, please reconsider. There's two parallel things going on here so I'll detail:

• Per PEP-517/8 you should define both your build dependencies and build endpoint into pyproject.toml.
• If you intend the package to be used with pip wheel must be mandatory, as pip always builds a wheel for all PEP-517/8 packages. Sure you can fallback distutils to build packages, but distutils is considered not maintained so you probably want setuptools. Furthermore distutils does not provide PEP-517 interface...

You can find more details in https://www.bernat.tech/pep-517-and-python-packaging/

[giampaolo]

I'm gonna re-open and take a look at this later. Sorry I didn't know about this but want to understand it first before merging.

[gaborbernat]

Ping 👍

[gaborbernat]

Any updates on this?

[giampaolo]

@gaborbernat would there be an advantage to move classifiers, description, author, etc. from setup.py to pyproject.toml? Note: there's some info such as version which are calculated at runtime so they should stay in setup.py.

I merged this PR because to my understanding with this in place pip can automatically download setuptools before installing psutil, but pyproject.toml seems to offer more than that. I am not sure what of that could be useful for psutil though.

There was a proposal some time ago: #1207 for adding a license to the wheel files. I wonder if adding license = 'BSD' to pyproject.toml would achieve the same effect.

[gaborbernat]

Depends on the build backend. setuptools does not allow specifying options in pyproject.toml, only flit and poetry does. You could put stuff into setup.cfg though, in the spirit of declarative configuration over imperative.

[giampaolo]

I would like to avoid adding yet another file to the root dir. I was already skeptical about adding pyproject.toml for this reason.

[gaborbernat]

If number of files is what concerns you, then you can modify contents of .coveragerc into setup.cfg.

[giampaolo]

This basically broke 5.6.4 release, see #1613, #1614 and #1615. I am going to revert this change and make a new release today.

[gaborbernat]

I've made comments on both PRs. I think one is a python interpreter bug (maybe in pip?) and the other is an inappropriate usage, more details on the tickets themselves. 🤷‍♂

[giampaolo]

This seems relevant:
pypa/pip#6163 (comment)
psutil does the same in setup.py (updates sys.path in order to import a psutil module before install) so that appears to be the problem.

Later on, sys.path is already mentioned:
pypa/pip#6163 (comment)

Later on it seems there is a workaround:
pypa/pip#6163 (comment)

But I'm getting lost here. It seems some older version of pip (or setuptools? or both?) are affected, meaning with pyproject.toml in place some users are gonna be bitten eventually.

[giampaolo]

Also CC-ing @pfmoore here =)

[giampaolo]

#1613 also seems related.

[pfmoore]

Not sure what input is needed from me here. There's not enough information for me to diagnose what the problem is with this PR just from reading the comments, and I don't have a lot of spare time at the moment, so I can't try to reproduce the issue.

[giampaolo]

Nevermind Paul, I just CC-ed you because I saw you were involved in the original pip issue in Jan. Sorry for the noise.

I just released psutil 5.6.5 which removed pyproject.toml and hopefully fixed #1613, #1614 and #1615.

[gaborbernat]

I don't think removing pyproject.toml and effectively opting out from PEP-517/518 is a fix, but 🤷‍♂ It's more like switching to the recommended and going ahead supported path of python packaging tools exposed a few bad workflows for your users; so you decided to revert. The fixes should happen within the broken workflows/tools; not by not using the pyproject.toml IMHO.

[pfmoore]

As far as I can see, the problem here is that you're still trying to support building with just pure distutils. That's not a supported approach any more (where "supported" is obviously a slippery term in open source, but here means "nobody in the packaging tool community really worries about making that scenario work") so you're going to increasingly find it hard to continue supporting that scenario.

At some point, changes to the packaging ecosystem (whether in pip, or in some other tool) will break distutils-only builds for you (if nothing else, at some point although not soon, pip will stop supporting the "legacy" direct setup.py method of building), so you should consider how you'll handle that. It looks like you already support building with setuptools, so simply adding a deprecation warning for the distutils-only case and deciding when you'll desupport it should be enough, if that's the route you want to take.

I've opened pypa/packaging.python.org#667 to get the position on pure distutils projects clarified.

[giampaolo]

I didn't know distutils was deprecated (or on the verge of being declared as such - FWIW Python 3.8 docs does not mention it). I'm not a package expert, but as far as psutil is concerned I always considered setuptools a "plus", mainly because psutil has no third-party deps. Also I'm not sure I understand what PEP-517/518 support will bring in practical terms for psutil right now. As far as psutil is concerned it seems that using pyproject.toml will result in making pip install setuptools before psutil (if not already installed) and install psutil as a wheel (still not sure I understand why that's supposed to be better - I'll have to read the PEPs more carefully). Are there other advantages I am missing here?
My point is that if that's the only benefit, considering there were 3 different reports for which the solution is not clear reported in less than 24 hours, then I prefer to stick with the status quo, wait for people to upgrade their old pip/setuptools versions and possibly reintroduce pyproject.toml later. AFAIC right now pyproject.toml only produced a broken psutil release for some users for no real benefit. Whether it was up to bad users workflow or old pip/setuptools version seem irrelevant to me, to be honest. Note that psutil still supports Python 2.6, CentOS-5 and Windows XP, meaning it is expected to be installed on systems that may not be up to date in terms of pip/setuptools.

[pfmoore]

OK, if you're supporting systems that old, you may well have to handle things differently (pip itself hasn't supported Python 2.6 for quite some time now, so presumably your users on that environment are either using very old versions of pip, or running setup.py manually).

Under PEP 517, pip installs setuptools (or rather, the appropriate build tools whatever they are), yes, but only temporarily so that should not affect the final environment.

I agree 100% that reverting to address user difficulties, and then looking at pyproject.toml without the pressure of an ongoing emergency, is the right solution. But of the issues I looked at, none of them seem directly related to pip, sothe fixes will need to come in the form of advising users what to change in their systems/workflows, not via code changes in your build system (see #1613 and #1614, and quite possibly #1615 as well).

The problem with distutils, that setuptools solves (at least, the one you will care about) is that distutils won't be updated to support newer systems and standards. I don't honestly know how often something like a new compiler or OS quirk comes along that's only handled in setuptools, but as you're using setuptools if it's available and only using pure distutils as a fallback, this may not cause you problems.

Ultimately, the issue you're facing here is that you support a wider range of systems and workflows than your build tools do. How you manage that is something you're going to have to figure out - and it's not urgent at this point but at some stage, you'll have to deal with it (tell your users the can't use newer versions of pip? drop support for Python 2.6?)

I guess that's as much as I can say right now, but if you have any questions, ping me and I'll be happy to advise.

[giampaolo]

Thanks a lot Paul. I have a better understanding now. I'll see you on python-dev. =)

[ncoghlan]

Direct usage of distutils is definitely deprecated, hence https://docs.python.org/3/library/distutils.html telling people to use setuptools instead, and https://docs.python.org/3/distutils/ saying outright that those docs are only sticking around until the setuptools documentation is complete unto itself, rather than assuming readers are also looking at the distutils documentation.

Eventually pypa/packaging-problems#127 will be implemented, and setuptools will be providing the only implementation of the distutils API (with no legacy partial implementation in the standard library).