Update package metadata

[ofek]

• I have added an entry to docs/changelog.md

Summary of changes

This implements PEP 621, obviating the need for setup.py, setup.cfg, and MANIFEST.in.

Support has not landed in setuptools, so builds will now use hatchling.

Backend

Hatchling is the minimal build backend of Hatch v1.0.0 (v1 rewrite of CLI will come out of beta around PyCon). You can think of it as the same distinction between flit-core & flit or poetry-core & poetry, with hatchling being more similar to flit-core in that only accepted standards are used, like for dependency specification (PEP 631). One of the central differentiators is the extensibility offered by its plugin system (see the showcase).

Note that the only reason the version is 0.x is because 1.0.0 will drop support for Python 2. It is quite stable, and I'm in the process of switching all of our Python packages here at Datadog. Here's one of the repos that we just completely switched (~150 packages): https://github.com/DataDog/integrations-core

Differences

In comparison to https://pypi.org/project/pipx/1.0.0/#files

Wheel

Hatchling already supports the final draft of PEP 639 so license files are stored in .dist-info/license_files/ rather than dumped directly in .dist-info/.

Source distribution

Even though the old MANIFEST.in says otherwise, *.md files weren't actually included. Here's a before & after:

[ofek]

There is a CI check that verifies source distributions contain everything in the repo, which is odd because the PyPI release definitely does not.

I can either remove that check (especially since the tool mentions a MANIFEST.in but that file no longer exists) or have it ship everything in the repo. Just let me know!

[cs01]

The check can be removed now since we're no longer using MANIFEST

[ofek]

Done!

[gaborbernat]

One feature gap compared to setuptools and setup.cfg I see here is that this file is now manually maintained and formatted while using setuptools one can use the excellent setup-cfg-fmt project for it. Does hatch have plans to add formatting support? Without it I'm tempted to keep using setuptools 🤔 Naturally if other maintainers even with that feature gap like hatch I'm not going to oppose 👍

[ofek]

I suppose I could add a command for that but:

1. Something like that is indeed better suited as a small, separate project that can be used by all.
2. I would personally never use it because I would like to retain comments, and tomlkit is extremely buggy.

Are you saying that until there is a pyproject.toml linter you're not keen on using PEP 621?

[gaborbernat]

Indeed 😊 I am loosing some functionality otherwise that I do have with the current framework, and the benefits of switching to PEP-621 in the short term are not that much compared to setup.cfg.

[henryiii]

Setup-cfg-fmt does a lot more than just formatting, it also keeps up the classifiers for new Python versions and cleans them up, sorts dependencies, etc. It also runs in pre-commit, can taplo?

It also strips comments, valid empty (sub) sections, and can’t be disabled for part of a file AFAIK…

[henryiii]

tomlkit is extremely buggy.

👍

An eventual tool should not be tied to a build backend, but be a generic tool; probably a pair of tools like taplo & (imaginary atm) pep621-fmt or something. Fmt is a little off since it would be more of a validator. Happy to have it live alongside hatch though. ;)

[henryiii]

There are some very good reasons to put comments in sometimes, like https://github.com/scikit-hep/awkward-1.0/blob/6d3c1d23d7ab492d0178b1798e179088a066e00a/pyproject.toml#L90-L157 - probably rare for the project metadata itself, but the other tool fields might need comments (flake8/pylint config being big ones for needed comments). Comments can be overdone, sure, but sometimes they are needed. setup-cfg-fmt doesn't support them for technical reasons, not primarily because they are a bad idea. I usually have managed without comments in setup.cfg, though, sometimes painfully with flake8, but really hope not to leak that issue over into pyproject.toml. :)

[abravalheri]

I would personally never use it because I would like to retain comments, and tomlkit is extremely buggy.

tomlkit has received a lot of improvements over the last 6 months. I personally have contributed several patches and @frostming has become a maintainer and has been doing an amazing job!

As an example of its potential, I can safely use it in ini2toml for translating the "AST" of .ini files to .toml, which includes preserving comments.

So if its being a while that you last checked tomlkit, maybe it is a good time to have a look if it improved for your use cases?

[gaborbernat]

For what's worth I've been using tomlkit for https://github.com/gaborbernat/pyproject-fmt and works well for me 👍

[frostming]

Run into this due to the ping, and I found @gaborbernat already made a pyproject.toml formatter, powered by tomlkit

[henryiii]

I tried to use toolkit in cibuildwheel (at least 6 months ago, probably more), and found it too buggy to use; it was a rather weird format, but it was important to keep it, which it failed to do. I ended up just writing a manual writer, since it was a very well structured limited file. Great to hear it is seeing improvements! I know it was not 1.0.0 compliant either last I checked, but is now. :)

And @gaborbernat, @abravalheri, and @ofek have all made TOML formatter/validators. :) pypyroject-fmt, pyproject-validate, and validate-pyproject.

[henryiii]

I think hatchling should be added to conda-forge before this goes in, or installer. This will break the current conda-forge package. Flit(+core) are on conda forge, but not hatchling.

Getting installer in will likely be easier after it joins the PyPA, probably in a week if all goes well.

[ofek]

Done!

• https://github.com/conda-forge/hatchling-feedstock
• https://anaconda.org/conda-forge/hatchling

[henryiii]

I no longer have any objections; I don't see a strong reason to use hatch over flit for something this simple, but now I don't see a strong reason to use flit over hatch either. The addition of proper PyPy testing on Windows in Hatch is also really nice to see. Edit: and good error messages.

[q0w]

I no longer have any objections; I don't see a strong reason to use hatch over flit for something this simple, but now I don't see a strong reason to use flit over hatch either. The addition of proper PyPy testing on Windows in Hatch is also really nice to see.

Choosing flit you can stick with one and only one tool and spend your time to improve one and only one tool.

[ofek]

Choosing flit you can stick with one and only one tool and spend your time to improve one and only one tool.

By that logic we all should just contribute to setuptools 😉

[henryiii]

we all should just contribute to setuptools 😉

Good luck, setuptools has 40K+ lines of Python code, flit-core has 2K. Most of the code is deeply entrenched in legacy usage and is a pain to touch, and it has a distutils / setuptools structure that is really irritating. In the future, other build systems might get there too, but we've learned a lot about limiting API in the years since (partially because we now have choice, and we don't need a "must be usable by everyone" build system).

I am a big fan of contributing to a project instead of starting a new one (which is why I'm part of many of the projects I'm part of!), but flit's goal is to stay simple, so that's why other "simple but not so simple" build systems can exist. That's why Flit can't support VCS versioning; there is no "build" step where you can add that.

I do think we are getting a few too many "simple" build systems, though - pdm's, hatchling, flit-core, trampolim, whey... I really like what PDM (and Hatch?) do by supporting any PEP 621 build system inside the environment manager, rather than requiring their own (though PDM does have one).

[ofek]

I really like what PDM (and Hatch?) do by supporting any PEP 621 build system inside the environment manager

Correct, Hatch does that too. Interoperability is very important to me https://ofek.dev/hatch/latest/meta/faq/#interoperability

[q0w]

So maybe PDM instead of hatchling?😉

[gaborbernat]

I'm definitely in the hatchling over pdm group as maintainer. 👌

[abravalheri]

Choosing flit you can stick with one and only one tool and spend your time to improve one and only one tool.

The whole point of PEP 517 was to make possible to have diversity in the ecosystem... This way of thinking seems to go in the opposite direction of the standardisation efforts.

[layday]

I don't know if it's appropriate for me to review a PR here but I don't see anything wrong implementation-wise.

[henryiii]

I don't think I need to say more than I've already said, no need for an explicit review. If this was my repo, I'd still be lightly biased to keep this "in house", and use flit-core - but it's practically no change, due to PEP 621, so it could even be done later if hatchling maintenance suddenly ends or something. Most users will use wheels, so various backends are pretty safe for pure Python projects. I don't have any reservations over using hatching. And the implementation looks good.

[henryiii]

PS: just for completeness, since I mentioned it before, I did check GHA/Azure's wheel setup, and they use pip to install pipx, so they should not be affected.

[cs01]

Thanks all for your participation. If any one is interested in becoming a maintainer, reviewing other pull requests, or triaging issues, please let me know! I can be reached at chadsmith.software@gmail.com.

[cs01]

Somehow this broke CI https://github.com/pypa/pipx/runs/5339892213?check_suite_focus=true. Is anyone able to investigate/fix?

[ofek]

Seems like a flaky test; I'd highly recommend https://github.com/pytest-dev/pytest-rerunfailures

[cs01]

Seems like a flaky test; I'd highly recommend https://github.com/pytest-dev/pytest-rerunfailures

Sounds like a good idea. If anyone has the bandwidth to try and fix this I would really appreciate it. I'm not sure when I can get to it unfortunately.

[dukecat0]

Somehow this broke CI https://github.com/pypa/pipx/runs/5339892213?check_suite_focus=true. Is anyone able to investigate/fix?

Seems related to ipython/ipython#13554

[ofek]

@cs01 I opened #822