♻️ REFACTOR: Source package metadata from PyPI

[chrisjsewell]

This refactor makes it so that package data is primarily sourced from PyPI,
if a pypi_name key is present if the pip_url points to a PyPI package,
with a fallback to the repository build file (setup.json, setup.cfg, pyproject.toml).

(a) this ensures the released metadata is shown (not unreleased, development metadata),
and (b) this handles dynamic metadata, such as the version in flit and setup.cfg.

Currently, the entry points are always read from the build file,
since PyPI JSON does not include entry points,
and so we would need to download and read the wheel file to obtain them.
when the bdist_wheel is available on PyPI, it is downloaded and the entry-points extracted from it

I diffed the original plugin_metadata.json to the new, and everything looked correct,
except for one thing:
The aiida_version is null for some more packages.
This is because PyPI only extracts/stores requires_dist data, if abdist_wheel distribution is available (see https://www.python.org/dev/peps/pep-0491/).
This is essentially a bug with those packages (and we should warn them),
since all best practice build frontends (https://github.com/pypa/build, flit publish, poetry publish)
build both the sdist and the wheel.
Therefore, there is now a new report warning for this: WARNING: No bdist_wheel available for PyPI release

[chrisjsewell]

Oh and it also adds reading of PEP621 compliant pyproject.toml,
which is basically the goal here: allow us to recommend https://github.com/pypa/flit, as the best practice for pure-python packages 😄

FYI, the latest flit + pip versions now support editable builds with no setup.py 🎉 (i.e. pip install -e .), which is something that had stopped me from using it previously

(I've now implemented it in https://github.com/aiidateam/aiida-firecrest, and it's working great)

[ltalirz]

Thanks @chrisjsewell !

If I were to play devil's advocate, instead of needing to fetch just the build file from the repo, we now need to fetch both from PyPI and from the repo...

I agree, however, that it is a step in the right direction - we want to show the properties of the published packages (if available); showing updated data from the repository that conflicts with the info on PyPI is undesirable.

I just have one main change request, after this happy to merge

[ltalirz]

should we open an issue for this then?

[chrisjsewell]

Yep 👍 (see the comment I just made)

[chrisjsewell]

actually now fixed 😄

[ltalirz]

great! Would you mind updating the comment as well?

[ltalirz]

Correct me if I'm wrong but I don't think this additional field is necessary.

As described in the README, the pip_url will contain either the package name or a URL to the repo.

One can argue about whether that is the best solution, but it has been in place since a while, so I suggest you simply get the package name from there.

[chrisjsewell]

Yep sure, will just have to add a bit of logic that decides if it is a pypi name (and so we can source metadata from there) or an external url (so we can only use the plugin_info url for metadata)

[chrisjsewell]

Ok fixed in 03f09bd (#201)

[chrisjsewell]

If I were to play devil's advocate, instead of needing to fetch just the build file from the repo, we now need to fetch both from PyPI and from the repo...

But yeh, as you said, the repo data is not what we should be referencing; it should be the latest release.
It's a shame that the pypi json does not directly include the entry points, then we absolutely would not need to read from the repo.
You can, download/parse the released wheel file (the url is supplied in the pypi json), then you wouldn't have to touch the source. The only thing is, obviously this would incur a cost in the amount of data you have to download.
Probably leave that for a separate PR though.

[chrisjsewell]

If I were to play devil's advocate, instead of needing to fetch just the build file from the repo, we now need to fetch both from PyPI and from the repo...

Ok now you don't 😝

The entry points are fetched from the PyPI bdist_wheel (if available) and, in this case, downloading/reading from the repo is entirely skipped.

tested the speed, and there is really not that much difference:

$ time aiida-registry fetch --no-fetch-pypi
...
aiida-registry fetch --no-fetch-pypi  23.48s user 0.57s system 27% cpu 1:26.61 total
$ time aiida-registry fetch --fetch-pypi
...
aiida-registry fetch --fetch-pypi  26.97s user 0.92s system 25% cpu 1:47.67 total

(and obviously this does not include request caching)

[ltalirz]

thanks Chris!

can you still update the comment; then it should be good to go.

I trust that you checked that the fetched metadata is at least as complete as before.

[ltalirz]

great! Would you mind updating the comment as well?

[chrisjsewell]

I trust that you checked that the fetched metadata is at least as complete as before.

yep it is indeed 👍

[chrisjsewell]

can you still update the comment; then it should be good to go.

done