Add discussion page that links to pypackaging-native.github.io

[viblo]

Adds a discussion page that briefly list some issues around binary wheels and links to pypackaging-native.github.io

Closes #1546

---

📚 Documentation preview 📚: https://python-packaging-user-guide--1555.org.readthedocs.build/en/1555/

[chrysle]

There is also https://packaging.python.org/en/latest/guides/packaging-binary-extensions/ which I believe should be linked to somewhere.

[viblo]

There is also https://packaging.python.org/en/latest/guides/packaging-binary-extensions/ which I believe should be linked to somewhere.

Oh, after reading there I wonder if it would be better to put this under https://packaging.python.org/en/latest/guides/packaging-binary-extensions/#additional-resources instead?

[webknjaz]

I know that @lwasser and @henryiii have more packaging documentation websites to link. Perhaps, we could aggregate them together somehow?

[henryiii]

I think this makes more sense under the existing page - a new page means more to maintain. Also, this page is really more of a list of "what's wrong with the PyPI model" written to try to convince PyPI to adopt more conda-forge style model than for people getting started with packaging their own extensions trying to follow existing best practices. I think the lead-in description here is pretty good, but I don't think it deserves its own page (and I'd love for the existing binary page to get a bit more love! Last updated 2013 is printed across the top!)

My page on packaging compiled extensions is at https://learn.scientific-python.org/development/guides/packaging-compiled (& https://learn.scientific-python.org/development/guides/gha-wheels/). It is focused on someone wanting to learn how to package an extension, but it's only focused on the basics of packaging and distribution, there's very little about how to write the code that is going into the binary.

[rgommers]

+1 for linking to @henryiii scientific-python guide and @lwasser's PyOpenSci guide (https://www.pyopensci.org/python-package-guide/index.html or a page under that).

this page is really more of a list of "what's wrong with the PyPI model" written to try to convince PyPI to adopt more conda-forge style model

This really isn't correct. https://pypackaging-native.github.io/ on purpose stays away from suggesting any kind of solution. The front page states the key goals:

"This site aims to provide an overview of the most important Python packaging issues for such projects, with in-depth explanations and references.

The content on this site is meant to provide insights and good reference material. This will hopefully provide common ground when discussing potential solutions for those problems or design changes in Python packaging as a whole or in individual packaging tools."

Any kind of improvement on any of the topics covered by pypackaging-native would be great to see.

[henryiii]

I think (could be wrong) the only page on compiled extensions is https://www.pyopensci.org/python-package-guide/package-structure-code/complex-python-package-builds.html.

This really isn't correct. https://pypackaging-native.github.io/ on purpose stays away from suggesting any kind of solution.

Apologies, I stated that too strongly. My impression comes from the fact that the topics are titles like "PyPI's author-led social model and its limitations" and "Lack of a build farm for PyPI", which seem to be negatively comparing to Conda - and almost every page mentions Conda. The build farm issue is largely mitigated by cibuildwheel and GitHub Actions, while this is just mentioned in a note on tooling. I think just slightly more neutral language would go a long way in making it seem more unbiased. For example, removing "and it's limitations" (this is a site about issues, after all), and "lack of a build farm" could be reframed into a more balanced discussion of pros and cons of cibuildwheel and build vs. a hypothetical build farm. "lack of a build farm" clearly implies that the solution is a build farm. It's not a pros-and-cons discussion.

All of this is "problems with PyPI wheels", which is fine and useful somewhere, I just think it needs to be clearly indicated - which it is in the description in this PR - and it doesn't deserve a dedicated page for all eternity (at least the URL will be eternal), it is better as a section at the bottom of an existing page (I think the one linked above is good).

I think the last thing we need is more pages, especially ones that lead off into discussions about problems.

[rgommers]

it doesn't deserve a dedicated page for all eternity (at least the URL will be eternal), it is better as a section at the bottom of an existing page (I think the one linked above is good).

Sure - I didn't review this PR in detail and have no opinion on this. Whatever seems best to the reviewers/maintainers here.

My impression comes from the fact that the topics are titles like "PyPI's author-led social model and its limitations" and "Lack of a build farm for PyPI", which seem to be negatively comparing to Conda - and almost every page mentions Conda.

A quick search shows that Conda has 15 pages with mentions, Spack has 11, Debian has 6, Nix has 6, Homebrew has 5. It just draws on examples from other distros and tools where the authors (mainly me, but not only me) deemed that useful.

I could equally write a set of pages on topics with the Conda ecosystem where PyPI, Pip & co compare favorably. E.g., the repodata fetching issue makes conda envs super slow to install anything from conda-forge on lower-bandwidth connections; Pip is significantly better in this respect. That is just not the topic of pypackaging-native, it's focused on PyPI and Python packaging standards & tooling.

The content is honestly meant constructively, because I have seen 15+ years of discussion where many Python packaging folks are simply repeating partial problem statements related to needs of scientific and data science needs over and over, and discussions on distutils-sig and now Discourse simply do not converge because most participants lack context.

The build farm issue is largely mitigated by cibuildwheel and GitHub Actions

This also really is not the case. We're just in the middle of a many months long NumPy 2.0 release process for example, which is extremely painful because for PyPI we have no ability whatsoever to deal with ABI changes or enforce consistent use of compiler toolchains. This just isn't nearly as much of a problem in any other distro with centralized infrastructure. cibuildwheel is awesome and it helps with a different part of the build farm issue, which is capacity and easy workflows to build binaries - but it does nothing for topics like dealing with ABI changes.

You seem to be ascribing motivations that I promise I don't have. Also, the topics weren't chosen by me, but by gathering and ranking input from ~25 people with varying backgrounds (as explained on the front page). The build farm topic was high on the list for quite a few people.

[lwasser]

hi everyone 👋🏻 !! it would be wonderful if you link to both our pyOpenSci guide and the scientific packaging guide as it makes sense!!

Also please note - we are working on a handful of actual examples of bare minimum packaging scenarios including one that has C + hatch -- many thanks to @ucodery and @kenseehart as they both spent time working on this at our sprint last week! The content will take some time to develop however.

Please let me know if I can be helpful in any way here!

[viblo]

Alright, I have pushed a commit that moves the link to pypackaging native to the additional resource section on https://packaging.python.org/en/latest/guides/packaging-binary-extensions/

As for the other sites brought up here, I feel that they are broader in scope than just native packaging, and doesnt really fit together on the binary extensions page. Therefor I suggest those can be handled in a separate PR.

In my mind I could either see the big key projects page https://packaging.python.org/en/latest/key_projects/ expanded to also include guides, or maybe better a new page with links to external helpful pages, for example under Guides could be a "Third Party Guides" page with outgoing links.