Better wheel hosting solution

[rth]

Now that we have switched to wheels for packages, our current distribution model reaches its limits as it doesn't allow,

• having multiple available versions of a package for a given Python/Emscripten version
• have the package accessible as soon as it's built
• allow users to upload wheels from their CI (e.g. using cibuildwheel once it works).

This issue aims to discuss what an improved hosting solution could look like and evaluate several possible directions. Of course, the question remains whether the added benefits would be worth the extra maintenance effort.

From a usage perspective, I think what we want is a PyPI like mirror, where users (us included) can upload packages with twine using an API key. Internally packages would still be uploaded to the same S3 bucket and distributed via JsDelivr. So the system distributing packages (i.e. S3 + JsDelivr) is separate from the website of the package index. This way we can guarantee very good package availability, even if there is some maintenance downtime time on the package index website.

Aside from the above-listed functionality, I think we would need,

• the ability to upload files that are not wheels (e.g. .zip files) -> though this needs to be discussed, maybe wrapping those files as wheels would be better
• users can create accounts to upload their packages -> to be discussed, this would add maintenance cost
• ability to upload repodata.json ideally without using S3 credentials directly
• need to figure out what we do with unvendored test packages, so they are still part of the dependency tree but maybe they shouldn't have a separate page

Bonus features,

• have some way to inject extra information on the package page (e.g. extra comments about what works and what doesn't from the meta.yaml, or automatic package analysis about potentially unsupported functionality)

If we want to use an existing OSS solution, there are the following possibilities,

• PyPI warehouse: it's fairly specific to PyPI.org, very complex, and is not intended to be used for anything else than PyPI.org
• devpi
• pypi-server
• code behind piwheels
• custom solution with only the subset of the functionality we need (still a lot of work)

I'll later add below my evaluation of these solutions.

At the same time, we should keep in mind that in the long term wasm/emscripten wheels might be supported by PyPI, so maybe it's not worth spending too much effort on this. However, even if PyPI does support them, it might not include web-specific optimization that we are able to do.

[bollwyvl]

Yeah, multiple versions are pretty sticky: in jupyterlite, we're currently working through the growing pains of a major version of a "special" upstream (ipywidgets) which has a javascript-level dependency (widgetsnbextension), and will need to do some weird shims.

It's not that bad, though, as we've been maintaining a "collapsed" warehouse-like file structure which doesn't have some of the features of the conda-like repodata.json, but does allow for multiple versions of multiple packages.

To that end: the eventual solution should:

• be something that speaks an existing, well-known, efficient (as possible) package management format and syntax
• allows for multiple sources of (multiple versions) of packages
• tracks ABI-level data
• can be automated to a great extent with existing tooling

The closest I've come to this, is, no surprise here, the conda, conda-build, and ultimately, conda-forge ecosystem.

This might work, today on anaconda.org: it will actually host wheels, and provide a pip-compatible (but probably not a warehouse-compatible) API:

https://docs.anaconda.com/anacondaorg/user-guide/tasks/work-with-packages/#using-package-managers

It offers free (as in beer), (basically) unlimited package hosting for open source, as well as private packages (though it gets a bit hairy). There is an on-going effort to provide an open source implementation of that site's capabilities at https://github.com/mamba-org/quetz, but to my knowledge, it doesn't offer wheels.

Of course... the real step there would then be to eschew the wheel ecosystem entirely, and use the conda(-forge) ecosystem more directly. The double-edged sword of the "real" repodata.json is that it works pretty well for channel a few thousand package * arch * python * os * version, but starts to fall apart at the hundreds of thousands (conda-forge/noarch/repodata.json is 200mb, uncompressed). Even making that 10x better would still be unreasonable for fully client-side building.

[hoodmane]

@bollwyvl I think we want to stick to wheels. Most Python packages can make wheels automatically with python -m build in the package directory. With pyodide build it's possible to build emscripten wheels for a lot of packages without any modification or with minor fixes. Our hope is to get package maintainers to build and test emscripten wheels in their CI. Would it be possible to get package maintainers to build and test .conda archives? As far as I can tell, none of the major packages do this.

[bollwyvl]

hope is to get package maintainers to

I salute your optimism!

possible to get package maintainers to

No, they won't do that, either, but a few thousand packages in, conda-forge has demonstrated that a community-based, distributed model with heavy automation on donated CI, and donated cold storage can achieve substantial, reproducible things at scale, and keep them up-to-date with best practices.

[rth]

Thanks for your summary about packaging on the jupyter side, it's helpful!

Emscripten-forge is going in the conda direction, with a planned contribution to conda-forge and I think it's great that people are exploring this idea.

For our side, we decided to go in the direction of wheels. Things have improved a lot there as well with cibuildwheels and related projects. A lot of packages are on PyPi only, and people still need the ability to install packages from there (or some private repositories that never going to be on conda-forge, particularly pure Python ones). About hosting I think we would rather rely on some community solution, we have a good agreement with JsDelivr CDN with no limits on the bandwidth so far.

[freakboy3742]

As a data point - BeeWare uses an anaconda.org repository to host iOS binary wheels, and it's been working really well. Having a similar collection available for Pyodide/emscripten wheels would be exceedingly helpful for some BeeWare's packaging workflows.

[rth]

Yes, I'm aware of that possibility, and I hear good things about it. For Pyodide we currently need more than wheels (there are also various .js files) for which we probably need JsDelivr in any case. Given the weight of Anaconda in this space, I'm also a bit reluctant to host our packages there to avoid too much centralization (and Pyscript might do something in that direction eventually I guess). As a for using it for hosting user-built wheel, certainly, we should probably mention this possibility.

So in the end I'm not sure what would be the outcome of this issue. There seem to be a growing sentiment that we don't want to manage or review a service where people can arbitrarily upload packages. We still need JsDelivr in any case for JS files. And PyPI support seem within reach on the medium term. Better support for third party hosting services is something we also need to improve in pyodide/micropip#62.

So maybe the outcome is just to recommend wheel hosting services people can use to host wheels (including anaconda.org)

[ryanking13]

So maybe the outcome is just to recommend wheel hosting services people can use to host wheels (including anaconda.org)

Yes, that was what I was thinking in pyodide/micropip#62. What we do is to support standard PyPI APIs (Simple API) + allow people to use alternative registries. Then people can choose any hosting solutions they want until PyPI supports hosting Emscripten wheels.

[freakboy3742]

@rth Totally understood that you might not want to visibly use an Anaconda service for this; I mention it only because (a) BeeWare is using it for other purposes, and (b) if the constraint was having access to a free simple index hosting option, the option is there.

In terms of my own personal wishlist - I acknowledge that there's a need for other non-wheel files to be hosted, but having the existing, officially published binary wheels available as a simple index would (AFAICT) do everything I'm currently looking for. Maintaining an unofficial mirror of the published Pyodide wheels is one of the options on the table, but I'd vastly prefer to avoid if I can. Longer term, having the ability for other users to upload wheels would also be great (but also moves into the territory of "build your own PyPI"); but in the short term, having pip-compatible access to the wheels that already exist would be more than sufficient.

[hoodmane]

Since we're already generating a simple index I agree that we should deploy it somewhere so you can use it.