Workaround to fix BPO-43112

[dvarrazzo]

What's the problem this feature will solve?

In this moment, because of BPO-43112, we are distributing broken musllinux_1_1 wheel packages for Python 3.9 and 3.10, which work on Docker images such as python:3.9-alpine but will not work on the alpine using the system Python.

When BPO-43112 will be solved, there will be a situation of incompatibility, resulting in newly built packages not working on the python-alpine image, for some time, and not working for current images.

Describe the solution you'd like

Pip is in the right place to save the day 🙂 because it could easily rename the broken file on install, if it detects the right conditions (tag musllinux_1_1, .so file suffix mismatching the sysconfig.get_config_vars("SOABI")) it might simply rename the wrongly named library. It has the file system permission to do so and it would work both to make broken wheels work on new images and fixed wheels to work on old images.

Alternative Solutions

A workaround bridge solution of distributing a compatibility symlink (see psycopg/psycopg#161) is hampered by the fact that symlinks in zip files are not handled by the zipfile module (https://bugs.python.org/issue18595, https://bugs.python.org/issue27318).

Additional context

• https://bugs.python.org/issue43112
• https://discuss.python.org/t/a-mess-with-soabi-tags-on-musllinux/11688
• Add image for musllinux_1_2 policy manylinux#1225
• bpo-43112: detect musl as a separate SOABI python/cpython#24502
• musllinux mess potential solution auditwheel#349
• Tracking issue for broken musllinux wheels cibuildwheel#934
• Some musllinux binary package not correctly working psycopg/psycopg#161

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[henryiii]

I think pip should make a symlink so both names work, otherwise this could become dependent on the Python patch version reading the installed copy. This should only be needed for Python < 3.11, as 3.11 should handle this correctly, I would expect.

As mentioned, ideally audithwheel should do this, but it can't make symlinks in wheels since CPython never fixed symlinks in zip files.

[pfmoore]

I don't quite understand the issue here. PEP 656 defines the tags for MUSL wheels. So when you say "we are distributing broken musllinux_1_1 wheel packages", why is the remedy not simply "don't do that, then"? I'm not being awkward here, I'm completely OK with fixing this issue, but the messages I've seen have been a bit confused, and I don't understand why this is suddenly an issue.

The inability to have symlinks in wheels is far from a new issue, and the patch referenced in BPO-43112 (python/cpython#24502) was submitted back in February, well before the musllinux PEP was approved - so why was the PEP submitted for approval if the PR hadn't been accepted? What was the expectation for how people would build musllinux wheels given that bpo-43112 was a known issue at the time (in particular, @tiran linked it in the PEP 656 thread here).

As I say, I'm fine with getting a fix for this, but the contract behind musllinux (and the perennial manylinux standard that it's based on) is that wheel producers are responsible for ensuring the wheels are compatible with all environments implied by the tag. How did that break down in this case?

[uranusjr]

I believe the problem is that the “non-broken” wheels don’t work on any of the currently released Python versions, so this request is based on an implicit engineering decision to make new wheels from this point on to work on those existing Python versions.

The wheel tag is also not technically related to the SOABI issue, because a platform-tagged wheel does not necessarily contain binary (it’s the other way around), and wheels can be built and deployed, it’s only Python is unable to load some code installed from it.

I’m wondering though, is it really the best idea to treat this as a bugfix? I also commented the same thought on bpo-43112; the current “wrong” -gnu suffix “works”, it’s just not disinguishing the libc implementations. I am missing information on how the current scheme is not working. Why must this behavioural change be backported?

[henryiii]

Quick visual summary: Python on musllinux_1_1, Alpine 3.12, and Alpine 3.13 both produces and expects the following file and internal .so (zip structure hidden for clarity):

psycopg_binary-3.0.4-cp310-cp310-musllinux_1_1_x86_64.whl
 - _psycopg.cpython-310-x86_64-linux-gnu.so

Hover, Alpine 3.14¹ produces² and expects only this:

psycopg_binary-3.0.4-cp310-cp310-musllinux_1_1_x86_64.whl
 - _psycopg.cpython-310-x86_64-linux-musl.so

For CPython 3.11, this is easy, just produce and read the -musl suffix. For older versions, or for abi3 extensions, or for wheels like cmake that don't have Python specific components that need to work on all Python versions, this suddenly becomes much more complex. A wheel built with -musl.so inside will not work anywhere but Alpine 3.14 (maybe some patch release version too? Or maybe 3.14.0?). A wheel build with -gnu.so is the reverse, it only works on Alpine 3.14.

Footnotes

1. Is this a patch they have added? It seems this might not be fixed in CPython yet? Wasn't following that. ↩

2. technically produces musllinux_1_2, since 3.14 has musl 1.2. But this can be ignored, it has no bearing on the expectation. ↩

[dvarrazzo]

@pfmoore The issue comes from Python having been broken for some time in ways that are not captured by the MUSL tag (and not adequately fixed by a new tag). Python distributed in the python:3.9-alpine docker image (and 10) is buggy (affected by BPO-43112):

$ docker run --rm python:3.9-alpine python -c 'import sysconfig; print(sysconfig.get_config_vars("SOABI"))'
['cpython-39-x86_64-linux-gnu']  <<< wrong

on this docker image gets built the manylinux, which creates libraries with -gnu in their name, which work on python:alpine, but:

• they don't work on alpine images, because their python goes:

$ docker run -ti --rm alpine
/ # apk update && apk add python3
/ # python3 -c 'import sysconfig; print(sysconfig.get_config_vars("SOABI"))'
['cpython-39-x86_64-linux-musl']

• they will not work on python:alpine either once the bug is fixed.
• after the bug is fixed, packages built on manylinux based on the fixed Python image will not work on currently released Python images.

The options to provide wheel packages that work for the whole Python 3.9 and 3.10 lifespan is either to build packages twice as big (and they get chubby pretty quickly already, see pydantic/pydantic#3439) or that the bug is addressed at install time, and pip happens to be in position to do so.

[uranusjr]

Python distributed in the python:3.9-alpine docker image (and 10) is buggy

This definition of “buggy” assumes that -gnu is wrong. But this is only a string the interpreter uses internally to find modules; if it’s any other value, say -linux, it suddenly looks much less wrong, does it not?

they don't work on alpine images, because their python goes

This can arguably be categorises as an Alpine bug instead, because they distribute Python built in a way that can’t load modules built from the official Python distribution. Using the same hypothetical -linux suffix, if a hypothetical Linux distribution builds Python to use -windows, would its inability to load modules even be remotely considered as a CPython bug? I doubt it.

[tiran]

The issue comes from Python having been broken for some time in ways that are not captured by the MUSL tag (and not adequately fixed by a new tag). Python distributed in the python:3.9-alpine docker image (and 10) is buggy (affected by BPO-43112)

There is some confusing around official Python container images on Dockerhub. The official Docker images on Dockerhub are not provided by the Python core team, Python release managers or PSF. They are created by the Docker community. We have no control what goes into the containers and how they are build.

[henryiii]

$ docker run --rm -it alpine:3.14
# apk add python3
...
# python3 -c 'import sysconfig; print(sysconfig.get_config_vars("SOABI"))'
['cpython-39-x86_64-linux-musl']

Is it currently supported to add multiple extensions in a single installed folder? I think that was the idea behind the tagging in the first place; you could have a shared directory accessible by multiple OSs; in this case, if it had worked, this could have been usable by both musl and gnu linux. Not sure how important this is (you can't mix manylinux versions, etc).

If not, then calling this "-gnu" isn't necessarily a bug, and Alpine's patching is at fault.

official Python container images on Dockerhub

These are not doing any special patching on CPython, only Alpine is, from what I understand. Ether -gnu is wrong, and therefore CPython is buggy, or it is correct, and Alpine is buggy.

[henryiii]

Here's the patch: https://git.alpinelinux.org/aports/tree/main/python3/bpo-43112.patch?h=3.14-stable (from https://git.alpinelinux.org/aports/tree/main/python3?h=3.14-stable )

And guessing that's @ncopa maintaining and adding the patch.

[dvarrazzo]

My definition of who is buggy and who is right came from reading the BPO issue, e.g. https://bugs.python.org/issue43112#msg386710

As far as I'm concerned I can let Python and Alpine people decide who is right and who is wrong.

The status quo is that, under the manylinux_1_1 tag, different incompatible images can be produced, and the risk is that, with a change of the build tools (manylinux issue), released packages will suddenly stop working (final packager issue).

As far as I am concerned, I can see that the -gnu suffix is more used than the -musl one (being python:alpine a popular base), so I would be totally fine if Alpine decided to go the -gnu way. However, as a packages creator I need:

• to be reassured that extensions will always be -gnu on 3.9 and 3.10 aka "43112 is not a bug, it's a feature", or
• the install tool is able to take care of the changes that may happen along the Python 3.9.x lifetime.

Without either of these reassurances, I cannot conscientiously release psycopg musllinux binary packages, and will stop doing so from the next bugfix release, in order to safeguard my users.