musllinux support #430

liath · 2021-09-21T03:58:10Z

musllinux spec landed, let's get these wheels rollin 😎

matthew-brett · 2021-09-21T22:44:01Z

These look reasonable - how can I help?

Do you know of a good way to test? I guess we need a muslinux testing docker image.

liath · 2021-09-22T01:12:09Z

Yeah looking at that now. Would you mind terribly if the alpine variants just pulled in pypa images like how the build step does? It'd really simplify things.

matthew-brett · 2021-09-22T08:32:45Z

The problem with that is that it is not a good test of whether the wheel really does work on another Linux.

Do you think it would be hard to make up another docker image like : https://github.com/multi-build/docker-images

liath · 2021-09-22T17:33:08Z

Yeah that's what I'm looking at. Getting multiple parallel versions of python on Alpine is tedious. I think the only routes to doing so are compiling them yourself, as the pypa images do, or pyenv, which I do not know much about as I've always used it through asdf which would be even less indicative of normal environments :P

liath · 2021-09-22T17:35:46Z

There's probably an argument there for just using what Alpine ships tbh, it is work to get other versions so most people will likely be using whatever is in apk.

liath · 2021-09-22T17:52:24Z

Or I could quit complaining and just get started on porting the build script. :P

matthew-brett · 2021-09-22T18:29:27Z

Actually, I think you'll find the build script will work fine, with not much modification - it's just you'll have to compile nearly all your Python versions, not just some of them - but that's just taking stuff out, and tweaking a bit.

liath · 2021-09-23T00:04:54Z

Yeah it seems fine (talking about this one right?). Just feels weird because it'll be exactly duplicating the pypa images and then why bother producing new images?

liath · 2021-09-23T00:06:46Z

Another option would be to just pull the official images. Which is probably where most musl packages produced here will end up being run

matthew-brett · 2021-09-23T07:52:36Z

I was thinking of https://github.com/multi-build/docker-images/tree/focal/docker - but any scripts that will build Pythons for you, or images where you have Python you can test on, will be good. The point is to have images that have different, newer kernels and libraries, than the baseline from the Manylinux build image, to test that the wheels work OK with these kernels and libraries, as well as the ones they were compiled for.

mattip · 2021-09-23T12:13:40Z

Do you think it would be hard to make up another docker image like : https://github.com/multi-build/docker-images

I think that makes sense. A PR to add alpine x86_64, i686, and arm64 images would be nice. I wouldn't worry about the build time to create python from source, after all these images are built only whenever a python version is released.

PyPy is not available for alpine linux, so attempts to use multibuild with it should report an error early on.

liath · 2021-09-28T07:07:35Z

Alright, so I've been noodling on this for a bit and I think this last commit is going to give the most relevant experience for end users. By using the official python images for this we get support for a whole range of versions that we don't have to maintain and more importantly, we get to test in one of the most likely environments users will be running in.

liath · 2021-09-28T07:46:30Z

The default interface is very simple and does what is expected at least. I think alpine:latest is probably the most likely end environment but that would mean we could only test against the major versions Alpine ships. It is possible, and probably should be documented as such, to use DOCKER_TEST_IMAGE="alpine:latest" with something in the config.sh hooks like:

function install_run {
    # get latest pip as apk delivers too old a version of packaging if you use theirs
    if [ -f "/etc/alpine-release" ]; then
      apk add python3 wget
      ln -s $(which python3) /bin/python
      wget "https://bootstrap.pypa.io/get-pip.py"
      python get-pip.py
    fi

    # Copypaste from multibuild/common_utils.sh:install_run
    install_wheel
    mkdir tmp_for_test
    (cd tmp_for_test && run_tests)
}

matthew-brett · 2021-09-28T09:18:52Z

I guess that Python is not installed by default in alpine:latest? It would be much better to have some alpine test image that was similar to the other test images, in that you didn't have to have custom stuff in your config, in order to run the standard tests. Did it turn out to be more difficult than we had predicted to make a corresponding docker image with different Python versions already installed?

liath · 2021-09-28T17:33:29Z

That's just to use alpine:latest, the commit I pushed does the right thing without having to mess with hooks.

liath · 2021-09-28T17:40:49Z

Building pythons shouldn't be hard but I feel it wouldn't be representative of the environments people actually use. Just from what I've seen at my orgs and how I use it, when people are using musl python they're using it in docker. If you need a version other than what Alpine ships, you're probably going to google it and get the official docker images. Only after everything else has failed would a normal user try to compile their own but even then they're likely to switch to one of the debian based official images and be done with it.

liath · 2021-10-01T18:23:32Z

Tested this with cffi builds and I think it's ready for for others to poke at so moving it outta draft now. For the record, this is how I've been testing:

#!/usr/bin/env bash

WORK_DIR=$(mktemp -d)
pushd "$WORK_DIR" || exit

git clone https://github.com/liath/cffi-travis-wheel .
hg clone https://foss.heptapod.net/pypy/cffi
git clone https://github.com/liath/multibuild --branch musllinux

source multibuild/common_utils.sh
source multibuild/travis_linux_steps.sh

export MB_PYTHON_VERSION=3.6 MB_ML_LIBC="musllinux" MB_ML_VER="_1_1"

before_install && build_wheel cffi

rm -rf "$WORK_DIR"

matthew-brett

A few first thoughts.

matthew-brett · 2021-10-04T10:45:16Z

common_utils.sh

@@ -308,6 +313,7 @@ function build_wheel_cmd {
    local repo_dir=${2:-$REPO_DIR}
    [ -z "$repo_dir" ] && echo "repo_dir not defined" && exit 1
    local wheelhouse=$(abspath ${WHEEL_SDIR:-wheelhouse})
+    mkdir -p "$wheelhouse"


Was this necessary? Did you get an error without it?

Yep, without it the wheel gets saved as a file at that path vs inside a folder.

matthew-brett · 2021-10-04T10:49:46Z

docker_test_wrap.sh

@@ -2,6 +2,17 @@
 # Install and test steps on Linux
 set -e

+# shim for python:alpine not having bash


This is the kind of thing that really should go in the test image, not in the invocation script. Does sh / ash not work?

It should but I was trying to go for as small a patch as possible instead of running around tweaking every script

Right - but the point is, here we have stuff which should be in the test image, and because it's not, it add maintenance burden to the multibuild code, as the reviewer tries to work out what it is about the test image you are trying to fix.

If I understand correctly, bash should be in the test image? Does multibuild assume /bin/sh is bash?

Is it required in the build image as well? Then the fix should be in pypya/manylinux. If it is only the test image, it should be fixed in multi-build/docker-images

docker_test_wrap.sh

manylinux_utils.sh

matthew-brett · 2021-10-04T10:55:45Z

travis_linux_steps.sh

@@ -132,5 +140,6 @@ function install_run {
        -e MANYLINUX_URL="$MANYLINUX_URL" \
        -e TEST_DEPENDS="$TEST_DEPENDS" \
        -v $PWD:/io \
+        --entrypoint sh \


Again - I think this should go in the test image itself. It should set its default entrypoint.

matthew-brett · 2021-10-13T13:19:01Z

@liath - there is now a docker test image for Alpine - it should soon be available at https://hub.docker.com/u/multibuild - it's being built now. Can you refactor to use that? Or would you like me to take this over?

mattip · 2021-10-13T14:28:42Z

The image is up, but the name focal_alpine feels weird
xref multi-build/docker-images#21.

liath · 2021-10-22T17:47:55Z

I still maintain that testing in the images people will be running these wheels on is the better solution, but I've switched to your test image as requested.

For the record, the hassle of installing bash on python:alpine gets you images that are:

exactly the same image where the wheels you're testing will likely run because they're the official images
updated weekly
available at any version of python with-in recent history
available for every platform that alpine runs on
you don't have to do a thing to manage

liath · 2021-10-25T18:07:57Z

Heck, if the shim is the issue we could just build from the official image inline with something like this in travis_linux_steps.sh:

     local plat=${1:-${PLAT:-x86_64}}
     if [ -z "$DOCKER_TEST_IMAGE" ]; then
         if [ "$MB_ML_LIBC" == "musllinux" ]; then
-          local docker_image="multibuild/alpine3.14_$plat"
+          docker pull "python:$MB_PYTHON_VERSION-alpine"
+          local docker_image="multibuild_alpine:latest"
+          docker build -t "$docker_image" -f "$MULTIBUILD_DIR/alpine.Dockerfile" \
+            --build-arg "PYTHON_VERSION=$MB_PYTHON_VERSION"  .
         else
           local bitness=$([ "$plat" == i686 ] && echo 32 || echo 64)
           local docker_image="matthewbrett/trusty:$bitness"
+          docker pull $docker_image
         fi
     else
         # aarch64 is called arm64v8 in Ubuntu
         local plat_subst=$([ "$plat" == aarch64 ] && echo arm64v8 || echo $plat)
         local docker_image="${DOCKER_TEST_IMAGE/\{PLAT\}/$plat_subst}"
+        docker pull $docker_image
     fi
# Moved the `docker pull` around because we want to pull `python:alpine` not the inline image.
-    docker pull $docker_image
     docker run --rm \
         -e PYTHON_VERSION="$MB_PYTHON_VERSION" \
         -e MB_PYTHON_VERSION="$MB_PYTHON_VERSION" \

And alpine.Dockerfile:

ARG PYTHON_VERSION="3.9"
FROM python:$PYTHON_VERSION-alpine

RUN apk update && apk add bash
ENTRYPOINT ["bash"]

manylinux_utils.sh

mattip · 2022-03-16T08:31:18Z

Is this waiting for more review? The NumPy builds use openblas compiled using multibuild, so in order to get numpy musl wheels we need musl support in multibuild.

matthew-brett · 2022-03-16T15:35:06Z

Looks good to me - let's get it in.

liath marked this pull request as ready for review October 1, 2021 18:23

matthew-brett reviewed Oct 4, 2021

View reviewed changes

stefanv mentioned this pull request Oct 12, 2021

Add Dockerfile to build Python 3.8, 3.9, and 3.10 on Alpine Linux multi-build/docker-images#20

Merged

liath added 6 commits October 22, 2021 09:48

initial pass at musllinux support

1062d81

clear up bash unary error

1cff255

use python:alpine as musllinux test image

3f00257

revert $BASH_SOURCE change in docker_test_wrapper.sh

e22a4db

more PR feedback

980d9b0

switch to multibuild provided alpine test images because... reasons?

7286847

fdintino reviewed Oct 26, 2021

View reviewed changes

manylinux_utils.sh Outdated Show resolved Hide resolved

fix typo

bc065e0

radarhere mentioned this pull request Nov 11, 2021

Musllinux wheels python-pillow/pillow-wheels#226

Closed

This was referenced Mar 16, 2022

Build for musl linux MacPython/openblas-libs#72

Closed

ENH: Enable musllinux wheels numpy/numpy#21200

Closed

matthew-brett merged commit 4b6eb06 into multi-build:devel Mar 16, 2022

radarhere mentioned this pull request Mar 17, 2022

Fixed installing cmake and wget on Alpine #458

Merged

liath deleted the musllinux branch March 19, 2022 06:13

radarhere mentioned this pull request Mar 30, 2022

Added musllinux python-pillow/pillow-wheels#271

Merged

musllinux support #430

musllinux support #430

Uh oh!

Conversation

liath commented Sep 21, 2021

Uh oh!

matthew-brett commented Sep 21, 2021

Uh oh!

liath commented Sep 22, 2021

Uh oh!

matthew-brett commented Sep 22, 2021

Uh oh!

liath commented Sep 22, 2021

Uh oh!

liath commented Sep 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liath commented Sep 22, 2021

Uh oh!

matthew-brett commented Sep 22, 2021

Uh oh!

liath commented Sep 23, 2021

Uh oh!

liath commented Sep 23, 2021

Uh oh!

matthew-brett commented Sep 23, 2021

Uh oh!

mattip commented Sep 23, 2021

Uh oh!

liath commented Sep 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liath commented Sep 28, 2021

Uh oh!

matthew-brett commented Sep 28, 2021 via email

Uh oh!

liath commented Sep 28, 2021

Uh oh!

liath commented Sep 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liath commented Oct 1, 2021

Uh oh!

matthew-brett left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matthew-brett commented Oct 13, 2021

Uh oh!

mattip commented Oct 13, 2021

Uh oh!

liath commented Oct 22, 2021

Uh oh!

liath commented Oct 25, 2021

Uh oh!

Uh oh!

mattip commented Mar 16, 2022

Uh oh!

matthew-brett commented Mar 16, 2022

Uh oh!

Uh oh!

liath commented Sep 22, 2021 •

edited

Loading

liath commented Sep 28, 2021 •

edited

Loading

liath commented Sep 28, 2021 •

edited

Loading