How to use matrix for multi-platform builds?

[felipecrs]

I know I can use the same runner to build all the platforms at the same time, but this causes my builds to take 2 hours instead of 20 minutes if I split to different runners.

I was able to achieve something similar with:

name: ci

on:
  push:
    branches:
      - "main"
  pull_request:
    branches:
      - "main"

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        platform:
          - linux/amd64
          - linux/386
          - linux/arm/v6
          - linux/arm/v7
          - linux/arm64
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Set cache name
        id: cache-name
        run: |
          echo 'cache-name=asterisk-cache-${{ matrix.platform }}' | sed 's:/:-:g' >> $GITHUB_OUTPUT

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: asterisk
          platforms: ${{ matrix.platform }}
          tags: asterisk
          cache-from: type=gha
          cache-to: type=local,dest=/tmp/asterisk-cache,mode=max

      - name: Upload cache
        uses: actions/upload-artifact@v3
        with:
          name: asterisk-cache-${{ steps.cache-name.outputs.cache-name }}
          path: /tmp/asterisk-cache
          if-no-files-found: error
          retention-days: 1

  push:
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v2
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Download cache
        uses: actions/download-artifact@v3
        with:
          path: /tmp/asterisk-cache

      - name: Get lowercase GitHub username
        id: repository_owner
        uses: ASzc/change-string-case-action@v5
        with:
          string: ${{ github.repository_owner }}

      - name: Docker meta
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: |
            ghcr.io/${{ steps.repository_owner.outputs.lowercase }}/asterisk-hass-addon
          tags: |
            type=ref,event=branch
            type=ref,event=pr
            type=semver,pattern={{version}}

      - name: Login to DockerHub
        if: github.event_name == 'push' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]'
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: asterisk
          platforms: |
            linux/amd64
            linux/386
            linux/arm/v6
            linux/arm/v7
            linux/arm64
          push: ${{ github.event_name == 'push' || github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
          cache-from: type=local,src=/tmp/.buildx-cache
          cache-to: type=gha,mode=max

The problem is that it takes 10 minutes to upload the cache and then more 5 minutes to download the cache again.

Is there any suggestion to circumvent this?

[K-shir0]

@felipecrs

I was able to handle individual jobs by using the method posted here.

Reference: #671 (comment)

[felipecrs]

That's very interesting!

I wonder if it would be possible to have buildx pushing different platforms without the need of being a single call.

Currently if I push a single --platform, it seems to override the previously pushed one.

[crazy-max]

docker/docs#17180 has been merged. See https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners

[felipecrs]

@crazy-max that's really cool but... last time I tried to use upload-artifact for the job, it took 20-30 minutes only to upload and download it.

It was definitely a showstopper for me.

Do you believe it has improved? Maybe I should try it again.

[crazy-max]

last time I tried to use upload-artifact for the job, it took 20-30 minutes only to upload and download it.

You were uploading all cache export which can be quite expensive to compress > upload > download > decompress. In https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners we are just uploading the resulting image tarball.

If you want to use cache in the first job you should consider the gha one with a scope affected for each platform. Let me know if you need some help for this.

[felipecrs]

I will do some testing. Thanks a lot!

[felipecrs]

This is how I'm adding GHA caching on top of the provided example:

- name: Prepare
  run: |
    mkdir -p /tmp/images
    platform=${{ matrix.platform }}
    platform=${platform//\//-}
    echo "TARFILE=${platform}.tar" >> $GITHUB_ENV
    echo "TAG=${{ env.TMP_LOCAL_IMAGE }}:${platform}" >> $GITHUB_ENV
    echo "SCOPE=${{ env.GITHUB_REF_NAME }}-${platform}" >> $GITHUB_ENV
- name: Build
  uses: docker/build-push-action@v4
  with:
    context: .
    platforms: ${{ matrix.platform }}
    tags: ${{ env.TAG }}
    outputs: type=docker,dest=/tmp/images/${{ env.TARFILE }}
    cache-from: type=gha,scope=${{ env.SCOPE }}
    cache-to: type=gha,scope=${{ env.SCOPE }},mode=max

The test is running now.

However, I wonder how can I integrate the push phase of the example with docker/metadata-action. I suppose I can map the tags to -t flags in docker buildx imagetools create with a shell script, but I wonder what should I do about the labels.

[sando38]

Hi, great to see that approach. I have been using this as well for a couple of months now. And can even adopt some of the commands to mine.

I did not find a solution, however, to annotate labels like "annotations": { "org.opencontainers.image.description": "DESCRIPTION" } to the resulting multi-arch image, yet. Is there a way how this could be achieved?

[crazy-max]

However, I wonder how can I integrate the push phase of the example with docker/metadata-action. I suppose I can map the tags to -t flags in docker buildx imagetools create with a shell script, but I wonder what should I do about the labels.

Following https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, I wonder if we could instead reuse the build push action with a temp dockerfile.

Not tested but here is the idea:

  push:
    runs-on: ubuntu-latest
    needs:
      - build
    services:
      registry:
        image: registry:2
        ports:
          - 5000:5000
    steps:
      -
        name: Download images
        uses: actions/download-artifact@v3
        with:
          name: images
          path: /tmp/images
      -
        name: Load images
        run: |
          for image in /tmp/images/*.tar; do
            docker load -i $image
          done
      -
        name: Push images to local registry
        run: |
          docker push -a ${{ env.TMP_LOCAL_IMAGE }}
      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          driver-opts: network=host
      -
        name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      -
        name: Temp Dockerfile
        run: |
          mkdir -p /tmp/dkfilectx
          echo "FROM ${{ env.TMP_LOCAL_IMAGE }}" > /tmp/dkfilectx/Dockerfile
      -
        name: Docker meta
        id: meta
        uses: docker/metadata-action@v4
        with:
          images: ${{ env.REGISTRY_IMAGE }}
      -
        name: Push
        uses: docker/build-push-action@v4
        with:
          context: /tmp/dkfilectx
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}
      -
        name: Inspect image
        run: |
          docker buildx imagetools inspect ${{ env.REGISTRY_IMAGE }}:${{ steps.meta.outputs.version }}

[felipecrs]

It's failing with:

https://github.com/TECH7Fox/asterisk-hass-addons/actions/runs/4949753398/jobs/8852483607?pr=251#step:11:157

Error: buildx failed with: ERROR: failed to solve: localhost:5000/asterisk-hass-addon: localhost:5000/asterisk-hass-addon:latest: not found

I think it's because the "Push" stage is missing the platforms. I'm trying with it now.

[felipecrs]

Oh no. That's not it. It's because we build the images with the platforms as tags, then later we try to access with :latest.

I think I know how to fix it. Trying now.