ci: use docker github builder to build binaries and images #6388
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -50,60 +50,63 @@ jobs: | |
| fields: platforms | ||
|
|
||
| binaries: | ||
| uses: docker/github-builder-experimental/.github/workflows/bake.yml@7643588149117bf0ca3a906caa3968c70484027a | ||
| permissions: | ||
| contents: read # same as global permission | ||
| id-token: write # for signing attestation(s) with GitHub OIDC Token | ||
| with: | ||
| runner: amd64 | ||
| setup-qemu: true | ||
| artifact-name: buildkit-binaries | ||
| artifact-upload: true | ||
| cache: true | ||
| cache-scope: binaries | ||
| target: release | ||
| output: local | ||
| sbom: true | ||
| sign: ${{ github.event_name != 'pull_request' }} | ||
|
|
||
| binaries-finalize: | ||
| runs-on: ubuntu-24.04 | ||
| needs: | ||
| - binaries | ||
| steps: | ||
| - | ||
| name: Download artifacts | ||
| uses: actions/download-artifact@v6 | ||
| with: | ||
| path: /tmp/buildx-output | ||
| name: ${{ needs.binaries.outputs.artifact-name }} | ||
| - | ||
| name: Rename provenance and sbom | ||
| run: | | ||
|
Member
There was a problem hiding this comment. Not needed atm but eventually we should provide some examples for github builder users for verifying the signatures of these artifacts. Eg. if for any reason the signature is not valid, that should error before creating the release.
Member
Author
There was a problem hiding this comment.
They have an example in the usage section using the They also have the cosign verify commands as output so they can invoke them locally if they want to: https://github.com/docker/github-builder-experimental/blob/7643588149117bf0ca3a906caa3968c70484027a/.github/workflows/build.yml#L133-L135 Looks like this: https://github.com/docker/github-builder-experimental/actions/runs/20895583887/job/60033611659#step:2:24 We could also print cosign commands in the build summary, this is tracked in docker/github-builder#11
We are also verifying the signature after signing blobs during build: https://github.com/docker/github-builder-experimental/blob/7643588149117bf0ca3a906caa3968c70484027a/.github/workflows/build.yml#L627-L630 Looks like this: https://github.com/docker/github-builder-experimental/actions/runs/20895583887/job/60033587793#step:12:62 
Member
There was a problem hiding this comment.
Yes but this is in our controlled repository. If the app is pushing a release from their own CI workflow, then ideally, they first verify the artifacts themselves instead of just trusting that the remote builder did it. There is also a case that something malicious gained capability to write artifacts (or overwrite the artifacts created by docker builder) so this way we can check that these were indeed the artifacts from the trusted builder. |
||
| for pdir in /tmp/buildx-output/*/; do | ||
| ( | ||
| cd "$pdir" | ||
| binname=$(find . -name 'buildkit-*') | ||
| filename=$(basename "$binname" | sed -E 's/\.(tar\.gz|zip)$//') | ||
| mv "provenance.json" "${filename}.provenance.json" | ||
| mv "sbom-binaries.spdx.json" "${filename}.sbom.json" | ||
| find . -name 'sbom*.json' -exec rm {} \; | ||
| if [ -f "provenance.sigstore.json" ]; then | ||
| mv "provenance.sigstore.json" "${filename}.sigstore.json" | ||
| fi | ||
| ) | ||
| done | ||
| mkdir -p "${{ env.DESTDIR }}" | ||
| mv /tmp/buildx-output/**/* "${{ env.DESTDIR }}/" | ||
| - | ||
| name: List artifacts | ||
| working-directory: ${{ env.DESTDIR }} | ||
| run: | | ||
| tree -nh . | ||
| - | ||
| name: Upload release binaries | ||
| uses: actions/upload-artifact@v6 | ||
| with: | ||
| name: buildkit-release | ||
| path: ${{ env.DESTDIR }}/* | ||
| if-no-files-found: error | ||
|
|
||
| test: | ||
| uses: ./.github/workflows/.test.yml | ||
|
|
@@ -164,128 +167,127 @@ jobs: | |
| with: | ||
| sarif_file: ${{ env.DESTDIR }}/govulncheck.out | ||
|
|
||
| image-prepare: | ||
| runs-on: ubuntu-24.04 | ||
| outputs: | ||
| includes: ${{ steps.set.outputs.includes }} | ||
| steps: | ||
| - | ||
| name: Set outputs | ||
| id: set | ||
| uses: actions/github-script@v8 | ||
| env: | ||
| INPUT_DEFAULT-BASE: alpine | ||
| INPUT_REF: ${{ github.ref }} | ||
| INPUT_IMAGE-NAME: ${{ env.IMAGE_NAME }} | ||
| with: | ||
| script: | | ||
| const defaultBase = core.getInput('default-base'); | ||
|
Member
There was a problem hiding this comment. These kinds of big inline blocks could have a comment in the beginning explaining the intention of the script. |
||
| const ref = core.getInput('ref'); | ||
| const imageName = core.getInput('image-name'); | ||
|
|
||
| function getTagSuffixAndLatest(base, target) { | ||
| let tagSuffix = ''; | ||
| if (target) { | ||
| tagSuffix += `-${target}`; | ||
| } | ||
| if (base && base !== defaultBase) { | ||
| tagSuffix += `-${base}`; | ||
| } | ||
| let tagLatest = ''; | ||
| if (ref && ref.startsWith('refs/tags/v')) { | ||
| const version = ref.replace('refs/tags/', ''); | ||
| if (/^v[0-9]+\.[0-9]+\.[0-9]+$/.test(version)) { | ||
| tagLatest = target ? target : 'latest'; | ||
| if (base && base !== defaultBase) { | ||
| tagLatest += `-${base}`; | ||
| } | ||
| } | ||
| } | ||
| return { tagSuffix, tagLatest }; | ||
| } | ||
|
|
||
| const matrix = [ | ||
| { base: 'alpine' }, | ||
| { base: 'alpine', target: 'rootless'}, | ||
| { base: 'ubuntu', buildTags: 'nvidia venus' } | ||
| ] | ||
|
|
||
| for (const entry of matrix) { | ||
| const { tagSuffix, tagLatest } = getTagSuffixAndLatest(entry.base, entry.target); | ||
| entry.imageName = imageName; | ||
| entry.tagSuffix = tagSuffix; | ||
| entry.tagLatest = tagLatest; | ||
| } | ||
|
|
||
| core.info(JSON.stringify(matrix, null, 2)); | ||
| core.setOutput('includes', JSON.stringify(matrix)); | ||
|
|
||
| image: | ||
| uses: docker/github-builder-experimental/.github/workflows/bake.yml@7643588149117bf0ca3a906caa3968c70484027a | ||
| needs: | ||
| - image-prepare | ||
| - test | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| include: ${{ fromJson(needs.image-prepare.outputs.includes) }} | ||
| permissions: | ||
| contents: read # same as global permission | ||
| id-token: write # for signing attestation(s) with GitHub OIDC Token | ||
| with: | ||
| runner: amd64 | ||
| setup-qemu: true | ||
| target: image-cross | ||
| cache: true | ||
| cache-scope: image | ||
| output: image | ||
| push: ${{ github.repository == 'moby/buildkit' && (github.event_name == 'schedule' || github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags/v')) }} | ||
| sbom: true | ||
| set: | | ||
| *.args.IMAGE_TARGET=${{ matrix.target }} | ||
| *.args.EXPORT_BASE=${{ matrix.base }} | ||
| *.args.BUILDKITD_TAGS=${{ matrix.buildTags }} | ||
|
Member
There was a problem hiding this comment. These |
||
| set-meta-annotations: true | ||
| meta-images: | | ||
| ${{ matrix.imageName }} | ||
| # versioning strategy | ||
| ## push semver tag v0.24.0 | ||
| ### moby/buildkit:v0.24.0 | ||
| ### moby/buildkit:latest | ||
| ### moby/buildkit:v0.24.0-rootless | ||
| ### moby/buildkit:rootless | ||
| ### moby/buildkit:v0.24.0-ubuntu | ||
| ### moby/buildkit:latest-ubuntu | ||
| ## push semver prerelease tag v0.24.0-rc1 | ||
| ### moby/buildkit:v0.24.0-rc1 | ||
| ### moby/buildkit:v0.24.0-rc1-rootless | ||
| ### moby/buildkit:v0.24.0-rc1-ubuntu | ||
| ## push on master | ||
| ### moby/buildkit:master | ||
| ### moby/buildkit:master-rootless | ||
| ### moby/buildkit:master-ubuntu | ||
| ## scheduled event on master | ||
| ### moby/buildkit:nightly | ||
| ### moby/buildkit:nightly-rootless | ||
| ### moby/buildkit:nightly-ubuntu | ||
| meta-tags: | | ||
| type=schedule,pattern=nightly,suffix=${{ matrix.tagSuffix }} | ||
| type=ref,event=branch,suffix=${{ matrix.tagSuffix }} | ||
| type=ref,event=pr,suffix=${{ matrix.tagSuffix }} | ||
| type=semver,pattern={{raw}},suffix=${{ matrix.tagSuffix }} | ||
| type=raw,value=${{ matrix.tagLatest }} | ||
| meta-flavor: | | ||
| latest=false | ||
| meta-annotations: | | ||
| org.opencontainers.image.title=BuildKit | ||
| org.opencontainers.image.vendor=Moby | ||
| meta-bake-target: meta-helper | ||
| secrets: | ||
| registry-auths: | | ||
| - registry: docker.io | ||
| username: ${{ secrets.DOCKERHUB_USERNAME }} | ||
| password: ${{ secrets.DOCKERHUB_TOKEN }} | ||
|
|
||
| scout: | ||
| runs-on: ubuntu-24.04 | ||
|
|
@@ -338,20 +340,15 @@ jobs: | |
| contents: write | ||
| needs: | ||
| - test | ||
| - binaries-finalize | ||
| - image | ||
| steps: | ||
| - | ||
| name: Download release binaries | ||
| uses: actions/download-artifact@v7 | ||
| with: | ||
| path: ${{ env.DESTDIR }} | ||
| name: buildkit-release | ||
| - | ||
| name: GitHub Release | ||
| if: startsWith(github.ref, 'refs/tags/v') | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a need for this or is amd64 just faster?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really a need but as we are doing cross compilation we can just use official GitHub-hosted runner and avoid relying on partner images: https://github.com/actions/partner-runner-images?tab=readme-ov-file#github-actions-partner-runner-images