Install the docker buildx plugin with a builder to build arm containers

[nitrocode]

In order to start building containers for amd64 and arm (and additional platforms), it would be nice to pre-install the buildx binary.

Install buildx

# install docker buildx globally
DOCKER_DIR=/usr/libexec/docker
## get latest version or pin it to v0.4.2
BUILDX_VERSION=$(curl --silent "https://api.github.com/repos/docker/buildx/releases/latest" | jq -r '.tag_name')
mkdir -p $DOCKER_DIR/cli-plugins

## check architecture
UNAME_ARCH=`uname -m`
case $UNAME_ARCH in
  aarch64)
    BUILDX_ARCH="arm64";
    ;;
  amd64)
    BUILDX_ARCH="amd64";
    ;;
  *)
    BUILDX_ARCH="amd64";
    ;;
esac

wget \
  -O $DOCKER_DIR/cli-plugins/docker-buildx \
  -nv https://github.com/docker/buildx/releases/download/$BUILDX_VERSION/buildx-$BUILDX_VERSION.linux-$BUILDX_ARCH
chmod a+x $DOCKER_DIR/cli-plugins/docker-buildx

## Install in the buildkite-agent user
# install arm
runuser -l buildkite-agent -c 'docker run --privileged --rm tonistiigi/binfmt --install all'
# create docker builder
runuser -l buildkite-agent -c 'docker buildx create --platform linux/amd64,linux/arm64,linux/arm/v7 --name mybuilder --use'
runuser -l buildkite-agent -c 'docker buildx inspect --bootstrap'

Sanity check: SSH into buildkite ec2 instance

$ sudo su - buildkite-agent
$ docker buildx version
github.com/docker/buildx v0.4.2 fb7b670b764764dc4716df3eba07ffdae4cc47b2
$ docker buildx ls
NAME/NODE    DRIVER/ENDPOINT             STATUS  PLATFORMS
mybuilder *  docker-container
  mybuilder0 unix:///var/run/docker.sock running linux/amd64*, linux/arm64*, linux/arm/v7*, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v6
default      docker
  default    default                     running linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

While ssh'ed, test container creation

mkdir test
cd test
cat <<EOF > Dockerfile
FROM ubuntu
RUN uname -m
EOF
docker buildx build --platform linux/amd64,linux/arm64 . --progress plain

Which then returns the expected architectures

#8 [linux/amd64 2/2] RUN uname -m
#8 0.153 x86_64
#8 DONE 0.2s

#6 [linux/arm64 2/2] RUN uname -m
#6 0.200 aarch64
#6 DONE 0.3s

Source: docker/buildx#57 (comment)

Misc

• EnableDockerUserNamespaceRemap=false should resolve this issue

  Error response from daemon: privileged mode is incompatible with user namespaces

  Related user namespace remap issue Document userns remapping #538

• EnableDockerExperimental=true is mandatory for buildx

• ECRAccessPolicy=poweruser is required for pushing containers to ecr

[yob]

This is an interesting suggestion. I know that buildx is distributed by default in most docker installs now, but the elastic stack AMI installs docker manually so doesn't have any of the bells and whistles.

I know we're keen to keep the tooling on the AMI fairly minimal, but I'll start a discussion about this one internally.

In the interim, I recommend installing buildx at instance boot time via a bootstrap script.

It might be best to install it to a system location rather than the home directory, as we set DOCKER_CONFIG to a random tmp dir for each build. Based on the docker source and my local docker install (on Debian), I think the plugins installed into /usr/libexec/docker/cli-plugins will be found automatically.

[nitrocode]

Yes, our buildkite_init.sh script is doing this currently. I modified it so it doesn't put the binary in the /var/lib/buildkite-agent/.docker/cli-plugins and instead puts the binary in /usr/libexec/docker/cli-plugins to install it globally. I checked this and it works correctly. Thank you.

[nitrocode]

One thing I cannot understand... I created the builder for buildkite-agent and I can see the builder for this user if I ssh to the instance and login as that user. However, when I run a build on the same EC2 instance as the same user, it doesn't seem to find my builder so I have to create the builder or it returns the following.

auto-push is currently not implemented for docker driver

Any idea why that is ? Seems like the docker buildx create commands are only created for the user that initiated it. I confirmed that the buildkite-agent user is the one that runs when I run the build. Not sure where to go from here. It does look like if I create the builder on-the-fly within the commands of my pipeline, then it works. I'd prefer to set this up when the instance starts using the BootstrapScriptUrl parameter.

[yob]

Unfortunately I'm not familiar enough with buildx to know the answer off the top of my head.

I created the builder for buildkite-agent and I can see the builder for this user if I ssh to the instance and login as that user

I'm not 100% sure what "created a builder" means in a buildx context. Do you mean that the docker-buildx executable was installed into /usr/libexec/docker/cli-plugins, but it's not available when docker runs within a buildkite job?

[nitrocode]

@yob here is how i created the builder using the bootstrapscripturl script which contains

## Install in the buildkite-agent user
# install arm
runuser -l buildkite-agent -c 'docker run --privileged --rm tonistiigi/binfmt --install all'
# create docker builder
runuser -l buildkite-agent -c 'docker buildx create --platform linux/amd64,linux/arm64,linux/arm/v7 --name mybuilder --use'
runuser -l buildkite-agent -c 'docker buildx inspect --bootstrap'

Even though this builder is available when i ssh into the instance as buildkite-agent, it's inaccessible when running a buildkite job.

I have to rerun these commands at the start of my pipeline.

[sj26]

I think docker buildx builders are defined in files within the docker config directory, and the elastic stack uses a pristine docker config for each build for isolation, and because otherwise state like docker login can leak between many agents on the same machine, or across jobs:

elastic-ci-stack-for-aws/packer/windows/conf/buildkite-agent/hooks/environment

Lines 7 to 10 in 374bcc5

	# a clean docker config for each job, for improved isolation
	BUILDKITE_DOCKER_CONFIG_TEMP_DIRECTORY=$(mktemp -d)
	export BUILDKITE_DOCKER_CONFIG_TEMP_DIRECTORY
	export DOCKER_CONFIG="$BUILDKITE_DOCKER_CONFIG_TEMP_DIRECTORY"

You might need to define a new builder in each pipeline which needs them, or add a bit of script in your environment hook or similar to do it for you, or copy the original docker config.

[nitrocode]

@sj26 that confirms why the builder needs to be recreated. For now I'm running this command prior to any buildx builds and it works quickly and doesn't require rerunning the qemu container or rerunning the bootstrap command.

docker buildx create --platform linux/amd64,linux/arm64,linux/arm/v7 --name mybuilder --use

Thank you very much

[v-kumar]

Any update on this issue?

buildx also makes distributed docker layer caching more easier as well as helping with multi-platform builds.

We are currently downloading the binary manually during build step (would rather not maintain a custom AMI).

[sj26]

docker buildx is included in the default docker packaging for the version we currently use:

https://docs.docker.com/engine/release-notes/#20106

I think we should be making sure it is available.

I notice we're custom installing docker:

https://github.com/buildkite/elastic-ci-stack-for-aws/blob/master/packer/linux/scripts/install-docker.sh

Should we still be doing this? Perhaps we should be upgrading to docker-ce from their official rpm channels instead? That would include buildx by default. Amazon Linux is compatible with centos 7, so this should work:

https://docs.docker.com/engine/install/centos/#install-using-the-repository
https://download.docker.com/linux/centos/

Otherwise we should probably fold in @nitrocode's example into the custom installer script.

[keithduncan]

Support for this has landed in the master branch 🎉 and you can update your stack’s template using the template published from master if you’d like to try it out before we publish the next tagged release.

We recommend using a CloudFormation Change Set when updating from the master template so you have the opportunity to review what changes will be made before applying them, happy build(x)ing!