arm64 - fork/exec /bin/sh: exec format error

[ricardorqr]

Hi team

I have a Jenkins pipeline which creates a multi-architecture image using Kaniko. I'm following this tutorial: https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-multi-architecture-image.html

This is my pipeline

pipeline {
  agent {
    kubernetes {
      inheritFrom 'jenkins-kaniko-tomcat'
      yaml """
        apiVersion: v1
        kind: Pod
        metadata:
          name: kaniko-tomcat
          namespace: jenkins
        spec:
          containers:
            - name: awscli
              image: amazon/aws-cli:latest
              command:
              - sleep
              args:
              - 99d
              volumeMounts:
                - name: docker-config
                  mountPath: /kaniko/.docker
                - name: jenkins-root
                  mountPath: /tmp/workspace
            
            - name: kaniko
              image: gcr.io/kaniko-project/executor:debug
              #image: 495633144232.dkr.ecr.us-west-2.amazonaws.com/jenkins-agent-rico:kaniko
              imagePullPolicy: Always
              command:
              - sleep
              args:
              - 99d
              volumeMounts:
                - name: docker-config
                  mountPath: /kaniko/.docker
                - name: jenkins-root
                  mountPath: /tmp/workspace       
          restartPolicy: Never
          volumes:
            - name: docker-config
              configMap:
                name: docker-config
            - name: jenkins-root
              emptyDir: {}
      """
    }
  }
  
  stages {
    // Other stages...

    stage('Build image - arm64') {
      steps {
        container(name: 'kaniko') {
          sh '''
            /kaniko/executor --context `pwd` --verbosity trace \
            --destination 123456789.dkr.ecr.us-east-2.amazonaws.com/my-repo:$my-image-arm64 \
            --custom-platform linux/arm64
          '''
        }
      }
    }
    
    stage('Build image - amd64') {
      steps {
        container(name: 'kaniko') {
          sh '''
            cd uber-cloud-tools
            modified_string=$(echo "$VERSION" | tr '/' '-')
          
            /kaniko/executor --context `pwd` --verbosity debug \
            --destination 123456789.dkr.ecr.us-east-2.amazonaws.com/my-repo:$my-image-amd64 \
            --custom-platform linux/amd64
          '''
        }
      }
    }

    stage('Build image - multi-architecture') {
      steps {
        container(name: 'awscli') {
          sh '''
            aws ecr get-login-password --region us-east-2 | docker login --username AWS \
            --password-stdin 123456789.dkr.ecr.us-east-2.amazonaws.com

            docker manifest create 123456789.dkr.ecr.us-east-2.amazonaws.com/my-repo:$my-image \
            123456789.dkr.ecr.us-east-2.amazonaws.com/my-repo:$my-image-arm64 \
            123456789.dkr.ecr.us-east-2.amazonaws.com/my-repo:$my-image-amd64

            docker manifest push 123456789.dkr.ecr.us-east-2.amazonaws.com/my-repo:$my-image
          '''
        }
      }
    }
    
  }
}

This is my Dockerfile. As you can see it is a very simple one.

FROM amazoncorretto:8u342

RUN yum install -y procps && yum clean all

And this is the error.

 ...
 [36mINFO [0m[0032] Cmd: /bin/sh                                 
 [36mINFO [0m[0032] Args: [-c yum install -y procps && yum clean all] 
 [36mINFO [0m[0032] Running: [/bin/sh -c yum install -y procps && yum clean all] 
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: exec format error

It fails when it executes the line RUN yum install -y procps && yum clean all. I have other pipelines that don't have any command (RUN) to perform, so they are working. If I remove the stage Build image - arm64, the stage Build image - amd64 works. I'm also using --custom-platform. It seems this error is related to arm64. I'm not sure.

Any idea how can I silver this?

[ricardorqr]

I tested removing the line RUN yum install -y procps && yum clean all and the stage Build image - arm64 worked. It seems for arm64, commands are not working.

[ricardorqr]

Any idea how I can solve this problem?

[cwboden]

I'm also running into this issue. I have a shell script which I copy in and attempt to run -- it works fine when building for --platform=linux/amd64 but not when I use --platform=linux/arm64 (nor linux/arm64/v8).

I get the same error message as @ricardorqr:

INFO[0011] Running: [/bin/sh -c chmod +x /tmp/wait_then_shutdown.sh] 
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/bash: exec format error

I'm using mysql:8.0.35, as compared to the amazoncorretto image above, but I would agree that this seems arm64 related. I'm not even able to RUN echo from that Dockerfile:

INFO[0014] Running: [/bin/sh -c echo "Hello, world!"]   
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: exec format error

Advice appreciated, let me know if I can add any more context.

[ricardorqr]

Hi @cwboden,

I have to say that you made me dig into my old repositories for this solution. Alright, let me try to explain step by step.

First, here are a few resources I had to read to understand this problem.

• https://www.reddit.com/r/docker/comments/13pcihm/cant_get_arm64_image4_to_build/
• https://drpdishant.medium.com/multi-arch-images-with-docker-buildx-and-qemu-141e0b6161e7
• https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/
• https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-multi-architecture-image.html

Second, since I have Docker running in this ec2 where Jenkins is deployed, I had to run QEMU as an image using this command.

sudo docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
sudo docker buildx ls

This was the output:

+ sudo docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
Unable to find image 'docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64' locally
a7996909642ee92942dcd6cff44b9b95f08dad64: Pulling from docker/binfmt
5d6ca6c8ba77: Pulling fs layer
b26a8e2c75fc: Pulling fs layer
3436361ddd98: Pulling fs layer
3436361ddd98: Download complete
b26a8e2c75fc: Verifying Checksum
b26a8e2c75fc: Download complete
5d6ca6c8ba77: Verifying Checksum
5d6ca6c8ba77: Download complete
5d6ca6c8ba77: Pull complete
b26a8e2c75fc: Pull complete
3436361ddd98: Pull complete
Digest: sha256:758ca0563f371b384cfd67b6590b5be2dc024fef45bc14a050ae104f0caad14e
Status: Downloaded newer image for docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
+ sudo docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT     PLATFORMS
default * docker
  default default         running v0.8+unknown linux/amd64, linux/386, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/arm/v7, linux/arm/v6

If you run this command sudo docker buildx ls without QEMU image, you will see way less architectures. Since you are running docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64, it adds/simulates the ARM platform.

After that, I was able to run the command to create the image for ARM64. Here is my code.

# Rum QEMU to emulate arm64
# https://www.reddit.com/r/docker/comments/13pcihm/cant_get_arm64_image4_to_build/
sudo docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64
sudo docker buildx ls

# Create images
modified_string=$(echo "${BUILD_ID}-${BRANCH}-${BUILD_NUM}" | tr '/' '-')
aws ecr get-login-password --region us-east-2 | sudo docker login --username AWS --password-stdin 123456789.dkr.ecr.us-east-2.amazonaws.com

# Create amd64 image
sudo docker buildx build --platform=linux/amd64 -t 123456789.dkr.ecr.us-east-2.amazonaws.com/XXXXX:$modified_string-amd64 -f Dockerfile . --push

# Create arm64 image
sudo docker buildx build --platform=linux/arm64 -t 123456789.dkr.ecr.us-east-2.amazonaws.com/XXXXX:$modified_string-arm64 -f Dockerfile . --push

# Create multi architecture image
sudo docker manifest create 123456789.dkr.ecr.us-east-2.amazonaws.com/XXXXX:$modified_string \
  123456789.dkr.ecr.us-east-2.amazonaws.com/XXXXX:$modified_string-arm64 \
  123456789.dkr.ecr.us-east-2.amazonaws.com/XXXXX:$modified_string-amd64
sudo docker manifest push 123456789.dkr.ecr.us-east-2.amazonaws.com/XXXXX:$modified_string

I hope it helps.

[shencan]

i have the same case,My temporary solution is to run 2 tasks one on an arm64 machine and one on an amd64 machine