EBS CSI Driver does not work on arm64 based instances

[gkrizek]

/kind bug

What happened?

arm support was added in #527 for the driver Docker images. However, when deploying the EBS CSI from this repo it references Docker images that are not multiarch. Thus the deployment gets stuck in a crash loop. So ARM still won't work.

These images:
https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/963eccfc404f56540f025a93fd4966fa6a73458f/deploy/kubernetes/overlays/stable/kustomization.yaml

As well as snapshotted and resize. The ones from qauy.io and not multi arch.

What you expected to happen?

A successful deployment of the EBS CSI Driver on ARM based servers like m6g.

How to reproduce it (as minimally and precisely as possible)?

Try to deploy the EBS CSI on ARM based systems.

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.8-eks-7c9bda", GitCommit:"7c9bda52c425d0d56d7b93f1377a826b4132c05c", GitTreeState:"clean", BuildDate:"2020-08-28T23:04:33Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Driver version:
  v0.7.0

[gkrizek]

Some Kubernetes output:

kube-system                             ebs-csi-controller-6ccc47b87c-9wgdz                      1/6     CrashLoopBackOff   29         3m36s
kube-system                             ebs-csi-node-sz75j                                       1/3     CrashLoopBackOff   18         8m8s

$ kubectl logs ebs-csi-controller-6ccc47b87c-9wgdz -n kube-system -c liveness-probe
standard_init_linux.go:211: exec user process caused "exec format error"

[Chili-Man]

this would be great as we want to run this on the gravitron ec2 instances

[gkrizek]

Totally. I rebuilt all the necessary images myself with ARM support to get them to work with Graviton2. But they're in my private ECR so I don't have a link to share

[StevenLou]

Any update for this issue?

[Chili-Man]

looks like with the recent commits, they are now publish ARM container images on DockerHub: https://hub.docker.com/r/amazon/aws-ebs-csi-driver/tags?page=1&ordering=last_updated

[gkrizek]

The EBS CSI images are, but that's not the issue here. It's the referenced Kubernetes Docker images that get deployed with this CSI that are not.

[Chili-Man]

Ahh, that's right; all of the images sourced from Quay.io do not have ARM64 images built. However, it does appear that newer versions of the CSI images do have ARM64 images; it appears that AWS should be upgrading their CSI dependencies to something more recent.

[ayberk]

We have a backlog item on this, but please thumbs up this issue if this is affecting you. We take those into account while prioritizing :)

[ayberk]

We've merged a new kustomization overlay with multiarch images: #653. Unfortunately some of the images are pulled from docker hub, but it should unblock you for now.

My initial tests were successful, please let me know if you encounter any issues.

[shrivastavshubham34]

I'm sorry, but I'm not sure if I'm supposed to pull in v0.8 or v0.7 of k8s.gcr.io/provider-aws/aws-ebs-csi-driver?
Does v0.7.0 work with Multiarch?

[ayberk]

@shrivastavshubham34 We're currently having issues with our v0.8, so I'd suggest using the arm overlay with v0.7 image for now.

[shrivastavshubham34]

    image_name: 'k8s.gcr.io/provider-aws/aws-ebs-csi-driver'
    image_tag: 'v0.7.0'

@ayberk It's still giving me

standard_init_linux.go:219: exec user process caused: exec format error

[ayberk]

@shrivastavshubham34 Yeah I was able to isolate the issue this morning and it seems like for some reason the images are not being pushed as multiarch to gcr.

For example I was able to run amazon/aws-ebs-csi-driver:v0.8.0-amazonlinux and amazon/aws-ebs-csi-driver:v0.7.1 on my graviton instance. You can pull them from dockerhub. Alternatively you can use 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-ebs-csi-driver:v0.7.1 if you prefer ECR.

I'll try to push v0.8.0 image to ECR today.

[shrivastavshubham34]

@ayberk doesn't work for ECR and amazon/aws-ebs-csi-driver:v0.7.1 for me.

helm upgrade --install aws-ebs-csi-driver \
    --namespace kube-system \
    --set enableVolumeScheduling=true \
    --set enableVolumeResizing=true \
    --set enableVolumeSnapshot=true \
    --set image.repository='amazon/aws-ebs-csi-driver' \
    --set image.tag='v0.7.1' \
    aws-ebs-csi-driver/aws-ebs-csi-driver