Minikube assumes remote docker host is same arch as local machine

[snickell]

What Happened?

I'm using a MacOS ARM64 ("apple silicon") laptop as my dev machine, but I do builds on a remote Docker Linux AMD64 host using DOCKER_HOST="tcp://192.168.1.44". If I create the minikube-in-docker while sshed into the Linux machine, minikube installs an AMD64 docker build. However, if I try to minikube start from my Mac initially, while it pulls an x86_64 docker image, the kubeadm and kubectl binaries in /var/lib/minikube/binaries/v1.22.3/kubeadm are NOT x86_64 but arm64, which won't run on my AMD64 docker host (it gets wrongly sniffed as a shell script and fails to run, but see below its an ELF just wrong architecture).

root@minikube:/var/lib/minikube/binaries/v1.22.3# file kubeadm
kubeadm: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=Ix-qGg0m4Y1_wbf6TIrB/IthoOfPXT3QgQ194kYq7/518O26hpL0jI16q3MMZ0/cj371ZbZEp94cJXf24Pn, stripped

Attach the log file

log.txt

Operating System

macOS (Default)

Driver

Docker

[afbjorklund]

Right, this is a "bug": (Using the docker driver has the same issues as the ssh driver)

• Add architecture to the binaries and images in the cache #9593

minikube doesn't have very good support for remote systems, and when it does it assumes that they are the same arch

you can still use kubernetes like that, but like multi-node before you might have to do some of the setup yourself (for now)

[klaases]

Hi @snickell, were you able to review @afbjorklund comments above, and did you have any additional comments?

It looks like this issue might have been identified per #9593.

[snickell]

Hi @klaases I don't know enough of the system internals to comment on @afbjorklund 's comment, but it seems very plausible to me? Fixing #9593 might not be enough to fix this, but it seems like a critical step at a min

[sharifelgamal]

#9593 should be fixed when minikube 1.26 is released, so we can revisit this when that happens.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[sharifelgamal]

This should now be doable, help wanted!

[mxey]

I was able to workaround this by using the x86_64 Minikube macOS binary and setting --preload=false.