Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images

[csabatuz-chess]

What Happened?

I was planning using Minikube with docker driver, on an Apple M1 chip.

My Docker Engine, when freshly started supports the following emulators:

% docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta​",
    "rosetta​​"
  ]
}
% minikube start
😄  minikube v1.32.0 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver. Other choices: qemu2, virtualbox, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
% docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64"
  ],
  "emulators": [
    "python3.10"
  ]
}

Minikube messes up the binfmt config, so from that point on, it's only possible to run images of the same architecture as minikube itself (in my case linux/arm64).

This invasive behaviour is a showstopper for me to use Minikube, despite it being lovely and I'd really like to switch my K3S based stack to Minikube.

My attempts to find a workaround, or try to restore the Binfmt settings manually failed, so reaching out with an issue.

Attach the log file

log.txt

Operating System

macOS (Default)

Driver

Docker

[csabatuz-chess]

Correction
My issue might be misplaced here

The problem is happening with OrbStack, but cannot reproduce with Docker Desktop. So the issue might be specific to Minikube on Orbstack with docker driver.

[phbelitz]

If I understand this correctly, then this also messes up the WSL interoperability -> anyone how uses WSL can normally run windows executables inside their WSL (e.g. vscode), but once you try to start minikube, this is no longer possible. Then this fix needs to be applied manually.

[afbjorklund]

I don't think the minikube container drivers do anything with binfmt_misc, do they? Should only be the VM that sets it up, and then not for the host...

[cotej]

Running into a similar issue here. My setup is on a Mac M2 using lima + vz + rosetta for the Docker VM.

With minikube 1.32.0 (with docker driver), once I start minikube it somehow breaks rosetta in the lima VM.
Same configuration works fine on minikube 1.31.2.

[afbjorklund]

Seems like it is our friend systemd, that is at it again:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1999275

So likely this started to happen after upgrading Ubuntu...

Python 3.10 now has a /usr/lib/binfmt.d/python3.10.conf

---

-ARG UBUNTU_JAMMY_IMAGE="ubuntu:jammy-20230624"
+ARG UBUNTU_JAMMY_IMAGE="ubuntu:jammy-20231004"

python3.10 (3.10.8-2) "Please add support for systemd-binfmt"

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1012377

RUN rm -f /usr/lib/binfmt.d/python3.10.conf

[afbjorklund]

cc @spowelljr @medyagh

[fbuetler]

I can confirm that the same configuration works with minikube v1.31.2 and fails with v1.32.0 (we are also using orbstack):

v1.31.2:

~
❯ minikube version
minikube version: v1.31.2
commit: fd7ecd9c4599bef9f04c0986c4a0187f98a4396e

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta​",
    "rosetta​​"
  ]
}

~
❮ minikube start
😄  minikube v1.31.2 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=6, Memory=24576MB) ...
🐳  Preparing Kubernetes v1.27.4 on Docker 24.0.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta​",
    "rosetta​​"
  ]
}

v1.32.0

~
❯ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/386",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-x86_64",
    "rosetta",
    "rosetta​",
    "rosetta​​"
  ]
}

~
❯ minikube start
😄  minikube v1.32.0 on Darwin 14.1.1 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=6, Memory=24576MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

~
❯ docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64"
  ],
  "emulators": [
    "python3.10"
  ]
}

[afbjorklund]

Unfortunately the base image version is not shown by default.

🚜 Pulling base image ...
🚜 Pulling base image ...

But the troublesome file has been added in the later version of it.

 const (
        // Version is the current version of kic
-       Version = "v0.0.40"
+       Version = "v0.0.42"

[spowelljr]

Curious, is it fixed in #17719 @fbuetler?

Without a patch release it seems that that current options are to use v1.31.2 or you should be able to specify to use v1.31.2's version of the base image during start with v1.32.0, I don't think there were any changes that would make the two incompatible.

minikube start --base-image gcr.io/k8s-minikube/kicbase:v0.0.40