Add multi-node support for QEMU

[AndriyDmytrenko]

What Happened?

Tried to start the brand new minikube cluster
$ minikube start -n 3 --driver qemu2
...
stderr:
W0708 07:41:05.949053 1157 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/var/run/cri-dockerd.sock". Please update your configuration!
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: couldn't validate the identity of the API Server: Get "https://control-plane.minikube.internal:8443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.0.2.15:8443: connect: connection refused

Attach the log file

logs.txt

Operating System

macOS (Default)

Driver

N/A

[spowelljr]

That's actually just a warning, the true error is:
error execution phase preflight: couldn't validate the identity of the API Server: Get "https://control-plane.minikube.internal:8443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": dial tcp 10.0.2.15:8443: connect: connection refused

The QEMU driver is quite new and has many currently unsupported features, and it seems based on your experience that multi-node is one of those features. I added this to our QEMU know issues list #14146 and we will add support for it when we have the time and resources to do so.

[afbjorklund]

It needs a different network, such as vde (or tap)

https://github.com/lima-vm/vde_vmnet

The "user" network only supports a single host, and very limited at that (no public ip address, needs tunneling)

https://wiki.qemu.org/Documentation/Networking

[AndriyDmytrenko]

I believe it's more than the qemu2 driver only issue, I tried hyperkit with the very similar result.
see the log from the command attached. The cgroup driver option has been added by suggestion of previously failed start command, but without any luck - just to note here.
minikube start --driver hyperkit -n 3 --extra-config=kubelet.cgroup-driver=systemd

logs.txt

Thanks.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mprimeaux]

Interestingly enough, multi-node seems to work on my M1 with 3 nodes and socket_vmnet.

❯ minikube version
minikube version: v1.30.1
commit: 08896fd

❯ minikube config view 
- memory: 16384
- container-runtime: containerd
- cpus: 6
- driver: qemu2

❯ minikube start -n 3
😄  minikube v1.30.1 on Darwin 13.4 (arm64)
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=6, Memory=16384MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.26.3 on Docker 20.10.23 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🔎  Verifying Kubernetes components...

👍  Starting worker node minikube-m02 in cluster minikube
🔥  Creating qemu2 VM (CPUs=6, Memory=16384MB, Disk=20000MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.105.24
🐳  Preparing Kubernetes v1.26.3 on Docker 20.10.23 ...
    ▪ env NO_PROXY=192.168.105.24
🔎  Verifying Kubernetes components...

👍  Starting worker node minikube-m03 in cluster minikube
🔥  Creating qemu2 VM (CPUs=6, Memory=16384MB, Disk=20000MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.105.24,192.168.105.25
🐳  Preparing Kubernetes v1.26.3 on Docker 20.10.23 ...
    ▪ env NO_PROXY=192.168.105.24
    ▪ env NO_PROXY=192.168.105.24,192.168.105.25
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

❯ kubectl get nodes
NAME           STATUS   ROLES           AGE     VERSION
minikube       Ready    control-plane   2m31s   v1.26.3
minikube-m02   Ready    <none>          118s    v1.26.3
minikube-m03   Ready    <none>          89s     v1.26.3

Well, at least the cluster starts up. I'll play around with this a bit more.

[caerulescens]

I would like to confirm that multi-node qemu driver works for macos 13 with socket_vmnet; it does not work on Debian 12 with the builtin network:

• fix: multi-node minikube does not start for qemu2 driver with builtin network #18067