New QEMU2 driver, minikube fork of the QEMU machine driver

[afbjorklund]

There was a docker-machine driver called "qemu", which was a portable and non-root version of the "kvm" (libvirt) driver:

https://github.com/machine-drivers/docker-machine-driver-qemu

Previously suggested as a driver for minikube, but rejected because of the limited network capabilities (i.e. no IP address)

• [FEATURE] Adding support for additional driver (QEMU/KVM) #2399

---

Since this driver supports all architectures, with hardware acceleration where available, and since it is Open Source - it's now on again.

It is somewhat similar to the "lima" (https://github.com/lima-vm/lima) instances, which also uses QEMU for running the virtual machines.

Note that there are two different versions of QEMU: "qemu-system" (like VirtualBox) and "qemu-user" (more similar to Rosetta)

Here we are discussing the hypervisor running the VM, but the other program is useful inside of it for running other arch images.

See #13639 for details.

It uses SSH* for tunneling.

* eventually it will be possible to add alternative networks such as Tap and VDE, if having enough privileges (root) on the host

But the initial setup just uses the built-in "user" network to provide SSH access: https://wiki.qemu.org/Documentation/Networking

[afbjorklund]

$ ./out/minikube start --driver=qemu2
😄  minikube v1.25.2 on Ubuntu 20.04
✨  Using the qemu2 (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=2, Memory=3900MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.23.5 on Docker 20.10.14 ...
❌  Unable to load cached images: loading cached images: stat /home/anders/.minikube/cache/images/amd64/docker.io/kubernetesui/dashboard_v2.5.1: no such file or directory
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Both SSH and Docker are forwarded from host, using the libmachine setup.

$ ./out/minikube docker-env
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://127.0.0.1:41041"
export DOCKER_CERT_PATH="/home/anders/.minikube/certs"
export MINIKUBE_ACTIVE_DOCKERD="minikube"

# To point your shell to minikube's docker-daemon, run:
# eval $(minikube -p minikube docker-env)

But the Kubernetes API Server (special minikube 8443 port) needs forwarding:

$ ./out/minikube kubectl cluster-info
Kubernetes control plane is running at https://localhost:39341
CoreDNS is running at https://localhost:39341/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

---

It was possible to work around the hard-coded port (2376) for Docker, using URL.

But the libmachine API does not allow for any other ports, like the docker API does.
This is a problem, if wanting to publish not only apiserver but also the other KIC ports.

        // control plane specific options
        params.PortMappings = append(params.PortMappings,
                oci.PortMapping{
                        ListenAddress: listAddr,
                        ContainerPort: int32(params.APIServerPort),
                },
                oci.PortMapping{
                        ListenAddress: listAddr,
                        ContainerPort: constants.SSHPort,
                },
                oci.PortMapping{
                        ListenAddress: listAddr,
                        ContainerPort: constants.DockerDaemonPort,
                },
                oci.PortMapping{
                        ListenAddress: listAddr,
                        ContainerPort: constants.RegistryAddonPort,
                },
                oci.PortMapping{
                        ListenAddress: listAddr,
                        ContainerPort: constants.AutoPauseProxyPort,
                },
        )

        // DockerDaemonPort is the port Docker daemon listening inside a minikube node (vm or container).
        DockerDaemonPort = 2376
        // APIServerPort is the default API server port
        APIServerPort = 8443
        // AutoPauseProxyPort is the port to be used as a reverse proxy for apiserver port
        AutoPauseProxyPort = 32443
        // SSHPort is the SSH serviceport on the node vm and container
        SSHPort = 22
        // RegistryAddonPort os the default registry addon port
        RegistryAddonPort = 5000

Currently these port mappings are stored outside of minikube, in the engine itself.

                if driver.NeedsPortForward(driverName) {
                        port, err = oci.ForwardedPort(driverName, cname, port)

This means that we need to add the host ports, not only for APIServerPort but also
for RegistryAddonPort and AutoPauseProxyPort, so we know where to dial those...

[afbjorklund]

Now that the driver has been forked from libmachine, it should be possible to add at least APIServerPort.

                startCmd = append(startCmd,
                        "-nic", fmt.Sprintf("user,model=virtio,hostfwd=tcp::%d-:22,hostfwd=tcp::%d-:2376,hostname=%s", d.SSHPort, d.EnginePort, d.GetMachineName()),
                )

Then we only need to use SSH for the service tunnels, and let the driver handles the ones that are published.

Similar to docker --publish: 127.0.0.1:49167->22/tcp, 127.0.0.1:49166->2376/tcp, 127.0.0.1:49165->5000/tcp, 127.0.0.1:49164->8443/tcp, 127.0.0.1:49163->32443/tcp but using the qemu user network hostfwd instead.