Minikube with driver "new Virtualization Framework" on Apple M1

[cscetbon]

As more and more people are thinking of moving to Minikube to replace "Docker For Mac" some work to enable virtualization on it should be a priority to get them onboard. Hyperkit being deprecated going with the new Virtualization Framework which uses higher API and is supposed to be easier to use probably makes more sense

See https://github.com/Code-Hex/vz

[cscetbon]

Did someone start any work on this somewhere that could be used as a base or needs to be completed ?

[afbjorklund]

Currently containerd (lima) uses QEMU and podman (machine) uses QEMU, but there is room for more hypervisor drivers.

• https://medium.com/nttlabs/containerd-and-lima-39e0b64d2a59

• https://www.redhat.com/sysadmin/podman-mac-machine-architecture

The main issue is actually not the CPU virtualization, at least not anymore. The problem is the network, especially VPN...

[cscetbon]

@afbjorklund but as Minikube already supports hyperkit I thought it should be easier to start from there and migrate it to the new framework which is supposed to be simple and support everything that is already supported.

[afbjorklund]

Yes, as far as I know both HyperKit and VpnKit are still Open Source even if Docker's new libraries might not be.

• https://github.com/moby/hyperkit

• https://github.com/moby/vpnkit

But it doesn't work on aarch64 (yet?)

[cscetbon]

migrate it to the new framework

What "new framework" are you talking about here ?

https://developer.apple.com/documentation/virtualization

The new one that deprecates hyperkit and there is a go binding for it at https://github.com/Code-Hex/vz

[afbjorklund]

Okay thanks, we will need someone to turn that into a "driver" for the libmachine framework:

https://github.com/docker/machine/blob/master/libmachine/drivers/drivers.go

[cscetbon]

@afbjorklund You mean writing something like https://github.com/kubernetes/minikube/tree/master/pkg/drivers/hyperkit, right ? Also do you know why the hyperkit doesn't work on M1 atm ?

[afbjorklund]

@afbjorklund You mean writing something like https://github.com/kubernetes/minikube/tree/master/pkg/drivers/hyperkit, right ?

Right (you will notice it imports the libmachine framework)

There is also an old driver for QEMU.

Also do you know why the hyperkit doesn't work on M1 atm ?

Same reason as VirtualBox, I suppose. No arm64 support.

See moby/hyperkit#303 (comment)

[cscetbon]

As you need help, @yoelb and I can tackle it

[riaz7se]

Any reference or links to follow for installing minikube on M1 using Apple Virtualization framework?
pls suggest

[sharifelgamal]

This is still something we're looking into. One prerequisite for this to work is a minikube ISO that will work on the arm64/aarch64 architecture, which is still in progress.

[xaocon]

This is still something we're looking into. One prerequisite for this to work is a minikube ISO that will work on the arm64/aarch64 architecture, which is still in progress.

Is that conversation going on somewhere public? I'd like to help. Not sure I could help much with the GO stuff but I could probably help with the ISO or helping track down how other projects are doing similar things (looking through lima for example).

[sharifelgamal]

The tracking issue for the arm64 ISO is #9228

[spowelljr]

We just released minikube v1.26.0-beta.0 that supports the QEMU driver --driver=qemu2.

https://github.com/kubernetes/minikube/releases/tag/v1.26.0-beta.0

[afbjorklund]

It would still be interesting to see a dedicated Mac driver (like the old HyperKit driver) using virtualization.framework

But the QEMU driver is also getting some real networking support, using the vmnet.framework (courtesy of lima-vm).

[afbjorklund]

• Optional support for Virtualization.framework (on macOS 13) lima-vm/lima#889

[tmoschou]

Looks like Podman v5.0.0-rc7 have switched their driver on MacOS over from QEMU to the native MacOS virtualization framework / hypervisor thanks to https://github.com/crc-org/vfkit and https://github.com/Code-Hex/vz

As a side, it would be good to know if the new framework potentially fixes the networking issue / firewall bugs we have with socket_vmnet - we are running MacOS's enrolled in MDM, which prohibit firewall modifications and our IT are reluctant to deploy a custom Firewall MDM payload. We basically have to stop/uninstall/reboot/reinstall socket_vmnet as a workaround, which is a shame as its limiting uptake of minikube/kubernetes.

[afbjorklund]

Now you can use gvproxy for the network, since it has integration with the vfkit program.

https://github.com/containers/gvisor-tap-vsock#run-with-vfkit

[afbjorklund]

Apparently using gvproxy is optional, since the built-in "nat" actually gives you an IP with dhcp.

So that can at least be used for ssh, by parsing the dhcpd_leases