Allow using Podman driver on Mac and Windows

[mateka]

Steps to reproduce the issue:

1. minikube start --driver=podman on Windows/Mac
2. minikube should start without errors

[afbjorklund]

Podman, just like Docker Engine, is a Linux program. So it needs a different offering, similar to Docker Desktop (i.e.with a VM)

Currently that is not ready, the offering from Red Hat is an "OpenShift Desktop" that runs OpenShift instead (it is called "CRC")

You can use "lima" perhaps:

• Issue Use lima as a driver #12508

The original report is here:

• Issue Podman Desktop driver on Mac or Win (and Linux) #8003

But using a VM driver (minikube start --vm) is currently recommended, over using one of those other alternatives for the VM.

In the future it might be possible to use the integrated CoreOS version, but it might be somewhat "bigger" than the Minikube OS.

• 230M minikube-v1.23.0.iso

• 1.5G fedora-coreos-34.20210904.1.0-qemu.x86_64.qcow2

[mateka]

I have created this issue because a few days ago Podman released a version, which works on macOS (they say, that it works on Windows too): https://podman.io/blogs/2021/09/06/podman-on-macs.html. From the user perspective, under the hood, it works similar to Docker Desktop: it creates a tiny virtual machine with qemu and runs in it. Even though it should not work on M1 Mac, I have run docker container with it with M1 Mac! Unfortunately, podman machine * commands are not working. The Mac M1 support should work soon (there is a working fork of it in brew: https://github.com/simnalamburt/homebrew-x/blob/main/Formula/podman-apple-silicon.rb and its author is working on PRs!).

If minikube uses Podman as an external command on Linux, then maybe it is possible to unblock Podman on other OSes?

[afbjorklund]

There are multiple ways to run podman-remote from windows and macos, but none that are ready for minikube just yet...

I'm not so sure about "tiny" though, this new machine is about 10x bigger than the old machine and 2x the size of Fedora

• old: https://podman.io/blogs/2019/01/14/podman-machine-and-boot2podman.html

• new: https://podman.io/blogs/2021/09/06/podman-on-macs.html

[afbjorklund]

If minikube uses Podman as an external command on Linux, then maybe it is possible to unblock Podman on other OSes?

Yeah, that is what that other story was about... We might do something similar for Docker, to use an Open Source version of it

[mateka]

There are multiple ways to run podman-remote from windows and macos, but none that are ready for minikube just yet...

What a pity!

I'm not so sure about "tiny" though, this new machine is about 10x bigger than the old machine and 2x the size of Fedora

Probably there will be a price to pay if You want to have an automated solution for Mac/Windows. Then users can decide if they want to manage their own VM or to use bundled on.

If Podman is not ready yet, I guess this issue should be closed?

[afbjorklund]

If Podman is not ready yet, I guess this issue should be closed?

We can leave it open, there are some new additions that makes it a bit different from the old one (#8003) which was more DIY

Right now there is no storage (shared with the host) in the new Podman Machine, so that feature probably needs to fixed first

But the ports publishing is (almost) fixed, or will be in the next release. And support for arm64 on Mac is coming, as you noted.

Meanwhile, there are lots of issues with both the container runtime (CRI-O) and the container engine (podman2) to be fixed...

[afbjorklund]

@mateka : it would also be useful to know, why you think that "podman" would be a better driver than "docker" or "hyperkit" ?

It is possible that the best solution would be a QEMU driver for minikube, talking directly to the Virtualization.framework (hvf)

The main reason why people like these "docker-in-docker" drivers, is because it goes faster* to delete and start new clusters.

When it comes to the casual user* with just one single-node cluster, it adds complexity and startup time compared to a VM.

* assuming that the virtual machine is running already...
(i.e. "pay no attention to that man behind the curtain!")

* on Mac or Win, that is. on Linux, it's another story...
(since it can run containers natively, without any VM)

[mateka]

I was a user of Docker Desktop. It was nice to not have to think about VM and have everything ready to use after booting my machine. After licensing changes, I have decided to look for an alternative to Docker Desktop (I am using my private MacBook when working remotely for my employer. On it I had Docker Desktop installed for private learning, but I do not want to think if it is legal or not if I am not paying for Docker Desktop).

If I understand correctly, to use docker I would have to create a separate VM and manage it. Maybe, when QEMU in brew would be working on M1, then it would be fine (but learning all correct parameters would be a task!). Now, for VMs I am using UTM, but having two (unneeded!) windows floating around to run a local Kubernetes cluster would be anoing!

Because I am using M1 Mac, "hyperkit" is not usable by me: moby/hyperkit#303 (comment).

So, I have thought: "If the preferred driver for minikube is Docker, then maybe Podman will be the best alternative". Moreover, maybe all settings (sharing directories and network configuration) will be working out of the box.

[afbjorklund]

Currently, there is only support for Docker Machine which makes it more similar to the previous generation (Docker Toolbox).

There are still some issues with M1, even if hvf support just got merged for QEMU. But Docker has deprecated* their HyperKit...
So it would need replacing anyway, and UTM/QEMU is the most likely - even if more work is needed on networking and VPN.

* https://www.docker.com/blog/the-magic-behind-the-scenes-of-docker-desktop/

On Mac (on Intel and M1 machines) we are currently transitioning away from our previous HyperKit implementation to use Apple’s new Virtualization framework to run this VM.

Paying the $7/month is probably the easiest short term, but there are some nice Open Source alternatives - like for next year

Note: the new Podman Machine is also based on QEMU, and this hvf acceleration for it also uses Virtualization.framework
We don't know yet if minikube will keep using a custom OS distribution, or transition over to one of the standard distributions.

• Issue MEP: Change from custom Buildroot ISO to standard Kubernetes OS #9992

[afbjorklund]

I added a PR to "allow" it, and some instructions on how to start the VM....

Apparently is currently broken*, but hopefully fixed in again Podman 3.4.0 ?

• Issue Error when trying to run minikube with podman machine #12581

All containers with networks are broken with this FCOS, not just minikube.