minikube with docker driver run as root not allowed but requires root anyway

[FlorianLudwig]

Steps to reproduce the issue:

1. Run:

   $ sudo minikube start --driver=docker` 
   [...]
   The "docker" driver should not be used with root privileges.
   

2. The solution is to create a user and grant it access to the docker socket. Granting access to the docker socket grants privileges equivalent to the root user.

Users will end up reading comments like here: #7903 (comment) and setup access to the docker socket without realizing they are granting root access to those users. By doing so minikube encourages less secure systems in practise.

[afbjorklund]

Hi @FlorianLudwig !

Hmm, the current situation is a bit confusing and the above seems to be as well:

Your step 1) is about "none" driver and step 2) is about "docker" driver ?

Maybe this happened because of some misguided suggestions on minikube's part:

❗ 'none' driver reported an issue: the 'none' driver must be run as the root user
💡 Suggestion: For non-root usage, try the newer 'docker' driver

Currently running minikube requires root, this is (or should be) documented: #7963
(The situation on macOS and Windows is a bit different, I'm going to ignore that here.)

And yes this also has security implications - which should be clearly documented - like so:

https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user

If you don’t want to preface the docker command with sudo, create a Unix group called docker and add users to it. When the Docker daemon starts, it creates a Unix socket accessible by members of the docker group.

Warning

The docker group grants privileges equivalent to the root user. For details on how this impacts security in your system, see Docker Daemon Attack Surface.

The podman driver is a bit more straight-forward, in that it still uses sudo rather than a root group.

---

Using a VM also requires admin privileges to set up*, but then root is inside the VM.
So this might be a better option, rather than running privileged containers (as root) ?

* the libvirt group is also root-equivalent

The current check against not running as root (with sudo), is mostly for the config files.
The files under the user account ended up being owned by root, or even under /root...

Only the "none" driver currently runs as root, since it is not intended for users but for CI
https://minikube.sigs.k8s.io/docs/drivers/none/

The "docker" driver is somewhere half-way, so it runs minikube as the user but docker as root.
Running all of Kubernetes as the user is an interesting project, but currently outside the scope.

You can check out the "Usernetes" project, if that is what you are after here ?

Ultimately we probably want to run also the none driver as non-root: #3760

Currently there are bunch of hacks and workarounds to move the config files afterwards.
And most of them don't work, due to hardcoded /root paths in for instance the certificates.

So it is more straight-forward to do everything as the user, and then call "sudo" as needed.
Even CI systems should not run as root by default, but rather use some functional user.

https://help.ubuntu.com/community/RootSudo

By default, the root account password is locked in Ubuntu

[afbjorklund]

But we should try to avoid misleading statements like these:

6276c1c

return registry.State{Error: fmt.Errorf("the 'none' driver must be run as the root user"), Healthy: false, Fix: "For non-root usage, try the newer 'docker' driver", Installed: true}

#3760 (comment)

I encourage that users of --driver=none consider using the newer --driver=docker, which does not require root access.

They seem to imply that running docker does not require root ?

---

I'm also not sure that the "docker" driver was ever intended as a substitute for "none" driver ?
It was more available as a new alternative on Linux, to run KIND (or KIC) rather than a VM.

Then it spread to other platforms, in case you were already running a Docker Desktop VM...
It requires less resources to run one VM than to run two VMs, but you could also choose ours.

[FlorianLudwig]

Hi @afbjorklund Thank you very much for your long explanation!

First of all, I indeed messed up my bug report - I meant the docker driver, not none. I fixed the bug report.

Some background: I am looking for a development environment without compromising (too much) on security nor usability - always a hard thing to do.

Options:

• Using the none driver (setup as root) together with a restricted kubernetes account (usable from non-root account)
• Using the docker driver (setup as root) but used from non-root account
• Using the kvm2 driver (setup as root) but used from non-root account
• Using the podman driver - I had some issues with this one but that's another story I guess

Using a VM also requires admin privileges to set up*, but then root is inside the VM.
So this might be a better option, rather than running privileged containers (as root) ?

• the libvirt group is also root-equivalent

Actually, this is what I ended up using. Without the libvirt group though, setup asked me for my password like 100 times but using kubectl now works fine.

---

But back to my original question: Why isn't it allowed to use the docker driver as root?

[FlorianLudwig]

But we should try to avoid misleading statements like these:

6276c1c

return registry.State{Error: fmt.Errorf("the 'none' driver must be run as the root user"), Healthy: false, Fix: "For non-root usage, try the newer 'docker' driver", Installed: true}

#3760 (comment)

I encourage that users of --driver=none consider using the newer --driver=docker, which does not require root access.

Exactly! This is what I mean with misleading :)

---

I'm also not sure that the "docker" driver was ever intended as a substitute for "none" driver ?
It was more available as a new alternative on Linux, to run KIND (or KIC) rather than a VM.

I didn't know about KIND, might be interesting to look at, thanks!

Then it spread to other platforms, in case you were already running a Docker Desktop VM...
It requires less resources to run one VM than to run two VMs, but you could also choose ours.

Actually, I don't run a VM normally. I try to stick to podman as much as I can as it plays a lot more nice with permissions etc.

[afbjorklund]

But back to my original question: Why isn't it allowed to use the docker driver as root?

There is no particular technical reason why the root user wouldn't be allowed to run minikube.
Theoretically you could log in as the root user, and have files end up under /root and be fine.

But since it is regarded as a bad practice (logging in as root), we try to "encourage" avoiding it.

The practical issue was that people run sudo minikube and ended up with unreadable config.

i.e. you would have files created by root, under ~/.minikube

So the next command would be unable to read e.g. config

Ending up with the ugly situation of having to sudo everything, even dashboard and kubectl
We wanted the user to be able to work from their usual account, instead of doing "ssh root@"

---

I didn't know about KIND, might be interesting to look at, thanks!

KIC (the "docker" driver) is our adapation of KIND, it uses the same docker image etc.

The main difference is that KIC is for developers, while KIND is for testing kubernetes.

Both are from Kubernetetes SIGs...

And both use kubeadm for setup.

Then it spread to other platforms, in case you were already running a Docker Desktop VM...
It requires less resources to run one VM than to run two VMs, but you could also choose ours.

Actually, I don't run a VM normally. I try to stick to podman as much as I can as it plays a lot more nice with permissions etc.

But podman is only available for linux, so there are no "other platforms" to talk about

Same problem though, so we expose podman-env to not have to use podman-machine
If we didn't, the user would end up with one VM for docker/podman and one VM for k8s ?

Once there is a Podman Desktop product, that could be considered as an option as well.

[FlorianLudwig]

Ending up with the ugly situation of having to sudo everything, even dashboard and kubectl
We wanted the user to be able to work from their usual account, instead of doing "ssh root@"

Isn't that still better than adding the user to the docker or kvm groups?

So maybe:

1. allow using sudo minikube and warn about that the config will be stored in root's home - maybe with a flag --yes-i-know-what-i-am-doing? (quickfix)
2. Rethink the security best practices recommended to the user in more details. (long term discussion)

[afbjorklund]

I think the sudo setup varies a little between the distributions, as well. Some use the same $HOME for both the user and the sudo user (root), while others change to put the sudo files under /root instead.

Others (looking at you, CentOS) even have different $PATH for sudo and root so it won't find things. Basically it is a hassle to support it, and most users seem fine without running minikube as root ?

Security could be better documented...

It's not really one of the strong points.

[FlorianLudwig]

What is the reasoning not allowing to run minikube as root? At least behind a flag --yes-i-know-what-i-am-doing? Setups are quite different and as you pointed out there is no technical reason to not allow it.

[jaybirdshain]

This is disappointing. Finally, I just did sudo chmod 0777 /var/run/docker.sock. And then minikube start and it started working.