none driver: support non systemd ways to restart docker

[medyagh]

currently if u run none driver on a system that does have docker but it is not systemd
it wont work !

because we expect docker restart to use systemctl (and init.d scripts wont work)

// Restart restarts Docker on a host
func (r *Docker) Restart() error {
	c := exec.Command("sudo", "systemctl", "restart", "docker")
	if _, err := r.Runner.RunCmd(c); err != nil {
		return errors.Wrap(err, "restarting docker.")
	}
	return nil
}

https://github.com/kubernetes/minikube/blob/master/pkg/minikube/cruntime/docker.go#L101

we could try do restart docker service using "service restart docker" if there is no systemctl service file for docker.

[afbjorklund]

we could try do restart docker service using "service restart docker" if there is no systemctl service file for docker.

sound good, if not running systemd then service should be available (or easy enough to add).

This issue applies to all the runtimes, not only docker. It is also used for the kubelet, for example.

So the functionality (start/stop/restart/is-active) should probably move into some library functions ?

[afbjorklund]

Apparently last one needs LSB status_of_proc https://wiki.debian.org/LSBInitScripts/StatusSupport

[afbjorklund]

@medyagh : do we have any supported environments remaining, that don't use systemd ?

it used to be mostly for the older versions of ubuntu, used when travis used containers not vms

[sayboras]

I tried to run integration test with ubuntu docker (as I am developing in Mac). Then came across this issue as https://hub.docker.com/_/ubuntu 18.04 is running with service only.

Understand that running docker in docker is not ideal, but having VM is just too much for my 13' mac.

[afbjorklund]

@sayboras : we have the "docker" driver for that, running the VM tests in docker won't be enough

this basically means running only the unit tests locally, and rely on the CI infrastructure for the rest

[afbjorklund]

@medyagh :

currently if u run none driver on a system that does have docker but it is not systemd

Do you have any examples of such an environment ? Both ubuntu and centos now have it

Just wondering it about the "soon" priority, as we already had a backlog item #2704 for this...

[afbjorklund]

Being implemented in #7081

[paddy-hack]

Do you have any examples of such an environment ? Both ubuntu and centos now have [systemd]

I do! Running Devuan with the docker-ce packages for the corresponding Debian release. I actually bent over backwards to resurrect sysv-style init support for that, see docker/for-linux#482.

Please do not assume systemd availability. Do also not assume service is available.

Please do provide proper fall-backs when things are different from the "My way or the highway" approach that some of the bigger distros try to enforce upon non-confirmists. Many thanks in advance. 🙇

[afbjorklund]

Can you supply an implementation for whatever sysinit that you are using instead ? Right now it has systemd and openrc

[paddy-hack]

I use openrc. If that means things are supposed to work, I'd like to point out the following

paddy-hack@boson:~$ minikube start
😄  minikube v1.11.0 on Debian 10.0
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🤷  docker "minikube" container is missing, will recreate.
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
🐳  Preparing Kubernetes v1.18.3 on Docker 19.03.2 ...
    ▪ kubeadm.pod-network-cidr=10.244.0.0/16
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"
💡  For best results, install kubectl: https://kubernetes.io/docs/tasks/tools/install-kubectl/
paddy-hack@boson:~$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

paddy-hack@boson:~$ minikube stop
✋  Stopping "minikube" in docker ...
🛑  Powering off "minikube" via SSH ...
🛑  Node "minikube" stopped.
paddy-hack@boson:~$ minikube start
😄  minikube v1.11.0 on Debian 10.0
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing docker container for "minikube" ...
🤦  StartHost failed, but will try again: driver start: start: docker start minikube: exit status 1
stdout:

stderr:
Error response from daemon: cgroups: cannot find cgroup mount destination: unknown
Error: failed to start containers: minikube

🔄  Restarting existing docker container for "minikube" ...
😿  Failed to start docker container. "minikube start" may fix it: driver start: start: docker start minikube: exit status 1
stdout:

stderr:
Error response from daemon: OCI runtime create failed: container with id exists: 53ac2f88bff8b8ea2db5cd4e9a3133ea9637cc8bd2e59c550008fba242ed74a7: unknown
Error: failed to start containers: minikube


💣  error provisioning host: Failed to start host: driver start: start: docker start minikube: exit status 1
stdout:

stderr:
Error response from daemon: OCI runtime create failed: container with id exists: 53ac2f88bff8b8ea2db5cd4e9a3133ea9637cc8bd2e59c550008fba242ed74a7: unknown
Error: failed to start containers: minikube


😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

[afbjorklund]

Please do not assume systemd availability.

I think that was the whole point of adding the "sysinit" abstraction later, rather than systemctl.

Not sure if Kubernetes and kubeadm supports any distro without it, but that is another story...

Do also not assume service is available.

Hmm, the current openrc.go does use "service". I guess this should have been rc-service ?

I use openrc

Then it should be all good! Someone else was using old CentOS (through AWS Linux), and it failed.

Mostly because the syntax of these init scripts are not (were not) compatible between distributions.

[afbjorklund]

Error response from daemon: cgroups: cannot find cgroup mount destination: unknown

This could be related to something else, like cgroupfs mounts not working properly or something ?

This script is rather nice: https://github.com/tianon/cgroupfs-mount

EDIT: It was already listed as a "suggests" dependency, for docker.io

[afbjorklund]

Confirm that on Alpine, it (service) is a symlink:

 docker run -it alpine
/ # which service
/ # which rc-service
/ # apk add openrc
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
(1/1) Installing openrc (0.41.2-r1)
Executing openrc-0.41.2-r1.post-install
Executing busybox-1.30.1-r2.trigger
OK: 8 MiB in 15 packages
/ # which service
/sbin/service
/ # which rc-service
/sbin/rc-service
/ # ls -l /sbin/*service
-rwxr-xr-x    1 root     root         26504 Jun 10  2019 /sbin/rc-service
lrwxrwxrwx    1 root     root            16 Jun  6 19:30 /sbin/service -> /sbin/rc-service

So maybe we should change minikube to use "rc-service".

https://wiki.alpinelinux.org/wiki/Alpine_Linux_Init_System

[afbjorklund]

I tested with Devuan Beowulf. Seems like "service" is missing, only service(8) man page remains.

Possibly we need some compat sysinit, for older system expecting to just run /etc/init.d/service

EDIT: It was not missing, it was in /usr/sbin/service (from init-system-helpers)

The main difference is that it is not in the PATH. Only /usr/bin is in the PATH here.

Running "sudo service" works just fine.

So let's keep using that (i.e. service)

[paddy-hack]

@afbjorklund Thanks for the feedback!

BTW, ifminikube supports openrc, I'd expect it to use rc-service. The service script from init-system-helpers effectively just runs /etc/init.d/$SERVICE.
Of course, if you claim that minikube supports init-system-helpers ... 😉

[paddy-hack]

As for cgroupfs-mount, the script you mention is part of Debian's and Devuan's cgroupfs-mount package which I have installed. The Docker.com docker-cepackage that I use lists it as a recommended package.

# docker.io is too old for my taste 😉

[afbjorklund]

Of course, if you claim that minikube supports init-system-helpers ... wink

If there is a backend added for sysv init, then openrc backend can go back to using rc-service

That would also "fix" systems without start-stop-daemon, such as Amazon Linux 1* (#8333)

And maybe implement Enable and Disable properly for openrc as well ? (with rc-update)

Probably 90% of it would be the same, except the actual init script itself. Can test on CentOS 6*.

* neither distribution release is supported next year, they both went to using systemd instead.
(i.e. Amazon Linux 2 and CentOS 7). So I guess most common are still Gentoo and Alpine.

https://en.wikipedia.org/wiki/Category:Linux_distributions_without_systemd

We still won't have any testing for them, but if it is just a "sysinit" (or a cgroup) needed ?

[afbjorklund]

docker.io is too old for my taste 😉

It was done for another reason...

See https://www.collabora.com/news-and-blog/blog/2018/07/04/docker-io-debian-package-back-to-life/

[paddy-hack]

You may want to take a look at the non-systemd distributions at distrowatch.com 😄.

As of writing, Devuan ranks between Alpine and Gentoo (but that may be due to the fact Devuan just released Beowulf).

[afbjorklund]

The important list is:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

The "none" driver is not the primary use case for minikube (more like the third or so)

As explained in #4733, we support "both kinds" of OS:
<see https://www.youtube.com/watch?v=bryvnZYiIaY>

Although, in practice we built our own Buildroot OS...
And "still working" on that CentOS support in CI (#3552)

[paddy-hack]

See https://www.collabora.com/news-and-blog/blog/2018/07/04/docker-io-debian-package-back-to-life/

Many thanks for pointing out that docker.io has been revived 🎉
I was still thinking it was stuck somewhere in the 1.x series 🤦
I may even give it a try, together with the docker-compose package when I switch to chimaera (the Devuan analog of Debian's bullseye, see https://devuan.org/os/releases)

[afbjorklund]

Kubernetes has very modest requirements on Docker versions, you just need the CRI basics. The main reason for upgrading is that the community edition does not receive any maintenance updates...

[tstromberg]

OpenRC support was added in minikube v1.10.

[paddy-hack]

Did you see #6954 (comment)?
It uses minikube v1.11.0 with OpenRC and it bombs when stop and try to start it again.

Whatever support was added in v1.10, doesn't work in v1.11.
Haven't tried since so don't know current status.

[afbjorklund]

I added some code in #8572

[afbjorklund]

If you think it is important to support distributions without systemd and without openrc, then please reopen...

The current supported distribution for OpenRC is Google CloudShell (#7539), so it might not work on others.
The canonical distribution is Gentoo, but there are some others as well: https://en.wikipedia.org/wiki/OpenRC

However, we don't have any good regression tests for this (non-systemd sysinit) so it will be largely untested.