none with containerd: make it possible to run without the docker CLI

[inductor]

Steps to reproduce the issue:

This is the current behavior regarding initizliation with none driver with containerd runtime.

# minikube start --driver=none --container-runtime='containerd'
😄  minikube v1.18.1 on Ubuntu 20.04 (amd64)
✨  Using the none driver based on user configuration

🤷  Exiting due to PROVIDER_NONE_NOT_FOUND: The 'none' provider was not found: exec: "docker": executable file not found in $PATH
💡  Suggestion: Install docker
📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

root@minikube:~# minikube start -h | grep provider
root@minikube:~# minikube start --driver=none --container-runtime='containerd'
😄  minikube v1.18.1 on Ubuntu 20.04 (amd64)
✨  Using the none driver based on user configuration

🤷  Exiting due to PROVIDER_NONE_NOT_FOUND: The 'none' provider was not found: exec: "docker": executable file not found in $PATH
💡  Suggestion: Install docker
📘  Documentation: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

Since Docker is no longer default runtime, I think even none driver should consider the runtime option if specified.

Full output of failed command:

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

[afbjorklund]

It's a bug, unfortunately: #5549

It should not look for "docker", when wanting to run with containerd.

Since Docker is no longer default runtime

Docker is still the default runtime, though.

[tharun208]

@afbjorklund I like to work on this

[afbjorklund]

I like to work on this

Please do, note that it is not the warning and the early exit that is the real issue here.

        if _, err := exec.LookPath("docker"); err != nil {
                return registry.State{Running: true, Error: err, Installed: false, Fix: "Install docker", Doc: "https://minikube.sigs.k8s.io/docs/reference/drivers/none/"}
        }

So just comment that out while debugging.

We also have some callouts to the Docker daemon, but those should just fail (for now)
So leave those as well, the main issue here is the requirement on the "docker" CLI client.

(the other is #8577)

[tstromberg]

/assign tharun208

[tstromberg]

I haven't looked at the none driver recently, but if it isn't possible to simply remove the docker checks when the none driver is used with non-Docker runtime, https://github.com/containerd/nerdctl may be a helpful alternative.

[inductor]

@tstromberg I'm aware of nerdctl(my friend made it), but I just wanted to setup kubernetes with local runtime without using kubeadm easily. So using nerdctl is not really an option for my use-case

[afbjorklund]

As far as I know, the "docker" issues are because the minikube driver registry is broken

Not because the Docker client is actually required for anything (beyond e.g. docker-env)
And we offer BuildKit as a replacement, if you want to build images for your containerd...

You will have to install crictl though, unfortunately that is not bundled with Kubernetes.

[sharifelgamal]

@tharun208 are you still interested in working on this? if so, reassign yourself.

Help wanted!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[abiosoft]

You can simply spoof it with a docker binary in path that does nothing.

sudo printf "#!/usr/bin/env bash\nexit 0" > /usr/local/bin/docker && sudo chmod +x /usr/local/bin/docker

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[medyagh]

@tharun208 I haven't heared from u...I will un-assign this issue is available for picking up

[staticdev]

@afbjorklund / @medyagh I want to debug this error to see if I could fix it so we don't need the workaround of:

ln -s /bin/false /usr/local/bin/docker

From TDD perspective where should I create an test on the codebase? Possibly we are talking about an e2e test, I also see there is this TestStartStop integration test but those tests does not have driver and container-runtime args.

PS.: plz go easy on me, I am new to minikube code =p

[afbjorklund]

The issue is somewhere in the driver registry, for some reason it calls every constructor twice...

https://github.com/kubernetes/minikube/tree/master/pkg/minikube/registry

[staticdev]

One thing that is confusing to me is that docker/podman are "--driver" options as well as "--container-runtime"s. This is not consistent with containerd. Maybe the logic of the code requires the creation of another driver option for containerd? Or else Podman and Docker should also use driver None.

https://github.com/kubernetes/minikube/blob/master/pkg/minikube/registry/global.go#L29 -> see IsKIC logic for example.

Am I missing something to understand? It is also not clear to me how these Kong Ingress Controller addons play a role in firing the runtime in K8s.

[afbjorklund]

@staticdev : we are running docker-in-docker, or crio-in-podman. So yes, it is confusing with the "driver" and "runtime".
So far nobody has bothered with containerd-in-nerdctl, but I suppose that it is technically possible as well. Plus permutations.

Kong Ingress Controller
Kubernetes In Container

But I think the docker "requirement" comes from the none driver, and not from the container runtime. IIRC.

🤷  Exiting due to PROVIDER_NONE_NOT_FOUND: The 'none' provider was not found: exec: "docker": executable file not found in $PATH

status for none: {Installed:false Healthy:false Running:true NeedsImprovement:false Error:exec: "docker": executable file not found in $PATH Reason: Fix:Install docker Doc:https://minikube.sigs.k8s.io/docs/reference/drivers/none/ Version:}

[afbjorklund]

So it is supposed to look in the registry for "none", but instead it runs the status command of "docker".

func status() (retState registry.State) {
	_, err := exec.LookPath(oci.Docker)
	if err != nil {
		return registry.State{Error: err, Installed: false, Healthy: false, Fix: "Install Docker", Doc: docURL}
	}

I think it is currently explicitly coded in both, but still broken even without it.

func status() registry.State {
	_, err := exec.LookPath("iptables")
	if err != nil {
		return registry.State{Running: true, Error: err, Fix: "iptables must be installed", Doc: "https://minikube.sigs.k8s.io/docs/reference/drivers/no
ne/"}
	}

	if _, err := exec.LookPath("docker"); err != nil {
		return registry.State{Running: true, Error: err, Installed: false, Fix: "Install docker", Doc: "https://minikube.sigs.k8s.io/docs/reference/driv
ers/none/"}
	}

---

Here is the libmachine error:

error starting host: creating host: create: precreate: exec: "docker": executable file not found in $PATH

https://github.com/kubernetes/minikube/blob/master/pkg/drivers/none/none.go#L82

So that's the part where it is using the wrong runtime (cruntime.Manager), docker instead of containerd.

If I remember correctly, it was due to "ContainerRuntime" being reset to the empty string somewhere ?

[staticdev]

@afbjorklund after digging a bit more I think the solution is to remove this docker check anyway since it is redundant with runtime available method for docker here https://github.com/kubernetes/minikube/blob/master/pkg/drivers/none/none.go#L82

The other path to call docker validation with though registry/global.go on L149 that calls

minikube/pkg/minikube/registry/drvs/none/none.go

Line 63 in 9f96701

if _, err := exec.LookPath("docker"); err != nil {

This ALWAYS checks if you have docker, independently of container runtime. This validation is completely unnecessary and redundant since runtime already does the same check. I created a PR to remove it.

[staticdev]

UPDATE: Still after removing the docker validation from status() of none.go we get:

X Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: precreate: exec: "docker": executable file not found in $PATH

After some debugging I see on precreate that the driver is wrongly being assigned to docker runtime instead of containerd. There reason for that is that the driver is created correctly the first time here with def.Config https://github.com/kubernetes/minikube/blob/master/pkg/minikube/machine/start.go#L150

Then, it is validated and for some reason, the driver is recreated (not sure if it is necessary) on api.NewHost(cfg.Driver, data) with def.Init and not def.Config! https://github.com/kubernetes/minikube/blob/master/pkg/minikube/machine/client.go#L95

When you create a none driver, you run func init() on none.go you have an empty config that generates a runtime "". This defaults to docker, making everything else fail.

Not sure how to improve here... options:

1. Send complete clusterConfig (instead of only drvName) and node to api.NewHost and then it is possible to perform def.Config.
2. Send the driver as parameter to api.NewHost instead of driver name and rawDriver, then you don't create it again. Not sure it will work, because it also needs driver name and rawDriver for api.legacyClient.NewHost (probably for backwards compatibility?)

Both options will break the API by changing the signature of api.NewHost, but I see no other references to it on code. Any thoughts @afbjorklund / @medyagh?

[lucastheisen]

Might be worth mentioning the workaround

sudo printf "#!/usr/bin/env bash\nexit 0" > /usr/local/bin/docker && sudo chmod +x /usr/local/bin/docker

Doesn't resolve the failure during minikube delete

[example@localhost ~]$ minikube delete --purge
:arrows_counterclockwise:  Uninstalling Kubernetes v1.25.2 using kubeadm ...
:fire:  Deleting "minikube" in none ...
:x:  Failed to delete cluster: host remove retry: kill: containers: docker: docker ps -a --filter=name=k8s_ --format={{.ID}}: exit status 125
stdout:

[nagstaku]

like @lucastheisen states, the work around has a failure on delete because of the filter looking for an underscore, where containerd uses 'k8s://' to prefix containers.... so, please let me know if this might be another workaround?

cat /usr/local/bin/docker
#!/bin/bash

# Check if the command being executed matches the desired pattern
if [[ "$1" == "ps" && "$2" == "-a" && "$3" == "--filter=name=k8s_" && "$4" == "--format={{.ID}}" ]]; then
    echo "Custom docker function triggered with arguments: $@"
    # Execute the desired command with the additional option
    nerdctl --namespace k8s.io "$@" --filter='name=k8s:'
else
    # Execute the original docker command
    echo "warning, this is actually containerd!"
    nerdctl --namespace k8s.io "$@"
fi

I had already made a bash_alias for nerdctl over docker, so I just made a shell script to modify the filter to also optionally do the containerd format.

It did result in a successful stop:

minikube stop
✋  Stopping node "minikube"  ...
✋  Stopping node "minikube"  ...
🛑  1 node stopped.

(and a successful start)