cri-dockerd v0.2.4 does not work out of the box with minikube

[mgabeler-lee-6rs]

What Happened?

https://github.com/Mirantis/cri-dockerd/releases/tag/v0.2.4 now defaults to using the kubenet network plugin (Mirantis/cri-dockerd#99). This does not work, at least with the none driver.

The cri-dockerd README suggests the cni driver as perhaps an alternative, but this doesn't work either.

Only by checking the old code from cri-dockerd v0.2.3 can one find that it used to run with "no" network plugin (--network-plugin= on the cri-dockerd command line) as a valid arg, and that is what is needed to make it work.

Attach the log file

With the kubenet driver, kubelet reports:

Aug 03 16:04:29 MYHOSTNAME kubelet[3512192]: E0803 16:04:29.502654 3512192 kubelet.go:2349] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: could not locate kubenet required CNI plugins [bridge host-local loopback] at [\"/opt/cni/bin\" \"/opt/cni/bin\"]"

and the node never becomes ready and so no pod is ever scheduled.

With the cni driver:

Aug 03 16:05:15 MYHOSTNAME kubelet[3517922]: E0803 16:05:15.295210 3517922 kubelet.go:2349] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"

Operating System

Ubuntu

Driver

None (Baremetal)

[afbjorklund]

This seems like a breaking change from upstream... No idea why cni would stop working too, though.

• Set a default network plugin now that we're on 1.24 Mirantis/cri-dockerd#99

Fortunately it doesn't affect the regular minikube deployment (0.2.2), until we update cri-dockerd next time...

There was another bug in 0.2.3, that cause the log to fill up with debugging output (as "error") - bug 85

[afbjorklund]

It is fixed by this PR, which always sets the network plugin: (avoids the previous shortcut, to not set it)

• Handle CRI config of NetworkPlugin and PauseImage #14703

This was the minikube code that got broken:

	if networkPlugin == "" {
		// no-op plugin
		return nil
	}

Now that default changed to "kubenet" 😔

[afbjorklund]

@mgabeler-lee-6rs the errors (with "cni") seems weird, it is like CNI plugins were not properly installed ?

network plugin is not ready: could not locate kubenet required CNI plugins [bridge host-local loopback] at [\"/opt/cni/bin\" \"/opt/cni/bin\"]

network plugin is not ready: cni config uninitialized

If not using the "no-op" (previous default network plugin), then both cri-tools and cni-plugins are required:

• https://github.com/kubernetes-sigs/cri-tools

• https://github.com/containernetworking/plugins

When installing kubernetes from packages, these are installed as requirements (of the kubelet, and kubeadm)

Installing and configuring cri and cni are now the reponsibility of the runtime, since Kubernetes v1.24.0-alpha.2

[afbjorklund]

Note that both dockershim and kubenet were removed from upstream, while recommending to use CRI and CNI.

• Dockershim deprecation side-effect : "kubenet" will be gone! kubernetes#100707

[mgabeler-lee-6rs]

If not using the "no-op" (previous default network plugin), then both cri-tools and cni-plugins are required:

Where is this documented?

• https://minikube.sigs.k8s.io/docs/drivers/none/ ? nope!
• Linked kubeadm docs https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ ? nope!
• Your link to the github repo for these plugins is actually the first time I've been able to find where to get them from. Searching based on the errors was unsuccessful for me (though now that I know more what to search for I can find this info ... though only in bug reports or indirect links from other documentation, that repo doesn't directly show up in any search results)

And actually, the "no-op" plugin does not seem to need the cni-plugins, it is the kubenet and cni plugin that need it?

When installing kubernetes from packages

But that isn't what happens here, unless I'm misunderstanding what you mean by "packages". Minikube is the one installing kubelet & kubeadm when using the none driver, looks like from binaries from its own release process. Minikube's on packages don't have any dependencies on these other tools, not least because only cri-dockerd even has .deb/.rpm packages available, crictl and the cni plugins do not.

---

Seems like the changes from #14555 should be extended to also check for the CNI plugins if they are required? Though with the PR you linked it seems like they may no longer be required?

---

#14703

Thanks for the link to this PR. I will try building from this branch and confirm if that fixes this issue for me or not!

---

I'm well aware the none driver is "advanced mode" and has extra requirements and caveats. I'd move away from it if the alternatives weren't horrifically worse for my team's use case ;)

It just seems the documentation on how to use it has degraded significantly with kubernetes 1.24 (in the sense that the documentation hasn't been fully updated to make it easy to setup as it was with 1.23). Granted, the lacking documentation is not just down to minikube, but also other kubernetes and peripheral projects too.

[afbjorklund]

The documentation hasn't really been updated (properly) for 1.24, the requirements have changed since 1.23

• docker
• cri-dockerd (new)
• cri-tools (new!)
• cni-plugins (new!)

[afbjorklund]

Granted, the lacking documentation is not just down to minikube, but also other kubernetes and peripheral projects too.

The documentation was actually deleted from Kubernetes, since it has been delegated to the container runtime providers.

• Dockershim removal feedback & issues kubernetes#106917 (comment)

So now it is up to Docker and Mirantis to provide this information, but they mostly do that for their paying customers.

[afbjorklund]

But that isn't what happens here, unless I'm misunderstanding what you mean by "packages"

It was a vague reference to the upstream documentation of installing kubernetes with apt or yum:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

Like you say, that is not what is happening with the "none" driver - it "assumes" that they are there.
The requirements are supposed to be checked by kubeadm, except for CNI which is "optional"*

* the "normal" behaviour is for Kubernetes to into CrashLoopBackoff, waiting for the installation...

The kubelet is now restarting every few seconds, as it waits in a crashloop for kubeadm to tell it what to do.

This is because CNI is a separate CNCF project (from Kubernetes), and lives at https://www.cni.dev/

After installing the required /opt/cni programs, it is also required to configure it using the networking plugins.
But this part is supposed to be handled by minikube, it does include some configuration for the various cni.

[afbjorklund]

And actually, the "no-op" plugin does not seem to need the cni-plugins, it is the kubenet and cni plugin that need it?

Correct.

This is why this was a breaking change, it seemed to work OK with the cri-dockerd 0.22 release for instance

[mgabeler-lee-6rs]

Like you say, that is not what is happening with the "none" driver - it "assumes" that they are there.

Sorry but no, it does assume they (kubectl and kubeadm) are present. Minikube itself downloads its own copies of them and installs them into /var/lib/minikube/binaries/${kubernetes-version}/ -- https://github.com/kubernetes/minikube/blob/master/pkg/minikube/download/binary.go

[afbjorklund]

I meant that minikube assumes that cri and cni are installed by the user, it does add the k8s components

For the supported OS, minikube pre-installs the cri and cni packages on the ISO and in the KIC image

[afbjorklund]

This needs some more documentation on the "none" page, beyond just some crictl and cni bullets.

Can perhaps link to the "without a package manager" section, of the upstream k8s.io documentation ?

Even though it is quite horrible. (for instance, the versions don't even match those used elsewhere)

BTW: This page also has the link to the available .deb and .rpm packages, in those other "tabs"...

• https://apt.kubernetes.io/

• https://yum.kubernetes.io/

[mgabeler-lee-6rs]

It is fixed by this PR, which always sets the network plugin: (avoids the previous shortcut, to not set it)

• Handle CRI config of NetworkPlugin and PauseImage #14703

This PR is not working for me as-is, but for a reason I was able to identify. Posted details there, but tl;dr it's just a typo and works after fixing that 👍

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mgabeler-lee-6rs]

/remove-lifecycle stale

The PRs are still going, this is still an issue

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[mgabeler-lee-6rs]

Minikube being broken due to this and linked issues for months on end has simply led to me & my team abandoning use of minikube. It's probably still broken, but I won't be able to help confirm any fixes. That said, the PRs are still out there 🤷