kubeadm debian package v1.25.6 fails against v1alpha2 API CRI due to dependency on cri-tools >= 1.26.0 #2866

mohag · 2023-01-19T16:00:45Z

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"ff2c119726cc1f8926fb0585c74b25921e866a28", GitTreeState:"clean", BuildDate:"2023-01-18T19:20:37Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.6", GitCommit:"ff2c119726cc1f8926fb0585c74b25921e866a28", GitTreeState:"clean", BuildDate:"2023-01-18T19:22:09Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration: VMware VMs
OS (e.g. from /etc/os-release): Ubuntu 22.04
Kernel (e.g. uname -a): Linux master-test-cluster.iotnxt.io 5.15.0-58-generic #-64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Container runtime (CRI) (e.g. containerd, cri-o): version=1.5.9-0ubuntu3.1
Container networking plugin (CNI) (e.g. Calico, Cilium): Calico
Others:

What happened?

When attempting to run kubeadm config images pull it fails compaining that the CRI runtime does not support the v1 API.

# kubeadm config images pull
I0119 17:36:05.929035   54330 version.go:256] remote version is much newer: v1.26.1; falling back to: stable-1.25
failed to pull image "registry.k8s.io/kube-apiserver:v1.25.6": output: time="2023-01-19T17:36:07+02:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint \"unix:///var/run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1

What you expected to happen?

A runtime supporting CRI apiVersion v1alpha2 should work, as documented here

How to reproduce it (as minimally and precisely as possible)?

On a Debian environment (Ubuntu 22.04 will work)
Configure the Kubernetes APT repo.
Install containerd 1.5.x
Set up pins in /etc/apt/preferences.d/kubeadm for kubeadm and cri-tools on 1.25.* (kubeadm v1.25.6 will fail with this, see above, so the pin for cri-tools might need to be removed to reproduce the error)
With the pins, attempt to install kubeadm (it will fail complaining that it depends on >= v1.26. If the pin is removed, it will install, but fail to communicate with containerd v1.5.9)

More minimal for the underlying issue:
On ubuntu 22.04:
Check dependencies for kubeadm v1.25.5 and v1.25.6 with apt show. Install the cri-tools version that it depends on.
Install containerd 1.5.9 (that is the default one from the OS repo)

Run crictl -r unix:///run/containerd/containerd.sock version.

For cri-tools v1.26.0 (depended on by kubeadm v1.25.6):

crictl -r unix:///run/containerd/containerd.sock version
FATA[0000] validate service connection: CRI v1 runtime API is not implemented for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService

For cri-tools v1.25.0 (which is within kubeadm v1.25.5's dependencies, but not v1.25.6's)

# crictl -r unix:///run/containerd/containerd.sock version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  1.5.9-0ubuntu3.1
RuntimeApiVersion:  v1alpha2

Note that it works when the cri-tools version that v1.25.5 depends on is installed, but fails if the version that v1.25.6 depends on is installed.

Anything else we need to know?

v1.25.5 worked correctly - cri-tools v1.25.x could be installed and with that installed the pull works.

The version of cri-tools depended on by v1.25.6 (>= 1.26.0) differs from that of v1.25.5 (>= 1.25.0).

# apt show kubeadm=1.25.5-00
Package: kubeadm
Version: 1.25.5-00
Priority: optional
Section: misc
Maintainer: Kubernetes Authors <kubernetes-dev+release@googlegroups.com>
Installed-Size: 43.8 MB
Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 1.1.1), cri-tools (>= 1.25.0)
Homepage: https://kubernetes.io/
Download-Size: 9,229 kB
APT-Sources: https://apt.kubernetes.io/ kubernetes-xenial/main amd64 Packages
Description: Kubernetes Cluster Bootstrapping Tool
The Kubernetes command line tool for bootstrapping a Kubernetes cluster.

# apt show kubeadm=1.25.6-00
Package: kubeadm
Version: 1.25.6-00
Priority: optional
Section: misc
Maintainer: Kubernetes Authors <kubernetes-dev+release@googlegroups.com>
Installed-Size: 43.8 MB
Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 1.2.0), cri-tools (>= 1.26.0)
Homepage: https://kubernetes.io/
Download-Size: 9,230 kB
APT-Manual-Installed: yes
APT-Sources: https://apt.kubernetes.io/ kubernetes-xenial/main amd64 Packages
Description: Kubernetes Cluster Bootstrapping Tool
The Kubernetes command line tool for bootstrapping a Kubernetes cluster.

The text was updated successfully, but these errors were encountered:

mohag · 2023-01-19T16:03:21Z

TLDR: kubeadm v1.25.6's package requires a newer cri-tools, which increases the CRI apiVersion requirement to v1, from v1alpha2 with kubeadm v1.25.5.

neolit123 · 2023-01-19T16:06:48Z

/transfer release

this is a package management issue.
moving to k/release

@afbjorklund i believe you investigated similar issues recently?

afbjorklund · 2023-01-19T16:32:47Z

Yes, the new kubeadm can only be used with the new container runtimes.

It does not support containerd 1.5 or cri-dockerd 0.2 (or dockershim). So the depends are wrong

https://kubernetes.io/blog/2022/12/09/kubernetes-v1-26-release/#cri-v1alpha2-removed

afbjorklund · 2023-01-19T16:35:29Z

@saschagrunert

f1da4c4
024e778

afbjorklund · 2023-01-19T16:57:32Z

The upgrade of CNI is also doubtful, for a maintenance release...

Haven't checked yet if the 1.2.0 packages are also mispackaged, like 1.0/1.1 were

Create test jobs to validate that debs/rpms can be installed #821 (comment)

EDIT: No, it looks OK:

anders@lima-k8s:~$ apt list | grep kubernetes-xenial

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

cri-tools/kubernetes-xenial,now 1.26.0-00 amd64 [installed]
docker-engine/kubernetes-xenial 1.11.2-0~xenial amd64
kubeadm/kubernetes-xenial,now 1.26.1-00 amd64 [installed]
kubectl/kubernetes-xenial,now 1.26.1-00 amd64 [installed]
kubelet/kubernetes-xenial,now 1.26.1-00 amd64 [installed]
kubernetes-cni/kubernetes-xenial,now 1.2.0-00 amd64 [installed]
rkt/kubernetes-xenial 1.29.0-1 amd64
anders@lima-k8s:~$ crictl --version
crictl version v1.26.0
anders@lima-k8s:~$ /opt/cni/bin/portmap --version
CNI portmap plugin v1.2.0
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
anders@lima-k8s:~$ /opt/cni/bin/flannel --version
CNI Plugin flannel version v1.1.0 (linux/amd64) commit 6e8bb113 built on 2022-05-20T12:20:06Z
CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
anders@lima-k8s:~$ containerd --version
containerd github.com/containerd/containerd v1.6.12 a05d175400b1145e5e6a735a6710579d181e7fb0
anders@lima-k8s:~$ sudo crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  v1.6.12
RuntimeApiVersion:  v1

afbjorklund · 2023-01-19T17:29:12Z

This is also a breaking change for minikube, so need to bundle CRI and CNI with k8s instead of with the OS

Update CRI and CNI installations to the ones used by default kubernetes minikube#15495

I haven't yet figured out the best way to get which versions to install, but maybe it needs a mapping table ?

https://github.com/kubernetes/minikube/blob/master/pkg/minikube/constants/constants_kubeadm_images.go

One can extract it from the debs, but I think it lives in the release documents somewhere.

k8s	kubeadm
1.20.0	Depends: kubelet (>= 1.13.0), kubectl (>= 1.13.0), kubernetes-cni (>= 0.8.7), cri-tools (>= 1.13.0)
1.21.0	Depends: kubelet (>= 1.13.0), kubectl (>= 1.13.0), kubernetes-cni (>= 0.8.7), cri-tools (>= 1.13.0)
1.22.0	Depends: kubelet (>= 1.13.0), kubectl (>= 1.13.0), kubernetes-cni (>= 0.8.7), cri-tools (>= 1.13.0)
1.23.0	Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 0.8.7), cri-tools (>= 1.19.0)
1.24.0	Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 0.8.7), cri-tools (>= 1.23.0)
1.25.0	Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 0.8.7), cri-tools (>= 1.24.2)
1.26.0	Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 1.1.1), cri-tools (>= 1.25.0)

Still need to figure out which release version to use. kubernetes/release packages/rpm/kubelet.spec

It seems hardcoded in the docs: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/

RELEASE_VERSION="v0.4.0"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubelet/lib/systemd/system/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service
sudo mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/kubepkg/templates/latest/deb/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

neolit123 · 2023-01-19T17:37:03Z

The upgrade of CNI is also doubtful, for a maintenance release...

i think this is due to the old problem where package updates land on all minor releases. i know one discussed fix was complicated - separate package repositories per minor.

afbjorklund · 2023-01-19T17:42:38Z

The package updates are fine, they are pinned anyway so requires picking a version to install.

anders@lima-k8s:~$ apt-mark showhold
kubeadm
kubectl
kubelet

Having to select the minor version of the repo is annoying, like how it is being done in cri-o.

These are the actual dependencies inside the debs, that has been bumped for the older too.

Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 1.2.0), cri-tools (>= 1.26.0)

Since it is no longer backwards compatible, it needs to be forked and updated separately...

[1.24] Remove dockershim from crictl code and docs kubernetes-sigs/cri-tools#870 (comment)

afbjorklund · 2023-01-19T19:16:05Z

Beyond kubeadm=1.25.6-00, it is also in kubeadm=1.24.10-00 and kubeadm=1.23.16-00

dlipovetsky · 2023-01-19T22:30:43Z

I was going to report this, but found this issue.

I think this affects all package-based new installations of kubeadm <= 1.26.0, because the cri-tools version has only a lower bound, as shown in the table above. (Also see the kubeadm.rpmspec).

Is it fair to assume that kubeadm <= 1.26.0 is guaranteed, until at least its end-of-life, to work with CRI v1alpha2? For argument's sake, I'll say yes. Then, since kubeadm requires cri-tools, kubeadm cannot depend on cri-tools v1.26..0.

Therefore, the fix is to add an upper bound to the cri-tools dependency for all kubeadm <= 1.26.0 packages that are still under support.

It is, in theory, possible to release new RPM and DEB packages for older versions of kubeadm by incrementing the release and revision, respectively.

(I'm not sure how that's could be done in this repository, because I only see a single branch. I would expect to modify the RPM and DEB metadata in release branches.)

afbjorklund · 2023-01-20T07:19:40Z

(I'm not sure how that's could be done in this repository, because I only see a single branch. I would expect to modify the RPM and DEB metadata in release branches.)

I think that is the real missing feature here.

packaging specs should be per-kubernetes version and in-tree (k/k) #1913

The current depends is an ongoing patchwork:

Clarification regarding versioning of cri-tools #1636

I'm not really sure why kubeadm requires cri-tools, and we had some package conflicts with kubernetes-cni
(some distributions are already shipping something like containernetworking-plugins, with the same contents)

But I think the latest update should be reverted meanwhile, so that it can stay on the 1.25.0 and 1.1.1 "for a while".
Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 1.1.1), cri-tools (>= 1.25.0)

It needs then to be configured/pinned by the user, but that's not worse than the rest of the OOTB failures.

WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
FATA[0000] getting the runtime version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"

Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock

You can still install the new versions of the packages, if you are running with a newer container runtime.

afbjorklund · 2023-01-20T07:28:37Z

@mohag if you ever want to upgrade Kubernetes 1.26 (and up), which requires CRI v1 to be used:

Then you need to change, from ubuntu's containerd/runc to docker's containerd.io package.

containerd.io/focal 1.6.15-1 amd64 [residual-config]
containerd/focal-updates,focal-security,now 1.5.9-0ubuntu1~20.04.6 amd64 [installed,automatic]
runc/focal-updates,now 1.1.0-0ubuntu1~20.04.2 amd64 [installed]

The only thing to watch out for, is that the Docker packaging disables the CRI plugin by default...

disabled_plugins = ["cri"]

This is because CRI talks to cri-dockerd, so there is no reason for cri-containerd to be enabled (with Docker)

This is documented upstream, but can be an issue when upgrading. Should be OK again with Ubuntu 24.04 ?

https://kubernetes.io/docs/setup/production-environment/container-runtimes/#containerd

The containerd.io packages in DEB and RPM formats are distributed by Docker (not by the containerd project).

The default configuration can be generated via containerd config default > /etc/containerd/config.toml.

afbjorklund · 2023-01-20T07:44:33Z

This issue probably needs fixing, before automating the updates in the future:

Automatically promote cri-tools / cni plugins releases #2693

saschagrunert · 2023-01-20T08:05:01Z

The upgrade of CNI is also doubtful, for a maintenance release...

i think this is due to the old problem where package updates land on all minor releases. i know one discussed fix was complicated - separate package repositories per minor.

Exactly, I think we can solve this with the new packaging approach (kubernetes/enhancements#3750), but we may require an intermediate fix as well.

mohag · 2023-01-20T08:13:43Z

@mohag if you ever want to upgrade Kubernetes 1.26 (and up), which requires CRI v1 to be used:

That is mainly why we are testing 1.25 for now. They might update containerd.

I'm also thinking adding a CRI-O role to our Ansible deployment as an option might be useful... (It also ensures that we don't get too used to one container runtime...)

Then you need to change, from ubuntu's containerd/runc to docker's containerd.io package.

The only thing to watch out for, is that the Docker packaging disables the CRI plugin by default...
disabled_plugins = ["cri"]

That is useful to know, thanks.

dlipovetsky · 2023-01-20T17:39:05Z

I'm not really sure why kubeadm requires cri-tools, and we had some package conflicts with kubernetes-cni
(some distributions are already shipping something like containernetworking-plugins, with the same contents)

kubeadm calls the crictl CLI to pull images, and list and remove containers. Please see https://github.com/kubernetes/kubernetes/blob/3eb7b7a48fea4c8bb9a3349f873b452ae6f0a1cc/cmd/kubeadm/app/util/runtime/runtime.go.

But I think the latest update should be reverted meanwhile, so that it can stay on the 1.25.0 and 1.1.1 "for a while".
Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 1.1.1), cri-tools (>= 1.25.0)

Given the number of installations this impacts, I agree.

IIUC, anyone that provisions Kubernetes <= 1.26.0 using our packages, and your container runtime does not expose the CRI v1 API, e.g., containerd <= 1.6.0. I suspect this describes many production clusters.

afbjorklund · 2023-01-21T08:09:03Z

But it could have been an "external" dependency, since it is checked during the preflight phase ?

The versions also seemed to be totally arbitrary, so maybe it could have been left at the names ?

k8s-ci-robot transferred this issue from kubernetes/kubeadm Jan 19, 2023

k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-priority labels Jan 19, 2023

ralgozino mentioned this issue Jan 20, 2023

Reintroduce cri-tools version pinning sighupio/fury-kubernetes-on-premises#43

Closed

dlipovetsky mentioned this issue Jan 20, 2023

feat: Install specific cri-tools package on debian derivatives mesosphere/konvoy-image-builder#631

Merged

xmudrii mentioned this issue Jan 25, 2023

Revert bumping kubernetes-cni to v1.2.0 and cri-tools to v1.26.0 #2875

Merged

k8s-ci-robot closed this as completed in #2875 Jan 25, 2023

afbjorklund mentioned this issue Apr 11, 2023

cri-tools 1.26 incompatible with containerd 1.5 kubernetes-sigs/cri-tools#1130

Closed

mohag mentioned this issue Oct 23, 2023

Install cri-tools awslabs/amazon-eks-ami#1235

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm debian package v1.25.6 fails against v1alpha2 API CRI due to dependency on cri-tools >= 1.26.0 #2866

kubeadm debian package v1.25.6 fails against v1alpha2 API CRI due to dependency on cri-tools >= 1.26.0 #2866

mohag commented Jan 19, 2023 •

edited

Loading

mohag commented Jan 19, 2023

neolit123 commented Jan 19, 2023

afbjorklund commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

neolit123 commented Jan 19, 2023

afbjorklund commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

dlipovetsky commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 20, 2023 •

edited

Loading

afbjorklund commented Jan 20, 2023

afbjorklund commented Jan 20, 2023

saschagrunert commented Jan 20, 2023

mohag commented Jan 20, 2023 •

edited

Loading

dlipovetsky commented Jan 20, 2023

afbjorklund commented Jan 21, 2023 •

edited

Loading

kubeadm debian package v1.25.6 fails against v1alpha2 API CRI due to dependency on cri-tools >= 1.26.0 #2866

kubeadm debian package v1.25.6 fails against v1alpha2 API CRI due to dependency on cri-tools >= 1.26.0 #2866

Comments

mohag commented Jan 19, 2023 • edited Loading

Is this a BUG REPORT or FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

mohag commented Jan 19, 2023

neolit123 commented Jan 19, 2023

afbjorklund commented Jan 19, 2023 • edited Loading

afbjorklund commented Jan 19, 2023 • edited Loading

afbjorklund commented Jan 19, 2023 • edited Loading

afbjorklund commented Jan 19, 2023 • edited Loading

neolit123 commented Jan 19, 2023

afbjorklund commented Jan 19, 2023 • edited Loading

afbjorklund commented Jan 19, 2023 • edited Loading

dlipovetsky commented Jan 19, 2023 • edited Loading

afbjorklund commented Jan 20, 2023 • edited Loading

afbjorklund commented Jan 20, 2023

afbjorklund commented Jan 20, 2023

saschagrunert commented Jan 20, 2023

mohag commented Jan 20, 2023 • edited Loading

dlipovetsky commented Jan 20, 2023

afbjorklund commented Jan 21, 2023 • edited Loading

mohag commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 19, 2023 •

edited

Loading

dlipovetsky commented Jan 19, 2023 •

edited

Loading

afbjorklund commented Jan 20, 2023 •

edited

Loading

mohag commented Jan 20, 2023 •

edited

Loading

afbjorklund commented Jan 21, 2023 •

edited

Loading