cri-tools 1.26 incompatible with containerd 1.5 #1130

yogeek · 2023-04-05T16:14:32Z

What happened:

We deploy K8S clusters on AWS EC2 instances with kubeadm
We are trying to follow an immutable pattern by :

building our K8S AMIs with packer from a CSI base image from AWS marketplace (ubuntu 1.18)
pinning versions for the packages we installed, currently
- kubeadm=1.22.15-00
- kubelet=1.22.15-00
- containerd=1.5.11-1
- kubecni=0.8.7-00
- ...
using terraform to launch EC2 with kubeadm commands in cloud-init to either init a new cluster or join/upgrade an existing one

Recently, we built a new base image to benefit from the latest CSI hardening without changing anything in our scripts (still installing the same pinned packaged versions, only the base ubuntu changed) and we were surprised to have a broken cluster as a result, resulting from error in kubeadm command.

After some investigations, we found out that "cri-tools" package (that we do not install explicitely) was implicitely installed as a kubeadm dependency, but apparently, even without modifying the kubeadm version, the installation of kubeadm in our new AMI installed a newer version of "cri-tools"... before we had cri-tools 1.25 package and we ended with cri-tools 1.26.
And cri-tools 1.26 is apparently not compatible with containerd 1.5... which seem to be the root cause of our kubeadm isssue

$ apt show kubeadm
Package: kubeadm
Version: 1.22.15-00
[...]
Depends: kubelet (>= 1.19.0), kubectl (>= 1.19.0), kubernetes-cni (>= 1.1.1), cri-tools (>= 1.25.0)

As soon as cri-tools 1.26 was released, any installation of kubeadm would pull it as a dependency and cause the issue with containerd.

To solve it, we thought about 2 solutions :

either upgrade containerd to 1.6 (but in theory it is not compatible with kubernetes 1.22 so we would have to upgrade to 1.23, which is planned but not immediately)
or downgrade cri-tools explicitely to 1.25 until we upgrade kubernetes to 1.26...

What you expected to happen:

I would expect :

either cri-tools to offer a compatibility with containerd 1.5
or kubeadm to pin cri-tools dependency to ensure not breaking things (because containerd 1.5 is needed for kubernetes 1.22

Anything else we need to know?:

I was not sure if I should open the issue here or in the kubeadm repository...

Environment:

Container runtime or hardware configuration: containerd 1.5.11-1
OS (e.g: cat /etc/os-release):

NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Kernel (e.g. uname -a):

Linux ip-10-10-101-228.eu-central-1.compute.internal 5.4.0-1097-aws #105~18.04.1-Ubuntu SMP Mon Feb 13 17:50:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

The text was updated successfully, but these errors were encountered:

afbjorklund · 2023-04-05T16:56:33Z

You need to pin crictl as well, once you figure out what version to use

Like v1.22.1 (for v1.22.15)

yogeek · 2023-04-05T21:52:28Z

@afbjorklund crictl is not included in cri-tools package...?

And regarding my initial question, do you confirm that cri-tools not being compatible with containers 1.5 is an bug ?

afbjorklund · 2023-04-06T07:04:24Z

Sorry, my bad. Sometimes the package for crictl is called cri-tools, sometimes crictl...

And regarding my initial question, do you confirm that cri-tools not being compatible with containers 1.5 is an bug ?

It is not a bug, support for CRI v1alpha2 was deliberately removed in Kubernetes 1.26

https://kubernetes.io/blog/2022/12/09/kubernetes-v1-26-release/#cri-v1alpha2-removed

The "bug"/missing feature, is pinning the kubeadm requirements on cri-tools to match

cri-tools (< 1.26.0)

afbjorklund · 2023-04-06T07:11:08Z

Trying to come up with a good way to match them, right now release repo is the best "guess"

Update CRI and CNI installations to the ones used by default kubernetes kubernetes/minikube#15495

We probably want to pre-compute a JSON file (or such) for caching purposes, like with the others

https://github.com/kubernetes/minikube/blob/master/pkg/minikube/constants/constants_kubeadm_images.go

kubeadm config images list

afbjorklund · 2023-04-06T07:18:41Z

The old version of cri-tools is having similar issues communicating with newer runtimes, such as CRI-O.

docker@minikube:~$ crictl --version
crictl version v1.21.0
docker@minikube:~$ sudo crictl version
FATA[0000] getting the runtime version: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

Fortunately the default cri-o version lags a bit, so it still "works" when using 1.24-1.25 instead of 1.26.

Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.25.3
RuntimeApiVersion:  v1alpha2

I guess we need to stop using packages for it, or at least change the repository away from Kubic.

docker@minikube:~$ apt list cri-tools 
Listing... Done
cri-tools/unknown,now 1.21.0~2 amd64 [installed]
cri-tools/unknown 1.21.0~2 arm64
cri-tools/unknown 1.21.0~2 armhf
cri-tools/unknown 1.21.0~2 s390x
docker@minikube:~$ apt list cri-o
Listing... Done
cri-o/unknown,now 1.25.3~0 amd64 [installed]
cri-o/unknown 1.25.3~0 arm64
cri-o/unknown 1.25.3~0 armhf
cri-o/unknown 1.25.3~0 s390x
docker@minikube:~$ apt-cache madison cri-tools
 cri-tools |   1.21.0~2 | http://downloadcontent.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04  Packages
docker@minikube:~$ apt-cache madison cri-o
     cri-o |   1.25.3~0 | https://downloadcontent.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/1.25/xUbuntu_20.04  Packages

Both docker and containerd still support alpha2, but I guess they too will drop support soon (v2.0).

https://containerd.io/releases/#kubernetes-support

afbjorklund · 2023-04-06T07:22:07Z

Same issue as:

crictl runtime connetion error #1073

afbjorklund · 2023-04-11T18:16:11Z

Original revert:

kubeadm debian package v1.25.6 fails against v1alpha2 API CRI due to dependency on cri-tools >= 1.26.0 kubernetes/release#2866

yogeek · 2023-04-11T20:54:05Z

@afbjorklund thanks for the link to the revert issue. But it seems that this was closed whereas the discussion was not finished... Or do I misunderstood something ?
What about the proposition to add a upper bound to the cri-tools dependency in kubeadm package ? It seemed like a valid option that could have prevented the current issue, isn't it?

afbjorklund · 2023-04-12T05:29:08Z

Yes, I don't think the underlying issue was resolved...

It is up to the Kubernetes user to set up and configure CRI and CNI, including finding a version that works and changing required configuration such as socket paths or cgroup version. In your case, I think it needs pinning the version (like the kube debs) ?

yogeek · 2023-04-12T06:59:29Z

Ok...it is a pity that the underlying dependency issue in kubeadm was not fixed but apparently it does not seem so easy (I do not understand all the implications to be honest)

For now, and for those who will have the same problem, here is the workaround we did to make it work when installing a 1.22 K8s cluster :

install containerd 1.5 (because the 1.6 is not meant to work with K8s < 1.23)
install kubeadm 1.22 (comes with cri-tools 1.26 by default)
downgrade cri-tools to 1.25

Thank you for your help !

afbjorklund · 2023-04-12T07:29:58Z

The versions are a bit hard to guess, Kubernetes 1.27.0 uses cri-tools 1.26.0 and cni-plugins 1.2.0

cri-tools/kubernetes-xenial,now 1.26.0-00 amd64 [installed]
docker-engine/kubernetes-xenial 1.11.2-0~xenial amd64
kubeadm/kubernetes-xenial,now 1.27.0-00 amd64 [installed]
kubectl/kubernetes-xenial,now 1.27.0-00 amd64 [installed]
kubelet/kubernetes-xenial,now 1.27.0-00 amd64 [installed]
kubernetes-cni/kubernetes-xenial,now 1.2.0-00 amd64 [installed]
rkt/kubernetes-xenial 1.29.0-1 amd64

Easy now (latest/greatest), but it might be harder to align in the future when it (v1.27) too is old

k8s-triage-robot · 2023-07-11T13:35:48Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-01-19T17:02:49Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-02-18T17:58:12Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot · 2024-02-18T17:58:16Z

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

yogeek · 2024-02-18T18:11:14Z

/reopen
/remove-lifecycle rotten

k8s-ci-robot · 2024-02-18T18:11:17Z

@yogeek: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

afbjorklund · 2024-02-18T20:09:17Z

Seems unlikely that this will get done, now that the old packages are removed:

https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/

The new packages have one repository per release, so uses different CRI and CNI

https://build.opensuse.org/project/subprojects/isv:kubernetes

yogeek · 2024-02-19T18:24:34Z

Ok @afbjorklund thanks for the update 👍
Let's close this

afbjorklund · 2024-02-19T19:02:17Z

I spoke too soon, the old pinning was just copied over to the new repos. So v1.25 still installs cri-tools 1.26...

cri-tools/unknown 1.26.0-1.1 amd64
cri-tools/unknown 1.26.0-1.1 arm64
cri-tools/unknown 1.26.0-1.1 ppc64el
cri-tools/unknown 1.26.0-1.1 s390x
kubeadm/unknown 1.25.16-1.1 amd64
kubeadm/unknown 1.25.16-1.1 arm64
kubeadm/unknown 1.25.16-1.1 ppc64el
kubeadm/unknown 1.25.16-1.1 s390x
kubectl/unknown 1.25.16-1.1 amd64
kubectl/unknown 1.25.16-1.1 arm64
kubectl/unknown 1.25.16-1.1 ppc64el
kubectl/unknown 1.25.16-1.1 s390x
kubelet/unknown 1.25.16-1.1 amd64
kubelet/unknown 1.25.16-1.1 arm64
kubelet/unknown 1.25.16-1.1 ppc64el
kubelet/unknown 1.25.16-1.1 s390x
kubernetes-cni/unknown 1.2.0-2.1 amd64
kubernetes-cni/unknown 1.2.0-2.1 arm64
kubernetes-cni/unknown 1.2.0-2.1 ppc64el
kubernetes-cni/unknown 1.2.0-2.1 s390x

But for the newer releases (1.28 and 1.29), they were supposed to be forked for each of the repositories.

https://github.com/kubernetes/enhancements/blob/master/keps/sig-release/1731-publishing-packages/README.md

cri-tools/unknown 1.28.0-1.1 amd64
kubeadm/unknown 1.28.7-1.1 amd64
kubectl/unknown 1.28.7-1.1 amd64
kubelet/unknown 1.28.7-1.1 amd64
kubernetes-cni/unknown 1.2.0-2.1 amd64

cri-tools/unknown 1.29.0-1.1 amd64
kubeadm/unknown 1.29.2-1.1 amd64
kubectl/unknown 1.29.2-1.1 amd64
kubelet/unknown 1.29.2-1.1 amd64
kubernetes-cni/unknown 1.3.0-1.1 amd64

yogeek added kind/bug Categorizes issue or PR as related to a bug. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Apr 5, 2023

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 11, 2023

afbjorklund mentioned this issue Aug 18, 2023

Debian packages stopped on 1.26? #1244

Closed

Calvinaud mentioned this issue Sep 1, 2023

chaos experiment container-kill hepler pod throws error unknown service runtime.v1alpha2.RuntimeService by helper pod. litmuschaos/litmus#4064

Closed

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 19, 2024

k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Feb 18, 2024

k8s-ci-robot reopened this Feb 18, 2024

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Feb 18, 2024

yogeek closed this as not planned Won't fix, can't repro, duplicate, stale Feb 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cri-tools 1.26 incompatible with containerd 1.5 #1130

cri-tools 1.26 incompatible with containerd 1.5 #1130

yogeek commented Apr 5, 2023

afbjorklund commented Apr 5, 2023 •

edited

Loading

yogeek commented Apr 5, 2023 •

edited

Loading

afbjorklund commented Apr 6, 2023 •

edited

Loading

afbjorklund commented Apr 6, 2023 •

edited

Loading

afbjorklund commented Apr 6, 2023 •

edited

Loading

afbjorklund commented Apr 6, 2023

afbjorklund commented Apr 11, 2023

yogeek commented Apr 11, 2023

afbjorklund commented Apr 12, 2023

yogeek commented Apr 12, 2023

afbjorklund commented Apr 12, 2023 •

edited

Loading

k8s-triage-robot commented Jul 11, 2023

k8s-triage-robot commented Jan 19, 2024

k8s-triage-robot commented Feb 18, 2024

k8s-ci-robot commented Feb 18, 2024

yogeek commented Feb 18, 2024

k8s-ci-robot commented Feb 18, 2024

afbjorklund commented Feb 18, 2024 •

edited

Loading

yogeek commented Feb 19, 2024

afbjorklund commented Feb 19, 2024 •

edited

Loading

cri-tools 1.26 incompatible with containerd 1.5 #1130

cri-tools 1.26 incompatible with containerd 1.5 #1130

Comments

yogeek commented Apr 5, 2023

What happened:

What you expected to happen:

Anything else we need to know?:

Environment:

afbjorklund commented Apr 5, 2023 • edited Loading

yogeek commented Apr 5, 2023 • edited Loading

afbjorklund commented Apr 6, 2023 • edited Loading

afbjorklund commented Apr 6, 2023 • edited Loading

afbjorklund commented Apr 6, 2023 • edited Loading

afbjorklund commented Apr 6, 2023

afbjorklund commented Apr 11, 2023

yogeek commented Apr 11, 2023

afbjorklund commented Apr 12, 2023

yogeek commented Apr 12, 2023

afbjorklund commented Apr 12, 2023 • edited Loading

k8s-triage-robot commented Jul 11, 2023

k8s-triage-robot commented Jan 19, 2024

k8s-triage-robot commented Feb 18, 2024

k8s-ci-robot commented Feb 18, 2024

yogeek commented Feb 18, 2024

k8s-ci-robot commented Feb 18, 2024

afbjorklund commented Feb 18, 2024 • edited Loading

yogeek commented Feb 19, 2024

afbjorklund commented Feb 19, 2024 • edited Loading

afbjorklund commented Apr 5, 2023 •

edited

Loading

yogeek commented Apr 5, 2023 •

edited

Loading

afbjorklund commented Apr 6, 2023 •

edited

Loading

afbjorklund commented Apr 6, 2023 •

edited

Loading

afbjorklund commented Apr 6, 2023 •

edited

Loading

afbjorklund commented Apr 12, 2023 •

edited

Loading

afbjorklund commented Feb 18, 2024 •

edited

Loading

afbjorklund commented Feb 19, 2024 •

edited

Loading