Upgrade to v18.4 failed, cannot install both kubelet-1.18.4-1.x86_64 and kubelet-1.18.4-0.x86_64

[StefanSa]

What happened:
on the master centos-8 there is the following error message when trying to update the packages.

Problem: cannot install both kubelet-1.18.4-1.x86_64 and kubelet-1.18.4-0.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.7.5-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.3-0.x86_64

We don't see this problem in the nodes centos-7
Likewise, we did not see this error in the master centtos-8 in the last update to v18.3.

How to reproduce it (as minimally and precisely as possible):
yum update / dnf update

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: vmware
• OS (e.g: cat /etc/os-release): centos-8

[StefanSa]

/sig Release

[StefanSa]

This helps me as a short-term solution.
yum update --skip-broken --nobest

but does not fix the actual packet conflict.

yum update
 Problem: cannot install both kubelet-1.18.4-0.x86_64 and kubelet-1.18.4-1.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.8.6-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.4-1.x86_64

[justaugustus]

@StefanSa -- What happens when you completely remove kubelet and kubernetes-cni and then attempt to install one of the following kubelets:

• kubelet-1.18.5-0
• kubelet-1.18.4-1

(Related to kubernetes/kubernetes#92242.)
/assign
/area release-eng
cc: @kubernetes/release-engineering

[StefanSa]

@justaugustus
i remove kubelet and kubernetes-cniand installed it again successfully with:
yum install kubelet-1.18.5 kubeadm-1.18.5, however, a yum update still fails.

yum update
Problem: cannot install both kubelet-1.18.4-0.x86_64 and kubelet-1.18.5-0.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.8.6-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.5-0.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken'
to skip uninstallable packages or '--nobest' to use not only best candidate packages)

As already mentioned, we don't see that on our nodes that are based on centos 7, currently we only see this problem with the master that is based on centos 8.

[mrbobbytables]

There does still seem to be an issue on centos8 I had the same issue when testing with 1.17.8, 1.18.1, 1.18.3, and 1.18.5

root@a77d16cf886f /]# yum info kubelet
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:06:28 ago on Thu Jul 30 12:22:37 2020.
Installed Packages
Name         : kubelet
Version      : 1.18.5
Release      : 0
Architecture : x86_64
Size         : 108 M
Source       : kubelet-1.18.5-0.src.rpm
Repository   : @System
From repo    : kubernetes
Summary      : Container cluster management
URL          : https://kubernetes.io
License      : ASL 2.0
Description  : The node agent of Kubernetes, the container cluster manager.

Available Packages
Name         : kubelet
Version      : 1.18.6
Release      : 0
Architecture : x86_64
Size         : 21 M
Source       : kubelet-1.18.6-0.src.rpm
Repository   : kubernetes
Summary      : Container cluster management
URL          : https://kubernetes.io
License      : ASL 2.0
Description  : The node agent of Kubernetes, the container cluster manager.

[root@a77d16cf886f /]# yum update
Failed to set locale, defaulting to C.UTF-8
Last metadata expiration check: 0:06:33 ago on Thu Jul 30 12:22:37 2020.
Error:
 Problem: cannot install both kubelet-1.18.6-0.x86_64 and kubelet-1.18.4-0.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.8.6-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.5-0.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

[huataihuang]

I still had same issue with 1.18.6 upgade to 1.19.0.

I use centos 8.

#dnf update
...
Error:
 Problem: cannot install both kubelet-1.19.0-0.x86_64 and kubelet-1.18.4-0.x86_64
  - cannot install the best update candidate for package kubernetes-cni-0.8.6-0.x86_64
  - cannot install the best update candidate for package kubelet-1.18.6-0.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

#dnf info kubelet
Last metadata expiration check: 0:39:13 ago on Wed 02 Sep 2020 09:18:41 AM CST.
Installed Packages
Name         : kubelet
Version      : 1.18.6
Release      : 0
Architecture : x86_64
Size         : 108 M
Source       : kubelet-1.18.6-0.src.rpm
Repository   : @System
From repo    : kubernetes
Summary      : Container cluster management
URL          : https://kubernetes.io
License      : ASL 2.0
Description  : The node agent of Kubernetes, the container cluster manager.

Available Packages
Name         : kubelet
Version      : 1.19.0
Release      : 0
Architecture : x86_64
Size         : 19 M
Source       : kubelet-1.19.0-0.src.rpm
Repository   : kubernetes
Summary      : Container cluster management
URL          : https://kubernetes.io
License      : ASL 2.0
Description  : The node agent of Kubernetes, the container cluster manager.

[huataihuang]

I add follow config to /etc/yum.conf to exclude conflict kubelet

exclude=kubelet-1.18.4-0 kubelet-1.17.7-0 kubelet-1.16.11-0

then use dnf upgrade to upgrade kubelet 1.19.0 without problem.

I hope it can help you.

[laundry-96]

@huataihuang What repository url do you have? I've been using https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 (which is what the kubernetes website tells me to use) and it doesn't seem to have kubelet-1.19 in it...

[huataihuang]

@huataihuang What repository url do you have? I've been using https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 (which is what the kubernetes website tells me to use) and it doesn't seem to have kubelet-1.19 in it...

https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

here is my /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

I was follow kubernetes website guide to deploy, and I add exclude config by my experience, maybe you not meet this version conflict.

[justaugustus]

/assign @saschagrunert
/unassign

Sascha -- can you take a look or assign to someone on @kubernetes/release-engineering to investigate?

The following issues are similar classes of issues for apt/yum:

• Kubeadm upgrade from 1.18.4-00 to 1.18.5-00 fails on Ubuntu 20.04 due to unmet dependencies and package conflicts kubernetes#92698
• Installation of kubeadm=1.13.5* and kubectl=1.13.5* apt packages is broken kubernetes#92587
• Installation of 1.13.5 kubeadm and kubelet on Centos/RHEL broken. kubeadm-1.13.5 is installing kubelet-1.18.5 as dependency kubernetes#92586
• Upgrade to v18.4 failed, cannot install both kubelet-1.18.4-1.x86_64 and kubelet-1.18.4-0.x86_64 #3044

[saschagrunert]

/help
Not sure if we can solve this without re-building those packages.

[k8s-ci-robot]

@saschagrunert:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help
Not sure if we can solve this without re-building those packages.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aimcod]

I add follow config to /etc/yum.conf to exclude conflict kubelet

exclude=kubelet-1.18.4-0 kubelet-1.17.7-0 kubelet-1.16.11-0

then use dnf upgrade to upgrade kubelet 1.19.0 without problem.

I hope it can help you.

It did help, quite a lot. Thanks for this.

What I ended up doing, was to exclude=kubelet-1.1*

I am adding this to my ansible kubernetes deployment playbooks. Sweet and short.

Thanks a ton!

[mchoeti]

This is it thanks !!!

[nickson]

I have the same issue with Alma Linux 9.1:

dnf update
...  
 Problem: cannot install both kubelet-1.26.1-0.x86_64 and kubelet-1.18.4-0.x86_64

The workaround with the exclude fixes it:

tail -1 /etc/yum.repos.d/kubernetes.repo  
exclude = kubelet-1.18.* kubelet-1.17.* kubelet-1.16.*

[delfws]

same queastion in CentOS 9 Stream

root in ~
≥ uname -a                                                                                                         9:01
Linux master 5.14.0-307.el9.x86_64 kubernetes/kubernetes#1 SMP PREEMPT_DYNAMIC Wed May 3 06:16:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

root in ~
≥ yum upgrade                                                                                                      8:44
上次元数据过期检查：0:06:42 前，执行于 2023年05月10日 星期三 08时38分13秒。
错误：
 问题: 无法同时安装 kubelet-1.18.4-0.x86_64 和 kubelet-1.27.1-0.x86_64
  - 无法为软件包安装最佳更新候选 kubernetes-cni-1.2.0-0.x86_64
  - 无法为软件包安装最佳更新候选 kubelet-1.27.1-0.x86_64
(尝试在命令行中添加 '--allowerasing' 来替换冲突的软件包 或 '--skip-broken' 来跳过无法安装的软件包 或 '--nobest' 来不只使用软件包的最佳候选)

[aojea]

1.18 reach the end of life and we are not supporting this version anymore.
I guess we can close this issue.

last supported version is 1.24 https://kubernetes.io/releases/

There is no support for upgrading since 1.18 to any of the supported versions https://kubernetes.io/releases/version-skew-policy/#supported-component-upgrade-order

/close

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

1.18 reach the end of life and we are not supporting this version anymore.
I guess we can close this issue.

last supported version is 1.24 https://kubernetes.io/releases/

There is no support for upgrading since 1.18 to any of the supported versions https://kubernetes.io/releases/version-skew-policy/#supported-component-upgrade-order

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[shawnz]

@aojea I think there's a misunderstanding. This bug applies to the latest versions even though the error message presented incorrectly mentions 1.18. The problem is that running "yum upgrade" will attempt to install 1.18 even if you are using a much newer version already.

[vrusinov]

Yes, the bug is not related to whether 1.18 is supported or not. It still happens when trying to update to latest Kubernetes. Please reopen.

PS: actually removing unsupported Kubernetes versions from the main yum repo may help. Another option is to have separate repositories for each of the major releases.

/reopen

[k8s-ci-robot]

@vrusinov: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

Yes, the bug is not related to whether 1.18 is supported or not. It still happens when trying to update to latest Kubernetes. Please reopen.

PS: actually removing unsupported Kubernetes versions from the main yum repo may help. Another option is to have separate repositories for each of the major releases.

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ViliusS]

@aojea please reopen. The title of this issue is incorrect, this happens when trying to update to any latest version. Even from 1.25.x to 1.26.x.

[aojea]

@aojea please reopen. The title of this issue is incorrect, this happens when trying to update to any latest version. Even from 1.25.x to 1.26.x.

all the snippets in the code are with 1.18, can you paste a snippet with those versions ?

[vrusinov]

@aojea #3044 is trying to install 1.26.

[aojea]

install 1.26 ... from 1.18

anything that is not on a supported version (latest 1.24) and within the supported skew (n-2 versions respect the control-plane) is not supported #3044

[shawnz]

@aojea No, the issue occurs no matter what the original and target versions are. No matter what version you are upgrading from or to, 1.18 will incorrectly get marked for installation. The issue does not require 1.18 to be the original or target version in the upgrade. Even if you are upgrading from 1.25 to 1.26 for example, it will still mark 1.18 to be installed if you use "yum update".

[aojea]

Even if you are upgrading from 1.25 to 1.26 for example, it will still mark 1.18 to be installed if you use "yum update".

lol, I didn't understand that , sorry

/reopen

do we have 1.18 hardcoded somewhere?
/sig release

[k8s-ci-robot]

@aojea: Reopened this issue.

In response to this:

Even if you are upgrading from 1.25 to 1.26 for example, it will still mark 1.18 to be installed if you use "yum update".

lol, I didn't understand that , sorry

/reopen

do we have 1.18 hardcoded somewhere?
/sig release

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aojea]

I think that the rpms sources are here https://github.com/kubernetes/release/tree/master/packages/rpm

/transfer kubernetes/release

[k8s-ci-robot]

@aojea: Something went wrong or the destination repo kubernetes/kubernetes/release does not exist.

In response to this:

I think that the rpms sources are here https://github.com/kubernetes/release/tree/master/packages/rpm

/transfer kubernetes/release

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aojea]

/transfer release

[ViliusS]

Even if you are upgrading from 1.25 to 1.26 for example, it will still mark 1.18 to be installed if you use "yum update".

lol, I didn't understand that , sorry

/reopen

do we have 1.18 hardcoded somewhere? /sig release

I could be wrong here, but I don't think it is an issue with RPM themselves, but rather how dependency tree is generated on Kubernetes' Yum repository.

Probably an easy way to check, just remove those 1.16.x, 1.17.x, and 1.18.x RPMs from repository, and regenerate yum dependencies. Those old versions should not be used anyway.

[jamison-maxwell]

Can confirm that the behavior suspected by @ViliusS above is indeed correct. I setup a local copy of the Kubernetes repository with only the latest packages. As demonstrated by the output below, when the repository only contains the 1.27.x packages, there is no dependency issue.

[root@tempk8s ~]# dnf list installed | grep @kubernetes
cri-tools.x86_64                      1.26.0-0                       @kubernetes
kubeadm.x86_64                        1.25.8-0                       @kubernetes
kubectl.x86_64                        1.25.8-0                       @kubernetes
kubelet.x86_64                        1.25.8-0                       @kubernetes
kubernetes-cni.x86_64                 1.2.0-0                        @kubernetes

-----------------------------------

[root@tempk8s ~]# dnf repo-pkgs kubernetes list | egrep -v ^Last | column -t
Installed              Packages
cri-tools.x86_64       1.26.0-0  @kubernetes
kubeadm.x86_64         1.25.8-0  @kubernetes
kubectl.x86_64         1.25.8-0  @kubernetes
kubelet.x86_64         1.25.8-0  @kubernetes
kubernetes-cni.x86_64  1.2.0-0   @kubernetes
Available              Packages
kubeadm.x86_64         1.27.3-0  kubernetes
kubectl.x86_64         1.27.3-0  kubernetes
kubelet.x86_64         1.27.3-0  kubernetes
rkt.x86_64             1.27.0-1  kubernetes

-----------------------------------

[root@tempk8s ~]# yum update
Last metadata expiration check: 0:25:15 ago on Fri 07 Jul 2023 09:40:45 PM EDT.
Dependencies resolved.
====================================================================================================================================================================================
 Package                                                 Architecture                                           Version                                                   Repository
====================================================================================================================================================================================
Upgrading:
 kubeadm                                                 x86_64                                                 1.27.3-0                                                  kubernetes
 kubectl                                                 x86_64                                                 1.27.3-0                                                  kubernetes
 kubelet                                                 x86_64                                                 1.27.3-0                                                  kubernetes

Also, using the official repository that contains all versions, I am able to reproduce the behavior initial indicated in this issue.

[ViliusS]

As new repositories do not include Kubernetes 1.16, 1.17, or 1.18 this issue can be closed.

[solutionstack]

I add follow config to /etc/yum.conf to exclude conflict kubelet

exclude=kubelet-1.18.4-0 kubelet-1.17.7-0 kubelet-1.16.11-0

then use dnf upgrade to upgrade kubelet 1.19.0 without problem.

I hope it can help you.

Still facing similar in 2024, I used a wild card exclude to ignore all 1.1 versions

exclude=kubelet-1.1*

[lumarel]

@solutionstack I don't expect there being any further change on the repo content here, if you didn't notice the location of the repos has been moved last year to a different location, with individual versioned repos, and the old content got frozen.