Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configuration issues with containerd failing Kubespray install with "unknown service runtime.v1alpha2.ImageService" error #9956

Closed
supertetelman opened this issue Apr 4, 2023 · 6 comments
Closed

Configuration issues with containerd failing Kubespray install with "unknown service runtime.v1alpha2.ImageService" error #9956

supertetelman opened this issue Apr 4, 2023 · 6 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@supertetelman
Copy link

supertetelman commented Apr 4, 2023 •
edited
Loading

I am attempting to bump a project I managed up to the latest version of Kubernetes. Along with this change comes the containerd as the default runtime and I am running into version errors across the board in my CI system. The last set of issues I ran into turned out to be actual bugs which have since been patched in commits to the release branch. But I am now seeing several errors

This looks very similar to an issue over at https://serverfault.com/questions/1074008/containerd-1-4-9-unimplemented-desc-unknown-service-runtime-v1alpha2-runtimese where the issue seemed to be a containerd plugin.

Wondering if anyone has any ideas, I'm installing onto a freshlly installed fairly vanilla set of OSs and I am not making any changes to the kubespray code or making any major configuration changes to the group_vars.

This exact same setup is working fine with both docker and containerd as underlying runtimes in a previuos kubespray release.

  • OS:
  • Ubuntu 20.04
  • Ubuntu 18.04
  • CentOS 7.9

Kubespray version (commit) (git rev-parse --short HEAD):
Latest commit to the release-2.21 branch

commit: 08467ad

group_vars
Full configuration file can be found here: https://github.com/supertetelman/deepops/blob/containerd-test-update/config.example/group_vars/k8s-cluster.yml

There are not many changes around containerd other than:

containerd_insecure_registries:
  "registry.local:31500": "http://registry.local:31500"

Error:

`FAILED - RETRYING: download_container | Download image if required (1 retries left).
fatal: [virtual-gpu01-0 -> virtual-gpu01-0]: FAILED! => changed=true 
  attempts: 4
  cmd:
  - /usr/local/bin/crictl
  - pull
  - quay.io/calico/node:v3.24.5
  delta: '0:00:00.039323'
  end: '2023-03-29 03:03:57.565413'
  msg: non-zero return code
  rc: 1
  start: '2023-03-29 03:03:57.526090'
  stderr: |-
    E0329 03:03:57.563464   15688 remote_image.go:222] "PullImage from image service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService" image="quay.io/calico/node:v3.24.5"
    time="2023-03-29T03:03:57Z" level=fatal msg="pulling image: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService"
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>`

Full PR can be found here: NVIDIA/deepops#1253
@supertetelman supertetelman added the kind/bug Categorizes issue or PR as related to a bug. label Apr 4, 2023
@Ghostwritten
Copy link

I had the same problem：

  • kubespray-offline: v2.21.0
  • kubelet: v1.25.6
  • containerd: v1.6.15
Apr 14 10:36:03 kube-control-plan01 systemd[1]: kubelet.service: Service RestartSec=10s expired, scheduling restart.
Apr 14 10:36:03 kube-control-plan01 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 805.
Apr 14 10:36:03 kube-control-plan01 systemd[1]: Stopped Kubernetes Kubelet Server.
Apr 14 10:36:03 kube-control-plan01 systemd[1]: Started Kubernetes Kubelet Server.
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384661   11870 flags.go:64] FLAG: --add-dir-header="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384697   11870 flags.go:64] FLAG: --address="0.0.0.0"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384701   11870 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384705   11870 flags.go:64] FLAG: --alsologtostderr="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384707   11870 flags.go:64] FLAG: --anonymous-auth="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384709   11870 flags.go:64] FLAG: --application-metrics-count-limit="100"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384711   11870 flags.go:64] FLAG: --authentication-token-webhook="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384713   11870 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384716   11870 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384719   11870 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384721   11870 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384723   11870 flags.go:64] FLAG: --azure-container-registry-config=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384725   11870 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384727   11870 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/bootstrap-kubelet.conf"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384730   11870 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384732   11870 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384734   11870 flags.go:64] FLAG: --cgroup-root=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384735   11870 flags.go:64] FLAG: --cgroups-per-qos="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384737   11870 flags.go:64] FLAG: --client-ca-file=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384739   11870 flags.go:64] FLAG: --cloud-config=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384741   11870 flags.go:64] FLAG: --cloud-provider=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384742   11870 flags.go:64] FLAG: --cluster-dns="[]"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384745   11870 flags.go:64] FLAG: --cluster-domain=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384747   11870 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet-config.yaml"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384749   11870 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384751   11870 flags.go:64] FLAG: --container-log-max-files="5"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384754   11870 flags.go:64] FLAG: --container-log-max-size="10Mi"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384756   11870 flags.go:64] FLAG: --container-runtime="remote"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384757   11870 flags.go:64] FLAG: --container-runtime-endpoint="unix:///var/run/containerd/containerd.sock"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384760   11870 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384764   11870 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384766   11870 flags.go:64] FLAG: --contention-profiling="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384768   11870 flags.go:64] FLAG: --cpu-cfs-quota="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384769   11870 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384771   11870 flags.go:64] FLAG: --cpu-manager-policy="none"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384773   11870 flags.go:64] FLAG: --cpu-manager-policy-options=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384778   11870 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384781   11870 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384782   11870 flags.go:64] FLAG: --enable-debugging-handlers="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384785   11870 flags.go:64] FLAG: --enable-load-reader="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384788   11870 flags.go:64] FLAG: --enable-server="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384792   11870 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384796   11870 flags.go:64] FLAG: --event-burst="10"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384799   11870 flags.go:64] FLAG: --event-qps="5"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384803   11870 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384805   11870 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384809   11870 flags.go:64] FLAG: --eviction-hard="imagefs.available<15%,memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384816   11870 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384818   11870 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384820   11870 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384822   11870 flags.go:64] FLAG: --eviction-soft=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384824   11870 flags.go:64] FLAG: --eviction-soft-grace-period=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384826   11870 flags.go:64] FLAG: --exit-on-lock-contention="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384828   11870 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384830   11870 flags.go:64] FLAG: --experimental-mounter-path=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384832   11870 flags.go:64] FLAG: --fail-swap-on="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384833   11870 flags.go:64] FLAG: --feature-gates=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384836   11870 flags.go:64] FLAG: --file-check-frequency="20s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384838   11870 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384840   11870 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384842   11870 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384844   11870 flags.go:64] FLAG: --healthz-port="10248"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384847   11870 flags.go:64] FLAG: --help="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384849   11870 flags.go:64] FLAG: --hostname-override="kube-control-plan01"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384851   11870 flags.go:64] FLAG: --housekeeping-interval="10s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384853   11870 flags.go:64] FLAG: --http-check-frequency="20s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384855   11870 flags.go:64] FLAG: --image-credential-provider-bin-dir=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384857   11870 flags.go:64] FLAG: --image-credential-provider-config=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384858   11870 flags.go:64] FLAG: --image-gc-high-threshold="85"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384860   11870 flags.go:64] FLAG: --image-gc-low-threshold="80"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384862   11870 flags.go:64] FLAG: --image-service-endpoint=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384864   11870 flags.go:64] FLAG: --iptables-drop-bit="15"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384866   11870 flags.go:64] FLAG: --iptables-masquerade-bit="14"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384867   11870 flags.go:64] FLAG: --keep-terminated-pod-volumes="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384869   11870 flags.go:64] FLAG: --kernel-memcg-notification="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384871   11870 flags.go:64] FLAG: --kube-api-burst="10"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384873   11870 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384875   11870 flags.go:64] FLAG: --kube-api-qps="5"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384878   11870 flags.go:64] FLAG: --kube-reserved=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384880   11870 flags.go:64] FLAG: --kube-reserved-cgroup=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384882   11870 flags.go:64] FLAG: --kubeconfig="/etc/kubernetes/kubelet.conf"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384884   11870 flags.go:64] FLAG: --kubelet-cgroups=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384886   11870 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384887   11870 flags.go:64] FLAG: --lock-file=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384889   11870 flags.go:64] FLAG: --log-backtrace-at=":0"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384892   11870 flags.go:64] FLAG: --log-cadvisor-usage="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384894   11870 flags.go:64] FLAG: --log-dir=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384895   11870 flags.go:64] FLAG: --log-file=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384897   11870 flags.go:64] FLAG: --log-file-max-size="1800"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384899   11870 flags.go:64] FLAG: --log-flush-frequency="5s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384901   11870 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384905   11870 flags.go:64] FLAG: --log-json-split-stream="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384907   11870 flags.go:64] FLAG: --logging-format="text"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384909   11870 flags.go:64] FLAG: --logtostderr="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384912   11870 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384914   11870 flags.go:64] FLAG: --make-iptables-util-chains="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384916   11870 flags.go:64] FLAG: --manifest-url=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384918   11870 flags.go:64] FLAG: --manifest-url-header=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384920   11870 flags.go:64] FLAG: --master-service-namespace="default"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384922   11870 flags.go:64] FLAG: --max-open-files="1000000"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384924   11870 flags.go:64] FLAG: --max-pods="110"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384926   11870 flags.go:64] FLAG: --maximum-dead-containers="-1"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384928   11870 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384930   11870 flags.go:64] FLAG: --memory-manager-policy="None"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384932   11870 flags.go:64] FLAG: --minimum-container-ttl-duration="0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384934   11870 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384936   11870 flags.go:64] FLAG: --node-ip="100.168.110.21"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384937   11870 flags.go:64] FLAG: --node-labels=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384940   11870 flags.go:64] FLAG: --node-status-max-images="50"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384942   11870 flags.go:64] FLAG: --node-status-update-frequency="10s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384944   11870 flags.go:64] FLAG: --one-output="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384947   11870 flags.go:64] FLAG: --oom-score-adj="-999"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384949   11870 flags.go:64] FLAG: --pod-cidr=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384951   11870 flags.go:64] FLAG: --pod-infra-container-image="registry.k8s.io/pause:3.8"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384953   11870 flags.go:64] FLAG: --pod-manifest-path=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384955   11870 flags.go:64] FLAG: --pod-max-pids="-1"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384957   11870 flags.go:64] FLAG: --pods-per-core="0"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384959   11870 flags.go:64] FLAG: --port="10250"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384961   11870 flags.go:64] FLAG: --protect-kernel-defaults="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384962   11870 flags.go:64] FLAG: --provider-id=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384964   11870 flags.go:64] FLAG: --qos-reserved=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384966   11870 flags.go:64] FLAG: --read-only-port="10255"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384968   11870 flags.go:64] FLAG: --register-node="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384970   11870 flags.go:64] FLAG: --register-schedulable="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384972   11870 flags.go:64] FLAG: --register-with-taints=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384974   11870 flags.go:64] FLAG: --registry-burst="10"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384976   11870 flags.go:64] FLAG: --registry-qps="5"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384979   11870 flags.go:64] FLAG: --reserved-cpus=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384981   11870 flags.go:64] FLAG: --reserved-memory=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384983   11870 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384985   11870 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384987   11870 flags.go:64] FLAG: --rotate-certificates="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384988   11870 flags.go:64] FLAG: --rotate-server-certificates="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384990   11870 flags.go:64] FLAG: --runonce="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384992   11870 flags.go:64] FLAG: --runtime-cgroups="/etc/systemd/system/kubelet.service"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384994   11870 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384996   11870 flags.go:64] FLAG: --seccomp-default="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.384998   11870 flags.go:64] FLAG: --serialize-image-pulls="true"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385000   11870 flags.go:64] FLAG: --skip-headers="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385001   11870 flags.go:64] FLAG: --skip-log-headers="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385003   11870 flags.go:64] FLAG: --stderrthreshold="2"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385005   11870 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385007   11870 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385011   11870 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385013   11870 flags.go:64] FLAG: --storage-driver-password="root"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385014   11870 flags.go:64] FLAG: --storage-driver-secure="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385016   11870 flags.go:64] FLAG: --storage-driver-table="stats"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385018   11870 flags.go:64] FLAG: --storage-driver-user="root"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385020   11870 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385022   11870 flags.go:64] FLAG: --sync-frequency="1m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385024   11870 flags.go:64] FLAG: --system-cgroups=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385026   11870 flags.go:64] FLAG: --system-reserved=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385028   11870 flags.go:64] FLAG: --system-reserved-cgroup=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385030   11870 flags.go:64] FLAG: --tls-cert-file=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385031   11870 flags.go:64] FLAG: --tls-cipher-suites="[]"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385034   11870 flags.go:64] FLAG: --tls-min-version=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385036   11870 flags.go:64] FLAG: --tls-private-key-file=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385038   11870 flags.go:64] FLAG: --topology-manager-policy="none"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385040   11870 flags.go:64] FLAG: --topology-manager-scope="container"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385043   11870 flags.go:64] FLAG: --v="2"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385046   11870 flags.go:64] FLAG: --version="false"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385048   11870 flags.go:64] FLAG: --vmodule=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385051   11870 flags.go:64] FLAG: --volume-plugin-dir="/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385053   11870 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.385088   11870 feature_gate.go:245] feature gates: &{map[]}
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.386326   11870 server.go:413] "Kubelet version" kubeletVersion="v1.25.6"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.386337   11870 server.go:415] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.386365   11870 feature_gate.go:245] feature gates: &{map[]}
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.386413   11870 feature_gate.go:245] feature gates: &{map[]}
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.386480   11870 server.go:825] "Client rotation is on, will bootstrap in background"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387259   11870 bootstrap.go:84] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387300   11870 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387416   11870 server.go:882] "Starting client certificate rotation"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387428   11870 certificate_manager.go:270] kubernetes.io/kube-apiserver-client-kubelet: Certificate rotation is enabled
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387583   11870 certificate_manager.go:270] kubernetes.io/kube-apiserver-client-kubelet: Certificate expiration is 2024-04-12 10:05:52 +0000 UTC, rotation deadline is 2024-02-25 08:51:10.513342613 +0000 UTC
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387613   11870 certificate_manager.go:270] kubernetes.io/kube-apiserver-client-kubelet: Waiting 7614h15m7.125735599s for next certificate rotation
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387666   11870 dynamic_cafile_content.go:119] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/ssl/ca.crt"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387754   11870 manager.go:163] cAdvisor running in container: "/system.slice/kubelet.service"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387779   11870 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/ssl/ca.crt"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.387989   11870 fs.go:133] Filesystem UUIDs: map[2da215e7-81e6-4cf5-a021-4c615bc68d5f:/dev/dm-0 63b8909c-4e91-4fac-b133-50e757adf655:/dev/dm-2 6858317f-1156-4bd8-a242-b12c1258a155:/dev/dm-1 b16c6145-8899-4c77-aa30-9d5d265515dc:/dev/nvme0n1p1]
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.388006   11870 fs.go:134] Filesystem partitions: map[/dev/mapper/rl-home:{mountpoint:/home major:253 minor:2 fsType:xfs blockSize:0} /dev/mapper/rl-root:{mountpoint:/ major:253 minor:0 fsType:xfs blockSize:0} /dev/nvme0n1p1:{mountpoint:/boot major:259 minor:1 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /run/user/0:{mountpoint:/run/user/0 major:0 minor:45 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:25 fsType:tmpfs blockSize:0}]
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.388025   11870 nvidia.go:54] NVIDIA GPU metrics disabled
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390141   11870 manager.go:212] Machine: {Timestamp:2023-04-14 10:36:03.389986606 +0800 CST m=+0.033238469 CPUVendorID:GenuineIntel NumCores:4 NumPhysicalCores:1 NumSockets:4 CpuFrequency:3792000 MemoryCapacity:8112312320 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:e9622bdcc20448f68e3da7961a207415 SystemUUID:98a10142-d36f-0178-5a75-5b97b28505eb BootID:06d67049-4167-4411-b140-8d9d31a4dd0b Filesystems:[{Device:/dev/mapper/rl-home DeviceMajor:253 DeviceMinor:2 Capacity:33466781696 Type:vfs Inodes:16349184 HasInodes:true} {Device:/run/user/0 DeviceMajor:0 DeviceMinor:45 Capacity:811229184 Type:vfs Inodes:990272 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:4056154112 Type:vfs Inodes:990272 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:4056154112 Type:vfs Inodes:990272 HasInodes:true} {Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:25 Capacity:4056154112 Type:vfs Inodes:990272 HasInodes:true} {Device:/dev/mapper/rl-root DeviceMajor:253 DeviceMinor:0 Capacity:68551770112 Type:vfs Inodes:33488896 HasInodes:true} {Device:/dev/nvme0n1p1 DeviceMajor:259 DeviceMinor:1 Capacity:1063256064 Type:vfs Inodes:524288 HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size:68585259008 Scheduler:none} 253:1:{Name:dm-1 Major:253 Minor:1 Size:4227858432 Scheduler:none} 253:2:{Name:dm-2 Major:253 Minor:2 Size:33483128832 Scheduler:none} 259:0:{Name:nvme0n1 Major:259 Minor:0 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:bond0 MacAddress:00:50:56:81:17:43 Speed:10000 Mtu:1500} {Name:ens160 MacAddress:00:50:56:81:17:43 Speed:10000 Mtu:1500} {Name:ens224 MacAddress:00:50:56:81:17:43 Speed:10000 Mtu:1500}] Topology:[{Id:0 Memory:8112312320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:262144 Type:Unified Level:2}] UncoreCaches:[{Id:0 Size:16777216 Type:Unified Level:3}] SocketID:0} {Id:0 Threads:[1] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:262144 Type:Unified Level:2}] UncoreCaches:[{Id:2 Size:16777216 Type:Unified Level:3}] SocketID:2} {Id:0 Threads:[2] Caches:[{Id:4 Size:32768 Type:Data Level:1} {Id:4 Size:32768 Type:Instruction Level:1} {Id:4 Size:262144 Type:Unified Level:2}] UncoreCaches:[{Id:4 Size:16777216 Type:Unified Level:3}] SocketID:4} {Id:0 Threads:[3] Caches:[{Id:6 Size:32768 Type:Data Level:1} {Id:6 Size:32768 Type:Instruction Level:1} {Id:6 Size:262144 Type:Unified Level:2}] UncoreCaches:[{Id:6 Size:16777216 Type:Unified Level:3}] SocketID:6}] Caches:[]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390202   11870 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390305   11870 manager.go:228] Version: {KernelVersion:4.18.0-425.13.1.el8_7.x86_64 ContainerOsVersion:Rocky Linux 8.7 (Green Obsidian) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390366   11870 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390565   11870 container_manager_linux.go:262] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390613   11870 container_manager_linux.go:267] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName:/etc/systemd/system/kubelet.service SystemCgroupsName: KubeletCgroupsName:/kube.slice/kubelet.service KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:true NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390632   11870 topology_manager.go:134] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390642   11870 container_manager_linux.go:302] "Creating device plugin manager" devicePluginEnabled=true
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390654   11870 manager.go:127] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390661   11870 server.go:64] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.390686   11870 state_mem.go:36] "Initialized new in-memory state store"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.391793   11870 remote_runtime.go:142] "Falling back to CRI v1alpha2 runtime API (deprecated)"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.392151   11870 remote_image.go:98] "Falling back to CRI v1alpha2 image API (deprecated)"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.392170   11870 server.go:1136] "Using root directory" path="/var/lib/kubelet"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.392209   11870 kubelet.go:381] "Attempting to sync node with API server"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.392217   11870 kubelet.go:270] "Adding static pod path" path="/etc/kubernetes/manifests"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.392225   11870 file.go:68] "Watching path" path="/etc/kubernetes/manifests"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.392231   11870 kubelet.go:281] "Adding apiserver pod source"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: I0414 10:36:03.392237   11870 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: W0414 10:36:03.394860   11870 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get "https://100.168.110.21:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkube-control-plan01&limit=500&resourceVersion=0": dial tcp 100.168.110.21:6443: connect: connection refused
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: E0414 10:36:03.394902   11870 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://100.168.110.21:6443/api/v1/nodes?fieldSelector=metadata.name%3Dkube-control-plan01&limit=500&resourceVersion=0": dial tcp 100.168.110.21:6443: connect: connection refused
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: W0414 10:36:03.394929   11870 reflector.go:424] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get "https://100.168.110.21:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 100.168.110.21:6443: connect: connection refused
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: E0414 10:36:03.394951   11870 reflector.go:140] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://100.168.110.21:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp 100.168.110.21:6443: connect: connection refused
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: E0414 10:36:03.395309   11870 remote_runtime.go:189] "Version from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: E0414 10:36:03.395338   11870 kuberuntime_manager.go:226] "Get runtime version failed" err="get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
Apr 14 10:36:03 kube-control-plan01 kubelet[11870]: E0414 10:36:03.395353   11870 run.go:74] "command failed" err="failed to run Kubelet: failed to create kubelet: get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
Apr 14 10:36:03 kube-control-plan01 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Apr 14 10:36:03 kube-control-plan01 systemd[1]: kubelet.service: Failed with result 'exit-code'.

config.toml

version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
oom_score = 0

[grpc]
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  level = "info"

[metrics]
  address = ""
  grpc_histogram = false

[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "100.168.110.199:35000/pause:3.8"
    max_container_log_line_size = -1
    enable_unprivileged_ports = false
    enable_unprivileged_icmp = false
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      snapshotter = "overlayfs"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          runtime_engine = ""
          runtime_root = ""
          base_runtime_spec = "/etc/containerd/cri-base.json"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            systemdCgroup = true
    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = "/etc/containerd/certs.d"
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."100.168.110.199:35000"]
          endpoint = ["http://100.168.110.199:35000"]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."100.168.110.199:35000".tls]
          insecure_skip_verify = true

if I run:

rm -f config.toml
systemctl restart containerd.service ;systemctl status containerd.service
systemctl restart kubelet.service ;systemctl status kubelet.service

of course,I am ok, But how does my private image repository work! so , I don't know what to do😫

@leonbartlett
Copy link

leonbartlett commented May 23, 2023 •
edited
Loading

I think the problem is this #9566
It has added config_path and creates config files under the new certs.d location, but it has left the deprecated config registry.configs and registry.mirrors, and you can't have both.

https://github.com/containerd/containerd/blob/release/1.6/docs/cri/config.md

# If config_path is not provided defaults are used.
#
# *** registry.configs and registry.mirrors that were a part of containerd 1.4
# are now DEPRECATED and will only be used if the config_path is not specified.
config_path = ""

I resolved the issue by removing the following lines from the config.toml template

{% if containerd_insecure_registries is defined and containerd_insecure_registries|length>0 %}
config_path = "{{ containerd_cfg_dir }}/certs.d"
{% endif %}

@troodes
Copy link

troodes commented Jun 2, 2023

Appears to be fixed in #9770

@ErikJiang
Copy link
Member

I'm having the same problem on the release-2.21 branch, and have now submitted a PR to that branch.
#10665

@VannTen
Copy link
Contributor

VannTen commented Jan 22, 2024

Both above PR being merged, the issues appears to have been fixed.
Closing
/close

@k8s-ci-robot
Copy link
Contributor

@VannTen: Closing this issue.

In response to this:

Both above PR being merged, the issues appears to have been fixed.
Closing
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@supertetelman @leonbartlett @ErikJiang @VannTen @Ghostwritten @k8s-ci-robot @troodes