CKS cluster doesn't come up if "cloud.kubernetes.cluster.experimental.features.enabled" setting is enabled

[kiranchavala]

problem

CKS cluster doesn't come up if "cloud.kubernetes.cluster.experimental.features.enabled" setting is enabled

versions

ACS 4.20.1, 4.21

The steps to reproduce the bug

1. Enable the global setting "cloud.kubernetes.cluster.experimental.features.enabled" setting

2. Launch a cks cluster, enable private registry and provide your docker hub credentials

dockerregistryurl= "https://index.docker.io/v1/"
dockerregistryusername= <>
dockerregistrypassword=<>

3. CKS cluster is struck in starting state

4. On logging into control node found the following errors

root@example-cluster-control-19927ba5b6e:/opt/bin# crictl images
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/opt/bin/crictl.yaml"
WARN[0000] Image connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///run/crio/crio.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/crio/crio.sock: connect: no such file or directory"
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: no such file or directory"
FATA[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: no such file or directory"

kublet logs

Sep 08 05:18:03 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 08 05:18:03 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 08 05:18:13 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 41.
Sep 08 05:18:13 example-cluster-control-19927ba5b6e systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:13 example-cluster-control-19927ba5b6e systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:14 example-cluster-control-19927ba5b6e kubelet[31660]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's -->
Sep 08 05:18:14 example-cluster-control-19927ba5b6e kubelet[31660]: E0908 05:18:14.034670   31660 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubele>
Sep 08 05:18:14 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 08 05:18:14 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 42.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e kubelet[32393]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's -->
Sep 08 05:18:24 example-cluster-control-19927ba5b6e kubelet[32393]: E0908 05:18:24.237711   32393 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubele>

The file

"/etc/containerd/config.toml" is not populated correctly

What to do about it?

Cloudstack should support private container registry

[weizhouapache]

@kiranchavala
can you share the kubelet config in /var/lib/kubelet ?

[kiranchavala]

@weizhouapache there is no file /var/lib/kubelet and didn't generate

Sep 09 05:51:19 test-private-control-1992cfb0b17 kubelet[50328]: E0909 05:51:19.086110   50328 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Sep 09 05:51:19 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 05:51:19 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 67.
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 09 05:51:29 test-private-control-1992cfb0b17 kubelet[51068]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:51:29 test-private-control-1992cfb0b17 kubelet[51068]: E0909 05:51:29.237105   51068 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 68.
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 09 05:51:39 test-private-control-1992cfb0b17 kubelet[51779]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:51:39 test-private-control-1992cfb0b17 kubelet[51779]: E0909 05:51:39.373116   51779 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Failed with result 'exit-code'

root@test-private-control-1992cfb0b17:/var/lib/containerd# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Tue 2025-09-09 05:50:58 UTC; 5s ago
       Docs: http://kubernetes.io/docs/
    Process: 48881 ExecStart=/opt/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code>
   Main PID: 48881 (code=exited, status=1/FAILURE)
        CPU: 99ms

Sep 09 05:50:58 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 05:50:58 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Failed with result 'exit-code'.