[Release-1.27] - Support for containerd 1.5+ config_path: On-the-fly container registry configuration

[brandond]

Backport fix for Support for containerd 1.5+ config_path: On-the-fly container registry configuration

• Support for containerd 1.5+ config_path: On-the-fly container registry configuration #5568

[aganesh-suse]

Validated on release-1.27 branch with version v1.27.10+k3s2

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent

Config.yaml:

system-default-registry: test-private-registry.com
debug: true
token: xxxx
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server

Registries.yaml

$ cat /etc/rancher/rke2/registries.yaml
mirrors:
  test-private-registry.com:
    endpoint:
      - https://test-private-registry.com  
  docker.io:
    endpoint:
      - https://test-private-registry.com
  k8s.gcr.io:
    endpoint:
      - https://test-private-registry.com
configs:
  test-private-registry.com:
    auth:
      username: user
      password: password
    tls:
      ca_file: /home/ubuntu/ca.pem

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

Copy registries.yaml to /etc/rancher/k3s/registries.yaml
copy ca.pem to the user home directory (as per path provided in the registries.yaml file).
2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_VERSION='v1.27.10+k3s2' sh -s - server

3. Verify content of config.toml file for config_path:

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml

4. Verify content of config_path directory and the corresponding host.toml files:

 $ sudo ls /var/lib/rancher/k3s/agent/etc/containerd/certs.d
 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/test-private-registry.com/hosts.toml
 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml 
 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/k8s.gcr.io/hosts.toml

Validation Results:

• k3s version used for validation:

$ k3s -v
k3s version v1.27.10+k3s2 (5b2ac881)
go version go1.20.13
'''
Cluster Status:
'''
$ kubectl get nodes
NAME               STATUS   ROLES                       AGE     VERSION
ip-172-31-24-154   Ready    control-plane,etcd,master   4m22s   v1.27.10+k3s2
ip-172-31-24-217   Ready    control-plane,etcd,master   3m22s   v1.27.10+k3s2
ip-172-31-24-94    Ready    <none>                      2m42s   v1.27.10+k3s2
ip-172-31-29-236   Ready    control-plane,etcd,master   5m12s   v1.27.10+k3s2

$ kubectl get pods -A
NAMESPACE        NAME                                      READY   STATUS      RESTARTS   AGE
auto-clusterip   test-clusterip-864b6877d9-gf4w7           1/1     Running     0          54s
auto-clusterip   test-clusterip-864b6877d9-rvdc7           1/1     Running     0          54s
auto-daemonset   test-daemonset-8rcnn                      1/1     Running     0          54s
auto-daemonset   test-daemonset-mm255                      1/1     Running     0          54s
auto-daemonset   test-daemonset-p4lms                      1/1     Running     0          54s
auto-daemonset   test-daemonset-xw5w2                      1/1     Running     0          54s
auto-dns         dnsutils                                  1/1     Running     0          54s
auto-ingress     test-ingress-78lhl                        1/1     Running     0          54s
auto-ingress     test-ingress-fhqcv                        1/1     Running     0          54s
auto-nodeport    test-nodeport-6867b6d5c7-5mptv            1/1     Running     0          54s
auto-nodeport    test-nodeport-6867b6d5c7-szrzm            1/1     Running     0          54s
default          clusterip-pod-demo                        1/1     Running     0          57s
default          clusterip-pod-demo-2                      1/1     Running     0          57s
default          clusterip-pod-demo-3                      1/1     Running     0          57s
kube-system      coredns-848b6f557d-cgr5t                  1/1     Running     0          4m17s
kube-system      helm-install-traefik-crd-479g9            0/1     Completed   0          4m17s
kube-system      helm-install-traefik-xxlqb                0/1     Completed   0          4m17s
kube-system      local-path-provisioner-79754ddd66-ttrv6   1/1     Running     0          4m17s
kube-system      metrics-server-544bdb6fc5-brmkm           1/1     Running     0          4m17s
kube-system      svclb-traefik-fbdd33f8-2mvjx              2/2     Running     0          2m42s
kube-system      svclb-traefik-fbdd33f8-4q9pw              2/2     Running     0          3m22s
kube-system      svclb-traefik-fbdd33f8-4vdmg              2/2     Running     0          4m1s
kube-system      svclb-traefik-fbdd33f8-z5n4w              2/2     Running     0          4m1s
kube-system      traefik-848dd846c-hlrrt                   1/1     Running     0          4m1s
pvt-reg-test     pvt-reg-test-6db5c56565-skl8s             1/1     Running     0          40s
pvt-reg-test     pvt-reg-test-6db5c56565-xg87m             1/1     Running     0          40s

config.toml content:

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml
# File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

.
.
.

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"




[plugins."io.containerd.grpc.v1.cri".registry.configs."test-private-registry.com".auth]
  username = "user"
  password = "password"

certs.d content:

 $ sudo ls /var/lib/rancher/k3s/agent/etc/containerd/certs.d
docker.io
test-private-registry.com
k8s.gcr.io

hosts.toml file contents:

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/test-private-registry.com/hosts.toml
# File generated by k3s. DO NOT EDIT.


[host."https://test-private-registry.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/docker.io/hosts.toml 
# File generated by k3s. DO NOT EDIT.
server = "https://registry-1.docker.io/v2"

[host."https://test-private-registry.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]

 $ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/k8s.gcr.io/hosts.toml 
# File generated by k3s. DO NOT EDIT.
server = "https://k8s.gcr.io/v2"

[host."https://test-private-registry.com/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/home/ubuntu/ca.pem"]