Containerd CRI registry mirrors and configs config via config.toml have been deprecated #8972

brandond · 2023-11-30T05:02:44Z

Newer release of containerd have deprecated use of config.toml to configure registry endpoints and TLS config, in favor of docker-style certs.d directory structure. See the comment at https://github.com/containerd/containerd/blob/release/1.7/docs/cri/registry.md

Other projects like https://github.com/XenitAB/spegel require use of this structure.

There are other benefits to this, such as allowing for dynamic reloading of config other than auth, which is still only available via config.toml.

There are other issues also open tracking this work:

Support for containerd 1.5+ config_path: On-the-fly container registry configuration #5568

The text was updated successfully, but these errors were encountered:

VestigeJ · 2024-01-19T00:08:13Z

$ sudo cat /etc/containerd/config.toml

# See containerd-config.toml(5) for documentation.

$ sudo journalctl -u k3s | grep -i "config.toml"

Jan 18 19:10:32 ip-117 k3s[2692]: time="2024-01-18T19:10:32Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"

OLD vs NEW (showing new config path entry)

$ cat old-format.toml

# File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true



[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/e5efd5aeb0cb8e4c1d802582bd7085968576e89d8f34ca22a450b4f4ae4d4c15/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

$ cat config.toml

# File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true



[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/41baf3230c69f60cbc3c461f264a8a80892b78b4841a8dbe43d2feb6e297c541/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"

VestigeJ · 2024-01-23T00:57:13Z

##Environment Details
Reproduced using VERSION=v1.29.0+k3s1
Validated using VERSION=v1.29.1-rc2+k3s1

Infrastructure

Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.55.44-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME              STATUS   ROLES                       AGE     VERSION
ip-1-1-2-7        Ready    control-plane,etcd,master   2m14s   v1.29.1-rc2+k3s1

Config.yaml:

token: YOUR_TOKEN_HERE
write-kubeconfig-mode: 644
debug: true
protect-kernel-defaults: true
cluster-init: true

Validation

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl

$ sudo cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

 version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/41baf3230c69f60cbc3c461f264a8a80892b78b4841a8dbe43d2feb6e297c541/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic"]
  runtime_type = "io.containerd.lunatic.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic".options]
  BinaryName = "/usr/local/bin/containerd-shim-lunatic-v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/local/bin/nvidia-container-runtime"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-experimental"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-experimental".options]
  BinaryName = "/usr/local/bin/nvidia-container-runtime-experimental"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin"]
  runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin".options]
  BinaryName = "/usr/local/bin/containerd-shim-spin-v1"

$ sudo INSTALL_K3S_VERSION=v1.29.1-rc2+k3s1 INSTALL_K3S_EXEC=server ./install-k3s.sh

[INFO]  Using v1.29.1-rc2+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.29.1-rc2+k3s1/sha256sum-amd64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.29.1-rc2+k3s1/k3s
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

$ set_kubefig //export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

$ kg runtimeclass

NAME                  HANDLER               AGE
crun                  crun                  102s
lunatic               lunatic               102s
nvidia                nvidia                102s
nvidia-experimental   nvidia-experimental   102s
slight                slight                102s
spin                  spin                  102s
wasmedge              wasmedge              102s
wasmer                wasmer                102s
wasmtime              wasmtime              102s
wws                   wws                   101s

get_wasi() {
    has_bin wget
    _arch=$(uname -m)
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v1-lunatic-linux-"${_arch}".tar.gz
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v1-slight-linux-"${_arch}".tar.gz
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v1-wws-linux-"${_arch}".tar.gz
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/containerd-wasm-shims-v2-spin-linux-"${_arch}".tar.gz
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/workload.yaml
    wget https://github.com/deislabs/containerd-wasm-shims/releases/download/v0.10.0/runtime.yaml

    tar -xvf containerd-wasm-shims-v1-lunatic-linux-"${_arch}".tar.gz
    tar -xvf containerd-wasm-shims-v1-slight-linux-"${_arch}".tar.gz
    tar -xvf containerd-wasm-shims-v1-wws-linux-"${_arch}".tar.gz
    tar -xvf containerd-wasm-shims-v2-spin-linux-"${_arch}".tar.gz

    chmod +x containerd-shim-lunatic-v1
    chmod +x containerd-shim-slight-v1
    chmod +x containerd-shim-wws-v1
    chmod +x containerd-shim-spin-v2

    sudo cp containerd-shim-lunatic-v1 /usr/local/bin/
    sudo cp containerd-shim-slight-v1 /usr/local/bin/
    sudo cp containerd-shim-wws-v1 /usr/local/bin/
    sudo cp containerd-shim-spin-v2 /usr/local/bin/
    # call host-config function to write config.toml file
    set_runtimes
}

set_runtimes() {
  cat <<EOF > config.toml.tmpl
  # File generated by k3s. DO NOT EDIT. Use config.toml.tmpl instead.
version = 2

[plugins."io.containerd.internal.v1.opt"]
  path = "/var/lib/rancher/k3s/agent/containerd"
[plugins."io.containerd.grpc.v1.cri"]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  enable_unprivileged_ports = true
  enable_unprivileged_icmp = true
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true

[plugins."io.containerd.grpc.v1.cri".cni]
  bin_dir = "/var/lib/rancher/k3s/data/41baf3230c69f60cbc3c461f264a8a80892b78b4841a8dbe43d2feb6e297c541/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/var/lib/rancher/k3s/agent/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic"]
  runtime_type = "io.containerd.lunatic.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."lunatic".options]
  BinaryName = "/usr/local/bin/containerd-shim-lunatic-v1"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia".options]
  BinaryName = "/usr/local/bin/nvidia-container-runtime"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-experimental"]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."nvidia-experimental".options]
  BinaryName = "/usr/local/bin/nvidia-container-runtime-experimental"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin"]
  runtime_type = "io.containerd.spin.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."spin".options]
  BinaryName = "/usr/local/bin/containerd-shim-spin-v1"

EOF
sudo mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/
sudo cp config.toml.tmpl /var/lib/rancher/k3s/agent/etc/containerd/
}

github-project-automation bot added this to K3s Development Nov 30, 2023

github-project-automation bot moved this to New in K3s Development Nov 30, 2023

brandond changed the title ~~Containerd CRI registry mirrors and configs config via config.toml are have been deprecated~~ Containerd CRI registry mirrors and configs config via config.toml have been deprecated Nov 30, 2023

brandond mentioned this issue Nov 30, 2023

Add support for containerd cri registry config_path #8973

Merged

2 tasks

caroline-suse-rancher moved this from New to Peer Review in K3s Development Jan 4, 2024

caroline-suse-rancher added this to the v1.29.1+k3s1 milestone Jan 4, 2024

brandond mentioned this issue Jan 17, 2024

Track containerd registry changes from K3s rancher/rke2#5298

Closed

VestigeJ self-assigned this Jan 18, 2024

VestigeJ moved this from Peer Review to To Test in K3s Development Jan 22, 2024

VestigeJ closed this as completed Jan 23, 2024

github-project-automation bot moved this from To Test to Done Issue in K3s Development Jan 23, 2024

brandond mentioned this issue Mar 6, 2024

Mirrors configured in /etc/rancher/k3s/registries.yaml do not take effect #9626

Closed

brandond mentioned this issue Mar 14, 2024

mirrors."docker.io" is ignored rancher/rke2#5612

Closed

brandond mentioned this issue Aug 1, 2024

Fail pull image from mirror rewrite on v1.28.11 version with airgap enviroment rancher/rke2#6447

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Containerd CRI registry mirrors and configs config via config.toml have been deprecated #8972

Containerd CRI registry mirrors and configs config via config.toml have been deprecated #8972

brandond commented Nov 30, 2023

VestigeJ commented Jan 19, 2024

VestigeJ commented Jan 23, 2024

Validation

Containerd CRI registry mirrors and configs config via config.toml have been deprecated #8972

Containerd CRI registry mirrors and configs config via config.toml have been deprecated #8972

Comments

brandond commented Nov 30, 2023

VestigeJ commented Jan 19, 2024

VestigeJ commented Jan 23, 2024

Validation