ISO upgrades to support user namespaces

[mbaynton]

What Happened?

A new feature is nearing readiness in kubernetes/containerd/runc that enables you to run your containers in their own linux user namespace. This is beneficial for security / container isolation and also enables you to do certain operations inside containers that were previously only possible with privileged host permissions.

This feature is set to be promoted to beta in kubernetes 1.30.

It would be great to be able to use minikube to prototype setups that use user namespaces, but lots of much newer software in the ISO will be needed:

• Modern linux kernel, at least 6.3
• Containerd 2.0 (only beta versions available today)
• runc 1.2 (not yet available, see release 1.2.0-rc.1 opencontainers/runc#4221)

Attach the log file

n/a, trying to submit a feature request but can't find a better form

Operating System

Ubuntu

Driver

KVM2

[afbjorklund]

Sounds like an experimental feature, that doesn't really fit with the LTS versions used in the minikube ISO (or KIC)

Theoretically there could be two image versions, but that means more maintenance (and it is already struggling)

[afbjorklund]

KEP 127 looks completely untested/unimplemented for Docker, which is still the minikube default container runtime...

For Beta, the feature is tested for containerd and CRI-O in cri-tools repo using critest

[rata]

@afbjorklund if it helps to make a decision, it doesn't seem it will be implemented in cri-dockerd anytime soon, as docker has some limitations. I've been trying them to implement it for years now, and although they like the idea, there was no progress so far: Mirantis/cri-dockerd#74

[mbaynton]

doesn't really fit with the LTS versions used in the minikube ISO

Yeah at the moment it's definitely too soon to build an updated ISO for general use that enables this. I'm optimistic releases of containerd and runc will be available soon though :). But everything needs to be marked as an LTS release as well?

[afbjorklund]

But everything needs to be marked as an LTS release as well?

It is not an absolute rule, but historically minikube used Ubuntu LTS as the base for decisions for the Buildroot versions.
Other distributions such as kind, were doing latest/greatest (or Debian) and rebuilding containerd on a nightly basis...

Ubuntu	Linux
16.04	4.4
18.04	4.15
20.04	5.4
22.04	5.15
24.04	6.8 (?)

But after that it was pretty arbitrary, stayed on 4.19 for a while since it was stable and then jumped to 5.10 (770d41f).
One could go for a similar jump to 6.x, perhaps timed with the OS update. The runtime versions are even more arbitrary.

• ISO: Update Buildroot from 2023.02.9 to 2024.02.7 #18433
• Allow specific version of container run time in minikube #6432

---

The versions of containerd and runc usually came (bundled) with Docker...

https://docs.docker.com/engine/install/binaries/ (or get.docker.com)

• dockerd 26.0.0
  • containerd 1.7.13
  • runc 1.1.12

[afbjorklund]

Note: Buildroot 2024.02 (LTS) supports these kernels:

# From https://www.kernel.org/pub/linux/kernel/v6.x/sha256sums.asc
sha256  4e43d8c5fba14f7c82597838011648056487b7550fd83276ad534559e8499b1d  linux-6.6.18.tar.xz
sha256  faa49ca22fb55ed4d5ca2a55e07dd10e4e171cfc3b92568a631453cd2068b39b  linux-6.1.79.tar.xz
# From https://www.kernel.org/pub/linux/kernel/v5.x/sha256sums.asc
sha256  bd84809a367eb400eb04e0e70294e6ba12fc03b6bfb5a7dfaca548f8947501b0  linux-5.15.149.tar.xz
sha256  4ea63c5a90fdc3c459ab35c11ee8c93d2364a7cdbfb101100f8cab70d490ef6d  linux-5.10.210.tar.xz
sha256  ff54bec6d053c7994f3bb8c45021de2858ff9f740d2ccbbcf072b87821a918cf  linux-5.4.269.tar.xz
# From https://www.kernel.org/pub/linux/kernel/v4.x/sha256sums.asc
sha256  83eeff613405d0045d0f717c6ac14c178678fe0a163c41d9dd8878ac0f73e352  linux-4.19.307.tar.xz

See https://buildroot.org/news.html (and 2024.02.x)

Going with Kernel 5.15 would be the more obvious choice.

[mbaynton]

The user namespace stuff needs features from newer kernels than 5.15. I've had success with 6.6, apparently you can go back as far as 6.3 but it looks like that's not an option for buildroot.

[rata]

@mbaynton it should work fine with 5.12+ but you can't use tmpfs volumes (so service account tokens can't really work) and you have a higher pod start latency and storage overhead if you use 5.12 too.

[rata]

/remove-lifecycle rotten

containerd 2.0 and runc 1.2 should be released this year with proper support. It seems worth keeping this open a little longer

[medyagh]

I agree we should have a tutorial that lets ppl build their own ISO easily
and basicly if they want containerd 2.0 can do minkube start --iso-url=file://....

and not part of released minikube ISOs