Custom Runtime Spec Defaults

[crosbymichael]

There are many different defaults that an operator would want to change when it comes to the container runtime. Anything from rlimits, capabilities, or default mounts are fields that could be configurable in a deployment. Right now, CRI does not have a way to specify this at the daemon level.

I would like to add defaults to the daemon for runtime configuration on how the default runtime spec is created. However, I would also like to prevent having many fields in the daemon's configuration for an ever growing number of runtime options that a operator could want to change.

To solve both these requirements, I would like to propose custom spec defaults by changing the default spec. Currently all specs are created from a static, compiled in struct with default options.
ref: https://github.com/containerd/cri/blob/master/pkg/server/container_create.go#L291

This function uses the default found in the oci package as the base spec that all container's are created from. I want to add a simple daemon level option called base_runtime_spec which is a path to a serialized runtime spec on disk.

base_runtime_spec = "/etc/containerd/cri-base.json"

This would allow an operator to change ANYTHING on the spec to suit their needs and is forward compatible with all future runtime spec changes. The only read code change we would need in this codebase is to read this spec from disk and subsitute it in pace of the GenerateSpec's default, compiled in struct.

Issues:

However, there could be a few issues because of the way CRI handles the spec. It looks like functions and opts will unset or replace a lot of options on the spec and will need to be accounted for.

ref: https://github.com/containerd/cri/blob/master/pkg/server/container_create_unix.go#L115

Comments, questions, concerns?

[jterry75]

What are you trying to set here? Container create opts / shim opts? The runtime handler opts as they are today are really about shim opts

[mikebrow]

Recommend having a command to generate a new default from the existing code, or similar. Use the code to generate default if none exists. Add debug showing a diff from default to actual.. more debug showing diff when there is an update. Maybe named defaults by runtime see pattern here https://github.com/containerd/cri/blob/master/docs/config.md in the [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] section.

[mikebrow]

with the current pattern we try to only allow the user to make somewhat useful changes...

Allowing ANYTHING has a lot of pros and cons...

[crosbymichael]

@jterry75 this would be more, OCI runtime opts for the container. Such as, remove NET_RAW from all containers by default in my deployment.

@mikebrow There are pros and cons but since this would be an ops level decision, I think it's ok. If i want to configure my deployment with tighter security requirements on capabilities, mounts, or create default ulimits then we should allow that.

[mikebrow]

@jterry75 this would be more, OCI runtime opts for the container. Such as, remove NET_RAW from all containers by default in my deployment.

@mikebrow There are pros and cons but since this would be an ops level decision, I think it's ok. If i want to configure my deployment with tighter security requirements on capabilities, mounts, or create default ulimits then we should allow that.

Yes. Will give ops/cloud/product teams a very nice point of control for customizing. Will have to make sure we highlight this in the release notes and give our upstream products a heads up. Just thinking out loud :-)

[alban]

I would like to use this to add or remove OCI hooks. My use case is for Inspektor Gadget, which drops two scripts for the OCI hook PreStart and PostStop. I have a patch to do that with cri-o: it drops the OCI hooks in /etc/containers/oci/hooks.d/. For containerd/cri, I don't have a proper solution.

Ideally for my use case, Inspektor Gadget should be able to add and remove the OCI hooks it needs without parsing a default config.json but it should just add / remove files in a .d/ directory such as /etc/containerd/hooks.d. Otherwise, there is a risk that multiple tools doing this kind of configuration change step over each other toes.

The configuration should also be taken into account without restarting the containerd daemon.

This would also be useful for a Kubernetes operator that takes OCI hook custom resources and apply them on the CRI. That operator could install & remove files in /etc/containerd/hooks.d (for containerd) or /etc/containers/oci/hooks.d/ (for cri-o).

/cc @mauriciovasquezbernal

[crosbymichael]

@alban I forgot about hooks. This is a perfect solution for that usecase as well.

[mikebrow]

yeah.. and hooks is almost golden (finally) runc review in progress :-)

[mikebrow]

hook issue here: containerd/containerd#6645 I was just waiting on the spec approved hooks to be implemented in runc(opencontainers/runc#2229) ... old wip here #1248

FYI crio has a schema/process for deciding when to add hooks. .. Over here: https://github.com/containers/libpod/tree/master/pkg/hooks