🐘 Switch to`opencontainers/runc` as a library (Was: Explore replacing opencontainers/runc with containerd/cgroups)

[dims]

Kubernetes use of opencontainers/runc as a library is placing undue burden on the runc team, for example:

• move most packages into internal opencontainers/runc#3028
• Investigate uses of cgroups.Manager.GetCgroups() (unused, perhaps can be deprecated) opencontainers/runc#3221 (comment)

We now have a cgroups specific library in containerd org that we can explore to start slowly replacing functionality we needed earlier from runc i think.
https://github.com/containerd/cgroups

As of right now k/k master shows the following imports of opencontainers/runc:

❯ rg '"github.com/opencontainers/runc' | grep -v vendor | cut -f 2 -d '"' | sort | uniq -c | sort
      1 github.com/opencontainers/runc/libcontainer/cgroups/systemd
      1 github.com/opencontainers/runc/libcontainer/utils
      2 github.com/opencontainers/runc/libcontainer/apparmor
      2 github.com/opencontainers/runc/libcontainer/cgroups/manager
      2 github.com/opencontainers/runc/libcontainer/configs
      2 github.com/opencontainers/runc/libcontainer/userns
      3 github.com/opencontainers/runc/libcontainer/cgroups/fscommon
     17 github.com/opencontainers/runc/libcontainer/cgroups

/sig node

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dims]

/area code-organization

[haircommander]

in both cases aren't we signing up a different project to support kubernetes and our needs/timelines? Obviously there's some overlap between containerd community and k8s but I don't see how moving to containerd package would solve this problem of k8s relying on a package outside of the k8s repo (and on contributors who may not have k8s as their priority/focus)

I personally think it's better to pull the public libcontainer pieces out of runc and have those versioned separately from runc, so we don't need to wait for runc verison bumps to have a tagged version of libcontainer.

[yujuhong]

in both cases aren't we signing up a different project to support kubernetes and our needs/timelines? Obviously there's some overlap between containerd community and k8s but I don't see how moving to containerd package would solve this problem of k8s relying on a package outside of the k8s repo (and on contributors who may not have k8s as their priority/focus)

Agree. Not sure why this would resolve the dependency issue.

(cc @samuelkarp for the suggestion on relying on containerd library)

[mrunalp]

Adding @kolyshkin as we have discussed this before.

[BenTheElder]

I personally think it's better to pull the public libcontainer pieces out of runc and have those versioned separately from runc, so we don't need to wait for runc verison bumps to have a tagged version of libcontainer.

Not necessarily disagreeing, but ... who will maintain these? This may be duplicating effort given https://github.com/containerd/cgroups is already being maintained for public import as a standalone library.

My understanding is the runc team is actively attempting to stop supporting reuse of this code, so we'd need someone else to maintain the fork that re-exports them. If containerd/cgroups already accomplishes this then ... (e.g. containerd/cgroups@6f4af8b)

in both cases aren't we signing up a different project to support kubernetes and our needs/timelines? Obviously there's some overlap between containerd community and k8s but I don't see how moving to containerd package would solve this problem of k8s relying on a package outside of the k8s repo (and on contributors who may not have k8s as their priority/focus)

Our needs: yes, to an extent, but the difference is containerd/cgroups is a seperate public library instead of some packages from runc's implementation. It already has a distinct version (currently v3.0.3) and timeline from containerd/containerd.

cc @thaJeztah as a recently prolific committer to containerd/cgroups.

[dims]

@BenTheElder yes, discussions have started in containerd project as well if they want to scope up or down to include what we may need.

[dims]

Here's another option!

• Extract only files we use from runc : https://github.com/dims/libcontainer/
• update cadvisor with above: https://github.com/dims/cadvisor/tree/try-dims-libcontainer-repo
• then update k/k with new cadvisor and that library - https://github.com/kubernetes/kubernetes/pull/128200/files

We drop 20K lines of code and a bunch of tangled dependencies ... but then we own the library!

[haircommander]

I like this approach :) @kolyshkin @AkihiroSuda @cyphar @lifubang @rata WDYT

[rata]

@haircommander thanks for tagging us! I've just asked in the runc issue for opinions to other maintainers (opencontainers/runc#3028 (comment)). Ideally, we should agree something and come-back with an opinion from the project (I'd like that, at least :)).

Just to understand, what is the context of this issue? Why do you want to remove the dependency on runc?

[dims]

The use of runc as a binary vs runc as a library has been a journey, in addition to the above linked issues, please see those listed below as well. If you look at issues/PRs in opencontainers/runc you will see more of the history. So far the burden was put on runc folks by k8s without discussion. Here we are trying to see if there is a better way going forward.

• Move cgroupv1 specific code out of libcontainers/cgroups pkg opencontainers/runc#2471 (comment)
• v1.1.6 regression: adding misc controller to cgroup v1 makes kubelet sad opencontainers/runc#3849 (comment)
• Upgrade Cilium's eBPF library version to 0.16 opencontainers/runc#4436 (comment)