runsc: add --cpu-num-fixed to pin the sandbox CPU count - #14142
Open
mayur-tolexo wants to merge 1 commit into
Open
runsc: add --cpu-num-fixed to pin the sandbox CPU count#14142mayur-tolexo wants to merge 1 commit into
mayur-tolexo wants to merge 1 commit into
Conversation
The number of CPUs exposed to the sandbox is derived at boot from the cgroup CPU quota and is not updated afterwards, so raising a container's CPU quota in place (for example via Kubernetes in-place pod resize) does not give the guest any more usable CPUs. Workloads that size their parallelism to the reported CPU count stay capped at the boot value until the sandbox is recreated. See google#14141. Add a --cpu-num-fixed flag that pins the CPU count to a fixed value, independent of both the cgroup quota and the host CPU count. A sandbox can then boot with headroom CPUs and have a later in-place quota increase take effect without a restart. The value may exceed the host CPU count; the cgroup quota still bounds real CPU usage. It applies whether the cgroup is managed externally or by the sentry, and takes precedence over --cpu-num-from-quota. Signed-off-by: Mayur Das <mayur.das@neevcloud.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The number of CPUs a sandbox sees is derived at boot from the cgroup CPU quota
and is not updated afterwards. When a container's CPU limit is raised in place
(for example with Kubernetes in-place pod resize) the host cgroup quota changes
but the guest keeps the CPU count it booted with, so workloads that size their
parallelism to the reported count (Go's GOMAXPROCS,
make -j$(nproc), the JVM,nginx
worker_processes auto) stay capped at the old value and can't use the CPUthat was just added without recreating the sandbox. This is the guest-visibility
side of #14141.
This adds a
--cpu-num-fixedflag. When set, it fixes the number of CPUs exposedto the sandbox to that value, independent of the cgroup quota and the host CPU
count, and takes precedence over
--cpu-num-from-quota. A sandbox can then bootwith headroom CPUs and have a later in-place quota increase take effect without a
restart; the cgroup quota still bounds real CPU usage, so the extra CPUs are only
usable up to the quota. The value may exceed the host CPU count. The flag is
honored whether or not runsc reads a host cgroup, so it works with both
--systemd-cgroupand--ignore-cgroups.Verified on kind (Kubernetes 1.35, containerd 2.2.0, systrap, arm64) with the
runsc runtime installed as
--cpu-num-fixed=8.With
--cpu-num-fixed=8 --systemd-cgroup=true:With
--cpu-num-fixed=8 --ignore-cgroups=true(runsc reads no host cgroup):And the motivating case, an in-place CPU resize actually becoming usable: a pod
booted with a 1-CPU limit and
--cpu-num-fixed=8reports 8 CPUs; with the quotaat 1 a CPU-bound, nproc-sized workload consumes ~1.0 cores, and after an in-place
resize of the CPU limit to 8 (no restart) the same workload consumes ~8.0 cores.
Without the flag the guest stays at the boot-time count of 2 and the workload
stays capped at ~2.0 cores.
The CPU-count derivation is factored into a small helper (
cpuNumForSandbox) witha unit test, and there is an end-to-end
TestNumCPUFixedthat installs a-cpunumfixedruntime and checks the guest CPU count is pinned regardless of thecontainer's cpuset.