Container level resource limitation (cgroups)

gvisor only support POD level resource limitation(cpu, mem etc.)
In CRI, there is no resource config in RunPodSandboxRequest. While gvisor only set the cgroup value of runsc-sandbox when creating the first pause container. Thus there is no way to set the resource limitation for container created in CRI situation.

Thanks for the issue. FYI #172 is also related.

Note that per-container resource isolation isn't necessarily related to CRI but is needed to fully support CRI.

Currently, runsc only supports cgroup configuration for the entire pod. Even when container cgroup isolation is implemented, you would still want to set cgroup around the pod, because you can't really trust limits set inside the sandbox as it can be compromised. One possibility would be to create a cgroup for the pod with no limits when the pause container is started, and then modify it as needed when more containers are added/removed to/from the pod.

I remember there was some talk of "Pod overhead" as part of the Pod API since Kata needed it as well. We should follow up on that and make sure we don't do anything that doesn't align with that.

Currently, runsc only supports cgroup configuration for the entire pod. Even when container cgroup isolation is implemented, you would still want to set cgroup around the pod, because you can't really trust limits set inside the sandbox as it can be compromised. One possibility would be to create a cgroup for the pod with no limits when the pause container is started, and then modify it as needed when more containers are added/removed to/from the pod.

Hi! I was hoping you could shed some light on this:

The kubernetes docs state:

A Pod resource request/limit for a particular resource type is the sum of the resource requests/limits of that type for each Container in the Pod."

My question, how does gvisor deal with CRI resource limits/requests?

I am running gvisor with kubernetes using the gvisor-containerd-shim, no matter if i specify a resource limit or not the container reports:

Total Host Memory = 2097152 KiB
Container CGroup Limit = 0 KiB

However, when setting resource limits the performance deteriorates massively, a lot more so than for other runtimes such as runc or kata(qemu), so my takeaway is that something(and something different from other runtimes) does happen.

I've encountered this issue while trying to use the --cpu-num-from-quota parameter.
The cgroup of pod has the CPU quota, but container cgroup is created only with cpu.shares, which is not triggering any cpu number modification.

@williamviktorsson

@williamviktorsson I'm also seeing the same behaviour regarding memory. Did you end up figuring out what was causing this and how resource requests and limits should be configured?

@williamviktorsson

@williamviktorsson I'm also seeing the same behaviour regarding memory. Did you end up figuring out what was causing this and how resource requests and limits should be configured?

I did not find an answer to why this is the case. My use of gVisor was primarily in a performance evaluation of alternate container runtimes for Kubernetes(gVisor, Kata, runC, Firecracker). To make the comparison as fair as possible, I decided to run with large pod limits to to avoid the issue discussed here. I was hoping to learn if this approach was naive and why the detoriation of performance occured so noticeably.

These were the limits i used for all runtimes:

resources:
  limits:
    memory: "2Gi"
    cpu: "1"

@ernado

As you mentioned, K8s will aggregate all container limits and apply them to the pod. However, there is no concept of a pod in OCI, which is the interface used to create actual containers. So K8s has no choice but to set cgroup limits directly for pods. Pod limits are never exposed to gVisor. gVisor gets only per-container limits, which are ignored for now. We could update the limits based on each container, similar to K8s, but it can get complicated with individual containers' lifecycle, e.g. when a container stops, the memory limit may need to be lowered below the current usage, but maybe the container is only restarting.

gVisor uses the init container (aka. pause container) to represent the pod limits. We'd need to change CRI to include pod limits and pass it down to the runtime in the init container. Or add annotations to convey this information to the runtime.

@ernado, --cpu-num-from-quota will have no effect because it applies to the init container limits, not the pod.

@williamviktorsson, do you have an example that shows the slow down you are seeing?

It means if we want to limit gVisor containers CPU, we have to use a LimitRange and this will impact all pods in the namespace ?

@almottier

@almottier No, this issue doesn't have to do with LimitRange or Kubernetes scheduling. LimitRange is only used to set default requests/limits on Pods in a namespace anyway.

This issue is that gVisor can't currently limit per-container which means that even if you specify limits on the containers, one container in the Pod could use potentially use all the CPU and starve the other container in the pod etc. However, Pod limits are also created and those do work so the resources of the containers together will still be limited.

I think I understood why it was not working. I had one of the containers without resources limits in this pod.
Please correct me if I am wrong: pods limits are containers limits aggregation, so, if one of the containers in this pod doesn't have resources limits, the pod will not be limited ?

That's exactly correct. You'd need to set limits to all containers in the Pod. This is covered here: https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods#additional_security_recommendations

Related #1906

Hi guys,
After reading this thread I think I am experiencing an issue which should not happen, since I understand it is not the correct behavior.
Basically when running for example htop inside the container, total container memory shown corresponds to node's not pod.
I am basically requesting and limiting memory when declaring pod. My pod has only one container.
As an important detail, the underlying container engine in containerd, k8s version 1.19

For example, I define a pod with the following:

Limits:
  cpu:                500m
  ephemeral-storage:  47Mi
  memory:             286Mi
Requests:
  cpu:                500m
  ephemeral-storage:  47Mi
  memory:             286Mi

However, top shows:
KiB Mem : 2097152 total, 2082340 free which is about 2GB

Any idea what I might be doing wrong here?

Thanks

@nsoubelet

@nsoubelet, this happens because Kubernetes creates the initial container (pause) with unlimited memory. The initial container triggers sandbox creation and that's the memory reported. Then individual containers are created with memory limits, but since gVisor doesn't yet support cgroups between containers, the memory limit is not taken into account.

pause container:

  "linux": {
    "resources": {
      "cpu": {
        "shares": 2
      }
    },
  },

second container:

  "linux": {
    "resources": {
      "memory": {
        "limit": 299892736
      },
      "cpu": {
        "shares": 512,
        "quota": 50000,
        "period": 100000
      }
    },

However, if you look at the cgroup where the sandbox is running, it does have the correct limit set:

cat /sys/fs/cgroup/memory/kubepods/pod1a4d3383-2c88-4f0b-924d-b85fc76a4466/memory.limit_in_bytes 
299892736

Therefore, the Pod's aggregate memory limit is respected, just not reported correctly inside the sandbox. I think we can do better to report the correct memory limit inside the sandbox in this case (even if it's the aggregate for the pod). I'll create another issue to track it.

@fvoznika

Hi @fvoznika, many thanks for confirming. It's clear now.
And yes, you are right, limit enforcement works ok, issue is only with the information displayed.

feat(api): per-container cpu and memory limits on ActorTemplate

Per-container CPU and memory limits for micro-VM actors (#859)

Navigation Menu

Repository navigation

Container level resource limitation (cgroups) #190

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Container level resource limitation (cgroups) #190

Description

Activity

ianlewis commented on Apr 6, 2019

ianlewis commented on Apr 6, 2019

fvoznika commented on Apr 12, 2019

ianlewis commented on May 31, 2019

williamviktorsson commented on Dec 2, 2019

ernado commented on Apr 16, 2020

nanzhong commented on Jun 10, 2020

williamviktorsson commented on Jun 10, 2020

fvoznika commented on Jun 11, 2020

almottier commented on Sep 14, 2020

ianlewis commented on Sep 14, 2020

almottier commented on Sep 14, 2020

fvoznika commented on Sep 15, 2020

ianlewis commented on Nov 21, 2020

nsoubelet commented on Jan 22, 2021

fvoznika commented on Jan 26, 2021

nsoubelet commented on Jan 26, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions