Determine if we should support cpuset-cpus and cpuset-mem

[derekwaynecarr]

Docker now supports cpuset-cpus and cpuset-mem arguments when running a container to control which cpus can execute a container. We have been asked by some users if we plan to support this feature in Kubernetes.

https://docs.docker.com/reference/run/#runtime-constraints-on-cpu-and-memory

Opening an issue to gather feedback on if its desired or not.

@erictune @dchen1107

[erictune]

Short answer

Not soon.

Long answer

Supporting both cpusets and cfs well is very complex.

Having to support both options will increase the development time of some features which we definitely want, such as:

• QoS Tiers and dealing with CPU overcommitment
• Auto-scaling

The complexity will become further evident if we later add features like:

• normalizing CPU for different platforms
• optimizing access to i/o devices
• dealing with processor cache interference.
• dealing with NUMA

If we expose the docker flags to users via the Pod spec today, it will be much harder to build these things.

However, we should being collecting the use cases where users think cpusets will help them. Then we can have a broader cost/benefit discussion about supporting cpusets. I think I know what many of these use cases are, but it would still to be good to hear users state them.

[derekwaynecarr]

@erictune - I am going to direct the original questioner to this issue for feedback on their needs, but my initial response was in-line with yours prior to opening the issue.

[resouer]

@erictune Is it possible for you to explain the main difficulty if we support cpuset?

My use case is: as k8s will support job controller soon, we want to fix Long Running Container to some cpu and let other jobs (batches) to compete for other cores. Does that make sense?

[jeremyeder]

Hi Eric,

By your initial comments, I'm sure we're on the same page in understanding the technical, performance benefits of this feature.

Indeed, the use-cases for exposing these knobs are precisely as you've stated.
Optimizing access to i/o devices, dealing with processor cache interference and dealing with NUMA.

If a users has an existing performance sensitive application that is lends itself well to microservices/containerization, they'll need all of these features in order to consider containers as a possibility.

These applications typically have their own init-scripts which apply tuning at both the host and app level, and (very often), automatically account for different hardware/virtual topologies.

We can approach containerizing this sort of application by simply migrating the init-scripts (hardware discovery, numactl, SCHED_FIFO and irq stuff) into each container. This of course is the definition of anti-pattern.

However, I'd argue that wanting to run performance-sensitive-pods (I call them "PSP's") in Kube is not an anti-pattern -- it's a pattern that's not implemented yet. Agree that usage may require some sort of nodeSelector matching, to avoid the complexities of having CFS workloads co-located with these HPC loads.

Exposing these knobs are one step in enabling a Kube stack to support workloads such as batch processing, HPC/scientific, big-data compute-bound analysis.

Examples of additional enablement would be exposing specific I/O devices to pods, RDMA, having a container "follow" a PCI device in terms of NUMA locality.

Thanks!

[erictune]

There are benefits to exposing this level of detail (cores, numa nodes) to users, and there are costs to exposing it as well.

Within the problem space of running web services, my experience is that on balance it is better not to expose these details to users.

I can see from your comments, Jeremy, that you are interested in high-frequency trading (HFT). With web services, latency is measured in milliseconds. I've heard that for HFT, it is measured in microseconds or even nanoseconds. For very short deadlines, like with HFT, I see how it it is important to expose these details.

So, I think the question is whether Kubernetes is a good fit for HFT, or if it can be adapted to be a good fit for HFT without compromising the web services use case. I'm not sure what the answer is to this question.