Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CPU manager proposal. #654

Merged
merged 15 commits into from
Aug 1, 2017
Merged

Add CPU manager proposal. #654

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
ae38e21
Added initial draft of CPU manager proposal.
ConnorDoyle May 23, 2017
a21b6ad
Added cache alloation implementation phase.
ConnorDoyle Jun 5, 2017
b961e4e
Fixed typos.
ConnorDoyle Jul 11, 2017
a6afac8
Updated code snippets to match PoC branch.
ConnorDoyle Jul 12, 2017
79a2bb5
Minor formatting fix.
ConnorDoyle Jul 12, 2017
ca32930
Fixed typos, described sched_relax_domain_level.
ConnorDoyle Jul 14, 2017
8714cae
Added CPU Manager block diagram.
ConnorDoyle Jul 21, 2017
6eece46
Updated topo discovery section with decision.
ConnorDoyle Jul 21, 2017
63d8db1
Tied up loose ends in proposal.
ConnorDoyle Jul 24, 2017
694d2f4
Updated CPU manager configuration details.
ConnorDoyle Jul 24, 2017
8c98636
Fixed review comments from @balajismaniam.
ConnorDoyle Jul 24, 2017
3dfe261
s/floor/ceiling, recommend integer allocatable.cpu
ConnorDoyle Jul 24, 2017
a04d7eb
Merge pull request #1 from ConnorDoyle/cpu-manager-proposal-config
ConnorDoyle Jul 24, 2017
5780b48
Updated staticPolicy.Start sketch.
ConnorDoyle Jul 24, 2017
6bc03ac
Fix review comments from @derekwaynecarr.
ConnorDoyle Jul 25, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updated topo discovery section with decision.
  • Loading branch information
@ConnorDoyle
ConnorDoyle authored Jul 21, 2017
commit 6eece46e694e70eb455c8ceee37b0f623fed1946
7 changes: 4 additions & 3 deletions contributors/design-proposals/cpu-manager.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,10 @@ hyper-threading turned on yields both sibling threads on the same
physical core. Likewise, allocating two CPUs on a non-hyper-threaded
system yields two cores on the same socket.
Copy link

@teferi teferi Jun 7, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What should happen if there are enough CPUs on a node, but they're far away from each other (I'm requesting 2 CPUs, but the ones left are on different cores). For some workloads admitting such Pod may be acceptable, for some not. Would it be a good idea to make this behaviour configurable?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For now, our plan is to have the first policy (static) provide the "best" allocation when assigning dedicated cores. So, at higher G pod density it would be possible to observe the topology fragmentation you described.

If it were configurable in the policy, do you have ideas about how you would like that to look (name for the knob, values, defaults etc.)?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The more I think about it — the more I realize how complicated things actually are =/
Generally my perception is that by default CPUs of a single container inside a Pod should be as close to each other as possible, while the CPUs of a different containers inside a Pod should be as far as possible from each other. The only case I can imagine when I would want to pack as much dedicated containers on a single socket as possible is when I would want to launch multiple small (1-CPU) containers and then launch a single giant one, that asks for half the CPUs of a core.

For a container if we want to allow to control how close the CPUs are, I imagine it might be a double parameter in the pod spec. Smth like:

corePolicy:
  placement: dense|separate
  strict: true|false

with dense and false as defaults. separate would mean to attempt to spread CPUs as much as possible and dense would mean to pack as close as possible. With strict: false meaning that workload can tolerate if the requirements are not met. And strict: true on the other side should mean that the container has to be rescheduled.

And then there is a whole new level with HT enabled. I can imagine smth like:

coreThreadPolicy:
  placement: dense|separate|separate_restrict
  strict: true|false

with dense and false as defaults. dense here would mean allow giving a container multiple sibling CPUs. separate would mean attempt to spread CPUs across non-siblings. spearate_restict would mean spread CPUs across non-siblings and reserve the sibling, so that it is not assigned to anyone.
I'm afraid that I might be over-thinking the whole problem =)

As a small disclaimer I'd like to mention that I come from OpenStack, so my perception migh be tainted with how OS does it =) Here is a link to a doc that describes how OS approaches similar problems.
(cc @ashish-billore)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@teferi I like the idea of corePolicy.
As for coreThreadPolicy - isn't it a little bit too much ? i.e. not all environments are homogeneous, some nodes may not have HT enabled (probably not much). Besides separate_restrict can be accomplished by requesting i.e. 2 cores. If node has HT and

corePolicy:
  placement: dense
  strict: true

then it would result in 2 CPUs on 1 physical core.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flyingcougar separate_restrict is more about not allowing others to use core's sibling. It probably would be very difficult to implement, since k8s would still think that we have those cores available for scheduling. So anyway it was probably a weird idea.

corePolicy is about packing the CPUs as tightly as possible on cores in the same socket, while coreThreadPolicy is about packing CPUs as tightly as possible within one core. So imo both of those have their uses. I think I need to draw a couple of diagrams to illustrate these ideas...

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @fabiand I guess you should participate in ^^ part of the discussion

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The more I think about it — the more I realize how complicated things actually are =/

Yes :)

I'm not an expert on this domain, but …
It's pretty clear that we - kubevirt - need the 'strict dense placement' option as a first step.
I actually wonder if additional differentiations are needed: Either you care, and you require an optimal placement, or you don't and get best effort.

Some workloads, like kubevirt (again), might want to manage parts of numa nodes themselfes, in a more fine granular fashion. In those cases it might make sense, to be able to tell the kubelet, that some nodes, sockets, core, should be reserved/exclusive to a specific container. A process can then take those reserved parts and manage them as needed (i.e. in our case, libvirt would take care of managing those nodes).
Long story short, we might want a mechanism to lend nodes to a process.

Another thing is that in future we need - and thus should already think of - to combine the placement with ResourceClasses #782, because in reality you want a process close to the device it uses, to avoid numa-node remote memory access.
Closely related to this is actually also the relationship to huge-pages.

I do not think that we need to solve all of this now, but we should understand the picture, to not create something which will already contradict with what we see coming up.

@berrange @mpolednik thoughts?

Copy link
Contributor Author

@ConnorDoyle ConnorDoyle Jul 11, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fabiand there's exactly that topic on the resource management workgroup agenda for this morning. Besides the device binding logic and the hugetlb controller settings, CNI plugins could also take topology into account on a multi-socket system with multiple network interfaces. I hope we can achieve agreement that a solution is needed to unify affinity somehow in the v1.9 or v1.10 releases.

On the topic of the CPU manager itself, policies that operators want can get really complicated. Hopefully we can get a good default scoped and implemented first and then extend with more advanced policies. In the resource management face-to-face where this component was conceived, we had a pretty long discussion about implementing a more flexible policy driven by explicit pool configuration. Maybe we could think of ways to represent the behaviors needed for the advanced cases in the config domain. In my mind, a rich configurable policy may buy enough flexibility so that we don't need another plugin system for external policies. At the same time, for v1.8 at least, changes to the pod spec API are probably out of scope.


##### Options for discovering topology
**Decision:** Initially the CPU Manager will re-use the existing discovery
mechanism in cAdvisor.

Alternate options considered for discovering topology:

1. Read and parse the virtual file [`/proc/cpuinfo`][procfs] and construct a
convenient data structure.
Expand All @@ -98,8 +101,6 @@ system yields two cores on the same socket.
contains code to build a ThreadSet from the output of `lscpu -p`.
1. Execute a mature external topology program like [`mpi-hwloc`][hwloc] --
potentially adding support for the hwloc file format to the Kubelet.
1. Re-use existing discovery functionality from cAdvisor. **(preferred initial
solution)**

#### CPU Manager interfaces (sketch)

Expand Down