CA-420968: avoid large performance hit on small NUMA nodes

[edwintorok]

NUMA optimized placement can have a large performance hit on machines with small NUMA nodes and VMs with a large number of vCPUs. For example a machine that has 2 sockets, which can run at most 32 vCPUs in a single socket (NUMA node), and a VM with 32 vCPUs.

Usually Xen would try to spread the load across actual cores, and avoid the hyperthread siblings (when the machine is sufficiently idle, or the workload is bursty), e.g. using CPUs 0,2,4,etc.
But when NUMA placement is used all the vCPUs must be in the same NUMA node. If that NUMA node doesn't have enough cores, then Xen will have no choice but to use CPUs 0,1,2,3,etc.

Hyperthread siblings share resources, and if you try to use both at the same time you get a big performance hit, depending on the workload. We've also seen this previously with Xen's core-scheduling support (which is off by default)

Avoid this by "requesting" threads_per_core times more vCPUs for each VM, which will make the placement algorithm choose the next size up in terms of NUMA nodes (i.e. instead of a single NUMA node use 2,3 as needed, falling back to using all nodes if needed).

The potential gain from reducing memory latency with a NUMA optimized placement (~20% on Intel Memory Latency Checker: Idle latency) is outweighed by the potential loss due to reduced CPU capacity (40%-75% on OpenSSL, POV-Ray, and OpenVINO), so this is the correct tradeoff.

If the NUMA node is large enough, or if the VMs have a small number of vCPUs then we still try to use a single NUMA node as we did previously.

The performance difference can be reproduced and verified easily by running openssl speed -multi 32 rsa4096 on a 32 vCPU VM on a host that has 2 NUMA nodes, with 32 PCPUs each, and 2 threads per core.

[edwintorok]

Draft PR because this requires more testing, and perhaps also introducing another NUMA policy enum that preserves the original behaviour, just in case we need it.

Eventually it might also be useful to introduce a NUMA policy for VMs, so that some VMs could be NUMA optimized, and not others. Now that we have proper memory reservation in the new version of Xen, and we know exactly where each VM will go we could do that (in the initial implementation for the old version of Xen we couldn't, because unless all VMs got balanced we couldn't predict how much memory would be left when booting a VM, unless it was all part of a known, small number of NUMA nodes). But that is a larger change that may come in a future PR.

[psafont]

Does the slowdown happen for any number of vcpus? For example, if a NUMA node has 4 SMT cores and the VM requests 4 vCPUs

[edwintorok]

Does the slowdown happen for any number of vcpus? For example, if a NUMA node has 4 SMT cores and the VM requests 4 vCPUs

We don't have any CPUs with threads_per_core = 4 in our lab, only 1 or 2. I think architectures other than x86-64 would have SMT4 cores. I used physinfo.threads_per_core to be future proof though (Xeon Phi had 4 threads per core, and maybe they'll try again with other microarchitectures).

I'd expect that the slowdown would be even worse with SMT4, although it really depends which resources are shared between the threads, and which resources are duplicated. For example here https://en.wikichip.org/wiki/intel/microarchitectures/cascade_lake#Scheduler_Ports_.26_Execution_Units there are 7 execution ports shared between all threads on a core. But out of order execution can also use those for instruction-level-parallelism.

If you have good code that takes full advantage of ILP (e.g. most low-level benchmarks, numeric code, etc.), then you might be able to saturate the core with just 1 hyperthread sibling, in this case attempting to use the hyperthread sibling(s) for anything else would slow down both. This is true regardless of whether NUMA optimization is in use or not (but NUMA optimization makes it more likely you hit this problem because you have fewer cores then).
OTOH if you have code that doesn't take full advantage of ILP (e.g. it branches a lot in unpredictable ways, or has a lot of data-dependencies) then you'd have free execution ports, and running something on the hyperthread sibling wouldn't affect the first one much (and overall result in a performance increase because you have now used also the hyperthread siblings so have 2x the processing power).

OTOH my fix only works if the other sibling is idle. If it gets used by a different VM then the situation could be even worse, e.g. the DTLB may not be duplicated for each thread on all microarchitectures. But that can also happen if NUMA optimization is completely turned off.

@mg12 suggested another optimization: try to still run the VM on a single NUMA node, and only when it is too busy spread (the CPUs) out. Although this may be too late because the memory is already bound to a single NUMA node, and Xen doesn't have runtime rebalancing like Linux would.
Anyway the optimization could be tried out if we change the soft affinity mask slightly to use only every threads_per_core in the set, i.e. instead of soft pinning to 32-63, we soft pin to 32,34,...,62. Then if the VM uses at most 16 of its vCPUs at once, it'd still benefit from the NUMA speedup, and once it starts using more, then it'll start running some of its vCPUs elsewhere. TBC whether this actually result in any better performance than just not pinning at all and spreading the memory (so that when vCPUs run elsewhere they have some chance to use local memory).

I can introduce a few more enums in the numa policy, and change the best_effort to be an alias that testing shows results in better performance in a wider set of configurations. The user can then override the policy to something else for different workloads (although this really needs per VM policy support too).

[last-genius]

We don't have any CPUs with threads_per_core = 4 in our lab, only 1 or 2. I think architectures other than x86-64 would have SMT4 cores. I used physinfo.threads_per_core to be future proof though (Xeon Phi had 4 threads per core, and maybe they'll try again with other microarchitectures).

I think Pau was asking about a NUMA node with 4 threads/2 cores; not an SMT with 4 threads per core - i.e. how do the performance numbers scale in case of smaller NUMA nodes than ones with 32 threads

[edwintorok]

We don't have any CPUs with threads_per_core = 4 in our lab, only 1 or 2. I think architectures other than x86-64 would have SMT4 cores. I used physinfo.threads_per_core to be future proof though (Xeon Phi had 4 threads per core, and maybe they'll try again with other microarchitectures).

I think Pau was asking about a NUMA node with 4 threads/2 cores; not an SMT with 4 threads per core - i.e. how do the performance numbers scale in case of smaller NUMA nodes than ones with 32 threads

I'd expect the performance regression to be about the same when looking at relative numbers (and assuming that you now use -multi 4 argument to the OpenSSL benchmark), but that 'd be interesting to confirm.

[edwintorok]

There is also potentially a problem due to the way we balance VMs across NUMA nodes, we currently only take into account memory. But CPU overload can have a much larger impact than memory latency due to using remote NUMA nodes.
Maybe we'd also need a policy that sorts by most free cpus in a NUMA node first when picking a node, and then by available memory (obviously excluding nodes without enough memory). And keep sorting by memory when picking the number of NUMA nodes to use.

E.g. if you have 1 large VM, 256GiB, and 4 smaller 64GiB, all with same vCPU count. We probably don't want to end up with all 4 of the small ones on the same NUMA node.

[edwintorok]

The previous small patch had a bug, it also tried to iterate beyond the number of vcpus assigned to the VM.
Also the sorting is not quite right, and we should sort by number of free CPUs first.
To avoid a combinatorial explosion in XAPI I've introduced the low level choices in xenopsd as separate record fields,
and at the XAPI level I've only introduced 1 new policy: prio_cpu_mem, which multiplies the number of vcpus a VM has by threads_per_core of the host.
best_effort becomes an alias to this one, and prio_mem_cpu is the old one, should we need it.

Sorting is not yet implemented, and this still needs some testing, so keeping as a draft.

[edwintorok]

We should also sort at the pool level by available CPUs first, and then by memory, otherwise a VM with a large amount of memory could create a very unbalanced pool, with some hosts having higher CPU oversubscription than others.
But I'll do that in a separate PR.

[edwintorok]

Probably better to avoid looking at threads_per_core, and count the number of cores we see in the topology instead for each NUMA node. I'm not sure what the threads_per_core setting would show in Xen when it is booted with smt=false, but in that case we don't need the doubling.

[edwintorok]

Starting to look too complex though, I'll try to simplify, so don't review this yet.

[edwintorok]

I simplified this by rewriting it from scratch, the policy is no longer exposed to XAPI. There is no new escape hatch, we use the previous escape hatch in xenopsd.conf: numa-placement=true which sets the policy to the behaviour prior to this commit. The previous behaviour is renamed Prio_mem_only.

I'll leave the sorting to future PRs, because xenopsd doesn't currently seem to have the required statistics available on how many cpus are free on a numa node.

The actual change is the oneliner in the last commit that changes what Best_effort does, all the previous commits introduce the mechanisms.

Going to test this now.

[edwintorok]

My idea of using the existing numa-placement setting didn't work due to some left-over tech debt. That (deprecated, according to its docs) setting is turned on by default in XS9 (and not the policy in XAPI), because that is how we turn numa on and off between the 2 Xen versions. So it isn't deprecated after all.

I'll introduce a new boolean instead.

[edwintorok]

I've tested this on a host and it produced the expected NUMA node assignment: with 32 vCPU VM on a 2*32 CPU host, with 2 NUMA nodes, it assigns the VM to all nodes. If I reduce the number of vCPUs to 16, it assigns it to just one.

I also ran the existing NUMA test suites, but they keep running into preexisting bugs (they're a bit too sensitive, and complain about a 10.2% imbalance when the threshold is 10%; or about running out of memory starting a VM when the host has 21.8 GiB free on a NUMA node, and the VM was 22 GiB, and so on). None of those seem to be related to my changes, because the VMs don't have enough vCPUs to hit the node limits on the machines it ran on.