underhill_core: serialize sidecar VP online #1443
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benhillis
added
release_2505
The Linux kernel serializes CPU online. If multiple sidecar VPs need to be onlined into OpenVMM simultaneously, they will all stop running the guest OS and wait for their turn to be onlined into Linux. This means the average blackout time for a VP that's onlined early in boot is linear in the number of early-onlined VPs. And thanks to typical device configurations, this is usually linear in the total number of VPs. To avoid this, explicitly serialize VP online _before_ the target VP is stopped. This allows the VP to continue running the guest until it reaches the front of the queue. This reduces the average blackout time to just the time to online one CPU, meaning this solution should scale to any number of VPs.
jstarks
force-pushed
the
wait_to_avoid_waiting
branch
from
5754a31 to
3ffda59
Compare
benhillis
approved these changes
May 30, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Linux kernel serializes CPU hotplug. If multiple sidecar VPs need to be onlined into OpenVMM simultaneously, they will all stop running the guest while associated Linux threads call into the Linux kernel to online the CPU (which will block on the CPU hotplug lock or whatever).
This means the average blackout time for a VP that's onlined early in boot is linear in the number of early-onlined VPs. And thanks to typical device configurations, this is usually linear in the total number of VPs. This is a performance problem.
To avoid this, explicitly serialize VP online before the target VP is stopped. This allows the VP to continue running the guest until it reaches the front of the online queue. This reduces the average blackout time to just the time to online one CPU, meaning this solution should scale to any number of VPs.