[virtio-mem] Dynamic slots and performance tests #5490

Manciukic · 2025-10-23T09:40:55Z

Changes

Add support for dynamic slots to make unplugged memory unaccessible from the guest
Add performance tests
Add mprotect to protect unplugged slots from the VMM device emulation.
Add tests for dirty page tracking and incremental snapshots.

Reason

virtio-mem feature work.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

This functionality cannot be added in rust-vmm.

Add a slot_cnt parameter to next_kvm_slot. This will be used to allocate multiple slots for a slotted hotpluggable region. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

To avoid boilerplate code in multiple places, let's just define it once and use it everywhere. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

In preparation to adding support for multiple memory slots, refactor the mincore_bitmap function to accept a pointer and length rather than an entire memory region object. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

codecov · 2025-10-23T15:05:50Z

Codecov Report

❌ Patch coverage is 97.34848% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.30%. Comparing base (f6e1740) to head (25fa432).

Files with missing lines	Patch %	Lines
src/vmm/src/vstate/memory.rs	97.17%	5 Missing ⚠️
src/vmm/src/builder.rs	75.00%	1 Missing ⚠️
src/vmm/src/vstate/vm.rs	98.21%	1 Missing ⚠️

Additional details and impacted files

@@                  Coverage Diff                   @@
##           feature/virtio-mem    #5490      +/-   ##
======================================================
+ Coverage               83.23%   83.30%   +0.07%     
======================================================
  Files                     276      276              
  Lines                   28666    28836     +170     
======================================================
+ Hits                    23861    24023     +162     
- Misses                   4805     4813       +8

Flag	Coverage Δ
5.10-m5n.metal	`83.51% <97.34%> (+0.07%)`	⬆️
5.10-m6a.metal	`82.83% <97.34%> (+0.07%)`	⬆️
5.10-m6g.metal	`80.29% <94.31%> (+0.10%)`	⬆️
5.10-m6i.metal	`83.52% <97.34%> (+0.07%)`	⬆️
5.10-m7a.metal-48xl	`82.82% <97.34%> (+0.07%)`	⬆️
5.10-m7g.metal	`80.29% <94.31%> (+0.11%)`	⬆️
5.10-m7i.metal-24xl	`83.48% <97.34%> (+0.06%)`	⬆️
5.10-m7i.metal-48xl	`83.48% <97.34%> (+0.06%)`	⬆️
5.10-m8g.metal-24xl	`80.29% <94.31%> (+0.10%)`	⬆️
5.10-m8g.metal-48xl	`80.29% <94.31%> (+0.10%)`	⬆️
6.1-m5n.metal	`83.55% <97.34%> (+0.07%)`	⬆️
6.1-m6a.metal	`82.87% <97.34%> (+0.08%)`	⬆️
6.1-m6g.metal	`80.29% <94.31%> (+0.10%)`	⬆️
6.1-m6i.metal	`83.54% <97.34%> (+0.07%)`	⬆️
6.1-m7a.metal-48xl	`82.85% <97.34%> (+0.08%)`	⬆️
6.1-m7g.metal	`80.29% <94.31%> (+0.10%)`	⬆️
6.1-m7i.metal-24xl	`83.55% <97.34%> (+0.07%)`	⬆️
6.1-m7i.metal-48xl	`83.55% <97.34%> (+0.06%)`	⬆️
6.1-m8g.metal-24xl	`80.28% <94.31%> (+0.10%)`	⬆️
6.1-m8g.metal-48xl	`80.29% <94.31%> (+0.11%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

A single GuestMemoryRegion can be split into multiple KVM slots. This is used for Hotplug type regions where we can dynamically remove access to the region from the guest. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

Dynamically plug/unplug KVM slots when any/all of the blocks are plugged/unplugged. This prevents the guest from accessing unplugged memory. However, this doesn't yet prevent the device emulation from accessing the slots. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

Add a performance test that measures the latency to hot(un)plug different amounts of memory. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

Add the memory hotplug tests to buildkite. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

This prevents the device emulation to be tricked into accessing unplugged memory ranges. If a malicious driver tries to do so, the VMM will crash with a memory error. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

These two functions were doing a similar thing. Let's extend the build_from_snapshot to support uffd and drop the newly introduced clone_vm. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

Extend the tests to also check that everything works with diff snapshots (dirty page tracking and mincore). Signed-off-by: Riccardo Mancini <mancio@amazon.com>

Add a test that verifies that incremental snapshots work as expected. Each generation will plug more memory and verify that the contents in memory from previous generations are persisted. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

kalyazin · 2025-10-27T11:40:28Z

src/vmm/src/vstate/vm.rs

    /// Obtain the next free kvm slot id
-    pub fn next_kvm_slot(&self) -> Option<u32> {
-        let next = self.common.next_kvm_slot.fetch_add(1, Ordering::Relaxed);
+    pub fn next_kvm_slot(&self, slot_cnt: u32) -> Option<u32> {


I think it'd be worth explaining what the new argument does.

kalyazin · 2025-10-27T11:45:27Z

src/vmm/src/vstate/memory.rs

+    pub slot_from: u32,
+    /// the size of the slots of this region
+    pub slot_size: usize,
+    /// a bitvec indicating whether region i is plugged into KVM (1) or not (0)


whether slot i is plugged?

correct, will fix

kalyazin · 2025-10-27T11:46:44Z

src/vmm/src/vstate/memory.rs

+    pub plugged: Mutex<BitVec>,
+}
+
+/// A snapshot of the state of a memory slot


what does "snapshot of the state" mean here?

I mean that it's a point-in-time view of the state of the memory slot. Maybe it'd be less confusing "The current state of a memory slot"

kalyazin · 2025-10-27T11:49:19Z

src/vmm/src/vstate/memory.rs

+        slot_cnt: usize,
+    ) -> Self {
+        // Caller should pass a valid slot size and count for this region
+        assert!(slot_size * slot_cnt == u64_to_usize(region.len()));


nit: can we infer slot_size or slot_cnt instead?

I could recompute the slot_cnt that is computed in the caller already. I agree it's cleaner.

kalyazin · 2025-10-27T11:55:35Z

resources/seccomp/aarch64-unknown-linux-musl.json

+                        "type": "dword",
+                        "op": "eq",
+                        "val": 1075883590,
+                        "comment": "KVM_SET_USER_MEMORY_REGION"


nit: can explain why we need it here

kalyazin · 2025-10-27T11:58:31Z

src/vmm/src/devices/virtio/mem/device.rs

+            .guest_memory()
+            .iter()
+            .find(|r| r.region_type == GuestRegionType::Hotpluggable)
+            .expect("there should be one and only one hotpluggable region");


how do we verify it's the only one?

It's not verified, if there are multiple it's gonna return the first one. I specified it in the same sentence because I didn't want to leave a separate comment.

kalyazin · 2025-10-27T12:08:05Z

src/vmm/src/vstate/memory.rs

 }

+impl<'a> GuestMemorySlot<'a> {
+    pub(crate) fn protect(&self, accessible: bool) -> Result<(), MemoryError> {


nit: the reverse logic of the boolean may read more smoothly: protect(true) would mean "protection enabled".

Manciukic added 3 commits October 23, 2025 15:46

refactor: allow reserving multiple kvm slots at once

df8c61f

Add a slot_cnt parameter to next_kvm_slot. This will be used to allocate multiple slots for a slotted hotpluggable region. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

refactor: define a set_user_memory_region function in Vm

1ae1596

To avoid boilerplate code in multiple places, let's just define it once and use it everywhere. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

refactor(mincore): pass addr and len to mincore_bitmap

16eaf6f

In preparation to adding support for multiple memory slots, refactor the mincore_bitmap function to accept a pointer and length rather than an entire memory region object. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

Manciukic force-pushed the virtio-mem/mprotect branch from a94732d to 418d9be Compare October 23, 2025 14:58

Manciukic added 2 commits October 23, 2025 16:06

feat(mem): introduce KVM slots per GuestMemoryRegion

555135e

A single GuestMemoryRegion can be split into multiple KVM slots. This is used for Hotplug type regions where we can dynamically remove access to the region from the guest. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

Manciukic force-pushed the virtio-mem/mprotect branch from 418d9be to 9f5740e Compare October 23, 2025 15:07

Manciukic added 2 commits October 23, 2025 16:10

test(virtio-mem): add performance test

45d927e

Add a performance test that measures the latency to hot(un)plug different amounts of memory. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

chore(bk): add memory-hotplug pipeline definition to perf tests

f272beb

Add the memory hotplug tests to buildkite. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

Manciukic force-pushed the virtio-mem/mprotect branch from 9f5740e to 84671d9 Compare October 23, 2025 15:18

Manciukic added 4 commits October 23, 2025 16:21

feat(virtio-mem): mprotect unplugged memory ranges

c926344

This prevents the device emulation to be tricked into accessing unplugged memory ranges. If a malicious driver tries to do so, the VMM will crash with a memory error. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

refactor: remove clone_vm and extend build_from_snapshot to accept uffd

c76904f

These two functions were doing a similar thing. Let's extend the build_from_snapshot to support uffd and drop the newly introduced clone_vm. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

test(virtio-mem): test multiple snapshot methods

fd6bcff

Extend the tests to also check that everything works with diff snapshots (dirty page tracking and mincore). Signed-off-by: Riccardo Mancini <mancio@amazon.com>

test(virtio-mem): test incremental snapshots

25fa432

Add a test that verifies that incremental snapshots work as expected. Each generation will plug more memory and verify that the contents in memory from previous generations are persisted. Signed-off-by: Riccardo Mancini <mancio@amazon.com>

Manciukic force-pushed the virtio-mem/mprotect branch from 84671d9 to 25fa432 Compare October 23, 2025 15:23

Manciukic marked this pull request as ready for review October 23, 2025 16:03

Manciukic added the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Oct 23, 2025

kalyazin reviewed Oct 27, 2025

View reviewed changes

Uh oh!

[virtio-mem] Dynamic slots and performance tests #5490

Are you sure you want to change the base?

[virtio-mem] Dynamic slots and performance tests #5490

Uh oh!

Conversation

Manciukic commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Reason

License Acceptance

PR Checklist

Uh oh!

codecov bot commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Manciukic commented Oct 23, 2025 •

edited

Loading

codecov bot commented Oct 23, 2025 •

edited

Loading