feat: reset KVM_REG_ARM_PTIMER_CNT on VM boot #4987

ShadowCurse · 2025-01-10T13:00:37Z

Reset KVM_REG_ARM_PTIMER_CNT physical counter register on VM boot to avoid passing through host physical counter. Note that resetting the register on VM boot does not guarantee that VM will see the counter value 0 at startup because there is a delta in time between register reset and VM boot during which counter continues to advance.

Reason

Prevent guest from reading host performance counter.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

I have read and understand CONTRIBUTING.md.
I have run tools/devtool checkstyle to verify that the PR passes the
automated style checks.
I have described what is done in these changes, why they are needed, and
how they are solving the problem in a clear and encompassing way.
I have updated any relevant documentation (both in code and in the docs)
in the PR.
I have mentioned all user-facing changes in CHANGELOG.md.
If a specific issue led to this PR, this PR closes the issue.
When making API changes, I have followed the
Runbook for Firecracker API changes.
I have tested all new and changed functionalities in unit tests and/or
integration tests.
I have linked an issue to every new TODO.

This functionality cannot be added in rust-vmm.

codecov · 2025-01-10T13:15:37Z

Codecov Report

Attention: Patch coverage is 93.75000% with 8 lines in your changes missing coverage. Please review.

Project coverage is 83.10%. Comparing base (3fb06e9) to head (d59728f).
Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
src/vmm/src/vstate/kvm.rs	94.36%	4 Missing ⚠️
src/vmm/src/arch/aarch64/vcpu.rs	50.00%	3 Missing ⚠️
src/vmm/src/builder.rs	97.43%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4987      +/-   ##
==========================================
+ Coverage   83.06%   83.10%   +0.04%     
==========================================
  Files         244      245       +1     
  Lines       26658    26723      +65     
==========================================
+ Hits        22144    22209      +65     
  Misses       4514     4514

Flag	Coverage Δ
5.10-c5n.metal	`83.57% <88.42%> (-0.01%)`	⬇️
5.10-m5n.metal	`83.55% <88.42%> (-0.01%)`	⬇️
5.10-m6a.metal	`82.76% <88.42%> (-0.02%)`	⬇️
5.10-m6g.metal	`79.57% <91.96%> (+0.13%)`	⬆️
5.10-m6i.metal	`83.54% <88.42%> (-0.02%)`	⬇️
5.10-m7g.metal	`79.57% <91.96%> (+0.13%)`	⬆️
6.1-c5n.metal	`83.57% <88.42%> (-0.01%)`	⬇️
6.1-m5n.metal	`?`
6.1-m6a.metal	`82.76% <88.42%> (-0.01%)`	⬇️
6.1-m6g.metal	`79.57% <91.96%> (+0.13%)`	⬆️
6.1-m6i.metal	`83.54% <88.42%> (-0.02%)`	⬇️
6.1-m7g.metal	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

src/vmm/src/arch/aarch64/vcpu.rs

src/vmm/src/arch/aarch64/regs.rs

CHANGELOG.md

src/vmm/src/arch/aarch64/vcpu.rs

tests/integration_tests/functional/test_snapshot_basic.py

roypat

feel free to ignore the nits, the changelog comment is the only one preventing me from hitting approve xP

src/vmm/src/arch/aarch64/vcpu.rs

src/vmm/src/builder.rs

CHANGELOG.md

docs/prod-host-setup.md

tests/integration_tests/functional/test_snapshot_basic.py

src/vmm/src/vstate/kvm.rs

CHANGELOG.md

`Vm` constructor was the only place where the `/dev/kvm` was open and only there we could do any KVM (not VM) specific checks. By moving this KVM logic into a separate struct we can can do KVM specific actions (like checking optional KVM capabilities) without needing to reopen the `/dev/kvm` again. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

tests/integration_tests/functional/test_snapshot_basic.py

Reset KVM_REG_ARM_PTIMER_CNT physical counter register on VM boot to avoid passing through host physical counter. Note that resetting the register on VM boot does not guarantee that VM will see the counter value 0 at startup because there is a delta in time between register reset and VM boot during which counter continues to advance. In order to check if the kernel supports the counter reset we query KVM_CAP_COUNTER_OFFSET capability and only reset the KVM_REG_ARM_PTIMER_CNT if it is present. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Update a note about physical counter on ARM being reset instead of directly passed through on kernels with `KVM_CAP_COUNTER_OFFSET` capability. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add an entry about physical counter reset to the CHANGELOG. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add a test to verify the reset of the physical counter on aarch64 VMs. To do this we check registers saved in the snapshot and verify the counter value is less than some reasonably small number we choose. The value is based on the observation of how much cycles it takes for a VM to boot and be snapshotted. The idea is that this value will always be smaller than the actual physical counter on the host. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse force-pushed the aarch64_counter_reset branch from b34d97d to b459ae3 Compare January 10, 2025 13:11

ShadowCurse force-pushed the aarch64_counter_reset branch 2 times, most recently from 1bd3f94 to 5ea2f4a Compare January 10, 2025 14:24

ShadowCurse changed the title ~~feat: reset SYS_CNTPCT_EL0 on VM boot~~ feat: reset KVM_REG_ARM_PTIMER_CNT on VM boot Jan 10, 2025

ShadowCurse force-pushed the aarch64_counter_reset branch 4 times, most recently from 75a7ec1 to 053f265 Compare January 10, 2025 15:19

ShadowCurse marked this pull request as ready for review January 10, 2025 15:20

ShadowCurse requested review from xmarcalx, kalyazin, pb8o and Manciukic as code owners January 10, 2025 15:20

ShadowCurse self-assigned this Jan 10, 2025

ShadowCurse added Status: Awaiting review Indicates that a pull request is ready to be reviewed Type: Fix Indicates a fix to existing code labels Jan 10, 2025

roypat reviewed Jan 10, 2025

View reviewed changes

src/vmm/src/arch/aarch64/vcpu.rs Outdated Show resolved Hide resolved

kalyazin reviewed Jan 10, 2025

View reviewed changes

ShadowCurse force-pushed the aarch64_counter_reset branch 2 times, most recently from d3abf67 to 57b458b Compare January 10, 2025 16:31

pb8o reviewed Jan 10, 2025

View reviewed changes

src/vmm/src/arch/aarch64/vcpu.rs Show resolved Hide resolved

ShadowCurse force-pushed the aarch64_counter_reset branch from f553778 to 0ab122a Compare January 13, 2025 18:30

pb8o reviewed Jan 14, 2025

View reviewed changes

tests/integration_tests/functional/test_snapshot_basic.py Outdated Show resolved Hide resolved

tests/integration_tests/functional/test_snapshot_basic.py Outdated Show resolved Hide resolved

ShadowCurse force-pushed the aarch64_counter_reset branch 6 times, most recently from e7d282b to 98592f7 Compare January 14, 2025 18:52

ShadowCurse requested a review from roypat January 15, 2025 17:12

roypat reviewed Jan 15, 2025

View reviewed changes

ShadowCurse force-pushed the aarch64_counter_reset branch 3 times, most recently from d767541 to 1801a5f Compare January 16, 2025 09:12

roypat previously approved these changes Jan 16, 2025

View reviewed changes

ShadowCurse force-pushed the aarch64_counter_reset branch 2 times, most recently from caf5d9d to bfc5a84 Compare January 16, 2025 13:24

ShadowCurse dismissed roypat’s stale review via 2b0aa20 January 16, 2025 15:02

ShadowCurse force-pushed the aarch64_counter_reset branch 2 times, most recently from 2b0aa20 to 406c2c4 Compare January 16, 2025 15:51

roypat previously approved these changes Jan 17, 2025

View reviewed changes

kalyazin reviewed Jan 17, 2025

View reviewed changes

ShadowCurse dismissed roypat’s stale review via 759d750 January 17, 2025 13:12

ShadowCurse force-pushed the aarch64_counter_reset branch from 406c2c4 to 759d750 Compare January 17, 2025 13:12

ShadowCurse force-pushed the aarch64_counter_reset branch 2 times, most recently from 649235d to 13b30a2 Compare January 17, 2025 13:21

kalyazin reviewed Jan 17, 2025

View reviewed changes

tests/integration_tests/functional/test_snapshot_basic.py Outdated Show resolved Hide resolved

tests/integration_tests/functional/test_snapshot_basic.py Outdated Show resolved Hide resolved

ShadowCurse force-pushed the aarch64_counter_reset branch 2 times, most recently from 355cc54 to 82afe20 Compare January 17, 2025 14:07

ShadowCurse added 4 commits January 17, 2025 14:08

chore: update prod-host-setup.md with arm physical counter info

b29ab67

Update a note about physical counter on ARM being reset instead of directly passed through on kernels with `KVM_CAP_COUNTER_OFFSET` capability. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

chore: add an entry to the CHANGELOG

a9a7e68

Add an entry about physical counter reset to the CHANGELOG. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse force-pushed the aarch64_counter_reset branch from 82afe20 to d59728f Compare January 17, 2025 14:08

kalyazin approved these changes Jan 17, 2025

View reviewed changes

JackThomson2 approved these changes Jan 17, 2025

View reviewed changes

ShadowCurse merged commit 525e686 into firecracker-microvm:main Jan 17, 2025
7 checks passed

ShadowCurse deleted the aarch64_counter_reset branch January 17, 2025 15:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: reset KVM_REG_ARM_PTIMER_CNT on VM boot #4987

feat: reset KVM_REG_ARM_PTIMER_CNT on VM boot #4987

ShadowCurse commented Jan 10, 2025 •

edited

Loading

codecov bot commented Jan 10, 2025 •

edited

Loading

roypat left a comment

feat: reset KVM_REG_ARM_PTIMER_CNT on VM boot #4987

feat: reset KVM_REG_ARM_PTIMER_CNT on VM boot #4987

Conversation

ShadowCurse commented Jan 10, 2025 • edited Loading

Reason

License Acceptance

PR Checklist

codecov bot commented Jan 10, 2025 • edited Loading

Codecov Report

roypat left a comment

Choose a reason for hiding this comment

ShadowCurse commented Jan 10, 2025 •

edited

Loading

codecov bot commented Jan 10, 2025 •

edited

Loading