Add support for Hyperlight KVM guest debugging using gdb #111

dblnz · 2024-12-13T17:54:02Z

The current implementation supports:

add/remove HW breakpoints (max 4)
add/remove SW breakpoints
read/write registers
read/write addresses
step/continue
get code offset from target

The overall architecture is designed to work like a Request - Response protocol from the gdb thread to the hypervisor handler over a DebugCommChannel.

All the functionality is implemented on the hypervisor side so it has access to the shared memory and VcpuFd
The gdb thread uses gdbstub to handle the communication with the gdb client
when the gdb client requests the supported features mentioned above, a request is sent forward to the hypervisor handler for the running sandbox to run

devigned

Super excited to see this landing. Nice work! My feedback mostly consists of nits and questions.

docs/how-to-debug-a-hyperlight-guest.md

src/hyperlight_host/src/hypervisor/gdb/mod.rs

src/hyperlight_host/src/hypervisor/gdb/target.rs

src/hyperlight_host/src/hypervisor/hypervisor_handler.rs

src/hyperlight_host/src/hypervisor/kvm.rs

src/hyperlight_host/src/hypervisor/hypervisor_handler.rs

ludfjig

This is awesome!! Looked briefly though everything and I will give a more in-depth review tomorrow

src/hyperlight_host/Cargo.toml

src/hyperlight_host/src/hypervisor/gdb/mod.rs

src/hyperlight_host/src/sandbox/config.rs

src/hyperlight_host/src/hypervisor/gdb/x86_64_target.rs

ludfjig

This is awesome. I tried it out and it works great. Left some more comments, feel free to dismiss if you disagree with any of them. I realize some of them might be hard to implement as well. GREAT WORK!!

src/hyperlight_host/src/sandbox/config.rs

docs/how-to-debug-a-hyperlight-guest.md

src/hyperlight_host/Cargo.toml

src/hyperlight_host/src/hypervisor/kvm.rs

src/hyperlight_host/src/hypervisor/gdb/event_loop.rs

src/hyperlight_host/src/hypervisor/gdb/mod.rs

docs/how-to-debug-a-hyperlight-guest.md

simongdavies

a few minor issues, but overall this is great. I don't know why the spelling issues are not getting picked up by the spell check in GH actions

docs/how-to-debug-a-hyperlight-guest.md

src/hyperlight_host/build.rs

src/hyperlight_host/src/hypervisor/gdb/mod.rs

src/hyperlight_host/src/hypervisor/kvm.rs

src/hyperlight_host/src/sandbox/mem_access.rs

syntactically

I really like how this looks overall! I've left a few small things in the code comments. I also have a couple of nits about the commit organization:

Commit message for 579b324: s/tread/thread/

Commits b175f41, af0a52a, 6cc6184, and 98ac60e: please convert
each one to a series of fixup! commits to the original commits
introducing code, so that we can autosquash before merging, or just
rebase the changes into the original commits now if nobody else
objects to that.

src/hyperlight_host/src/hypervisor/hypervisor_handler.rs

src/hyperlight_host/src/sandbox/config.rs

src/hyperlight_host/src/sandbox/uninitialized.rs

src/hyperlight_host/src/hypervisor/gdb/event_loop.rs

src/hyperlight_host/src/hypervisor/gdb/x86_64_target.rs

src/hyperlight_host/src/sandbox/mem_access.rs

src/hyperlight_host/src/hypervisor/gdb/mod.rs

src/hyperlight_host/src/hypervisor/gdb/x86_64_target.rs

src/hyperlight_host/src/hypervisor/kvm.rs

ludfjig

LGTM!

docs/how-to-debug-a-hyperlight-guest.md

devigned

lgtm! Great work, @dblnz!

src/hyperlight_host/src/hypervisor/gdb/event_loop.rs

src/hyperlight_host/src/hypervisor/handlers.rs

src/hyperlight_host/src/hypervisor/kvm.rs

…en gdb feature is on Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

- this type will be used by the gdb and the hypervisor handler to send requests and receive responses Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

- the target implements the traits to provide callbacks for the gdb commands Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

- adds a specific handler for the vcpu exit debug that waits for debug messages and processes them Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

- also adds handling of gdb client disconnect by resuming execution Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

- sometimes gdb tries to read from an address where the vcpu translate method fails, so we need to handle that in such a way that the hypervisor handler doesn't crash Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

- this helps with separation and testing Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

- replace `expect` statements with logic to propagate error up the call chain - improve log messages and comments Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

dblnz added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Dec 13, 2024

dblnz self-assigned this Dec 13, 2024

dblnz marked this pull request as draft December 13, 2024 17:54

dblnz force-pushed the gdb-support-latest branch 2 times, most recently from be21b85 to ca61def Compare December 19, 2024 11:41

devigned requested changes Jan 22, 2025

View reviewed changes

dblnz force-pushed the gdb-support-latest branch from 4a44b2e to 16492de Compare January 22, 2025 15:40

dblnz marked this pull request as ready for review January 22, 2025 16:07

dblnz changed the title ~~Initial support for gdb debugging in Hyperlight KVM guest~~ Add support for Hyperlight KVM guest debugging using gdb Jan 22, 2025

danbugs reviewed Jan 28, 2025

View reviewed changes

src/hyperlight_host/src/hypervisor/hypervisor_handler.rs Outdated Show resolved Hide resolved

src/hyperlight_host/src/hypervisor/hypervisor_handler.rs Outdated Show resolved Hide resolved

dblnz force-pushed the gdb-support-latest branch from 2a36f06 to d0db26e Compare February 4, 2025 21:07

ludfjig previously approved these changes Feb 5, 2025

View reviewed changes

dblnz dismissed ludfjig’s stale review via 6cc6184 February 5, 2025 12:57

dblnz force-pushed the gdb-support-latest branch from 17a5d14 to 98ac60e Compare February 5, 2025 16:06

ludfjig previously approved these changes Feb 7, 2025

View reviewed changes

simongdavies requested changes Feb 11, 2025

View reviewed changes

syntactically reviewed Feb 11, 2025

View reviewed changes

dblnz dismissed ludfjig’s stale review via 459b1c9 February 13, 2025 17:29

dblnz force-pushed the gdb-support-latest branch from 98ac60e to 459b1c9 Compare February 13, 2025 17:29

dblnz requested a review from devigned February 13, 2025 17:59

dblnz force-pushed the gdb-support-latest branch 6 times, most recently from 4e8d0d1 to 21d3079 Compare February 18, 2025 17:13

ludfjig previously approved these changes Feb 19, 2025

View reviewed changes

docs/how-to-debug-a-hyperlight-guest.md Show resolved Hide resolved

simongdavies previously approved these changes Feb 19, 2025

View reviewed changes

devigned previously approved these changes Feb 19, 2025

View reviewed changes

syntactically reviewed Feb 19, 2025

View reviewed changes

src/hyperlight_host/src/hypervisor/gdb/event_loop.rs Show resolved Hide resolved

src/hyperlight_host/src/hypervisor/handlers.rs Show resolved Hide resolved

src/hyperlight_host/src/hypervisor/kvm.rs Outdated Show resolved Hide resolved

dblnz added 17 commits February 21, 2025 12:16

disable timeouts for messaging between host and hypervisor handler wh…

ca9bb89

…en gdb feature is on Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add guest debug port to sandbox configuration

793f66c

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add gdb debug thread creation method

9d3e0b1

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add debug communication channel type

3ae6d7d

- this type will be used by the gdb and the hypervisor handler to send requests and receive responses Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add hyperlight sandbox target to handle gdb commands support

0079d7f

- the target implements the traits to provide callbacks for the gdb commands Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

handle debug vcpu exit on kvm side

6d1871d

- adds a specific handler for the vcpu exit debug that waits for debug messages and processes them Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add kvm debug configuration that handles vCPU interaction

1d2bdca

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add read/write registers support

9a73d85

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

hw breakpoint and resume support added

5479831

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add stepping support

ad4b582

- also adds handling of gdb client disconnect by resuming execution Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add a way to read/write host shared mem when debugging

323c6c5

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add read/write address and text section offset support

b60b808

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add sw breakpoint support

5ee059d

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add ci checks for the gdb feature

a0bd86d

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add documentation about the gdb debugging feature

6440a19

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add a way to report a non-fatal error to gdb

b18ff56

- sometimes gdb tries to read from an address where the vcpu translate method fails, so we need to handle that in such a way that the hypervisor handler doesn't crash Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

add CI gdb test

cfaefc3

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

dblnz dismissed stale reviews from devigned, simongdavies, and ludfjig via c722d03 February 21, 2025 12:34

dblnz force-pushed the gdb-support-latest branch from 21d3079 to c722d03 Compare February 21, 2025 12:34

dblnz added 3 commits February 21, 2025 14:43

add support for interrupts

0a03f81

Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

move gdb thread creation in hypervisor handler

251cdb4

- this helps with separation and testing Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

improve error handling and avoid panicking

7ab2149

- replace `expect` statements with logic to propagate error up the call chain - improve log messages and comments Signed-off-by: Doru Blânzeanu <dblnz@pm.me>

dblnz force-pushed the gdb-support-latest branch from c722d03 to 7ab2149 Compare February 21, 2025 12:47

syntactically approved these changes Feb 21, 2025

View reviewed changes

ludfjig approved these changes Feb 21, 2025

View reviewed changes

dblnz merged commit 757fb9b into hyperlight-dev:main Feb 21, 2025
21 checks passed

dblnz mentioned this pull request Mar 4, 2025

Hyperlight Debugging (KVM) #221

Closed

dblnz deleted the gdb-support-latest branch March 4, 2025 12:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Hyperlight KVM guest debugging using gdb #111

Add support for Hyperlight KVM guest debugging using gdb #111

dblnz commented Dec 13, 2024 •

edited

Loading

devigned left a comment

ludfjig left a comment

ludfjig left a comment

simongdavies left a comment

syntactically left a comment

ludfjig left a comment

devigned left a comment

Add support for Hyperlight KVM guest debugging using gdb #111

Add support for Hyperlight KVM guest debugging using gdb #111

Conversation

dblnz commented Dec 13, 2024 • edited Loading

devigned left a comment

Choose a reason for hiding this comment

ludfjig left a comment

Choose a reason for hiding this comment

ludfjig left a comment

Choose a reason for hiding this comment

simongdavies left a comment

Choose a reason for hiding this comment

syntactically left a comment

Choose a reason for hiding this comment

ludfjig left a comment

Choose a reason for hiding this comment

devigned left a comment

Choose a reason for hiding this comment

dblnz commented Dec 13, 2024 •

edited

Loading