kvm_pvm: Inject event in non-PVM mode

[HangShuYH]

Hello, while testing with the code from the pvm-612 branch, the command
sudo nerdctl run --net=none --cpu-shares 512 --runtime "io.containerd.kata.v2" --rm -t --name test-kata "$image" date

failed with the following error:
FATA[0003] failed to create shim task: Failed to Check if grpc server is working: ttrpc: closed: unknown

The host kernel produced the following warnings:

[  196.823963] kvm_pvm: Inject event in non-PVM mode
[  196.823999] ------------[ cut here ]------------
[  196.825039] WARNING: CPU: 0 PID: 314 at arch/x86/kvm/../../../virt/kvm/pfncache.c:298 __kvm_gpc_refresh+0x3b8/0x3d0 [kvm]
[  196.826171] Modules linked in: vhost_vsock vmw_vsock_virtio_transport_common vsock vhost vhost_iotlb kvm_pvm overlay ppdev kvm irqbypass crct10dif_pclmul crct10dif_common crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd serio_raw parport_pc parport [last unloaded: kvm_intel]
[  196.828616] CPU: 0 UID: 0 PID: 314 Comm: qemu-system-x86 Not tainted 6.12.33+ #6
[  196.829338] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[  196.830301] RIP: 0010:__kvm_gpc_refresh+0x3b8/0x3d0 [kvm]
[  196.830903] Code: 7d 01 4d 89 6c 24 70 49 8b 44 24 10 25 ff 0f 00 00 48 01 c5 49 89 6c 24 68 4c 89 ef e8 01 33 ff ff 49 8b 5c 24 70 31 ed eb 87 <0f> 0b bd ea ff ff ff eb b3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00
[  196.832691] RSP: 0018:ff3847b0c093bb78 EFLAGS: 00010246
[  196.833211] RAX: ff1b0f09826e1701 RBX: ff1b0f09826e1740 RCX: ff1b0f09826e1700
[  196.833915] RDX: ff1b0f0880000000 RSI: ffffffffffffffff RDI: ff1b0f09826e16f8
[  196.834611] RBP: 00000000ffffffea R08: 0000000000000000 R09: ffffffff9b448f40
[  196.835310] R10: 0000000000005ffd R11: 0000000000000004 R12: ff1b0f09826e16f8
[  196.836022] R13: 0000000000000000 R14: 0000000000001000 R15: ff1b0f09826e16f8
[  196.836714] FS:  00007f0823ffd700(0000) GS:ff1b0f0d13ee2000(0000) knlGS:0000000000000000
[  196.837500] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  196.838070] CR2: 00007f08240befff CR3: 000000010882a005 CR4: 0000000000771ef0
[  196.838818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  196.839511] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[  196.840203] PKRU: 55555554
[  196.840483] Call Trace:
[  196.840739]  <TASK>
[  196.840969]  kvm_gpc_refresh+0x60/0x80 [kvm]
[  196.841443]  __do_pvm_event+0x56/0x320 [kvm_pvm]
[  196.841911]  pvm_inject_irq+0x4e/0xf0 [kvm_pvm]
[  196.842366]  vcpu_run+0x1067/0x1be0 [kvm]
[  196.842822]  ? fpu_swap_kvm_fpstate+0x30/0x1e0
[  196.843265]  ? __local_bh_enable_ip+0x58/0x80
[  196.843704]  ? fpu_swap_kvm_fpstate+0x1c3/0x1e0
[  196.844159]  ? fpu_swap_kvm_fpstate+0x30/0x1e0
[  196.844603]  kvm_arch_vcpu_ioctl_run+0x482/0x650 [kvm]
[  196.845160]  ? kvm_set_irq+0x13f/0x1f0 [kvm]
[  196.845634]  ? __pfx_kvm_set_ioapic_irq+0x10/0x10 [kvm]
[  196.846195]  ? __pfx_kvm_set_pic_irq+0x10/0x10 [kvm]
[  196.846736]  ? __count_memcg_events+0x80/0x110
[  196.847866]  ? handle_mm_fault+0x1306/0x1860
[  196.848293]  ? kvm_vm_ioctl_irq_line+0x28/0x40 [kvm]
[  196.848854]  kvm_vcpu_ioctl+0x53b/0x600 [kvm]
[  196.849333]  __se_sys_ioctl+0x66/0xc0
[  196.850336]  do_syscall_64+0x5a/0x120
[  196.850726]  ? clear_bhb_loop+0x70/0xc0
[  196.851119]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  196.851652] RIP: 0033:0x7f08ba11453b
[  196.852022] Code: 0f 92 c0 84 c0 75 b0 49 8d 3c 1c e8 df 40 04 00 85 c0 78 b1 48 83 c4 08 4c 89 e0 5b 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a0 ff ff ff f7 d8 64 89 01 48
[  196.853793] RSP: 002b:00007f0823ffc788 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  196.854529] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f08ba11453b
[  196.855215] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000014
[  196.855916] RBP: 0000555558bcca90 R08: 00007f08ba259610 R09: 00000000000000ff
[  196.856607] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
[  196.857327] R13: 00007f08baa98e20 R14: 0000000000000001 R15: 0000000000000000
[  196.858051]  </TASK>
[  196.858294] ---[ end trace 0000000000000000 ]---

I used QEMU to boot the host kernel and then launched a guest kernel inside the host kernel.
Both the host kernel and guest kernel were compiled from the PVM-612 branch.
However, the execution succeeded when using kvm-intel.

I would like to know whether this is due to incomplete support in version 6.12 or if this is a bug.

[HangShuYH]

I retried pvm-612 following the guide. The host kernel warning disappreared. However, the error FATA[0003] failed to create shim task: Failed to Check if grpc server is working: ttrpc: closed: unknown still exists.

[HangShuYH]

I tried to use kvm_intel to run the guest kernel. I found that the problem still exists.
Then I replaced the vmlinux.pvm with vmlinux.container. I found that the problem disappeared.
I wonder if there is something wrong with the guest kernel(compiled with pvm-guest-6.12.33.config and pvm-612 branch)?

[bysui]

I tried to use kvm_intel to run the guest kernel. I found that the problem still exists. Then I replaced the vmlinux.pvm with vmlinux.container. I found that the problem disappeared. I wonder if there is something wrong with the guest kernel(compiled with pvm-guest-6.12.33.config and pvm-612 branch)?

Hello, while testing with the code from the pvm-612 branch, the command sudo nerdctl run --net=none --cpu-shares 512 --runtime "io.containerd.kata.v2" --rm -t --name test-kata "$image" date

failed with the following error: FATA[0003] failed to create shim task: Failed to Check if grpc server is working: ttrpc: closed: unknown

The host kernel produced the following warnings:

[  196.823963] kvm_pvm: Inject event in non-PVM mode
[  196.823999] ------------[ cut here ]------------
[  196.825039] WARNING: CPU: 0 PID: 314 at arch/x86/kvm/../../../virt/kvm/pfncache.c:298 __kvm_gpc_refresh+0x3b8/0x3d0 [kvm]
[  196.826171] Modules linked in: vhost_vsock vmw_vsock_virtio_transport_common vsock vhost vhost_iotlb kvm_pvm overlay ppdev kvm irqbypass crct10dif_pclmul crct10dif_common crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd serio_raw parport_pc parport [last unloaded: kvm_intel]
[  196.828616] CPU: 0 UID: 0 PID: 314 Comm: qemu-system-x86 Not tainted 6.12.33+ #6
[  196.829338] Hardware name: QEMU Ubuntu 24.04 PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[  196.830301] RIP: 0010:__kvm_gpc_refresh+0x3b8/0x3d0 [kvm]
[  196.830903] Code: 7d 01 4d 89 6c 24 70 49 8b 44 24 10 25 ff 0f 00 00 48 01 c5 49 89 6c 24 68 4c 89 ef e8 01 33 ff ff 49 8b 5c 24 70 31 ed eb 87 <0f> 0b bd ea ff ff ff eb b3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00
[  196.832691] RSP: 0018:ff3847b0c093bb78 EFLAGS: 00010246
[  196.833211] RAX: ff1b0f09826e1701 RBX: ff1b0f09826e1740 RCX: ff1b0f09826e1700
[  196.833915] RDX: ff1b0f0880000000 RSI: ffffffffffffffff RDI: ff1b0f09826e16f8
[  196.834611] RBP: 00000000ffffffea R08: 0000000000000000 R09: ffffffff9b448f40
[  196.835310] R10: 0000000000005ffd R11: 0000000000000004 R12: ff1b0f09826e16f8
[  196.836022] R13: 0000000000000000 R14: 0000000000001000 R15: ff1b0f09826e16f8
[  196.836714] FS:  00007f0823ffd700(0000) GS:ff1b0f0d13ee2000(0000) knlGS:0000000000000000
[  196.837500] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  196.838070] CR2: 00007f08240befff CR3: 000000010882a005 CR4: 0000000000771ef0
[  196.838818] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  196.839511] DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
[  196.840203] PKRU: 55555554
[  196.840483] Call Trace:
[  196.840739]  <TASK>
[  196.840969]  kvm_gpc_refresh+0x60/0x80 [kvm]
[  196.841443]  __do_pvm_event+0x56/0x320 [kvm_pvm]
[  196.841911]  pvm_inject_irq+0x4e/0xf0 [kvm_pvm]
[  196.842366]  vcpu_run+0x1067/0x1be0 [kvm]
[  196.842822]  ? fpu_swap_kvm_fpstate+0x30/0x1e0
[  196.843265]  ? __local_bh_enable_ip+0x58/0x80
[  196.843704]  ? fpu_swap_kvm_fpstate+0x1c3/0x1e0
[  196.844159]  ? fpu_swap_kvm_fpstate+0x30/0x1e0
[  196.844603]  kvm_arch_vcpu_ioctl_run+0x482/0x650 [kvm]
[  196.845160]  ? kvm_set_irq+0x13f/0x1f0 [kvm]
[  196.845634]  ? __pfx_kvm_set_ioapic_irq+0x10/0x10 [kvm]
[  196.846195]  ? __pfx_kvm_set_pic_irq+0x10/0x10 [kvm]
[  196.846736]  ? __count_memcg_events+0x80/0x110
[  196.847866]  ? handle_mm_fault+0x1306/0x1860
[  196.848293]  ? kvm_vm_ioctl_irq_line+0x28/0x40 [kvm]
[  196.848854]  kvm_vcpu_ioctl+0x53b/0x600 [kvm]
[  196.849333]  __se_sys_ioctl+0x66/0xc0
[  196.850336]  do_syscall_64+0x5a/0x120
[  196.850726]  ? clear_bhb_loop+0x70/0xc0
[  196.851119]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[  196.851652] RIP: 0033:0x7f08ba11453b
[  196.852022] Code: 0f 92 c0 84 c0 75 b0 49 8d 3c 1c e8 df 40 04 00 85 c0 78 b1 48 83 c4 08 4c 89 e0 5b 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a0 ff ff ff f7 d8 64 89 01 48
[  196.853793] RSP: 002b:00007f0823ffc788 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  196.854529] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f08ba11453b
[  196.855215] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000014
[  196.855916] RBP: 0000555558bcca90 R08: 00007f08ba259610 R09: 00000000000000ff
[  196.856607] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000000
[  196.857327] R13: 00007f08baa98e20 R14: 0000000000000001 R15: 0000000000000000
[  196.858051]  </TASK>
[  196.858294] ---[ end trace 0000000000000000 ]---

I used QEMU to boot the host kernel and then launched a guest kernel inside the host kernel. Both the host kernel and guest kernel were compiled from the PVM-612 branch. However, the execution succeeded when using kvm-intel.

I would like to know whether this is due to incomplete support in version 6.12 or if this is a bug.

Hi, @HangShuYH, sorry for reply later.

The warning here indicates that an interrupt is injected in non-PVM mode, where the PVM performs instruction emulation instead of executing the instruction directly on the CPU. Which L1 VMM are you using? Since RustVMMs like CloudHypervisor and Firecracker usually boot directly into 64-bit mode, I assume you are using QEMU as the L1 VMM. Have you configured qboot as the BIOS in the configuration file, since PVM currently only supports qboot instead of the default SeaBIOS?

[bysui]

I tried to use kvm_intel to run the guest kernel. I found that the problem still exists. Then I replaced the vmlinux.pvm with vmlinux.container. I found that the problem disappeared. I wonder if there is something wrong with the guest kernel(compiled with pvm-guest-6.12.33.config and pvm-612 branch)?

Uh, can you use QEMU to boot the guest kernel with the kvm_intel module? If so, which version of Kata Containers are you using? I'm not sure if the guest configuration is incompatible with your Kata Containers.

[HangShuYH]

I tried to use kvm_intel to run the guest kernel. I found that the problem still exists. Then I replaced the vmlinux.pvm with vmlinux.container. I found that the problem disappeared. I wonder if there is something wrong with the guest kernel(compiled with pvm-guest-6.12.33.config and pvm-612 branch)?

Uh, can you use QEMU to boot the guest kernel with the kvm_intel module? If so, which version of Kata Containers are you using? I'm not sure if the guest configuration is incompatible with your Kata Containers.

I used the following command to boot the guest kernel with kvm_intel module. Everything is okay.

sudo qemu-system-x86_64 \
        -m 2048M -smp 8 \
        -enable-kvm -cpu host -nographic -no-reboot \
        -kernel /home/sh/.vmlinux.pvm_guest \
        -initrd /home/sh/rootfs.img \
        -append "nokaslr console=ttyS0 earlyprintk=serial" \
        -net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10019-:44 -net nic \
        -gdb tcp::1367

However, then I run the following command:

sudo rmmod kvm_intel
sudo modprobe kvm_pvm
image="docker.m.daocloud.io/library/busybox:latest"
sudo nerdctl run --net=none --runtime "io.containerd.kata.v2" --rm -t --name test-kata "$image" date

Then I still got the error message:

FATA[0005] failed to create shim task: Failed to Check if grpc server is working: ttrpc: closed: unknown

Here is the kata version:

kata-runtime --version
kata-runtime  : 3.2.0
   commit   : 45687e3251604ccc71b595d37f14253c4584cd5f
   OCI specs: 1.0.2-dev

[bysui]

I tried to use kvm_intel to run the guest kernel. I found that the problem still exists. Then I replaced the vmlinux.pvm with vmlinux.container. I found that the problem disappeared. I wonder if there is something wrong with the guest kernel(compiled with pvm-guest-6.12.33.config and pvm-612 branch)?

Uh, can you use QEMU to boot the guest kernel with the kvm_intel module? If so, which version of Kata Containers are you using? I'm not sure if the guest configuration is incompatible with your Kata Containers.

I used the following command to boot the guest kernel with kvm_intel module. Everything is okay.

sudo qemu-system-x86_64
-m 2048M -smp 8
-enable-kvm -cpu host -nographic -no-reboot
-kernel /home/sh/.vmlinux.pvm_guest
-initrd /home/sh/rootfs.img
-append "nokaslr console=ttyS0 earlyprintk=serial"
-net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10019-:44 -net nic
-gdb tcp::1367
However, then I run the following command:

sudo rmmod kvm_intel
sudo modprobe kvm_pvm
image="docker.m.daocloud.io/library/busybox:latest"
sudo nerdctl run --net=none --runtime "io.containerd.kata.v2" --rm -t --name test-kata "$image" date
Then I still got the error message:

FATA[0005] failed to create shim task: Failed to Check if grpc server is working: ttrpc: closed: unknown
Here is the kata version:

kata-runtime --version
kata-runtime : 3.2.0
commit : 45687e3251604ccc71b595d37f14253c4584cd5f
OCI specs: 1.0.2-dev

It may be that the version of Kata Containers doesn't support the 6.12 guest kernel; I'm not sure about that. However, I see that 6.12 is supported starting from version 3.14 in the release note. The example 6.12 guest configuration file in our repo is based on the config-6.12.36-160 file in version 3.19.1. Therefore, you may want to try using the same version of Kata Containers or the latest version and apply the guest configuration from the release package.

[HangShuYH]

I tried to use kvm_intel to run the guest kernel. I found that the problem still exists. Then I replaced the vmlinux.pvm with vmlinux.container. I found that the problem disappeared. I wonder if there is something wrong with the guest kernel(compiled with pvm-guest-6.12.33.config and pvm-612 branch)?

Uh, can you use QEMU to boot the guest kernel with the kvm_intel module? If so, which version of Kata Containers are you using? I'm not sure if the guest configuration is incompatible with your Kata Containers.

I used the following command to boot the guest kernel with kvm_intel module. Everything is okay.
sudo qemu-system-x86_64
-m 2048M -smp 8
-enable-kvm -cpu host -nographic -no-reboot
-kernel /home/sh/.vmlinux.pvm_guest
-initrd /home/sh/rootfs.img
-append "nokaslr console=ttyS0 earlyprintk=serial"
-net user,host=10.0.2.10,hostfwd=tcp:127.0.0.1:10019-:44 -net nic
-gdb tcp::1367
However, then I run the following command:
sudo rmmod kvm_intel
sudo modprobe kvm_pvm
image="docker.m.daocloud.io/library/busybox:latest"
sudo nerdctl run --net=none --runtime "io.containerd.kata.v2" --rm -t --name test-kata "$image" date
Then I still got the error message:
FATA[0005] failed to create shim task: Failed to Check if grpc server is working: ttrpc: closed: unknown
Here is the kata version:
kata-runtime --version
kata-runtime : 3.2.0
commit : 45687e3251604ccc71b595d37f14253c4584cd5f
OCI specs: 1.0.2-dev

It may be that the version of Kata Containers doesn't support the 6.12 guest kernel; I'm not sure about that. However, I see that 6.12 is supported starting from version 3.14 in the release note. The example 6.12 guest configuration file in our repo is based on the config-6.12.36-160 file in version 3.19.1. Therefore, you may want to try using the same version of Kata Containers or the latest version and apply the guest configuration from the release package.

Thanks for your help! kata 3.19.1 really works!
Fantastic work on pvm!