Add perf kvm support for PVM

[ljrcore]

Improve the handling of exit reasons for PVM hypercalls by adding more specific exit reasons based on the type of PVM hypercall. This enables a more fine-grained analysis of the performance and behavior of PVM-based virtual machines.

The new exit reasons cover various PVM hypercall types, including IRQ_WIN, IRQ_HALT, LOAD_PGTBL, TLB_FLUSH, TLB_FLUSH_CURRENT, TLB_INVLPG, LOAD_GS, RDMSR, WRMSR, and LOAD_TLS. Providing specific exit reasons for each of these hypercall types allows for better insights into the performance characteristics of PVM-based virtual machines and helps identify potential areas for optimization.

Moreover, this change benefits performance analysis tools, such as perf, by allowing them to provide more detailed information about the execution of PVM hypercalls. This improvement helps users gain a better understanding of the performance bottlenecks in their virtualized environments and make informed decisions about optimizing their systems.

[bysui]

Hi, @ljrcore , thanks for your PR!
Actually, I concur that the current exit reason is too simplistic. I have also added more information when I need more debug information during debugging. However, I would prefer to keep the reason as PVM_EXIT_REASONS_HYPERCALL and instead put the hypercall number into info2. The reason for this is that I believe PVM_EXIT_REASONS_HYPERCALL should be in the same level as other reasons (e.g., PVM_SYSCALL_VECTOR). Then, we can put additional information into info2, as it is a u64 and the hypercall number is allowed to be bigger than a u32 (Although it is not now). Since for hypercall, pvm->exit_error_code is invalid information, we can put the hypercall number here. Additionally, for syscall, we can also put the syscall number into info2, allowing us to trace the guest userspace behavior. However, I am not familiar with the Perf tools. Is it easy for you to use the information in info2 to do the same analysis in Perf? And what do you prefer?

[ljrcore]

Hi, @ljrcore , thanks for your PR! Actually, I concur that the current exit reason is too simplistic. I have also added more information when I need more debug information during debugging. However, I would prefer to keep the reason as PVM_EXIT_REASONS_HYPERCALL and instead put the hypercall number into info2. The reason for this is that I believe PVM_EXIT_REASONS_HYPERCALL should be in the same level as other reasons (e.g., PVM_SYSCALL_VECTOR). Then, we can put additional information into info2, as it is a u64 and the hypercall number is allowed to be bigger than a u32 (Although it is not now). Since for hypercall, pvm->exit_error_code is invalid information, we can put the hypercall number here. Additionally, for syscall, we can also put the syscall number into info2, allowing us to trace the guest userspace behavior. However, I am not familiar with the Perf tools. Is it easy for you to use the information in info2 to do the same analysis in Perf? And what do you prefer?

Hi, @bysui, thanks for your suggestion. I agree that putting the hypercall number into info2 is a good idea. However, I would like to inform you that I am unable to easily use the information in info2 for the same analysis in Perf. Therefore, could we consider this as a new feature request?

Our main goal at the moment is to fix the issue of perf kvm being unavailable on PVM. Would it be possible to prioritize this issue and make perf kvm work on PVM first, as this feature has minimal impact on PVM?

Once we have addressed the current issue, we can then discuss and implement the improvements you suggested, such as putting the hypercall number and the syscall number into info2 for tracing guest userspace behavior.

[ljrcore]

How to use:

• Apply these four patches
• Compile PVM
• Compile perf

cd $LINUX_DIR
cd tools/perf
# Install perf required dependencies
make
make install

Validation results:

./perf kvm stat live
# Start the virtual machine

There will be output similar to this：

Analyze events for all VMs, all VCPUs:

                                 VM-EXIT    Samples  Samples%     Time%    Min Time    Max Time         Avg time

                                HC_WRMSR        354    33.94%     1.35%      0.28us  32004.80us    124.96us ( +-  77.21% )
                                   ERETS        197    18.89%     1.59%      0.57us  28004.05us    264.67us ( +-  70.56% )
                                 PF excp        185    17.74%     2.16%      0.79us  35560.46us    382.64us ( +-  69.91% )
                             HC_IRQ_HALT        155    14.86%    94.16%     16.66us 511981.18us  19869.99us ( +-  19.94% )
                               INTERRUPT         42     4.03%     0.72%      0.68us  23397.49us    558.96us ( +-  99.66% )
                                   ERETU         38     3.64%     0.00%      0.46us     16.11us      1.04us ( +-  39.38% )
                                HC_RDMSR         22     2.11%     0.00%      0.34us      0.65us      0.40us ( +-   3.43% )
                                 GP excp         18     1.73%     0.00%      0.61us      1.76us      0.82us ( +-   9.58% )
                           HC_TLB_INVLPG         18     1.73%     0.00%      0.76us     14.11us      4.52us ( +-  24.22% )
                           HC_LOAD_PGTBL          6     0.58%     0.00%      3.21us     13.25us      8.42us ( +-  16.76% )
                              HC_IRQ_WIN          5     0.48%     0.00%      0.67us      0.93us      0.75us ( +-   6.29% )
                    HC_TLB_FLUSH_CURRENT          2     0.19%     0.00%      0.46us      5.52us      2.99us ( +-  84.55% )
                                 SYSCALL          1     0.10%     0.00%      0.62us      0.62us      0.62us ( +-   0.00% )

Total Samples:1043, Total events handled time:3270695.00us.

[ljrcore]

It is worth mentioning that even with these patches applied, there are still unknowns that require additional fixes when using ./perf kvm --host --guest record -a and ./perf kvm --host --guest report.

Samples: 56K of event 'cpu-clock:ppp', Event count (approx.): 14130000000
Overhead  Command          Shared Object     Symbol
  66.64%  swapper          [kernel.vmlinux]  [k] default_idle
   5.24%  :229235          [unknown]         [u] 0x000000000043cffb
   5.11%  :229235          [unknown]         [u] 0xffffd97f81a01d40
   4.80%  :229236          [unknown]         [g] 0xffffd97f81a01d40
   4.60%  :229236          [unknown]         [g] 0x000000000043cffb
   1.41%  :229237          [unknown]         [g] 0xffffd97f81a01d40
   1.25%  :229237          [unknown]         [g] 0x000000000043cffb
   0.35%  :229238          [unknown]         [g] 0xffffd97f81a01d40
   0.30%  :229238          [unknown]         [g] 0x000000000043cffb
   0.29%  :229236          [unknown]         [g] 0x0000000000400af7
   0.29%  :229235          [unknown]         [u] 0x0000000000400af3
   0.29%  :229235          [unknown]         [u] 0x0000000000400af7
   0.24%  :229236          [unknown]         [g] 0x0000000000400af3
...

[laijs]

I don't think arch/x86/include/uapi/asm/pvm.h is part of PVM "architecture" ABI.
Those tracing ABI have been also in arch/x86/include/uapi/asm/pvm.h for several years and then moved into arch/x86/kvm/pvm/pvm.h when the code were made public for the same reason.

I think we can put those tracing ABI to arch/x86/include/uapi/asm/pvm_trace.h instead.

[laijs]

Please don't move this code. It is part of switcher API (linux kernel internal API) rather than any ABI.

SWITCH_EXIT_REASONS_SYSCALL is not used for PVM_EXIT_REASONS. (PVM_EXIT_REASONS_SYSCALL and several other code are used for PVM_EXIT_REASONS rather than SWITCH_EXIT_REASONS_SYSCALL nor PVM_SYSCALL_VECTOR)

pvm_get_syscall_exit_reason() should handle SWITCH_EXIT_REASONS_FAILED_VMETNRY explicitly and assign the reason to a code defined in pvm_trace.h. (arch/x86/include/uapi/asm/pvm.h should be renamed to pvm_trace.h). (SWITCH_EXIT_REASONS_FAILED_VMETNRY might be deleted in future)

[ljrcore]

Hi, @laijs, thanks for your suggestion. I modified the code as you suggested. Explicitly handle PVM_FAILED_VMENTRY_VECTOR in pvm_get_syscall_exit_reason() and assign the reason to the code defined in pvm_trace.h.

[bysui]

Hi, @ljrcore , thanks for your PR! Actually, I concur that the current exit reason is too simplistic. I have also added more information when I need more debug information during debugging. However, I would prefer to keep the reason as PVM_EXIT_REASONS_HYPERCALL and instead put the hypercall number into info2. The reason for this is that I believe PVM_EXIT_REASONS_HYPERCALL should be in the same level as other reasons (e.g., PVM_SYSCALL_VECTOR). Then, we can put additional information into info2, as it is a u64 and the hypercall number is allowed to be bigger than a u32 (Although it is not now). Since for hypercall, pvm->exit_error_code is invalid information, we can put the hypercall number here. Additionally, for syscall, we can also put the syscall number into info2, allowing us to trace the guest userspace behavior. However, I am not familiar with the Perf tools. Is it easy for you to use the information in info2 to do the same analysis in Perf? And what do you prefer?

Hi, @bysui, thanks for your suggestion. I agree that putting the hypercall number into info2 is a good idea. However, I would like to inform you that I am unable to easily use the information in info2 for the same analysis in Perf. Therefore, could we consider this as a new feature request?

Our main goal at the moment is to fix the issue of perf kvm being unavailable on PVM. Would it be possible to prioritize this issue and make perf kvm work on PVM first, as this feature has minimal impact on PVM?

If the current goal is to enable "perf kvm stat" for PVM, then I understand that exporting the exit reason information to userspace and allowing perf to parse it would be sufficient. My concern is that the hypercall number and exit reason number are not the same thing, mainly in terms of number allocation, so we cannot directly use the hypercall number as the exit reason number. If we need to differentiate the hypercall, then we may need to convert each hypercall number into a new exit reason number. This may become part of the PVM ABI. Additionally, in VMX, the hypercall (VMCALL) also does not differentiate specific hypercall numbers. Therefore, if we want to differentiate, we should use a uniform method to achieve this.

Once we have addressed the current issue, we can then discuss and implement the improvements you suggested, such as putting the hypercall number and the syscall number into info2 for tracing guest userspace behavior.

Actually, I believe perf kvm isn't fully utilizing the information provided by the kvm_exit() tracepoint. I can see that there are other tools, such as kvmon; maybe we can extend perf kvm tools like it. Additionally, I have noticed that the community is more likely to use eBPF for statistics rather than tracepoints.

analyzing-kvm-hypercalls-with-ebpf-tracing
PATCH v2 0/4 KVM: x86: tracepoint updates

[ljrcore]

Hi, @ljrcore , thanks for your PR! Actually, I concur that the current exit reason is too simplistic. I have also added more information when I need more debug information during debugging. However, I would prefer to keep the reason as PVM_EXIT_REASONS_HYPERCALL and instead put the hypercall number into info2. The reason for this is that I believe PVM_EXIT_REASONS_HYPERCALL should be in the same level as other reasons (e.g., PVM_SYSCALL_VECTOR). Then, we can put additional information into info2, as it is a u64 and the hypercall number is allowed to be bigger than a u32 (Although it is not now). Since for hypercall, pvm->exit_error_code is invalid information, we can put the hypercall number here. Additionally, for syscall, we can also put the syscall number into info2, allowing us to trace the guest userspace behavior. However, I am not familiar with the Perf tools. Is it easy for you to use the information in info2 to do the same analysis in Perf? And what do you prefer?

Hi, @bysui, thanks for your suggestion. I agree that putting the hypercall number into info2 is a good idea. However, I would like to inform you that I am unable to easily use the information in info2 for the same analysis in Perf. Therefore, could we consider this as a new feature request?
Our main goal at the moment is to fix the issue of perf kvm being unavailable on PVM. Would it be possible to prioritize this issue and make perf kvm work on PVM first, as this feature has minimal impact on PVM?

If the current goal is to enable "perf kvm stat" for PVM, then I understand that exporting the exit reason information to userspace and allowing perf to parse it would be sufficient. My concern is that the hypercall number and exit reason number are not the same thing, mainly in terms of number allocation, so we cannot directly use the hypercall number as the exit reason number. If we need to differentiate the hypercall, then we may need to convert each hypercall number into a new exit reason number. This may become part of the PVM ABI. Additionally, in VMX, the hypercall (VMCALL) also does not differentiate specific hypercall numbers. Therefore, if we want to differentiate, we should use a uniform method to achieve this.

Is it possible to assign the exit reason number of HYPERCALL split like this?

#define PVM_EXIT_REASONS_HC_IRQ_WIN		(PVM_EXIT_REASONS_HYPERCALL + 1)
#define PVM_EXIT_REASONS_HC_IRQ_HALT		(PVM_EXIT_REASONS_HYPERCALL + 2)
#define PVM_EXIT_REASONS_HC_LOAD_PGTBL		(PVM_EXIT_REASONS_HYPERCALL + 3)
#define PVM_EXIT_REASONS_HC_TLB_FLUSH		(PVM_EXIT_REASONS_HYPERCALL + 4)
#define PVM_EXIT_REASONS_HC_TLB_FLUSH_CURRENT	(PVM_EXIT_REASONS_HYPERCALL + 5)
#define PVM_EXIT_REASONS_HC_TLB_INVLPG		(PVM_EXIT_REASONS_HYPERCALL + 6)
#define PVM_EXIT_REASONS_HC_LOAD_GS		(PVM_EXIT_REASONS_HYPERCALL + 7)
#define PVM_EXIT_REASONS_HC_RDMSR		(PVM_EXIT_REASONS_HYPERCALL + 8)
#define PVM_EXIT_REASONS_HC_WRMSR		(PVM_EXIT_REASONS_HYPERCALL + 9)
#define PVM_EXIT_REASONS_HC_LOAD_TLS		(PVM_EXIT_REASONS_HYPERCALL + 10)

Do you have any suggestions?

Once we have addressed the current issue, we can then discuss and implement the improvements you suggested, such as putting the hypercall number and the syscall number into info2 for tracing guest userspace behavior.

Actually, I believe perf kvm isn't fully utilizing the information provided by the kvm_exit() tracepoint. I can see that there are other tools, such as kvmon; maybe we can extend perf kvm tools like it. Additionally, I have noticed that the community is more likely to use eBPF for statistics rather than tracepoints.

analyzing-kvm-hypercalls-with-ebpf-tracing PATCH v2 0/4 KVM: x86: tracepoint updates

Thanks for your knowledge. I'll try perf kvm to take advantage of the information provided by the kvm_exit() tracepoint.

[ljrcore]

Thanks for your valuable feedback on my previous patch submission. I've revised the patches accordingly and tested these changes thoroughly.

If you have any further suggestions or need more clarification, please let me know. I look forward to your response and hope these patches can be merged soon.

Thank you for your time and attention.

Best regards

[laijs]

@ljrcore Thank you for your contribution. We will merge your code into next pvm iteration sending upstream. And since PVM is not yet in the linux tree, the commits might be subjected to be modified/split/squashed and we are trying our best to mark your contribution in the new commits.

[laijs]

@ljrcore The reason we change commits (modified/split/squashed/rewritted) for next iteration is that PVM is not included in the upstream tree yet. PVM is still being developed and improved, mainly on the switcher code and PIE support and PV mmu. After the code is more mature, we will send V2. There is not time table for it, it might be sent soon for earlier review or procrastinated or bit.

[ljrcore]

@ljrcore The reason we change commits (modified/split/squashed/rewritted) for next iteration is that PVM is not included in the upstream tree yet. PVM is still being developed and improved, mainly on the switcher code and PIE support and PV mmu. After the code is more mature, we will send V2. There is not time table for it, it might be sent soon for earlier review or procrastinated or bit.

Got it.

Thanks