[BUG] Live migration fail when upgrade v1.2.1 to v1.2.2-rc2 due to virError

[albinsun]

Describe the bug
During v1.2.1 to v1.2.2-rc2 upgrade, hit live migration fail in the pre-drain phase due to continue looping following

VirtualMachineInstance migration uid 46c9f7af-1f77-4f94-bf78-e8320c9a9b24 failed. reason:Live migration failed error encountered during MigrateToURI3 libvirt api call: virError(Code=9, Domain=31, Message='operation failed: guest CPU doesn't match specification: missing features: waitpkg')

To Reproduce

1. 🟢 Create 3 node v1.2.1 cluster
2. 🟢 Create a VM (here located on node-2)
3. 🔴 Upgrade to v1.2.2-rc2
   • node-0 and node-1 are Succeeded
   • node-2 stuck in Pre-draining due to live migration fail
     
     
     

Expected behavior
Upgrade successfully

Support bundle
supportbundle_stuck_predraining.zip

Upgrade log
hvst-upgrade-gvmcm-upgradelog-archive-stuck_predraining.zip

Environment

• Harvester
  • Version: v1.2.1 -> v1.2.2-rc2
  • Profile: QEMU/KVM, 3 nodes (8C/16G/500G)
  • ui-source: Auto

[albinsun]

FYI
Can also reproduce in 2 nodes cluster.
Note that there is no migration in single node and we hit VM forcely shutdown instead.

Two nodes (8C/20G per node)

1. 🟢 Setup 2 nodes harvester-v1.2.1

   

2. 🟢 Create a VM (VM locates on node-1)

   

3. 🔴 Upgrade to v1.2.2-rc2

   node-1 is Cordoned and stuck in Pre-draining
   
   VM live migration fail
   
   

   VirtualMachineInstance migration uid 5de2134c-25e2-404e-88b2-9307f54866c8 failed. reason:Live migration failed error encountered during MigrateToURI3 libvirt api call: virError(Code=9, Domain=31, Message='operation failed: guest CPU doesn't match specification: missing features: waitpkg')
   

• Upgrade log: hvst-upgrade-4v6ct-upgradelog-archive-2nodes.zip
• Support bundle: supportbundle_2nodes.zip

Single node (16C/32G)

Note

No migration since only one node.

Upgrade successfully but VM was forced off.

• Upgrade log: hvst-upgrade-2xn2c-upgradelog-archive-single.zip
• Support bundle: supportbundle_single.zip

[w13915984028]

could this be same root cause ase #5756, both have (rke2) VM stucking on migrating and further blocks upgrading.

[bk201]

The cause is some QEMU change between SLES SP4 and SP5. The issue happens when harvester nodes are in VMs and guests are in nested VMs. Here is the words from virtualization team:

but the bug is rather that you see the waitpkg flag in SP4, more than the fact that you don't see it in SP5

yes, SP5's QEMU behavior is correct, i.e., on your particular hardware, it's ok to not advertise that flag in a nested VM. It's actually SP4's QEMU that is at fault, i.e., it shouldn't advertise it in the first place, while instead it did. As I said, I can backport the fix to SP's QEMU, but this won't probably help you for that particular VM (or it would break it in even worse way, when/if the updated QEMU would reach SP4's KubeVirt)

Moving the harvester VM cpumode from host-passthrough to host-mode workaround the issue.

[albinsun]

Test can pass after let QEMU using default cpu_mode host-model.
Ref. harvester/ipxe-examples#82

In short,

• host-passthrough will copy all feature from host cpu even libvirt does not understand.
• host-model will choose from supportted list which closest to host CPU.

So
For Performance: host-passthrough > host-model
For Reliability: host-model > host-passthrough

Ref. https://libvirt.org/formatdomain.html#cpu-model-and-topology

[albinsun]

Do not observe live migration fail after taking the workaround mentioned in harvester/ipxe-examples#82.
Let's close this issue first and discuss whether to commit the workaround.