nested virt: kvm crash "kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed."

[shoffmeister]

Environment:

Minikube version (use minikube version): 0.28.0

• OS: Fedora 28 (Workstation)
• VM Driver: kvm2 (out of 0.28.0 release)
• ISO version:
  "Boot2DockerURL": "file:///home/stefan/.minikube/cache/iso/minikube-v0.28.0.iso",
  "ISO": "/home/stefan/.minikube/machines/minikube/boot2docker.iso",
• Others:
  -- VMware Workstation 12 as type 2 hypervisor running on Windows 10 Home host; guest OS (see above) is intended to be the container of minikube.
  -- qemu-kvm -> QEMU emulator version 2.11.2(qemu-2.11.2-1.fc28)

What happened:
Failure to start minikube with error message "kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.", after

minikube delete
minikube start

What you expected to happen:
Successful launch of minikube, no error messages

How to reproduce it (as minimally and precisely as possible):

• set up clean Fedora 28 Workstation inside VMware Workstation 12
  -- enable emulation of CPU virtualization instructions
  -- in the .vmx, add line apic.xapic.enabled = "FALSE"
• sudo dnf install libvirt-daemon-kvm qemu-kvm
• edit /etc/modprobe.d/kvm.conf to enable nested KVM: options kvm_intel nested=1 (I have an Intel CPU)
• minikube config set vm-driver kvm2
• minikube delete
• minikube start

Anything else do we need to know:

This is a pristine setup where I am doing as little as possible to get kubernetes minikube to run inside KVM of Fedora 28 inside VMware Workstation. All configuration is intended to be as default and as controlled as humanly possible, so that I can maximize my chance of success for learning to use minikube.

minikube starts successfully with a changed KVM virtual machine CPU configuration, specifically

--- minikube-original.txt       2018-07-09 11:48:52.436387668 +0200
+++ hypervisor-default.txt      2018-07-09 11:49:20.776574169 +0200
@@ -15,7 +15,6 @@
     <apic/>
     <pae/>
   </features>
-  <cpu mode='host-passthrough' check='none'/>
   <clock offset='utc'/>
   <on_poweroff>destroy</on_poweroff>
   <on_reboot>restart</on_reboot>

i.e. switching from the default minikube value to "Hypervisor default" avoids the error message.

minikube KVM virtual machine configuration can be dumped using sudo virsh dumpxml minikube

Another working KVM virtual machine CPU configuration is

--- minikube-original.txt       2018-07-09 11:48:52.436387668 +0200
+++ copy-host-CPU-configuration.txt     2018-07-09 11:49:40.980707124 +0200
@@ -15,7 +15,9 @@
     <apic/>
     <pae/>
   </features>
-  <cpu mode='host-passthrough' check='none'/>
+  <cpu mode='host-model' check='partial'>
+    <model fallback='allow'/>
+  </cpu>
   <clock offset='utc'/>
   <on_poweroff>destroy</on_poweroff>
   <on_reboot>restart</on_reboot>

virt-manager (sudo dnf install virt-manager) offers a convenient means to inspect the virtual machine.

[shoffmeister]

Refer to https://libvirt.org/formatdomain.html#elementsCPU for documentation on the XML

I am logging this defect against minikube, in part because I cannot find a means to control (via minikube configuration) the configuration of the VM that minikube creates on minikube start. Allowing configuring some CPU via minikube config set would work; on top, "hypervisor default" as the default CPU would appear to be the right thing (instead of, implicitly, hard-coding passthrough as is the case right now).

[shoffmeister]

FWIW, I am not deeply familiar with kvm and qemu and native virtualization on Linux. Please provide pointers where to go and which incantations to use, to get the problem addressed at a more fundamental level.

[shoffmeister]

https://bugs.launchpad.net/qemu/+bug/1661386 is an item tracked by qemu; qemu believe that this is fundamentally due to a VMware problem (https://communities.vmware.com/thread/592140)

Still, with minikube it would be interesting to configure the type of CPU for the virtualization - right now, minikube implicitly uses "host-passthrough" while hyper-visor default CPU or just host-model (see above) would work.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten