kvm2 v1.2.0: Failed to delete cluster: Requested operation is not valid: domain is not running

[carroarmato0]

There was an existing cluster created by Minikube 1.0.1 and shutdown.
After upgrading minikube to 1.1.0 and the corresponding docker-machine-driver-kvm2, deleting the existing cluster didn't work.

Starting the existing cluster triggered minikube to perform a successful upgrade, although deleting the upgraded cluster afterwards still doesn't work.

Upgrading the cluster:

[carroarmato0@neon-flower:~] 70 $ minikube start
😄  minikube v1.1.0 on linux (amd64)
👍  minikube will upgrade the local cluster from Kubernetes 1.14.1 to 1.14.2
💿  Downloading Minikube ISO ...
 131.28 MB / 131.28 MB [============================================] 100.00% 0s
2019/05/22 09:26:30 No matching credentials were found, falling back on anonymous
2019/05/22 09:26:30 No matching credentials were found, falling back on anonymous
2019/05/22 09:26:30 No matching credentials were found, falling back on anonymous
2019/05/22 09:26:30 No matching credentials were found, falling back on anonymous

⚠️  Ignoring --vm-driver=virtualbox, as the existing "minikube" VM was created using the kvm2 driver.
⚠️  To switch drivers, you may create a new VM using `minikube start -p <name> --vm-driver=virtualbox`
⚠️  Alternatively, you may delete the existing VM using `minikube delete -p minikube`

🔄  Restarting existing kvm2 VM for "minikube" ...
⌛  Waiting for SSH access ...
🐳  Configuring environment for Kubernetes v1.14.2 on Docker 18.06.3-ce
💾  Downloading kubeadm v1.14.2
💾  Downloading kubelet v1.14.2
🚜  Pulling images ...
🔄  Relaunching Kubernetes v1.14.2 using kubeadm ... 
⌛  Verifying: apiserver proxy etcd scheduler controller dns
🏄  Done! kubectl is now configured to use "minikube"
[carroarmato0@neon-flower:~] $ minikube stop
✋  Stopping "minikube" in kvm2 ...
🛑  "minikube" stopped.

Deleting the cluster:

[carroarmato0@neon-flower:~] 70 $ minikube delete -v 10
Found binary path at /home/carroarmato0/.local/bin/docker-machine-driver-kvm2
Launching plugin server for driver kvm2
Plugin server listening at address 127.0.0.1:33831
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .DriverName
🔥  Deleting "minikube" from kvm2 ...
(minikube) Calling .Remove
(minikube) DBG | Removing machine...
(minikube) DBG | Trying to delete the networks (if possible)
(minikube) DBG | Checking if network minikube-net exists...
(minikube) DBG | Checking if the domain needs to be deleted
(minikube) Deleting of networks failed: network %s does not exist: virError(Code=43, Domain=19, Message='Network not found: no network with matching name 'minikube-net'')
(minikube) Domain minikube exists, removing...

💣  Failed to delete cluster: virError(Code=55, Domain=20, Message='Requested operation is not valid: domain is not running')

😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new

[carroarmato0]

I deleted the VM manually from libvirt. Had to explicitly re-delete in minikube before minikube was able to recreate the cluster:

[carroarmato0@neon-flower:~] $ minikube start --cpus 4 --memory 8192 --vm-driver kvm2 --network-plugin=cni --enable-default-cni  --container-runtime=cri-o --insecure-registry '127.0.0.1:5000'
😄  minikube v1.1.0 on linux (amd64)
💡  Tip: Use 'minikube start -p <name>' to create a new cluster, or 'minikube delete' to delete this one.
E0522 09:34:45.291275   14492 start.go:529] StartHost: Error getting state for host: getting connection: looking up domain: virError(Code=42, Domain=10, Message='Domain not found: no domain with matching name 'minikube'')

💣  Unable to start VM: Error getting state for host: getting connection: looking up domain: virError(Code=42, Domain=10, Message='Domain not found: no domain with matching name 'minikube'')

😿  Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉  https://github.com/kubernetes/minikube/issues/new
[carroarmato0@neon-flower:~] 70 $ minikube delete
🔥  Deleting "minikube" from kvm2 ...
💔  The "minikube" cluster has been deleted.
[carroarmato0@neon-flower:~] $ minikube start --cpus 4 --memory 8192 --vm-driver kvm2 --network-plugin=cni --enable-default-cni  --container-runtime=cri-o --insecure-registry '127.0.0.1:5000'
😄  minikube v1.1.0 on linux (amd64)
🔥  Creating kvm2 VM (CPUs=4, Memory=8192MB, Disk=20000MB) ...
🎁  Configuring environment for Kubernetes v1.14.2 on CRI-O 1.14.1
🚜  Pulling images ...
🚀  Launching Kubernetes ... 
⌛  Verifying: apiserver etcd
...

[tstromberg]

Very strange:

(minikube) Domain minikube exists, removing...
💣  Failed to delete cluster: virError(Code=55, Domain=20, Message='Requested operation is not valid: domain is not running')

Do you mind sharing the output of libvirtd --version? I haven't yet been able to replicate this using libvirtd 5.0.0. The message seems to indicate a race condition or filtering being applied, but you were able to delete it.

Do you mind sharing the command you used to delete the "minikube" domain?

[carroarmato0]

Ow, I forgot to add my OS and version information:

libvirtd (libvirt) 5.0.0
Ubuntu 19.04

As per deleting the VM, I used virt-manager, but the equivalent would be virsh undefine minikube I guess.

Re-reading the error message, seems that the deletion should work regardless of the VM is running or not?

[josedonizetti]

@tstromberg This isn't that strange. Deleting a domain in kvm is a two step action. Eg:

virsh destroy minikube
virsh undefine minikube

(minikube) Domain minikube exists, removing...
💣  Failed to delete cluster: virError(Code=55, Domain=20, Message='Requested operation is not valid: domain is not running')

The error above will happen if one destroys the running VM instance by virsh destroy minikube, but fail to undefine the domain definition that actually created it. When minikube do a lookup to check if the domain exists, it is still there just the running VM isn't.

It would also be a problem if one only did a virsh undefine and ignored destroying the running VM, minikube delete would also fail, maybe we can check if libvirt-go returns a specific error code for non existing VM, and non existing domains definition, and instead of raising an error we skip, like the network change, recently merged.

I'll take a look on this.

[amruthar]

Hi, Can someone please comment if there is a workaround for this temporarily?

[carroarmato0]

Hi, Can someone please comment if there is a workaround for this temporarily?
If you are ok with starting from scratch with a fresh new cluster instead of upgrading, you can use the commands @josedonizetti listed.

[josedonizetti]

@amruthar I'm not sure lastest kvm driver has my changes, but if you can build it from source, this problem is fixed.

[medyagh]

this issue still happens as on the integration tests : #4418

[medyagh]

here is logs from integration test for KVM:

--- FAIL: TestVersionUpgrade (0.13s)
    util.go:130: Error running command: delete . Output: * Deleting "minikube" from kvm2 ...
FAIL
minikube: FAIL
 

more :

+++ cat /home/jenkins/minikube-integration/linux-amd64-kvm2-4526-5210-693ae42dffce125c500e9ea0fe3ead3caf5bd441/kubeconfig
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /home/jenkins/minikube-integration/linux-amd64-kvm2-4526-5210-693ae42dffce125c500e9ea0fe3ead3caf5bd441/.minikube/ca.crt
    server: https://192.168.39.30:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /home/jenkins/minikube-integration/linux-amd64-kvm2-4526-5210-693ae42dffce125c500e9ea0fe3ead3caf5bd441/.minikube/client.crt
    client-key: /home/jenkins/minikube-integration/linux-amd64-kvm2-4526-5210-693ae42dffce125c500e9ea0fe3ead3caf5bd441/.minikube/client.key
+++ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Unable to connect to the server: dial tcp 192.168.39.30:8443: i/o timeout
+++ MINIKUBE=out/minikube-linux-amd64
+++ out/minikube-linux-amd64 status
host: Stopped
kubelet: 
apiserver: 
kubectl: 
+++ out/minikube-linux-amd64 ip
 
X Error getting IP: host is not running
 
* Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
  - https://github.com/kubernetes/minikube/issues/new
+++ [[ kvm2 == \n\o\n\e ]]

[medyagh]

@josedonizetti I confirm this issue is still happening with kvm2 driver and latest

What happens :

when trying to delete a stopped minikube a with kvm2 driver it fails with this error:

virError(Code=55, Domain=20, Message='Requested operation is not valid: domain is not running')

This started to show up when I killed a minikube stop command and i didn't let it finish gracefully. which I think what happens in our integration tests too.

versions:

$ docker-machine-driver-kvm2 --version
v1.2.0

$ minikube version
minikube version: v1.2.0

OS :

Debian rodete

[josedonizetti]

@medyagh Thanks. I'll work on this. :)