kvm2: minikube network does not exist

[rnmhdn]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version): v0.33.1

• OS (e.g. from /etc/os-release): Linux 4.20.6-arch1-1-ARCH Need a reliable and low latency local cluster setup for Kubernetes  #1 SMP PREEMPT Thu Jan 31 08:22:01 UTC 2019 x86_64 GNU/Linux
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): "DriverName": "kvm2",
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): "Boot2DockerURL": "file:///home/aran/.minikube/cache/iso/minikube-v0.33.1.iso",
  "ISO": "/home/aran/.minikube/machines/minikube/boot2docker.iso"

$ minikube ssh cat /etc/VERSION
E0207 22:36:29.854750    2550 ssh.go:54] Error attempting to ssh/run-ssh-command: Error getting state of host: getting connection: looking up domain: virError(Code=42, Domain=10, Message='Domain not found: no domain with matching name 'minikube'')

• Install tools:
• Others:
  The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):

minikube version
echo "";
echo "OS:";
cat /etc/os-release
echo "";
echo "VM driver:"; 
grep DriverName ~/.minikube/machines/minikube/config.json
echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:
minikube network does not exist
What you expected to happen:
minikube network exists.
How to reproduce it (as minimally and precisely as possible):

    torify curl -LO https://storage.googleapis.com/kubernetes-release/release/$(torify curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x ./kubectl && sudo mv ./kubectl /usr/local/bin/kubectl 
    torify curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.33.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
    sudo pacman -Sy libvirt qemu-headless ebtables dnsmasq 
    sudo systemctl enable libvirtd.service
    sudo pacman -Sy docker-machine
    installed minikube-bin kubectl-bin docker-machine-driver-kvm2 from AUR
    sudo pacman -S dmidecode
    sudo usermod -aG libvirt aran
    sudo reboot 
    torify wget "https://storage.googleapis.com/minikube/iso/minikube-v0.33.1.iso"
    python -m http.server

Output of minikube logs (if applicable):

F0207 22:45:38.784909    3156 logs.go:50] Error getting cluster bootstrapper: getting kubeadm bootstrapper: getting ssh client: Error creating new ssh host from driver: Error getting ssh host name for driver: machine in unknown state: getting connection: looking up domain: virError(Code=42, Domain=10, Message='Domain not found: no domain with matching name 'minikube'')

Anything else do we need to know:

$ sudo virsh net-list --all Name      State    Autostart   Persistent
--------------------------------------------
 default   active   no          yes

**Update: **
I removed .minikube and restarted libvirtd and ran the command again and now I get:
virError(Code=9, Domain=20, Message='operation failed: domain 'minikube' already exists with uuid f718bd7b-3f63-4151-a89d-02cca2a39c9b')
**Update: **
I did minikube delete
and then:

$ minikube start --iso-url "http://localhost:8000/minikube-v0.33.1.iso" --vm-driver kvm2
Starting local Kubernetes v1.13.2 cluster...
Starting VM...

and it's stuck there.
here is the last msgs in my journal:

Feb 07 23:11:43 Christopher audit[606]: VIRT_RESOURCE pid=606 uid=0 auid=4294967295 ses=4294967295 msg='virt=kvm resrc=vcpu reason=start vm="minikube" uuid=6130b1e6-5c16-454c-93ca-b6c46edfbf70 old-vcpu=0 new-vcpu=2 exe="/usr/bin/libvirtd" hostname=? addr=? terminal=? res=success'
Feb 07 23:11:43 Christopher audit[606]: VIRT_CONTROL pid=606 uid=0 auid=4294967295 ses=4294967295 msg='virt=kvm op=start reason=booted vm="minikube" uuid=6130b1e6-5c16-454c-93ca-b6c46edfbf70 vm-pid=2910 exe="/usr/bin/libvirtd" hostname=? addr=? terminal=? res=success'
Feb 07 23:11:45 Christopher kernel: virbr1: port 2(vnet1) entered learning state
Feb 07 23:11:45 Christopher kernel: virbr0: port 2(vnet0) entered learning state
Feb 07 23:11:47 Christopher NetworkManager[592]: <info>  [1549568507.1642] device (virbr0): carrier: link connected
Feb 07 23:11:47 Christopher NetworkManager[592]: <info>  [1549568507.1644] device (virbr1): carrier: link connected
Feb 07 23:11:47 Christopher kernel: virbr0: port 2(vnet0) entered forwarding state
Feb 07 23:11:47 Christopher kernel: virbr0: topology change detected, propagating
Feb 07 23:11:47 Christopher kernel: virbr1: port 2(vnet1) entered forwarding state
Feb 07 23:11:47 Christopher kernel: virbr1: topology change detected, propagating
Feb 07 23:12:08 Christopher dnsmasq-dhcp[696]: DHCPDISCOVER(virbr0) 6c:66:db:0f:21:13
Feb 07 23:12:08 Christopher dnsmasq-dhcp[696]: DHCPOFFER(virbr0) 192.168.122.215 6c:66:db:0f:21:13
Feb 07 23:12:08 Christopher dnsmasq-dhcp[696]: DHCPDISCOVER(virbr0) 6c:66:db:0f:21:13
Feb 07 23:12:08 Christopher dnsmasq-dhcp[696]: DHCPOFFER(virbr0) 192.168.122.215 6c:66:db:0f:21:13
Feb 07 23:12:08 Christopher dnsmasq-dhcp[696]: DHCPREQUEST(virbr0) 192.168.122.215 6c:66:db:0f:21:13
Feb 07 23:12:08 Christopher dnsmasq-dhcp[696]: DHCPACK(virbr0) 192.168.122.215 6c:66:db:0f:21:13 minikube
Feb 07 23:12:08 Christopher dnsmasq-dhcp[2899]: DHCPDISCOVER(virbr1) 70:8a:20:5b:bf:09
Feb 07 23:12:08 Christopher dnsmasq-dhcp[2899]: DHCPOFFER(virbr1) 192.168.39.44 70:8a:20:5b:bf:09
Feb 07 23:12:08 Christopher dnsmasq-dhcp[2899]: DHCPDISCOVER(virbr1) 70:8a:20:5b:bf:09
Feb 07 23:12:08 Christopher dnsmasq-dhcp[2899]: DHCPOFFER(virbr1) 192.168.39.44 70:8a:20:5b:bf:09
Feb 07 23:12:08 Christopher dnsmasq-dhcp[2899]: DHCPREQUEST(virbr1) 192.168.39.44 70:8a:20:5b:bf:09
Feb 07 23:12:08 Christopher dnsmasq-dhcp[2899]: DHCPACK(virbr1) 192.168.39.44 70:8a:20:5b:bf:09 minikube

[tstromberg]

Hey @aranmohyeddin - Do you mind sharing the output of: minikube start -v=8 --alsologtostderr --vm-driver=kvm2

It should show us exactly which step it's blocked on, but probably something related to networking.

Also, what release of libvirt are you using?

[rnmhdn]

$ minikube start -v=8 --alsologtostderr --vm-driver=kvm2 > output
I0213 15:02:41.001125   31231 notify.go:121] Checking for updates...
I0213 15:02:41.974045   31231 start.go:120] Viper configuration:
I0213 15:02:41.977092   31231 cluster.go:74] Skipping create...Using existing machine configuration
Found binary path at /usr/bin/docker-machine-driver-kvm2
Launching plugin server for driver kvm2
Plugin server listening at address 127.0.0.1:44173
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .DriverName
(minikube) Calling .GetState
I0213 15:02:42.012188   31231 cluster.go:86] Machine state:  Running
(minikube) Calling .DriverName
Getting to WaitForSSH function...
(minikube) Calling .GetSSHHostname
(minikube) Calling .GetSSHPort
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHKeyPath
(minikube) Calling .GetSSHUsername
Using SSH client type: native
&{{{<nil> 0 [] [] []} docker [0x83a9a0] 0x83a970  [] 0s}  22 <nil> <nil>}
About to run SSH command:
exit 0
Error dialing TCP: dial tcp :22: connect: connection refused
Error dialing TCP: dial tcp :22: connect: connection refused

and

$ cat output 
Aliases:
map[string]string{}
Override:
map[string]interface {}{"v":"8", "alsologtostderr":"true"}
PFlags:
map[string]viper.FlagValue{"docker-env":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc140)}, "enable-default-cni":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dca00)}, "hyperv-virtual-switch":viper.pflagValue{flag:(*pflag.Flag)(0xc000445e00)}, "keep-context":viper.pflagValue{flag:(*pflag.Flag)(0xc0004457c0)}, "memory":viper.pflagValue{flag:(*pflag.Flag)(0xc000445b80)}, "apiserver-ips":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc460)}, "apiserver-port":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc280)}, "dns-domain":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc500)}, "hyperkit-vsock-ports":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dcdc0)}, "service-cluster-ip-range":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc5a0)}, "apiserver-names":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc3c0)}, "cri-socket":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc820)}, "uuid":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dcc80)}, "profile":viper.pflagValue{flag:(*pflag.Flag)(0xc000444dc0)}, "cache-images":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dcb40)}, "gpu":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dce60)}, "hyperkit-vpnkit-sock":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dcd20)}, "kubernetes-version":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc8c0)}, "kvm-network":viper.pflagValue{flag:(*pflag.Flag)(0xc000445ea0)}, "apiserver-name":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc320)}, "cpus":viper.pflagValue{flag:(*pflag.Flag)(0xc000445c20)}, "disable-driver-mounts":viper.pflagValue{flag:(*pflag.Flag)(0xc0004459a0)}, "insecure-registry":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc640)}, "nfs-share":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc000)}, "xhyve-disk-driver":viper.pflagValue{flag:(*pflag.Flag)(0xc000445f40)}, "bootstrapper":viper.pflagValue{flag:(*pflag.Flag)(0xc000444e60)}, "container-runtime":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc780)}, "extra-config":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dcbe0)}, "feature-gates":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dcaa0)}, "host-only-cidr":viper.pflagValue{flag:(*pflag.Flag)(0xc000445d60)}, "iso-url":viper.pflagValue{flag:(*pflag.Flag)(0xc000445a40)}, "network-plugin":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc960)}, "docker-opt":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc1e0)}, "mount":viper.pflagValue{flag:(*pflag.Flag)(0xc000445860)}, "registry-mirror":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc6e0)}, "disk-size":viper.pflagValue{flag:(*pflag.Flag)(0xc000445cc0)}, "mount-string":viper.pflagValue{flag:(*pflag.Flag)(0xc000445900)}, "nfs-shares-root":viper.pflagValue{flag:(*pflag.Flag)(0xc0001dc0a0)}, "vm-driver":viper.pflagValue{flag:(*pflag.Flag)(0xc000445ae0)}}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"wantreporterrorprompt":false}
Defaults:
map[string]interface {}{"alsologtostderr":"false", "log_dir":"", "wantnonedriverwarning":true, "showdriverdeprecationnotification":true, "wantreporterror":false, "wantreporterrorprompt":true, "wantkubectldownloadmsg":true, "showbootstrapperdeprecationnotification":true, "v":"0", "wantupdatenotification":true, "reminderwaitperiodinhours":24}
Starting local Kubernetes v1.13.2 cluster...
Starting VM...
Waiting for SSH to be available...

and also:

$ libvirtd --version
libvirtd (libvirt) 5.0.0

[anselvo]

Hello, i have the same issue. After 120 second i get this error

E0214 13:40:20.008272   15961 start.go:205] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds.

OS: Ubuntu 18.10
libvirtd (libvirt) 4.6.0

$ minikube status
Error getting cluster bootstrapper: getting kubeadm bootstrapper: getting ssh client: Error dialing tcp via ssh client: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

[tstromberg]

minikube v0.34.1 includes an updated DHCP configuration. @aranmohyeddin - do you mind testing against this release to see how the error messages are different?

Please allow minikube the 5 minutes to time out, and paste the entire output of minikube start so that I can see if there are any other clues. Are there any firewall rules on this machine? I wonder if it simply isn't allowed to connect to the newly started VM.

@aelsergeev - That may be #3434 or #3566

[j1cs]

I did have the same issue.
I installed libvirtd and firewalld (in Artix a OpenRC implementation for Archlinux) For some reason firewalld does not give IP to the VM, maybe missed some configuration (i really dont know if you need to do something in firewalld for that).
After all this the init script of libvirtd needs firewalld as a dependency so i removed it then i restarted the libvirtd services and minkube finally start up.
I hope this is useful to someone.

[tstromberg]

Can someone confirm whether or not minikube v1.1 with the latest kvm2 driver addresses this bug? I suspect it might. To upgrade the kvm2 driver:

curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-kvm2 && sudo install docker-machine-driver-kvm2 /usr/local/bin/