dnsmasq: during machine creation: Machine didn't return an IP after 120 seconds #3434

tunggad · 2018-12-09T10:52:45Z

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

Please provide the following details:

Environment:

Minikube version : v0.30.0

OS (e.g. from /etc/os-release):

[root@archserver ~]# cat /etc/os-release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
ID_LIKE=archlinux
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"

VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):
"DriverName": "kvm2",
ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):

 "Boot2DockerURL": "file:///home/cryto/.minikube/cache/iso/minikube-v0.30.0.iso",
        "ISO": "/home/cryto/.minikube/machines/minikube/boot2docker.iso",

Install tools: yay -S kubectl-bin minikube-bin docker-machine-driver-kvm2
Others:

[root@archserver ~]# virsh version
Compiled against library: libvirt 4.9.0
Using library: libvirt 4.9.0
Using API: QEMU 4.9.0
Running hypervisor: QEMU 3.0.0

[root@archserver ~]# dnsmasq --version
Dnsmasq version 2.80  Copyright (c) 2000-2018 Simon Kelley
Compile time options: IPv6 GNU-getopt DBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSE-detect inotify dumpfile

This software comes with ABSOLUTELY NO WARRANTY.
Dnsmasq is free software, and you are welcome to redistribute it
under the terms of the GNU General Public License, version 2 or 3.

● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2018-12-04 22:04:52 UTC; 4 days ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 440 (libvirtd)
    Tasks: 23 (limit: 32768)
   Memory: 52.3M
   CGroup: /system.slice/libvirtd.service
           ├─ 440 /usr/bin/libvirtd
           ├─ 734 /usr/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/docker-machines.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
           ├─ 735 /usr/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/docker-machines.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
           ├─ 764 /usr/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
           ├─ 765 /usr/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
           ├─9801 /usr/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/minikube-net.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper
           └─9802 /usr/bin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/minikube-net.conf --leasefile-ro --dhcp-script=/usr/lib/libvirt/libvirt_leaseshelper

[root@archserver ~]# virsh net-list
 Name              State    Autostart   Persistent
----------------------------------------------------
 default           active   yes         yes
 docker-machines   active   yes         yes
 minikube-net      active   yes         yes

[root@archserver ~]# ls /var/lib/libvirt/dnsmasq
default.addnhosts  default.hostsfile          docker-machines.conf       minikube-net.addnhosts  minikube-net.hostsfile  virbr0.status  virbr1.status  virbr2.status
default.conf       docker-machines.addnhosts  docker-machines.hostsfile  minikube-net.conf       virbr0.macs             virbr1.macs    virbr2.macs
[root@archserver ~]# cat /var/lib/libvirt/dnsmasq/virbr2.status
[root@archserver ~]# cat /var/lib/libvirt/dnsmasq/virbr1.status
[root@archserver ~]# cat /var/lib/libvirt/dnsmasq/virbr0.status
[root@archserver ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
docker0         8000.02422574b795       no
virbr0          8000.5254004c2a7a       yes             virbr0-nic
                                                        vnet0
virbr1          8000.525400f245f1       yes             virbr1-nic
virbr2          8000.5254001a2828       yes             virbr2-nic
                                                        vnet1

What happened:
minikube fails to start with message: 'Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds'
What you expected to happen:
minikube starts successully
How to reproduce it (as minimally and precisely as possible):
execute minikube start --vm-driver kvm2
Output of minikube logs (if applicable):

[cryto@archserver ~]$ minikube start --vm-driver kvm2 --v=10 --alsologtostderr
I1209 09:55:49.814065    9761 notify.go:121] Checking for updates...
I1209 09:55:50.049275    9761 start.go:99] Viper configuration:
Aliases:
map[string]string{}
Override:
map[string]interface {}{"v":"10", "alsologtostderr":"true"}
PFlags:
map[string]viper.FlagValue{"kvm-network":viper.pflagValue{flag:(*pflag.Flag)(0xc420025ea0)}                                                                                                                                                  , "nfs-shares-root":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a80a0)}, "registry-mirror":v                                                                                                                                                  iper.pflagValue{flag:(*pflag.Flag)(0xc4203a85a0)}, "profile":viper.pflagValue{flag:(*pflag.                                                                                                                                                  Flag)(0xc420024dc0)}, "host-only-cidr":viper.pflagValue{flag:(*pflag.Flag)(0xc420025d60)},                                                                                                                                                   "hyperkit-vpnkit-sock":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a8aa0)}, "iso-url":viper.                                                                                                                                                  pflagValue{flag:(*pflag.Flag)(0xc420025a40)}, "network-plugin":viper.pflagValue{flag:(*pfla                                                                                                                                                  g.Flag)(0xc4203a8780)}, "apiserver-ips":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a83c0)},                                                                                                                                                   "apiserver-name":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a8280)}, "insecure-registry":v                                                                                                                                                  iper.pflagValue{flag:(*pflag.Flag)(0xc4203a8500)}, "kubernetes-version":viper.pflagValue{fl                                                                                                                                                  ag:(*pflag.Flag)(0xc4203a86e0)}, "dns-domain":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a8                                                                                                                                                  460)}, "gpu":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a8be0)}, "hyperkit-vsock-ports":vip                                                                                                                                                  er.pflagValue{flag:(*pflag.Flag)(0xc4203a8b40)}, "nfs-share":viper.pflagValue{flag:(*pflag.                                                                                                                                                  Flag)(0xc4203a8000)}, "apiserver-names":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a8320)},                                                                                                                                                   "container-runtime":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a8640)}, "cpus":viper.pflag                                                                                                                                                  Value{flag:(*pflag.Flag)(0xc420025c20)}, "disk-size":viper.pflagValue{flag:(*pflag.Flag)(0x                                                                                                                                                  c420025cc0)}, "uuid":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a8a00)}, "xhyve-disk-driver                                                                                                                                                  ":viper.pflagValue{flag:(*pflag.Flag)(0xc420025f40)}, "vm-driver":viper.pflagValue{flag:(*p                                                                                                                                                  flag.Flag)(0xc420025ae0)}, "cache-images":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a88c0)                                                                                                                                                  }, "docker-opt":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a81e0)}, "mount":viper.pflagValu                                                                                                                                                  e{flag:(*pflag.Flag)(0xc420025860)}, "mount-string":viper.pflagValue{flag:(*pflag.Flag)(0xc                                                                                                                                                  420025900)}, "docker-env":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a8140)}, "feature-gate                                                                                                                                                  s":viper.pflagValue{flag:(*pflag.Flag)(0xc4203a8820)}, "keep-context":viper.pflagValue{flag                                                                                                                                                  :(*pflag.Flag)(0xc4200257c0)}, "disable-driver-mounts":viper.pflagValue{flag:(*pflag.Flag)(                                                                                                                                                  0xc4200259a0)}, "hyperv-virtual-switch":viper.pflagValue{flag:(*pflag.Flag)(0xc420025e00)},                                                                                                                                                   "bootstrapper":viper.pflagValue{flag:(*pflag.Flag)(0xc420024e60)}, "extra-config":viper.pf                                                                                                                                                  lagValue{flag:(*pflag.Flag)(0xc4203a8960)}, "memory":viper.pflagValue{flag:(*pflag.Flag)(0x                                                                                                                                                  c420025b80)}}
Env:
map[string]string{}
Key/Value Store:
map[string]interface {}{}
Config:
map[string]interface {}{"wantreporterror":true}
Defaults:
map[string]interface {}{"wantupdatenotification":true, "reminderwaitperiodinhours":24, "als                                                                                                                                                  ologtostderr":"false", "log_dir":"", "wantreporterror":false, "wantreporterrorprompt":true,                                                                                                                                                   "wantkubectldownloadmsg":true, "wantnonedriverwarning":true, "showdriverdeprecationnotific                                                                                                                                                  ation":true, "showbootstrapperdeprecationnotification":true, "v":"0"}
Starting local Kubernetes v1.10.0 cluster...
Starting VM...
I1209 09:55:50.050000    9761 utils.go:100] retry loop 0
I1209 09:55:50.050043    9761 cluster.go:69] Machine does not exist... provisioning new mac                                                                                                                                                  hine
I1209 09:55:50.050073    9761 cluster.go:70] Provisioning machine with config: {MinikubeISO                                                                                                                                                  :https://storage.googleapis.com/minikube/iso/minikube-v0.30.0.iso Memory:2048 CPUs:2 DiskSi                                                                                                                                                  ze:20000 VMDriver:kvm2 HyperkitVpnKitSock: HyperkitVSockPorts:[] XhyveDiskDriver:ahci-hd Do                                                                                                                                                  ckerEnv:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtual                                                                                                                                                  Switch: KvmNetwork:default Downloader:{} DockerOpt:[] DisableDriverMounts:false NFSShare:[]                                                                                                                                                   NFSSharesRoot:/nfsshares UUID: GPU:false}
I1209 09:55:50.050202    9761 downloader.go:56] Not caching ISO, using https://storage.goog                                                                                                                                                  leapis.com/minikube/iso/minikube-v0.30.0.iso
Found binary path at /usr/bin/docker-machine-driver-kvm2
Launching plugin server for driver kvm2
Plugin server listening at address 127.0.0.1:39359
() Calling .GetVersion
Using API Version  1
() Calling .SetConfigRaw
() Calling .GetMachineName
(minikube) Calling .GetMachineName
(minikube) Calling .DriverName
Reading certificate data from /home/cryto/.minikube/certs/ca.pem
Decoding PEM data...
Parsing certificate...
Reading certificate data from /home/cryto/.minikube/certs/cert.pem
Decoding PEM data...
Parsing certificate...
Running pre-create checks...
(minikube) Calling .PreCreateCheck
(minikube) Calling .GetConfigRaw
Creating machine...
(minikube) Calling .Create
(minikube) Creating machine...
(minikube) Creating network...
(minikube) Setting up minikube home directory...
(minikube) DBG | Setting executable bit set on /home/cryto/.minikube/machines/minikube
(minikube) DBG | Setting executable bit set on /home/cryto/.minikube/machines
(minikube) DBG | Setting executable bit set on /home/cryto/.minikube
(minikube) Building disk image...
(minikube) DBG | Setting executable bit set on /home
(minikube) Downloading /home/cryto/.minikube/cache/boot2docker.iso from file:///home/cryto/                                                                                                                                                  .minikube/cache/iso/minikube-v0.30.0.iso...
(minikube) DBG | Writing magic tar header
(minikube) DBG | Writing SSH key tar header
(minikube) Creating domain...
(minikube) DBG | Finished creating machine, now starting machine...
(minikube) Creating network...
(minikube) Ensuring networks are active...
(minikube) Ensuring network default is active
(minikube) Ensuring network minikube-net is active
(minikube) Getting domain xml...
(minikube) Creating domain...
(minikube) Waiting to get IP...
(minikube) DBG | Waiting for machine to come up 0/40
(minikube) DBG | Waiting for machine to come up 1/40
(minikube) DBG | Waiting for machine to come up 2/40
(minikube) DBG | Waiting for machine to come up 3/40
(minikube) DBG | Waiting for machine to come up 4/40
(minikube) DBG | Waiting for machine to come up 5/40
(minikube) DBG | Waiting for machine to come up 6/40
(minikube) DBG | Waiting for machine to come up 7/40
(minikube) DBG | Waiting for machine to come up 8/40
(minikube) DBG | Waiting for machine to come up 9/40
(minikube) DBG | Waiting for machine to come up 10/40
(minikube) DBG | Waiting for machine to come up 11/40
(minikube) DBG | Waiting for machine to come up 12/40
(minikube) DBG | Waiting for machine to come up 13/40
(minikube) DBG | Waiting for machine to come up 14/40
(minikube) DBG | Waiting for machine to come up 15/40
(minikube) DBG | Waiting for machine to come up 16/40
(minikube) DBG | Waiting for machine to come up 17/40
(minikube) DBG | Waiting for machine to come up 18/40
(minikube) DBG | Waiting for machine to come up 19/40
(minikube) DBG | Waiting for machine to come up 20/40
(minikube) DBG | Waiting for machine to come up 21/40
(minikube) DBG | Waiting for machine to come up 22/40
(minikube) DBG | Waiting for machine to come up 23/40
(minikube) DBG | Waiting for machine to come up 24/40
(minikube) DBG | Waiting for machine to come up 25/40
(minikube) DBG | Waiting for machine to come up 26/40
(minikube) DBG | Waiting for machine to come up 27/40
(minikube) DBG | Waiting for machine to come up 28/40
(minikube) DBG | Waiting for machine to come up 29/40
(minikube) DBG | Waiting for machine to come up 30/40
(minikube) DBG | Waiting for machine to come up 31/40
(minikube) DBG | Waiting for machine to come up 32/40
(minikube) DBG | Waiting for machine to come up 33/40
(minikube) DBG | Waiting for machine to come up 34/40
(minikube) DBG | Waiting for machine to come up 35/40
(minikube) DBG | Waiting for machine to come up 36/40
(minikube) DBG | Waiting for machine to come up 37/40
(minikube) DBG | Waiting for machine to come up 38/40
(minikube) DBG | Waiting for machine to come up 39/40
(minikube) DBG | Waiting for machine to come up 40/40
E1209 09:57:58.546098    9761 start.go:168] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds.

Anything else do we need to know:
The dnsmasq status file for minikube-net at /var/lib/libvirt/dnsmasq/virbr2.status is empty! Actually all three status files for the three available virtual networks (default - virbr0.status, docker-machines - virbr1.status, minikube-net - virbr2.status) are empty. I think i got the same problem with minishift with the kvm driver (not kvm2) and it has been reported minishift#3028 - but still no solution though

I think the created VM has troubles to lease IP from dnsmasq but i dont know how to get it solved. Thanks you!

The text was updated successfully, but these errors were encountered:

viteksafronov · 2018-12-17T10:26:37Z

It looks like this strange bug was added by me when I tried to fix a problem with dnsmasq in #2745.
I don't know why, but on some machines the VM cannot get the IP from dnsmasq dhcp when this setting is enabled. I tried to discover the problem by tcpdumping the bridge, but I don't even see dhcp requests from VM. Anyway, I will try to fix it.
You could try to use the driver binary from 0.25.0 version, it should work for you.

alexdyukov · 2019-01-16T16:53:49Z

@viteksafronov Hey! The same error with 0.25.0 driver:
https://gist.githubusercontent.com/AlexDyukov/9e59afd73dbe35d7783219f1d0abe7bd/raw/3291ff35c3c7a17e2b1e3c4ce81ed77098d06333/gistfile1.txt

irasnyd · 2019-01-30T03:52:49Z

I was able to get minikube v0.33.1 and docker-machine-driver-kvm2 v0.31.0 to work correctly on CentOS 7. More recent versions of docker-machine-driver-kvm2 don't work for me.

onhubgg · 2019-02-01T15:08:40Z

Link to docker-machine-driver-kvm2 v0.31.0 to share ? No track found. Thanks

irasnyd · 2019-02-01T16:36:46Z

The docker-machine-driver-kvm2 is on the minikube release page for v0.31.0:
https://github.com/kubernetes/minikube/releases/tag/v0.31.0

Here is a direct link:
https://github.com/kubernetes/minikube/releases/download/v0.31.0/docker-machine-driver-kvm2

anselvo · 2019-02-20T02:36:09Z

I have the same problem with kvm2 and kvm drivers. Also i check new version 0.34.1 and version 0.25.0 it's didn't work for me ;(

If i can give you some information tell me what to do.

pratikpparikh · 2019-03-20T01:02:50Z

@aelsergeev did you solve this problem? I am facing the same.

anselvo · 2019-03-20T11:05:55Z

@aelsergeev did you solve this problem? I am facing the same.

No, I use VB driver now, you can better to write about the problem in this issue #3566

pratikpparikh · 2019-03-21T03:24:29Z

@aelsergeev it is fixed with the latest version of minikube. Will fail once you minikube stop && minikube delete && rm -rf ~/.minikube && rm -rf /tmp/*

anselvo · 2019-03-21T12:50:25Z

latest version not help me

pratikpparikh · 2019-03-21T21:17:30Z

@aelsergeev i am using debian.

tstromberg · 2019-05-22T22:55:27Z

I'm closing this issue as it hasn't seen activity in awhile, and it's unclear if this issue still exists. If this issue does continue to exist in the most recent release of minikube, please feel free to re-open it.

Do make sure to upgrade first to the latest kvm driver:

curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-kvm2 && sudo install docker-machine-driver-kvm2 /usr/local/bin/

Thank you for opening the issue!

tstromberg added co/kvm2-driver KVM2 driver related issues failed/local-networking startup failures due to networking issues labels Dec 18, 2018

tstromberg added priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. cause/vm-networking Startup failures due to VM networking and removed failed/local-networking startup failures due to networking issues labels Jan 24, 2019

tstromberg changed the title ~~kvm2: Error in driver during machine creation: Machine didn't return an IP after 120 seconds~~ kvm2 w/ dnsmasq: Machine didn't return an IP after 120 seconds Jan 24, 2019

tstromberg added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Jan 24, 2019

tstromberg mentioned this issue Feb 20, 2019

kvm2: minikube network does not exist #3636

Closed

tstromberg changed the title ~~kvm2 w/ dnsmasq: Machine didn't return an IP after 120 seconds~~ dnsmasq: during machine creation: Machine didn't return an IP after 120 seconds May 14, 2019

tstromberg closed this as completed May 22, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dnsmasq: during machine creation: Machine didn't return an IP after 120 seconds #3434

dnsmasq: during machine creation: Machine didn't return an IP after 120 seconds #3434

tunggad commented Dec 9, 2018 •

edited

Loading

viteksafronov commented Dec 17, 2018 •

edited

Loading

alexdyukov commented Jan 16, 2019

irasnyd commented Jan 30, 2019

onhubgg commented Feb 1, 2019

irasnyd commented Feb 1, 2019

anselvo commented Feb 20, 2019

pratikpparikh commented Mar 20, 2019

anselvo commented Mar 20, 2019

pratikpparikh commented Mar 21, 2019

anselvo commented Mar 21, 2019

pratikpparikh commented Mar 21, 2019

tstromberg commented May 22, 2019 •

edited

Loading

dnsmasq: during machine creation: Machine didn't return an IP after 120 seconds #3434

dnsmasq: during machine creation: Machine didn't return an IP after 120 seconds #3434

Comments

tunggad commented Dec 9, 2018 • edited Loading

viteksafronov commented Dec 17, 2018 • edited Loading

alexdyukov commented Jan 16, 2019

irasnyd commented Jan 30, 2019

onhubgg commented Feb 1, 2019

irasnyd commented Feb 1, 2019

anselvo commented Feb 20, 2019

pratikpparikh commented Mar 20, 2019

anselvo commented Mar 20, 2019

pratikpparikh commented Mar 21, 2019

anselvo commented Mar 21, 2019

pratikpparikh commented Mar 21, 2019

tstromberg commented May 22, 2019 • edited Loading

tunggad commented Dec 9, 2018 •

edited

Loading

viteksafronov commented Dec 17, 2018 •

edited

Loading

tstromberg commented May 22, 2019 •

edited

Loading