fix stale hyperkit.pid making minikube start hang #3593

balopat · 2019-01-26T01:12:30Z

tstromberg · 2019-01-28T17:45:45Z

Is some additional work required here?

--- FAIL: TestDocker (190.68s)
	util.go:246: got state: "(minikube) Found hyperkit pid: 12365\nRunning", expected "Running"

pkg/drivers/hyperkit/driver.go

tstromberg · 2019-01-28T18:13:09Z

This change is wonderful and I can't wait to see it merged. Thank you!

k8s-ci-robot · 2019-01-28T21:43:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: balopat

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [balopat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

balopat · 2019-01-28T21:49:35Z

Is some additional work required here?

--- FAIL: TestDocker (190.68s)
	util.go:246: got state: "(minikube) Found hyperkit pid: 12365\nRunning", expected "Running"

That was a bug that an additional debugging line introduced! I removed the offending log.

tstromberg · 2019-01-29T00:08:17Z

pkg/drivers/hyperkit/driver.go

+			Driver: "virtio-blk",
+		},
+	}
+	//}


Remove comment

tstromberg · 2019-01-29T00:09:00Z

Looks great. Please merge once test failures are addressed. The most recent one looks novel, but unfortunately there isn't much in the way of info:

 15:47:58 | > (minikube) clean start, hyperkit pid file doesn't exist: /Users/jenkins/minikube-integration/darwin-amd64-hyperkit-3593-77386-5af2f1eafc7a5d00238d05637682bf4264343497/.minikube/machines/minikube/hyperkit.pid
 15:47:58 | > (minikube) Using UUID 23979220-2357-11e9-ac78-38c98626f627
 15:48:00 | ! E0128 15:48:00.546915   77513 start.go:205] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: getting MAC address from UUID: vmnet: error from vmnet.framework: -1
 15:48:00 | ! .
 15:48:00 | !  Retrying.
 15:48:00 | ! I0128 15:48:00.546971   77513 utils.go:114] non-retriable error: Error creating host: Error creating machine: Error in driver during machine creation: getting MAC address from UUID: vmnet: error from vmnet.framework: -1
 15:48:00 | > minikube failed :( exiting with error code 1
 15:48:00 | ! E0128 15:48:00.546993   77513 start.go:211] Error starting host:  Error creating host: Error creating machine: Error in driver during machine creation: getting MAC address from UUID: vmnet: error from vmnet.framework: -1

k8s-ci-robot · 2019-01-29T01:29:20Z

New changes are detected. LGTM label has been removed.

balopat · 2019-01-29T01:37:26Z

It's an odd error that doesn't occur locally. I reverted a weird change in the disk setup - not sure why it was moved (this was an older code of mine that I was playing with - might have been an accidental mouse move?) - the issue might have been related.

balopat · 2019-01-29T04:01:25Z

It seems that something (maybe the first bad build here?) broke the hyperkit build - and it is now breaking on the master builds too with the same vmnet error. I am suspecting environmental/host level state to be responsible, we'll have to look tomorrow.

balopat · 2019-01-29T04:06:45Z

in fact there is a running hyperkit instance:

55199   ??  S     18:06.13 /usr/local/bin/hyperkit -A -u -F /Users/jenkins/minikube-integration/darwin-amd64-hyperkit-master-54872-00be38dadc7c4301471c9d6657853252bc729e4a/.minikube/machines/minikube/hyperkit.pid -c 2 -m 2048M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-net -U 304f5632-22ae-11e9-846f-38c98626f627 -s 2:0,virtio-blk,/Users/jenkins/minikube-integration/darwin-amd64-hyperkit-master-54872-00be38dadc7c4301471c9d6657853252bc729e4a/.minikube/machines/minikube/minikube.rawdisk -s 3,ahci-cd,/Users/jenkins/minikube-integration/darwin-amd64-hyperkit-master-54872-00be38dadc7c4301471c9d6657853252bc729e4a/.minikube/machines/minikube/boot2docker.iso -s 4,virtio-rnd -l com1,autopty=/Users/jenkins/minikube-integration/darwin-amd64-hyperkit-master-54872-00be38dadc7c4301471c9d6657853252bc729e4a/.minikube/machines/minikube/tty,log=/Users/jenkins/minikube-integration/darwin-amd64-hyperkit-master-54872-00be38dadc7c4301471c9d6657853252bc729e4a/.minikube/machines/minikube/console-ring -f kexec,/Users/jenkins/minikube-integration/darwin-amd64-hyperkit-master-54872-00be38dadc7c4301471c9d6657853252bc729e4a/.minikube/machines/minikube/bzimage,/Users/jenkins/minikube-integration/darwin-amd64-hyperkit-master-54872-00be38dadc7c4301471c9d6657853252bc729e4a/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube

I'm going to kill it and retry the job.

balopat · 2019-01-29T04:28:47Z

Some curious experiments:
while the rouge hyperkit server was running I tried to spin up a new hyperkit server manually.

$ sudo /usr/local/bin/hyperkit -A -u -F /Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/hyperkit.pid -c 2 -m 2048M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-net -U 304f5632-22ae-11e9-846f-38c98626f627 -s 2:0,virtio-blk,/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/minikube.rawdisk -s 3,ahci-cd,/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/boot2docker.iso -s 4,virtio-rnd -l com1,autopty=/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/tty,log=/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/console-ring -f kexec,/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/bzimage,/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube
virtio_net: Could not create vmnet interface, permission denied or no entitlement?

When I killed the server, for a while I still got virtio_net: Could not create vmnet interface, permission denied or no entitlement? and just a couple of minutes later the expected:

$ sudo /usr/local/bin/hyperkit -A -u -F /Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/hyperkit.pid -c 2 -m 2048M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-net -U 304f5632-22ae-11e9-846f-38c98626f627 -s 2:0,virtio-blk,/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/minikube.rawdisk -s 3,ahci-cd,/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/boot2docker.iso -s 4,virtio-rnd -l com1,autopty=/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/tty,log=/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/console-ring -f kexec,/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/bzimage,/Users/jenkins/minikube-integration/btest/.minikube/machines/minikube/initrd,earlyprintk=serial loglevel=3 user=docker console=ttyS0 console=tty0 noembed nomodeset norestore waitusb=10 systemd.legacy_systemd_cgroup_controller=yes base host=minikube
Could not open backing file: No such file or directory
Could not open backing file: No such file or directory

Which is expected as the directory did not contain the initrd, bzimage. rawdisk and boot2docker.iso files.

I bounced the jenkins job - let's see where that gets us.

balopat · 2019-01-29T04:34:01Z

I wonder where the collision was - one candidate is host=minikube - and would make sense that virtio-net wouldn't allow two hosts with the same name...

tstromberg · 2019-01-29T18:30:53Z

@minikube-bot OK to test

tstromberg · 2019-01-30T14:40:12Z

@minikube-bot OK to test

balopat added 2 commits January 25, 2019 17:08

fix stale hyperkit.pid making minikube start hang

9b97a85

lint

933356e

tstromberg suggested changes Jan 28, 2019

View reviewed changes

fixing review remarks

f70e6a0

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 28, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 28, 2019

formatting / removed extra comment

5af2f1e

balopat added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 28, 2019

tstromberg reviewed Jan 29, 2019

View reviewed changes

pkg/drivers/hyperkit/driver.go Outdated

Driver: "virtio-blk",

},

}

//}

Copy link

Contributor

tstromberg Jan 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove comment

reverting unnecessary change

17c0244

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 29, 2019

balopat added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 29, 2019

k8s-ci-robot merged commit fd38580 into kubernetes:master Jan 30, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix stale hyperkit.pid making minikube start hang #3593

fix stale hyperkit.pid making minikube start hang #3593

balopat commented Jan 26, 2019

tstromberg commented Jan 28, 2019 •

edited

Loading

tstromberg commented Jan 28, 2019 •

edited

Loading

k8s-ci-robot commented Jan 28, 2019

balopat commented Jan 28, 2019

tstromberg Jan 29, 2019

tstromberg commented Jan 29, 2019 •

edited

Loading

k8s-ci-robot commented Jan 29, 2019

balopat commented Jan 29, 2019

balopat commented Jan 29, 2019

balopat commented Jan 29, 2019

balopat commented Jan 29, 2019

balopat commented Jan 29, 2019

tstromberg commented Jan 29, 2019

tstromberg commented Jan 30, 2019

fix stale hyperkit.pid making minikube start hang #3593

fix stale hyperkit.pid making minikube start hang #3593

Conversation

balopat commented Jan 26, 2019

tstromberg commented Jan 28, 2019 • edited Loading

tstromberg commented Jan 28, 2019 • edited Loading

k8s-ci-robot commented Jan 28, 2019

balopat commented Jan 28, 2019

tstromberg Jan 29, 2019

Choose a reason for hiding this comment

tstromberg commented Jan 29, 2019 • edited Loading

k8s-ci-robot commented Jan 29, 2019

balopat commented Jan 29, 2019

balopat commented Jan 29, 2019

balopat commented Jan 29, 2019

balopat commented Jan 29, 2019

balopat commented Jan 29, 2019

tstromberg commented Jan 29, 2019

tstromberg commented Jan 30, 2019

tstromberg commented Jan 28, 2019 •

edited

Loading

tstromberg commented Jan 28, 2019 •

edited

Loading

tstromberg commented Jan 29, 2019 •

edited

Loading