Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hyperv: "minikube stop" fails (Shutting down 0% until it eventually times out) #2914

Closed
shahiddev opened this issue Jun 19, 2018 · 52 comments
Closed

hyperv: "minikube stop" fails (Shutting down 0% until it eventually times out) #2914

shahiddev opened this issue Jun 19, 2018 · 52 comments
Labels
co/hyperv HyperV related issues good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. r/2019q2 Issue was last reviewed 2019q2
Milestone
v1.3.0

Comments

@shahiddev
Copy link
Contributor

shahiddev commented Jun 19, 2018 •
edited
Loading

BUG REPORT
Please provide the following details:

Environment: Windows 10 Pro 1803

Minikube version (use minikube version): v0.28.0

  • OS (e.g. from /etc/os-release): Windows 10 Pro 1803
  • VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): Hyper-v
  • ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): v0.28.0
  • Install tools:
  • Others:

What happened: Minikube stop fails to stop minikube VM

What you expected to happen: VM stops

How to reproduce it (as minimally and precisely as possible): With running minikube, perform minikube stop

Output of minikube logs (if applicable):

Anything else do we need to know:
Clean install of Minikube 0.28.0 (deleted .minikube folder)
Command sits with powershell overlay showing Shutting down 0% until it eventually times out
Trying to shutdown via Hyper-V manager/Powershell doesn't work either
Opening VM in hyper-v shows command prompt - may not be relevant?
image
@shahiddev
Copy link
Contributor Author

Have rolled back to 0.27.0 to verify that minikube stop works in that version which it does

@gitfool
Copy link

gitfool commented Jun 19, 2018

Same environment and same problem here. Also, if I try to stop minikube from Hyper-V Manager I get the following error:

image

@gbraad
Copy link
Contributor

gbraad commented Jun 20, 2018 via email

@F21
Copy link

F21 commented Jun 20, 2018

I am seeing this issue. Shutdown takes a very long time and fails with this error:

Stopping local Kubernetes cluster...
Error stopping machine:  Error stopping host: minikube: exit status 1
================================================================================
An error has occurred. Would you like to opt in to sending anonymized crash
information to minikube to help prevent future errors?
To opt out of these messages, run the command:
        minikube config set WantReportErrorPrompt false
================================================================================

Once the shutdown fails, the machine reverts to "running" inside the Hyper-V manager. I then use Hyper-V to turn off the machine.

@shahiddev
Copy link
Contributor Author

shahiddev commented Jun 20, 2018 •
edited
Loading

@gbraad I'll dig a little deeper later but I reverted back to 0.27.0 and it works fine, upgrade to 0.28.0 and it stops working. So there seems to be something unusual going on with 0.28.0

Edited to add, if I connect to the VM and login I can then issue sudo poweroff -p and it shuts down fine.

Nothing blatantly obvious in the hyper-v logs, probably closest thing of interest I've found is

'minikube' has encountered a fatal error.  The guest operating system reported that it failed with the following error codes: ErrorCode0: 0x8, ErrorCode1: 0x81000004100E0000, ErrorCode2: 0x7FB9E02E643F, ErrorCode3: 0x40000000, ErrorCode4: 0x7FFDF78C75A0.  If the problem persists, contact Product Support for the guest operating system.

EDIT: Actually same error is logged for 0.27.0 but it doesn't impact shutdown on that version

@poagester
Copy link

poagester commented Jun 21, 2018 •
edited
Loading

i can confirm this is occuring with windows 10 pro 1703 with minikube .28 as well. My error message is:

C:\WINDOWS\system32>minikube stop minikube
Stopping local Kubernetes cluster...
Error stopping machine: Error stopping host: minikube: exit status 1

I will try to update this pc and see if that works, if not i will use the suggested downgrade for minikube.

upgrading to 1803 with Minikube v28 had the same results
downgrading to Minikube v27 works as normal

@shahiddev shahiddev mentioned this issue Jun 28, 2018
Closed
@discosultan
Copy link

discosultan commented Jun 30, 2018 •
edited
Loading

Quite annoying indeed. I've found

$ minikube ssh
$ sudo poweroff

to be the fastest workaround.

@LazerFX
Copy link

LazerFX commented Jul 3, 2018 •
edited
Loading

Thanks for the minikube ssh | sudo poweroff workaround for this issue - even telling Hyper-V to shut the device down did not work (Presumably because it was in a frozen // not responding state).

Is there any further information those of us who have this problem can provide to assist in resolving? I've not seen anything particularly nasty in the event logs or similar...

Edit
I have noticed that the sudo shutdown in minikube 0.27.0 does an immediate shutdown, whereas in minikube 0.28.0, it schedules the shutdown for one minute in the future. I don't know if the infrastructure issues a sudo shutdown in the background, but if it does and gets into a hang or race condition with itself re-issuing the command, it will never actually shutdown...

@wlei07
Copy link

wlei07 commented Jul 16, 2018

The same issue with minikube 0.28.0 on windows 10

@gbraad
Copy link
Contributor

gbraad commented Jul 16, 2018

@LazerFX @wlei07 can any of you try to use minishift to verify if we have the same issue? If this issue is reproducible we might be able to dig into this deeper...

@gbraad
Copy link
Contributor

gbraad commented Jul 16, 2018 •
edited
Loading

@shahiddev I noticed @LazerFX has observed the command shutdown gets scheduled for after one minute. This could mean the shutdown will happen...

In code, Stop-VM is triggered from a powershell session:

minikube/vendor/github.com/docker/machine/drivers/hyperv/hyperv.go

Line 346 in 33fce13

if err := cmd("Hyper-V\\Stop-VM", d.MachineName); err != nil {

So, I wonder what causes this difference in behaviour...

@shahiddev
Copy link
Contributor Author

@gbraad I've certainly waited for over 1min (in fact over 5mins) and it doesn't shutdown.
My suspicions were around how Hyper-V interacts with the Linux VM but I couldn't find anything useful to help with this. I did experiment with disabling the hyper-v integration services which includes an option for Guest OS shutdown and disabling this causes an immediate error.

image

I'm preparing for a talk tomorrow so don't want to mess with my minikube installation today but can try and have another go at this later this week.

@LazerFX
Copy link

LazerFX commented Jul 16, 2018

@gbraad I'm sorry, I don't know what minishift is, but I can certainly try and Google it when I'm in the office on the machine tomorrow, see if I can diagnose the issue. Any tips on what you're looking for, I'd appreciate :-D

I'm quite a neophyte to the whole kubernetes infrastructure, so I "don't know what I don't know" as the saying goes - I don't know the holes in my knowledge where I should be able to fill in.

@gbraad
Copy link
Contributor

gbraad commented Jul 17, 2018 via email

@LazerFX
Copy link

LazerFX commented Jul 17, 2018 •
edited
Loading

Just rolled up a minishift environment, based on minishift version v1.21.0+a8c8b37, and that is capable of successfully shutting down using minishift stop.

Based on other comments, I also tried the Stop-VM minikube command mentioned above. It puts a 0% progress-bar in the PowerShell window, and nothing continues. Looking in the SSH, I see there is a long-running -bash command:

$ sudo ps all
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
4     0  2052     1  20   0   9240   448 -      Ss+  tty1       0:00 /sbin/getty
4     0  2069     1  20   0   9240   452 -      Ss+  ttyS0      0:00 /sbin/getty
0  1001  4065  4064  20   0  16236  3432 -      Ss   pts/0      0:00 -bash
4     0  4267  4065  20   0  20316   688 -      R+   pts/0      0:00 ps all

That wasn't there prior to the minikube stop command being triggered.

@gbraad
Copy link
Contributor

gbraad commented Jul 17, 2018 via email

@shahiddev
Copy link
Contributor Author

looks like minikube 0.28.1 also has this issue.

@LazerFX
Copy link

LazerFX commented Jul 18, 2018

Was there an update to the ISO for 0.28? Can this be isolated as a potential source - i.e. the OS not responding properly? Can an 0.27 ISO be used on 0.28?

@shahiddev
Copy link
Contributor Author

@LazerFX I just did the same thing you did with minikube ssh but I can see that Bash command is already running before the shutdown was initiated. I ran minikube stop in a different window whilst connected via minikube SSH
image

@shahiddev
Copy link
Contributor Author

@gbraad Looks like MiniShift is using the latest version of Docker Machine (0.15) where as Minikube isn't. I wanted to update to the latest version anyway so would it make sense to just update to that? I'm a bit of a noob on this project so don't know the best way of doing that.

@gbraad
Copy link
Contributor

gbraad commented Jul 18, 2018 via email

@saostad
Copy link

saostad commented Aug 20, 2018 •
edited
Loading

Same Problem here:

win 10 enterprice verion 1803
Docker 18.06.0-ce
kubectl 1.11.0
minikube v0.28.2

@kognise
Copy link

kognise commented Aug 22, 2018

Same problem with the same exact information as @saostad. Is there a way to temporarily fix this?

@shahiddev
Copy link
Contributor Author

only minikube ssh and then sudo poweroff

@kognise
Copy link

kognise commented Aug 22, 2018

@shahiddev That works. I discovered that downgrading works too, though.

@idrm
Copy link

idrm commented Aug 22, 2018

You can combine the commands to issue a shutdown in one line:

minikube ssh "sudo poweroff"

@kognise
Copy link

kognise commented Aug 23, 2018

Could minikube just be updated to run that command for minikube stop?

@gbraad
Copy link
Contributor

gbraad commented Aug 23, 2018 via email

@zmbq
Copy link

zmbq commented Sep 4, 2018

I have exactly the same issue on Windows 10 pro. Stopping the cluster with poweroff worked, but then the cluster couldn't be started again. I had to uninstall, remove ~/.minikube and install version 0.27 for everything to work.

@tstromberg tstromberg self-assigned this Feb 13, 2019
@tstromberg tstromberg reopened this Feb 16, 2019
@vxavictor513
Copy link

Still happening on v0.35.0

@tstromberg
Copy link
Contributor

@vxavictor513 - Do you mind attaching the output of:

minikube stop --alsologtostderr -v=8

Thanks!

@vxavictor513
Copy link

vxavictor513 commented Mar 7, 2019 •
edited
Loading

@tstromberg - This is what I got after running the command and waiting for 10 minutes.

$ minikube stop --alsologtostderr -v=8
W0308 07:02:16.215679   20084 root.go:145] Error reading config file at C:\Users\user\.minikube\config\config.json: open C:\Users\user\.minikube\config\config.json: The system cannot find the file specified.
I0308 07:02:16.219677   20084 notify.go:121] Checking for updates...
:   Stopping "minikube" in hyperv ...
Stopping "minikube"...
[executing ==>] : C:\windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Hyper-V\Stop-VM minikube

I force-cancelled the minikube stop command by Ctrl+C, then minikube status shows it is still up.

$ minikube status
host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.1.213

@inteledyne
Copy link

FWIW, I experienced a lot of these errors when setting up minikube via chocolatey w/ powershell on windows 10. minikube delete followed by recreating eventually worked after a few tries.

@bjorn-ali-goransson
Copy link

Just had a scare with this when the kube wouldn't stop. Just followed the simple steps from the tutorial. Thought a computer restart would do it, but it got stuck at "Restarting your computer" for about 10 minutes. Luckily, though, in the end, it actually restarted.

@tstromberg tstromberg mentioned this issue Apr 2, 2019
Closed
@tstromberg tstromberg modified the milestones: v0.34.0, v1.1.0-candidate Apr 16, 2019
@tstromberg tstromberg removed their assignment Apr 30, 2019
@FrauleinGela
Copy link

yeah, I deleted my ~/.minikube folder + minikube virtual machine and run process : minikube start --vm-driver=x --kubernetes-version="vx.xx.x" and works 👍

@tstromberg tstromberg modified the milestones: v1.1.0, v1.2.0-candidate May 22, 2019
@tstromberg tstromberg changed the title hyperv: "minikube stop" fails hyperv: "minikube stop" fails (Shutting down 0% until it eventually times out) May 22, 2019
@tstromberg tstromberg added the r/2019q2 Issue was last reviewed 2019q2 label May 22, 2019
@tstromberg tstromberg removed this from the v1.2.0-candidate milestone May 29, 2019
@tstromberg tstromberg added this to the v1.2.0 milestone Jun 10, 2019
@Enoch-Ye
Copy link

Same problem here

Windows 10 Enterprise 1803
Docker version 18.09.2
Kubectl v1.10.11
Minikube v1.0.0

@blueelvis
Copy link
Contributor

@tstromberg I have also seen this where the machine in Hyper-V refuses to Power off even from the manager. I tried both Reset and ACPI Shutdown and it still didn't work. Strangely this has happened only with minikube for me till now. I would like to take a stab at this once I push the documentation of all the available arguments.

@blueelvis blueelvis mentioned this issue Jul 2, 2019
Closed
@Valgard
Copy link

Valgard commented Jul 2, 2019

Same problem here

Windows 10 Pro 1903
Docker v18.09.2
Kubectl v1.15.0
Minikube v1.2.0

Output from minikube stop:

* Stopping "minikube" in hyperv ...
* Stopping "minikube" in hyperv ...
* Stopping "minikube" in hyperv ...
* Stopping "minikube" in hyperv ...
* Stopping "minikube" in hyperv ...

X Unable to stop VM: Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1

* Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
  - https://github.com/kubernetes/minikube/issues/new

Output from minikube stop --alsologtostderr -v=8:

I0702 14:50:50.622640   20504 notify.go:128] Checking for updates...                                                                                            * Stopping "minikube" in hyperv ...                                                                                                                             Stopping "minikube"...
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Hyper-V\Stop-VM minikube
[stdout =====>] :
[stderr =====>] : Hyper-V\Stop-VM : Der Status von "minikube" konnte nicht geändert werden.
In Zeile:1 Zeichen:1
+ Hyper-V\Stop-VM minikube
+ ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Stop-VM], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.HyperV.PowerShell.Commands.StopVM
                                                                                                                                                                                                                                                                                                                                I0702 15:00:52.605171   20504 utils.go:126] error: Temporary Error: Stop: minikube: exit status 1 - sleeping 1s                                                 I0702 15:00:53.611320   20504 utils.go:115] retry loop 1                                                                                                        * Stopping "minikube" in hyperv ...                                                                                                                             Stopping "minikube"...
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Hyper-V\Stop-VM minikube
[stdout =====>] :
[stderr =====>] : Hyper-V\Stop-VM : Der Status von "minikube" konnte nicht geändert werden.
In Zeile:1 Zeichen:1
+ Hyper-V\Stop-VM minikube
+ ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Stop-VM], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.HyperV.PowerShell.Commands.StopVM
                                                                                                                                                                                                                                                                                                                                I0702 15:10:55.198539   20504 utils.go:126] error: Temporary Error: Stop: minikube: exit status 1 - sleeping 1s                                                 I0702 15:10:56.201604   20504 utils.go:115] retry loop 2                                                                                                        * Stopping "minikube" in hyperv ...                                                                                                                             Stopping "minikube"...
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Hyper-V\Stop-VM minikube
[stdout =====>] :
[stderr =====>] : Hyper-V\Stop-VM : Der Status von "minikube" konnte nicht geändert werden.
In Zeile:1 Zeichen:1
+ Hyper-V\Stop-VM minikube
+ ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Stop-VM], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.HyperV.PowerShell.Commands.StopVM


I0702 15:20:57.826478   20504 utils.go:126] error: Temporary Error: Stop: minikube: exit status 1 - sleeping 1s
I0702 15:20:58.831673   20504 utils.go:115] retry loop 3
* Stopping "minikube" in hyperv ...
Stopping "minikube"...
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Hyper-V\Stop-VM minikube
[stdout =====>] :
[stderr =====>] : Hyper-V\Stop-VM : Der Status von "minikube" konnte nicht geändert werden.
In Zeile:1 Zeichen:1
+ Hyper-V\Stop-VM minikube
+ ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Stop-VM], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.HyperV.PowerShell.Commands.StopVM


I0702 15:31:00.199707   20504 utils.go:126] error: Temporary Error: Stop: minikube: exit status 1 - sleeping 1s
I0702 15:31:01.204811   20504 utils.go:115] retry loop 4
* Stopping "minikube" in hyperv ...
Stopping "minikube"...
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive Hyper-V\Stop-VM minikube
[stdout =====>] :
[stderr =====>] : Hyper-V\Stop-VM : Der Status von "minikube" konnte nicht geändert werden.
In Zeile:1 Zeichen:1
+ Hyper-V\Stop-VM minikube
+ ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Stop-VM], VirtualizationException
    + FullyQualifiedErrorId : OperationFailed,Microsoft.HyperV.PowerShell.Commands.StopVM


I0702 15:41:02.817565   20504 utils.go:126] error: Temporary Error: Stop: minikube: exit status 1 - sleeping 1s
W0702 15:41:03.819005   20504 exit.go:100] Unable to stop VM: Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1

X Unable to stop VM: Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1
Temporary Error: Stop: minikube: exit status 1

* Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
  - https://github.com/kubernetes/minikube/issues/new

@thebjorn
Copy link

thebjorn commented Jul 7, 2019 •
edited
Loading

I have the same versions and issues as @Valgard above, in addition.. I was following the first tutorial (https://kubernetes.io/docs/tutorials/hello-minikube/) and I could enable the heapster addon, but minikube crashed when trying to disable it:

c:\srv> minikube addons enable heapster
* heapster was successfully enabled

c:\srv> minikube addons disable heapster

X disable failed: [disabling addon deploy/addons/heapster/influx-grafana-rc.yaml.tmpl: Process exited with status 1]

* Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
  - https://github.com/kubernetes/minikube/issues/new

c:\srv> minikube addons disable heapster --alsologtostderr -v=8
I0707 18:47:39.551945   13496 notify.go:128] Checking for updates...
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
[stdout =====>] : Running

[stderr =====>] :
[executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Hyper-V\Get-VM minikube ).networkadapters[0]).ipaddresses[0]
[stdout =====>] : 192.168.1.148

[stderr =====>] :
W0707 18:47:42.377956   13496 exit.go:100] disable failed: [disabling addon deploy/addons/heapster/influx-grafana-rc.yaml.tmpl: Process exited with status 1]

X disable failed: [disabling addon deploy/addons/heapster/influx-grafana-rc.yaml.tmpl: Process exited with status 1]

* Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
  - https://github.com/kubernetes/minikube/issues/new

@tstromberg tstromberg modified the milestones: v1.2.0, v1.3.0-candidate Jul 12, 2019
@blueelvis blueelvis mentioned this issue Jul 14, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
co/hyperv HyperV related issues good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. r/2019q2 Issue was last reviewed 2019q2
Projects
None yet
Milestone
v1.3.0
Development

No branches or pull requests