dashboard: Add Node condition check (DiskPressure and pod status checks) before opening #5815
Comments
|
ping? |
|
@GarretSidzaka
This will help us isolate the problem further. Thank you! and additionally, I wonder do you use a corp network or VPN or proxy ? |
medyagh
added
triage/needs-information
|
Bullet one: Bullet two: Verifying dashboard health ... ^C Bullet three:
Bullet Four: The answer to your last question is no. |
|
@GarretSidzaka would you please share the full output of |
|
@GarretSidzaka I am curious when you said it hangs forever, did you mean in terminal it is stuck at this ? if that is the case, then that is the expected behaviour, minikube will hang there and run a webserver, so you can access the dashboard in your browser |
medyagh
changed the title
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-66-generic x86_64)
System information as of Wed Nov 6 00:33:39 UTC 2019 System load: 0.0 Processes: 207
6 packages can be updated. Last login: Mon Nov 4 23:29:43 2019 from 71.209.166.96
I1106 00:34:13.503829 183139 start.go:236] hostinfo: {"hostname":"cloudstack","uptime":931694,"bootTime":1572068759,"procs":266,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"18.04","kernelVersion":"4.15.0-66-generic","virtualizationSystem":"","virtualizationRole":"","hostid":"73fa3e87-061f-4111-9c29-9a2074fc4bec"}
[Service] [Install]
! The 'none' driver provides limited isolation and may reduce system security and reliability. ! kubectl and minikube configuration will be stored in /home/garretsidzaka
|
i cannot load a local browser on a production headless VM, there is no X-server garretsidzaka@cloudstack:~$ sudo minikube dashboard
|
the point of the dashboard is actually a UI experience for kubernetes, if you dont have a browser, you might not actually need dashboard. do you mind sharing the output of curl for the dashboard url in a separate terminal ? btw do you happen to use vpn or proxies ? |
tstromberg
changed the title
|
The output of this command would be helpful for us to help with debugging:
It's worth noting that with sudo, the dashboard command will only output a URL rather than attempting to open a browser. |
tstromberg
added
co/dashboard
no unusual network. this is a front end bridged VM, production style. this network port has a static IP that is IANA, not NAT. there is no proxy or VPN. and yes its very nice to have this kind of research VM |
yes and when i click such a link after obviously replacing 127 with the actual IP address, it gives 503. attached is the log you requested |
|
Ping :3 |
|
Hey @GarretSidzaka -- looks like there are a couple of related issues (#4352 and #4749). I see you already commented on #4352, and I'm guessing none of those solutions fixed your issue? #4749 suggests increasing memory/cpu allocation to minikube. Perhaps you could give that a try? Please let us know the results of any of these experiments! |
sudo minikube dashboard --alsologtostderr -v=1
I1113 18:16:48.105064 30332 none.go:257] checking for running kubelet ...
I1113 18:16:48.105097 30332 exec_runner.go:42] (ExecRunner) Run: systemctl is-active --quiet service kubelet
🤔 Verifying dashboard health ...
I1113 18:16:48.136357 30332 service.go:236] Found service: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kubernetes-dashboard,GenerateName:,Namespace:kubernetes-dashboard,SelfLink:/api/v1/namespaces/kubernetes-dashboard/services/kubernetes-dashboard,UID:50ffb80e-1e61-41a3-8ee6-15fec08c9d0c,ResourceVersion:384,Generation:0,CreationTimestamp:2019-11-13 18:08:36 +0000 GMT,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,k8s-app: kubernetes-dashboard,kubernetes.io/minikube-addons: dashboard,},Annotations:map[string]string{kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"kubernetes-dashboard","kubernetes.io/minikube-addons":"dashboard"},"name":"kubernetes-dashboard","namespace":"kubernetes-dashboard"},"spec":{"ports":[{"port":80,"targetPort":9090}],"selector":{"k8s-app":"kubernetes-dashboard"}}}
,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],},Spec:ServiceSpec{Ports:[{ TCP 80 {0 9090 } 0}],Selector:map[string]string{k8s-app: kubernetes-dashboard,},ClusterIP:10.109.8.235,Type:ClusterIP,ExternalIPs:[],SessionAffinity:None,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}
🚀 Launching proxy ...
I1113 18:16:48.136574 30332 dashboard.go:167] Executing: /usr/bin/kubectl [/usr/bin/kubectl --context minikube proxy --port=0]
I1113 18:16:48.136855 30332 dashboard.go:172] Waiting for kubectl to output host:port ...
I1113 18:16:48.294744 30332 dashboard.go:190] proxy stdout: Starting to serve on 127.0.0.1:43299
🤔 Verifying proxy health ...
I1113 18:16:48.304599 30332 dashboard.go:227] http://127.0.0.1:43299/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cache-Control:[no-cache, private] Content-Length:[188] Content-Type:[application/json] Date:[Wed, 13 Nov 2019 18:16:48 GMT]] Body:0xc00035d840 ContentLength:188 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc0004aa800 TLS:<nil>}
I1113 18:16:49.412722 30332 dashboard.go:227] http://127.0.0.1:43299/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cache-Control:[no-cache, private] Content-Length:[188] Content-Type:[application/json] Date:[Wed, 13 Nov 2019 18:16:49 GMT]] Body:0xc000297100 ContentLength:188 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc00012ed00 TLS:<nil>}
I1113 18:16:51.579128 30332 dashboard.go:227] http://127.0.0.1:43299/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cache-Control:[no-cache, private] Content-Length:[188] Content-Type:[application/json] Date:[Wed, 13 Nov 2019 18:16:51 GMT]] Body:0xc0002971c0 ContentLength:188 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc00012ee00 TLS:<nil>}
I1113 18:16:54.225172 30332 dashboard.go:227] http://127.0.0.1:43299/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cache-Control:[no-cache, private] Content-Length:[188] Content-Type:[application/json] Date:[Wed, 13 Nov 2019 18:16:54 GMT]] Body:0xc0003cf940 ContentLength:188 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc000486300 TLS:<nil>}
I1113 18:16:57.412075 30332 dashboard.go:227] http://127.0.0.1:43299/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cache-Control:[no-cache, private] Content-Length:[188] Content-Type:[application/json] Date:[Wed, 13 Nov 2019 18:16:57 GMT]] Body:0xc00035d9c0 ContentLength:188 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc0004aa900 TLS:<nil>}
I1113 18:17:02.101728 30332 dashboard.go:227] http://127.0.0.1:43299/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cache-Control:[no-cache, private] Content-Length:[188] Content-Type:[application/json] Date:[Wed, 13 Nov 2019 18:17:02 GMT]] Body:0xc0003cfa80 ContentLength:188 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc0004aaa00 TLS:<nil>}
I1113 18:17:11.123045 30332 dashboard.go:227] http://127.0.0.1:43299/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:And if I hit the URL which is giving 503, I get: {
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "no endpoints available for service \"http:kubernetes-dashboard:\"",
"reason": "ServiceUnavailable",
"code": 503
} |
|
Thanks for the update @sebinsua Based on what you've shared, I now suspect that part of the issue may be that we are attempting to check the URL before checking if the pod is actually running. Why the dashboard service isn't healthy though, we still need to investigate. Do you mind helping us root cause this? Once you see the dashboard hanging at "Verifying proxy health ...", can you get the output of and share it with us?
Depending on what you share, I believe that part of the solution may be to insert a new health check that blocks until the pod is in 'Running' state, by calling minikube/cmd/minikube/cmd/dashboard.go Line 112 in 72016f1 What the |
|
these results were taking after the bug was replicated, in a separate SSH terminal. At the same time the proxy message was hanging in the other SSH window this this outputted eventually in the main window: |
|
Interesting. Do you mind adding the output of 'kubectl describe node' as
well? Thanks!
…On Wed, Nov 13, 2019, 6:19 PM GarretSidzaka ***@***.***> wrote:
output.txt
<https://github.com/kubernetes/minikube/files/3844296/output.txt>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#5815?email_source=notifications&email_token=AAAYYMA76IJ6FPNETGAYGETQTSYTTA5CNFSM4JIBWDFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEALAAA#issuecomment-553693184>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAYYMBP527WNFL3THBZR4DQTSYTTANCNFSM4JIBWDFA>
.
|
sudo kubectl describe node MemoryPressure False Thu, 14 Nov 2019 03:36:59 +0000 Wed, 06 Nov 2019 00:34:43 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available kube-system coredns-5644d7b6d9-hpm6t 100m (2%) 0 (0%) 70Mi (0%) 170Mi (2%) 8d cpu 755m (18%) 0 (0%) |
|
hi 😍 |
|
have this issue solved？i got the same problem. |
|
thoughts on this issue? |
|
/remove-lifecycle stale |
k8s-ci-robot
removed
the
lifecycle/stale
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
lifecycle/rotten
medyagh
removed
the
lifecycle/rotten
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
k8s-ci-robot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
k8s-ci-robot
added
lifecycle/rotten
sharifelgamal
added
lifecycle/frozen
sharifelgamal
changed the title
|
This is still an issue we'd like to resolve. Help wanted! |
|
In my case canceling (ctrl+C) and executing the same command ( |
|
Had the same problem. Tried turning off my VPN, didn’t help. Tried |
That worked for me. |
|
I had this same experience in macOS Big Sur 11.5.2 with docker desktop 3.3.3. In the same terminal session I installed minikube using default curl command copied from https://minikube.sigs.k8s.io/docs/start/:
I then ran:
I then run:
Then the process appeared to hang at
Then ctrl-c, open a new terminal:
And browser window opens as expected with dashboard showing. |
|
I met the same problem. Then it is stuck. Even if I stop it and execute it from another terminal. It didn't work |
|
Ping |
|
Nearly ditto what @jpshackelford said in #5815 (comment) If I run I installed v1.24.0 on Darwin 11.6 via homebrew on Docker 20.10.8, then |
|
Also experiencing this. Running It hangs for a while, and then produces this final line of output: If I run |
|
I got |
|
Any solution please? Update: My workaround
I have minikube version v1.26.0, If you need to figure out which exactly dashboard image tag you need go to |
|
Any solution please? |
Share my solution: |
|
Running this command in a PowerShell /w privilege worked. |
and others
when i run it always hangs forever and sometimes throws 503
garretsidzaka@$$$$$:~$ sudo minikube dashboard
^C
garretsidzaka@$$$$$:~$
garretsidzaka@$$$$$$:/usr/bin$ minikube version
minikube version: v1.4.0
commit: 7969c25
ubuntu vm 1804
vm-driver=none
The text was updated successfully, but these errors were encountered: