Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dashboard on none: unexpected response code: 503 #4352

Closed
irrgit opened this issue May 24, 2019 · 10 comments
Closed

dashboard on none: unexpected response code: 503 #4352

irrgit opened this issue May 24, 2019 · 10 comments
Labels
co/dashboard dashboard related issues co/none-driver help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@irrgit
Copy link

irrgit commented May 24, 2019

I cannot get minikube dashboard to work. Possible related issue #2380.
Minikube was started with these options:

sudo -E  minikube start --vm-driver=none --extra-config=kubelet.resolv-conf=/run/systemd/resolve/resolv.conf --extra-config=kubelet.cgroup-driver=systemd
kube@kube:~$ minikube version
minikube version: v1.1.0
kube@kube:~$ kubectl get pods -n kube-system kubernetes-dashboard-5f7b999d65-jmv67 
NAME                                    READY   STATUS    RESTARTS   AGE
kubernetes-dashboard-5f7b999d65-jmv67   1/1     Running   1          15d
kube@kube:~$ kubectl get svc -n kube-system kubernetes-dashboard 
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes-dashboard   ClusterIP   10.97.218.152   <none>        80/TCP    15d
kube@kube:~$ kubectl get deployments -n kube-system kubernetes-dashboard 
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
kubernetes-dashboard   1/1     1            1           15d
kube@kube:~$ sudo -E minikube dashboard --logtostderr --v=2
I0524 19:19:50.398017   11405 none.go:231] checking for running kubelet ...
I0524 19:19:50.398044   11405 exec_runner.go:39] Run: systemctl is-active --quiet service kubelet
* Enabling dashboard ...
I0524 19:19:50.404894   11405 none.go:231] checking for running kubelet ...
I0524 19:19:50.404906   11405 exec_runner.go:39] Run: systemctl is-active --quiet service kubelet
* Verifying dashboard health ...
I0524 19:19:50.448181   11405 service.go:225] Found service: &Service{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kubernetes-dashboard,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/services/kubernetes-dashboard,UID:75c9cad3-7259-11e9-9cee-9829a6fadae6,ResourceVersion:1674809,Generation:0,CreationTimestamp:2019-05-09 12:53:36 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{addonmanager.kubernetes.io/mode: Reconcile,app: kubernetes-dashboard,kubernetes.io/minikube-addons: dashboard,kubernetes.io/minikube-addons-endpoint: dashboard,},Annotations:map[string]string{kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","app":"kubernetes-dashboard","kubernetes.io/minikube-addons":"dashboard","kubernetes.io/minikube-addons-endpoint":"dashboard"},"name":"kubernetes-dashboard","namespace":"kube-system"},"spec":{"ports":[{"port":80,"targetPort":9090}],"selector":{"app":"kubernetes-dashboard"}}}
,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:ServiceSpec{Ports:[{ TCP 80 {0 9090 } 0}],Selector:map[string]string{app: kubernetes-dashboard,},ClusterIP:10.97.218.152,Type:ClusterIP,ExternalIPs:[],SessionAffinity:None,LoadBalancerIP:,LoadBalancerSourceRanges:[],ExternalName:,ExternalTrafficPolicy:,HealthCheckNodePort:0,PublishNotReadyAddresses:false,SessionAffinityConfig:nil,},Status:ServiceStatus{LoadBalancer:LoadBalancerStatus{Ingress:[],},},}
* Launching proxy ...
I0524 19:19:50.448629   11405 dashboard.go:139] Executing: /usr/local/bin/kubectl [/usr/local/bin/kubectl --context minikube proxy --port=0]
I0524 19:19:50.449049   11405 dashboard.go:144] Waiting for kubectl to output host:port ...
I0524 19:19:50.506108   11405 dashboard.go:162] proxy stdout: Starting to serve on 127.0.0.1:40167
* Verifying proxy health ...
I0524 19:19:50.524408   11405 dashboard.go:199] http://127.0.0.1:40167/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[182] Content-Type:[application/json] Date:[Fri, 24 May 2019 19:19:50 GMT]] Body:0xc0004b9600 ContentLength:182 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc000406400 TLS:<nil>}
I0524 19:19:50.524546   11405 utils.go:125] error: Temporary Error: unexpected response code: 503 - sleeping 1s
I0524 19:19:51.524706   11405 utils.go:114] retry loop 1
I0524 19:19:51.528851   11405 dashboard.go:199] http://127.0.0.1:40167/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[182] Content-Type:[application/json] Date:[Fri, 24 May 2019 19:19:51 GMT]] Body:0xc0002e4680 ContentLength:182 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc00011d600 TLS:<nil>}
I0524 19:19:51.528977   11405 utils.go:125] error: Temporary Error: unexpected response code: 503 - sleeping 1s
I0524 19:19:52.529131   11405 utils.go:114] retry loop 2
I0524 19:19:52.534004   11405 dashboard.go:199] http://127.0.0.1:40167/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[182] Content-Type:[application/json] Date:[Fri, 24 May 2019 19:19:52 GMT]] Body:0xc0002e4740 ContentLength:182 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc00011dd00 TLS:<nil>}
I0524 19:19:52.534103   11405 utils.go:125] error: Temporary Error: unexpected response code: 503 - sleeping 1s
I0524 19:19:53.534307   11405 utils.go:114] retry loop 3
I0524 19:19:53.539371   11405 dashboard.go:199] http://127.0.0.1:40167/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ response: <nil> &{Status:503 Service Unavailable StatusCode:503 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[182] Content-Type:[application/json] Date:[Fri, 24 May 2019 19:19:53 GMT]] Body:0xc0004b9900 ContentLength:182 TransferEncoding:[] Close:false Uncompressed:false Trailer:map[] Request:0xc00011de00 TLS:<nil>}
I0524 19:19:53.539473   11405 utils.go:125] error: Temporary Error: unexpected response code: 503 - sleeping 1s
kube@kube:~$ sudo -E minikube dashboard --url
* Enabling dashboard ...
* Verifying dashboard health ...
* Launching proxy ...
* Verifying proxy health ...
X http://127.0.0.1:34183/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ is not responding properly: Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
...
kube@kube:~$ 

OS: CentOS Linux release 7.6.1810 (Core)

@tstromberg tstromberg changed the title Minikube dashboard 503 error when trying to bring it up, --vm-driver=none dashboard on none: unexpected response code: 503 May 24, 2019
@tstromberg tstromberg added co/dashboard dashboard related issues co/none-driver help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/backlog Higher priority than priority/awaiting-more-evidence. labels May 24, 2019
@winds6206
Copy link

winds6206 commented May 27, 2019

Hi

I am encounter the same problem with --vm-driver=none

CentOS 7 minimal install

root@minikube[/tmp/Minikube]# minikube dashboard
* Enabling dashboard ...
* Verifying dashboard health ...
* Launching proxy ...
* Verifying proxy health ...
X http://127.0.0.1:33959/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ is not responding properly: Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
...
root@minikube[/tmp/Minikube]# minikube version
minikube version: v1.1.0

@lianghuiyuan
Copy link

+1
Ubuntu 18.04
$ minikube version
minikube version: v1.1.0

$ sudo minikube start --image-mirror-country=cn
  ......

$ sudo minikube dashboard
🔌  Enabling dashboard ...
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
💣  http://127.0.0.1:40339/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ is not responding properly: Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
......

@NovikovEvgeny
Copy link

NovikovEvgeny commented May 27, 2019

I had the same issue

But managed to make it work using --docker-env parameters and setting HTTP_PROXY, HTTPS_PROXY and NO_PROXY env variables as proposed here

minikube stop
minikube delete
rm -rf  ~/.minikube
minikube start --vm-driver=none --docker-env HTTP_PROXY=http://myproxy.com:8080 --docker-env HTTPS_PROXY=http://myproxy.com:8080 --docker-env NO_PROXY=127.0.0.1

Ubuntu 16.04
minikube 1.1.0

@zosocanuck
Copy link

zosocanuck commented May 31, 2019

Also having the same issue on CentOS 7.6 with minikube 1.1.0:

minikube start --vm-driver=none --extra-config=apiserver.authorization-mode=RBAC

minikube dashboard
🔌  Enabling dashboard ...
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
💣  http://127.0.0.1:36410/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ is not responding properly: Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503
Temporary Error: unexpected response code: 503

UPDATE
Managed to solve this by following suggest in this link

kubectl create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

@irrgit
Copy link
Author

irrgit commented Jun 5, 2019

Here are the logs for the pod of the dashboard failing. I guess after trying to enable it too many times, it has entered a failed state now.

kube@kube:~$ kubectl logs -f kubernetes-dashboard-79dd6bfc48-rbll6 -n kube-system
2019/06/05 20:20:33 Starting overwatch
2019/06/05 20:20:33 Using in-cluster config to connect to apiserver
2019/06/05 20:20:33 Using service account token for csrf signing
2019/06/05 20:20:33 Successful initial request to the apiserver, version: v1.14.2
2019/06/05 20:20:33 Generating JWE encryption key
2019/06/05 20:20:33 New synchronizer has been registered: kubernetes-dashboard-key-holder-kube-system. Starting
2019/06/05 20:20:33 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kube-system
2019/06/05 20:20:33 Synchronizer kubernetes-dashboard-key-holder-kube-system exited with error: unexpected object: &Secret{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Data:map[string][]byte{},Type:,StringData:map[string]string{},}
2019/06/05 20:20:33 Storing encryption key in a secret
panic: secrets is forbidden: User "system:serviceaccount:kube-system:default" cannot create resource "secrets" in API group "" in the namespace "kube-system"

goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/auth/jwe.(*rsaKeyHolder).init(0xc42040a2e0)
	/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/auth/jwe/keyholder.go:131 +0x35e
github.com/kubernetes/dashboard/src/app/backend/auth/jwe.NewRSAKeyHolder(0x1367500, 0xc42046e180, 0xc42046e180, 0x1213a6e)
	/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/auth/jwe/keyholder.go:170 +0x64
main.initAuthManager(0x13663e0, 0xc420520120, 0xc4204dbcd8, 0x1)
	/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/dashboard.go:185 +0x12c
main.main()
	/home/travis/build/kubernetes/dashboard/.tmp/backend/src/github.com/kubernetes/dashboard/src/app/backend/dashboard.go:103 +0x26b

@anibalardid
Copy link

I fixed issue deleting .kube and .minikube directories from my home , maybe it happens because I previously testing using virtualbox let some problems there.

try it ;)

@renhe134880
Copy link

I fixed issue .
just:
minikube stop
minikube start

@cc189
Copy link

cc189 commented Aug 15, 2019

resolved follow Minikube RBAC Quick Start
https://gist.github.com/F21/08bfc2e3592bed1e931ec40b8d2ab6f5

@tstromberg
Copy link
Contributor

tstromberg commented Sep 20, 2019

@irrgit - I believe this issue is now addressed by minikube v1.4, as it upgrades the dashboard to 2.0 and uses a different RBAC configuration. If you still see this issue with minikube v1.4 or higher, please reopen this issue by commenting with /reopen

Thank you for reporting this issue!

@GarretSidzaka
Copy link

GarretSidzaka commented Oct 31, 2019

/reopen
when i run it sometimes hangs forever and sometimes throws 503

garretsidzaka@cloudstack:/usr/bin$ minikube version
minikube version: v1.4.0
commit: 7969c25
ubuntu vm 1804
vm-driver=none

garretsidzaka@cloudstack:~$ sudo minikube dashboard

  • Verifying dashboard health ...
  • Launching proxy ...
  • Verifying proxy health ...

^C
garretsidzaka@cloudstack:~$

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/dashboard dashboard related issues co/none-driver help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Development

No branches or pull requests

10 participants