Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xhyve: lookup k8s.gcr.io ... read: connection refused #3726

Closed
3 tasks done
rsahu001 opened this issue Feb 21, 2019 · 13 comments
Closed
3 tasks done

xhyve: lookup k8s.gcr.io ... read: connection refused #3726

rsahu001 opened this issue Feb 21, 2019 · 13 comments
Labels
area/dns DNS issues co/xhyve kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. r/2019q2 Issue was last reviewed 2019q2

Comments

@rsahu001
Copy link

rsahu001 commented Feb 21, 2019

If this is a bug report, please include:

  • How to replicate the error, including the exact command-lines used.
  • The full output of the command that failed
  • The operating system name and version used
    Rajs-MacBook-Pro:~ rajsah$ sw_vers
    ProductName: Mac OS X
    ProductVersion: 10.13.6
    BuildVersion: 17G4015

Rajs-MacBook-Pro:~ rajsah$ sudo chown -R $(whoami) /usr/local/etc/bash_completion.d /usr/local/share/doc /usr/local/share/man /usr/local/share/man/man1 /usr/local/share/zsh /usr/local/share/zsh/site-functions
Rajs-MacBook-Pro:~ rajsah$ brew install kubectl
Updating Homebrew...
==> Downloading https://homebrew.bintray.com/bottles/kubernetes-cli-1.12.2.high_
######################################################################## 100.0%
==> Pouring kubernetes-cli-1.12.2.high_sierra.bottle.tar.gz
Error: The brew link step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/kubectl
Target /usr/local/bin/kubectl
already exists. You may want to remove it:
rm '/usr/local/bin/kubectl'

To force the link and overwrite all conflicting files:
brew link --overwrite kubernetes-cli

To list all files that would be deleted:
brew link --overwrite --dry-run kubernetes-cli

Possible conflicting files are:
/usr/local/bin/kubectl -> /Applications/Docker.app/Contents/Resources/bin/kubectl
==> Caveats
Bash completion has been installed to:
/usr/local/etc/bash_completion.d

zsh completions have been installed to:
/usr/local/share/zsh/site-functions
==> Summary
🍺 /usr/local/Cellar/kubernetes-cli/1.12.2: 208 files, 50.6MB
Rajs-MacBook-Pro:~ rajsah$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Rajs-MacBook-Pro:~ rajsah$ brew cask install minikube
Updating Homebrew...
==> Satisfying dependencies
All Formula dependencies satisfied.
==> Downloading https://storage.googleapis.com/minikube/releases/v0.34.1/minikub
######################################################################## 100.0%
==> Verifying SHA-256 checksum for Cask 'minikube'.
==> Installing Cask minikube
==> Linking Binary 'minikube-darwin-amd64' to '/usr/local/bin/minikube'.
🍺 minikube was successfully installed!
Rajs-MacBook-Pro:~ rajsah$ brew install docker-machine-driver-xhyve
==> Installing dependencies for docker-machine-driver-xhyve: docker-machine
==> Installing docker-machine-driver-xhyve dependency: docker-machine
==> Downloading https://homebrew.bintray.com/bottles/docker-machine-0.16.0.high_
######################################################################## 100.0%
==> Pouring docker-machine-0.16.0.high_sierra.bottle.tar.gz
Error: The brew link step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/docker-machine
Target /usr/local/bin/docker-machine
already exists. You may want to remove it:
rm '/usr/local/bin/docker-machine'

To force the link and overwrite all conflicting files:
brew link --overwrite docker-machine

To list all files that would be deleted:
brew link --overwrite --dry-run docker-machine

Possible conflicting files are:
/usr/local/bin/docker-machine -> /Applications/Docker.app/Contents/Resources/bin/docker-machine
==> Caveats
Bash completion has been installed to:
/usr/local/etc/bash_completion.d

zsh completions have been installed to:
/usr/local/share/zsh/site-functions

To have launchd start docker-machine now and restart at login:
brew services start docker-machine
Or, if you don't want/need a background service you can just run:
docker-machine start
==> Summary
🍺 /usr/local/Cellar/docker-machine/0.16.0: 11 files, 35.7MB
==> Installing docker-machine-driver-xhyve
==> Downloading https://homebrew.bintray.com/bottles/docker-machine-driver-xhyve
######################################################################## 100.0%
==> Pouring docker-machine-driver-xhyve-0.3.3_1.high_sierra.bottle.tar.gz
==> Caveats
This driver requires superuser privileges to access the hypervisor. To
enable, execute
sudo chown root:wheel /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
==> Summary
🍺 /usr/local/Cellar/docker-machine-driver-xhyve/0.3.3_1: 7 files, 9MB
==> Caveats
==> docker-machine
Bash completion has been installed to:
/usr/local/etc/bash_completion.d

zsh completions have been installed to:
/usr/local/share/zsh/site-functions

To have launchd start docker-machine now and restart at login:
brew services start docker-machine
Or, if you don't want/need a background service you can just run:
docker-machine start
==> docker-machine-driver-xhyve
This driver requires superuser privileges to access the hypervisor. To
enable, execute
sudo chown root:wheel /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
Rajs-MacBook-Pro:~ rajsah$ sudo chown root:wheel /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
Rajs-MacBook-Pro:~ rajsah$ sudo chmod u+s /usr/local/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
Rajs-MacBook-Pro:~ rajsah$ minikube start --vm-driver=xhyve
😄 minikube v0.34.1 on darwin (amd64)
⚠️ The xhyve driver is deprecated and support for it will be removed in a future release.
Please consider switching to the hyperkit driver, which is intended to replace the xhyve driver.
See https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#hyperkit-driver for more information.
To disable this message, run [minikube config set WantShowDriverDeprecationNotification false]
🔥 Creating xhyve VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
💿 Downloading Minikube ISO ...
184.30 MB / 184.30 MB [============================================] 100.00% 0s
📶 "minikube" IP address is 192.168.64.2
🐳 Configuring Docker as the container runtime ...
✨ Preparing Kubernetes environment ...
💾 Downloading kubeadm v1.13.3
💾 Downloading kubelet v1.13.3
🚜 Pulling images required by Kubernetes v1.13.3 ...
❌ Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout:
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.3": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:58821->192.168.64.1:53: read: connection refused
, error: exit status 1
: Process exited with status 1
🚀 Launching Kubernetes v1.13.3 using kubeadm ...
💣 Error starting cluster: kubeadm init:
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI

[init] Using Kubernetes version: v1.13.3
[preflight] Running pre-flight checks
[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:34174->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:59884->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:58617->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:42575->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:35733->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:37210->192.168.64.1:53: read: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.2:57388->192.168.64.1:53: read: connection refused
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...

: Process exited with status 1

😿 Sorry that minikube crashed. If this was unexpected, we would love to hear from you:
👉 https://github.com/kubernetes/minikube/issues/new
Rajs-MacBook-Pro:~ rajsah$

@tstromberg
Copy link
Contributor

xhyve is deprecated, and will be removed from the next release. I suggest trying the virtualbox or hyperkit drivers instead.

That said, this looks like an issue talking to the DNS service that xhyve starts up -- which I believe is dnsmasq based.

@tstromberg tstromberg added co/xhyve area/dns DNS issues kind/bug Categorizes issue or PR as related to a bug. labels Feb 22, 2019
@tstromberg tstromberg changed the title minikube crashed (while starting minikube start --vm-driver=xhyve) xhyve: lookup k8s.gcr.io ... read: connection refused Feb 22, 2019
@tstromberg tstromberg added the priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. label Feb 22, 2019
@yinzara
Copy link
Contributor

yinzara commented Mar 7, 2019

I am not using the Xhyve driver. I'm using the Hyperkit driver and I still get the following:

minikube start
😄  minikube v0.34.1 on darwin (amd64)
🔥  Creating hyperkit VM (CPUs=7, Memory=8192MB, Disk=20000MB) ...
📶  "minikube" IP address is 192.168.64.13
🐳  Configuring Docker as the container runtime ...
✨  Preparing Kubernetes environment ...
🚜  Pulling images required by Kubernetes v1.13.3 ...
❌  Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout: 
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.13.3": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.13:40683->192.168.64.1:53: read: connection refused
, error: exit status 1
: Process exited with status 1
🚀  Launching Kubernetes v1.13.3 using kubeadm ... 
💣  Error starting cluster: kubeadm init: 
sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data-minikube --ignore-preflight-errors=Port-10250 --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap --ignore-preflight-errors=CRI 

[init] Using Kubernetes version: v1.13.3
[preflight] Running pre-flight checks
	[WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
	[WARNING Swap]: running with swap on is not supported. Please disable swap
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.13:60019->192.168.64.1:53: read: connection refused
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.13:58962->192.168.64.1:53: read: connection refused
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.13:51737->192.168.64.1:53: read: connection refused
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.3: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.13:49418->192.168.64.1:53: read: connection refused
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.13:47727->192.168.64.1:53: read: connection refused
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.13:48391->192.168.64.1:53: read: connection refused
, error: exit status 1
	[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.13:57870->192.168.64.1:53: read: connection refused
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

: Process exited with status 1

@j4k
Copy link

j4k commented Mar 27, 2019

I'm also experiencing this with latest hyperkit and minikube

@tstromberg
Copy link
Contributor

tstromberg commented Mar 27, 2019

If you are running into this with hyperkit:

  • Install the latest minikube hyperkit driver:

curl -LO https://storage.googleapis.com/minikube/releases/latest/docker-machine-driver-hyperkit && sudo install -o root -g wheel -m 4755 docker-machine-driver-hyperkit /usr/local/bin/

  • Run minikube delete to remove the old busted VM
  • If it still doesn't work, open a new issue and I will help you track it down. This issue is specific to xhyve.

@bertramn
Copy link

Same here, for some reason minikube cannot pull images:

$ minikube start --vm-driver hyperkit
😄  minikube v1.0.0 on darwin (amd64)
🤹  Downloading Kubernetes v1.14.0 images in the background ...
🔥  Creating hyperkit VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
📶  "minikube" IP address is 192.168.64.6
🐳  Configuring Docker as the container runtime ...
🐳  Version of container runtime is 18.06.2-ce
⌛  Waiting for image downloads to complete ...
✨  Preparing Kubernetes environment ...
🚜  Pulling images required by Kubernetes v1.14.0 ...
❌  Unable to pull images, which may be OK: running cmd: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml: command failed: sudo kubeadm config images pull --config /var/lib/kubeadm.yaml
stdout: 
stderr: failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.0": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.6:48849->192.168.64.1:53: read: connection refused
, error: exit status 1
: Process exited with status 1
🚀  Launching Kubernetes v1.14.0 using kubeadm ... 
⌛  Waiting for pods: apiserver

@datajerk
Copy link

I only have this problem when running Cisco VPN. If I shutdown the VPN, no problems. This has been an issue for a while now.

@tstromberg tstromberg added the r/2019q2 Issue was last reviewed 2019q2 label May 22, 2019
@LennyCastaneda
Copy link

I'm getting a similar error when running: minikube start --vm-driver=hyperkit

failed to pull image "k8s.gcr.io/kube-apiserver:v1.14.3": output: Error response from daemon: Get https://k8s.gcr.io/v2/: dial tcp: lookup k8s.gcr.io on 192.168.64.1:53: read udp 192.168.64.6:46095->192.168.64.1:53: read: connection refused

@medyagh
Copy link
Member

medyagh commented Jun 19, 2019

@LennyCastaneda are you also using a vpn ?

@LennyCastaneda
Copy link

LennyCastaneda commented Jun 19, 2019 via email

@medyagh
Copy link
Member

medyagh commented Jun 19, 2019

@LennyCastaneda
I suspect the proxy is not allowing access to https://k8s.gcr.io/
Do you mind pasting the the output of
curl -vvv https://k8s.gcr.io/v2/

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 17, 2019
@tstromberg
Copy link
Contributor

Closing as xhyve support was removed.

@ashirbadomm
Copy link

@LennyCastaneda
I suspect the proxy is not allowing access to https://k8s.gcr.io/
Do you mind pasting the the output of
curl -vvv https://k8s.gcr.io/v2/

[administrator@localhost ~]$ curl -vvv https://k8s.gcr.io/v2/

  • About to connect() to k8s.gcr.io port 443 (#0)
  • Trying 172.217.194.82...
  • Connected to k8s.gcr.io (172.217.194.82) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
  • Server certificate:
  • subject: CN=*.gcr.io,O=Google LLC,L=Mountain View,ST=California,C=US
  • start date: Dec 03 14:55:05 2019 GMT
  • expire date: Feb 25 14:55:05 2020 GMT
  • common name: *.gcr.io
  • issuer: CN=GTS CA 1O1,O=Google Trust Services,C=US

GET /v2/ HTTP/1.1
User-Agent: curl/7.29.0
Host: k8s.gcr.io
Accept: /

< HTTP/1.1 401 Unauthorized
< Docker-Distribution-API-Version: registry/2.0
< WWW-Authenticate: Bearer realm="https://k8s.gcr.io/v2/token",service="k8s.gcr.io"
< Content-Type: application/json
< Date: Mon, 30 Dec 2019 05:20:57 GMT
< Server: Docker Registry
< Cache-Control: private
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< Alt-Svc: quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000
< Accept-Ranges: none
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
<

  • Connection #0 to host k8s.gcr.io left intact
    {"errors":[{"code":"UNAUTHORIZED","message":"Unauthorized access."}]}[administrator@localhost ~]$

Done, what does this mean now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dns DNS issues co/xhyve kind/bug Categorizes issue or PR as related to a bug. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/awaiting-more-evidence Lowest priority. Possibly useful, but not yet enough support to actually get it done. r/2019q2 Issue was last reviewed 2019q2
Projects
None yet
Development

No branches or pull requests