Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed pulling image: Get https://k8s.gcr.io/v2/: Client.Timeout exceeded while awaiting headers #3115

Closed
WhoisZihan opened this issue Sep 5, 2018 · 3 comments
Assignees
Labels
cause/firewall-or-proxy When firewalls or proxies seem to be interfering ev/hung-start kind/bug Categorizes issue or PR as related to a bug. triage/unresolved Indicates an issue that can not or will not be resolved.

Comments

@WhoisZihan
Copy link

WhoisZihan commented Sep 5, 2018

BUG REPOR

I know this is the same as #2018, but with latest version it still doesn't work and I can only start with --bootstrapper=localkube, other bootstrappers never work.

Minikube version : v0.28.2

  • OS : Ubuntu 16.04.5 LTS (Xenial Xerus)
  • VM Driver : kvm2 (But virtualbox does not work either)
  • ISO version : minikube-v0.28.1.iso, boot2docker.iso
  • Install tools:
    • kubectl version: v1.11.2
    • kubeadm version: v1.10.0
    • kubelet version: v1.10.0
  • Others:
    The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):
OS:
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

VM driver:
    "DriverName": "kvm2",

ISO version
        "Boot2DockerURL": "file:///home/fma/.minikube/cache/iso/minikube-v0.28.1.iso",
        "ISO": "/home/fma/.minikube/machines/minikube/boot2docker.iso",

What happened:
The command minikube start --vm-driver kvm2 --bootstrapper=kubeadm will get stuck at Starting cluster components.... By checking minikube logs, I find it gets refused when trying to connect to 127.0.0.1:8443: getsockopt: connection refused

What you expected to happen:
It starts normally.

How to reproduce it (as minimally and precisely as possible):

  1. Download minikube v0.28.2 and kubelet v1.11.2
  2. type minikube start --vm-driver kvm2 --bootstrapper=kubeadm
  3. Waiting for it to get stuck

Output of minikube logs (if applicable):
Here is part of it

Sep 05 12:24:15 minikube kubelet[2686]: E0905 12:24:15.468704 2686 pod_workers.go:186] Error syncing pod 31cf0ccbee286239d451edb6fb511513 ("kube-scheduler-minikube_kube-system(31cf0ccbee286239d451edb6fb511513)"), skipping: failed to "CreatePodSandbox" for "kube-scheduler-minikube_kube-system(31cf0ccbee286239d451edb6fb511513)" with CreatePodSandboxError: "CreatePodSandbox for pod "kube-scheduler-minikube_kube-system(31cf0ccbee286239d451edb6fb511513)" failed: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause-amd64:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

Sep 05 12:24:15 minikube kubelet[2686]: E0905 12:24:15.608771 2686 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Service: Get https://localhost:8443/api/v1/services?limit=500&resourceVersion=0: dial tcp 127.0.0.1:8443: getsockopt: connection refused

Anything else do we need to know:

  1. I can start with --bootstrapper=localkube but not with kubeadm, will it be the issue of RBAC?
  2. I can do ping k8s.io, but I can NOT ping k8s.gcr.io, will this cause start failure too?
  3. I also tried specifying --kubernetes-version=v1.9.4, but not working either.
@tstromberg tstromberg changed the title Using kubeadm as bootstrapper still gets stuck at starting cluster components start --bootstrapper=kubeadm: Hung at: Starting cluster components... (connection refused?) Sep 18, 2018
@tstromberg tstromberg added os/linux co/kvm2-driver KVM2 driver related issues ev/hung-start co/kubeadm Issues relating to kubeadm kind/bug Categorizes issue or PR as related to a bug. labels Sep 18, 2018
@tstromberg tstromberg changed the title start --bootstrapper=kubeadm: Hung at: Starting cluster components... (connection refused?) failed pulling image: Get https://k8s.gcr.io/v2/: Client.Timeout exceeded while awaiting headers Sep 19, 2018
@tstromberg
Copy link
Contributor

Do you mind attaching a full log file? There seems to be two main issues here:

  • Unable to deploy kube-scheduler because it can't access k8s.gcr.io.
  • The apiserver isn't running (:8443 connect refused)

The second can be many things. The first is indicative of a connectivity failure to GCR: is this minikube instance running within China, by any chance?

We should find a way to install minikube so that it doesn't require external connectivity.

@tstromberg tstromberg added failed/local-networking startup failures due to networking issues cause/poor-internet Failed due to internet access issues cause/firewall-or-proxy When firewalls or proxies seem to be interfering and removed failed/local-networking startup failures due to networking issues labels Sep 19, 2018
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 20, 2018
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 19, 2019
@tstromberg tstromberg added triage/obsolete Bugs that no longer occur in the latest stable release and removed more-info-needed labels Jan 22, 2019
@tstromberg tstromberg added triage/unresolved Indicates an issue that can not or will not be resolved. and removed cause/poor-internet Failed due to internet access issues co/kubeadm Issues relating to kubeadm co/kvm2-driver KVM2 driver related issues lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. os/linux triage/obsolete Bugs that no longer occur in the latest stable release labels Jan 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cause/firewall-or-proxy When firewalls or proxies seem to be interfering ev/hung-start kind/bug Categorizes issue or PR as related to a bug. triage/unresolved Indicates an issue that can not or will not be resolved.
Projects
None yet
Development

No branches or pull requests

4 participants