minikube start doesn't stop after download failure due to network issues

[lxzhang000]

I'm testing the minikube network connection issues in China and happened to find there is a potential bug:

After all image fallback addresses failed, minikube didn't stop to the process as expected, and failed later due to lack of valid images.

Steps to reproduce the issue:
os version: macOs catalina 10.15.7
minikube version: v1.16.0 commit: 9f1e482

minikube start --driver=virtualbox --v=1 --alsologtostderr

Full output of failed command:
I0114 22:46:46.233396 10344 cache.go:159] failed to download docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.15-snapshot4, will try fallback image if available: GET https://docker.pkg.github.com/v2/kubernetes/minikube/kicbase/manifests/v0.0.15-snapshot4: UNAUTHORIZED: GitHub Docker Registry needs login
E0114 22:46:46.233427 10344 cache.go:180] Error downloading kic artifacts: failed to download kic base image or any fallback image
I0114 22:46:46.233444 10344 cache.go:185] Successfully downloaded all kic artifacts
I0114 22:46:46.233473 10344 start.go:314] acquiring machines lock for minikube: {Name:mkb987fd896c2845336bd726163c7ebf1c4ecf2e Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I0114 22:46:46.233601 10344 start.go:318] acquired machines lock for "minikube" in 114.554µs
I0114 22:46:46.233634 10344 start.go:90] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.15-snapshot4@sha256:ef1f485b5a1cfa4c989bc05e153f0a8525968ec999e242efff871cbb3

I checked the master code:

In pkg/minikube/node/cache.go, in function "beginDownloadKicBaseImage", if all fallback addresses fails, it returns:

fmt.Errorf("failed to download kic base image or any fallback image")

I think it should wrap the type of the error here, so the function "waitDownloadKicBaseImage" could go to the desired branch and exit.

[sharifelgamal]

Yeah, I agree we could do better here and fail more gracefully. Help wanted!

[bartekzurawski]

Hi,

Can I take this one as my first issue?

[vishjain]

/assign

[andriyDev]

@vishjain Are you still working on this?

[vishjain]

@vishjain Are you still working on this?

Yes, I'm working on this one.

[andriyDev]

Awesome!

[medyagh]

I'm testing the minikube network connection issues in China and happened to find there is a potential bug:

After all image fallback addresses failed, minikube didn't stop to the process as expected, and failed later due to lack of valid images.

Steps to reproduce the issue:
os version: macOs catalina 10.15.7
minikube version: v1.16.0 commit: 9f1e482

minikube start --driver=virtualbox --v=1 --alsologtostderr

Full output of failed command:
I0114 22:46:46.233396 10344 cache.go:159] failed to download docker.pkg.github.com/kubernetes/minikube/kicbase:v0.0.15-snapshot4, will try fallback image if available: GET https://docker.pkg.github.com/v2/kubernetes/minikube/kicbase/manifests/v0.0.15-snapshot4: UNAUTHORIZED: GitHub Docker Registry needs login
E0114 22:46:46.233427 10344 cache.go:180] Error downloading kic artifacts: failed to download kic base image or any fallback image I0114 22:46:46.233444 10344 cache.go:185] Successfully downloaded all kic artifacts
I0114 22:46:46.233473 10344 start.go:314] acquiring machines lock for minikube: {Name:mkb987fd896c2845336bd726163c7ebf1c4ecf2e Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I0114 22:46:46.233601 10344 start.go:318] acquired machines lock for "minikube" in 114.554µs
I0114 22:46:46.233634 10344 start.go:90] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:registry.cn-hangzhou.aliyuncs.com/google_containers/kicbase:v0.0.15-snapshot4@sha256:ef1f485b5a1cfa4c989bc05e153f0a8525968ec999e242efff871cbb3

I checked the master code:

In pkg/minikube/node/cache.go, in function "beginDownloadKicBaseImage", if all fallback addresses fails, it returns:

fmt.Errorf("failed to download kic base image or any fallback image")

I think it should wrap the type of the error here, so the function "waitDownloadKicBaseImage" could go to the desired branch and exit.

@lxzhang000 this seems to be a very old minikube version ! and also this is virtualbox driver this should have nothing to do with KIC or docker driver

I am not convinceced what this issue is purposing to do

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten