Skaffold checking cache hangs due to network

[dfang]

skaffold build -v debug
INFO[0000] Skaffold &{Version:v1.15.0 ConfigVersion:skaffold/v2beta8 GitVersion: GitCommit:c4dce04b0d692dce61051d14cf4a2c04c8c688bd GitTreeState:dirty BuildDate:2020-10-02T16:35:42Z GoVersion:go1.15.2 Compiler:gc Platform:darwin/amd64}
DEBU[0000] Update check not enabled, skipping.
INFO[0000] Loaded Skaffold defaults from "/Users/mj/.skaffold/config"
DEBU[0000] could not parse date ""
INFO[0000] Using kubectl context: docker-desktop
DEBU[0000] Using builder: local
DEBU[0000] Running command: [/usr/local/bin/minikube profile list -o json]
DEBU[0000] Command output: [{"invalid":[],"valid":[{"Name":"minikube","Status":"","Config":{"Name":"minikube","KeepContext":false,"EmbedCerts":false,"MinikubeISO":"","KicBaseImage":"kicbase/stable:v0.0.12-snapshot3@sha256:1d687ba53e19dbe5fafe4cc18aa07f269ecc4b7b622f2251b5bf569ddb474e9b","Memory":1991,"CPUs":2,"DiskSize":20000,"VMDriver":"","Driver":"docker","HyperkitVpnKitSock":"","HyperkitVSockPorts":[],"DockerEnv":null,"ContainerVolumeMounts":null,"InsecureRegistry":null,"RegistryMirror":["https://registry.docker-cn.com"],"HostOnlyCIDR":"192.168.99.1/24","HypervVirtualSwitch":"","HypervUseExternalSwitch":false,"HypervExternalAdapter":"","KVMNetwork":"default","KVMQemuURI":"qemu:///system","KVMGPU":false,"KVMHidden":false,"DockerOpt":null,"DisableDriverMounts":false,"NFSShare":[],"NFSSharesRoot":"/nfsshares","UUID":"","NoVTXCheck":false,"DNSProxy":false,"HostDNSResolver":true,"HostOnlyNicType":"virtio","NatNicType":"virtio","KubernetesConfig":{"KubernetesVersion":"v1.18.3","ClusterName":"minikube","APIServerName":"minikubeCA","APIServerNames":null,"APIServerIPs":null,"DNSDomain":"cluster.local","ContainerRuntime":"docker","CRISocket":"","NetworkPlugin":"","FeatureGates":"","ServiceCIDR":"10.96.0.0/12","ImageRepository":"registry.cn-hangzhou.aliyuncs.com/google_containers","LoadBalancerStartIP":"","LoadBalancerEndIP":"","ExtraOptions":null,"ShouldLoadCachedImages":true,"EnableDefaultCNI":false,"CNI":"","NodeIP":"","NodePort":8443,"NodeName":""},"Nodes":[{"Name":"","IP":"172.17.0.2","Port":8443,"KubernetesVersion":"v1.18.3","ControlPlane":true,"Worker":true},{"Name":"m02","IP":"172.17.0.3","Port":0,"KubernetesVersion":"v1.18.3","ControlPlane":false,"Worker":true},{"Name":"m03","IP":"172.17.0.4","Port":0,"KubernetesVersion":"v1.18.3","ControlPlane":false,"Worker":true}],"Addons":{"ambassador":false,"csi-hostpath-driver":false,"dashboard":true,"default-storageclass":true,"efk":false,"freshpod":false,"gcp-auth":false,"gvisor":false,"helm-tiller":false,"ingress":false,"ingress-dns":false,"istio":false,"istio-provisioner":false,"kubevirt":false,"logviewer":false,"metallb":false,"metrics-server":false,"nvidia-driver-installer":false,"nvidia-gpu-device-plugin":false,"olm":false,"pod-security-policy":false,"registry":false,"registry-aliases":false,"registry-creds":false,"storage-provisioner":true,"storage-provisioner-gluster":false,"volumesnapshots":false},"VerifyComponents":{"apiserver":true,"system_pods":true},"StartHostTimeout":360000000000}}]}]
DEBU[0000] setting Docker user agent to skaffold-v1.15.0
DEBU[0000] push value not present, defaulting to false because localCluster is true
Generating tags...
 - echo -> DEBU[0000] Running command: [git describe --tags --always]
DEBU[0000] Command output: [e343c5a
]
DEBU[0000] Running command: [git status . --porcelain]
DEBU[0000] Command output: [ M k8s/deploy.yaml
 M skaffold.yaml
]
echo:e343c5a-dirty
INFO[0000] Tags generated in 19.006085ms
Checking cache...

Expected behavior

Actual behavior

it hangs here

Information

• Skaffold version:

λ  skaffold version
v1.15.0

λ  which skaffold
/usr/local/bin/skaffold

• Operating system: macOS 10.15.7
• Docker version: 2.4.0.0
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta8
kind: Config
metadata:
  name: echo
build:
  local: {}
  artifacts:
  - image: echo
deploy:
  kubectl:
    manifests:
    - k8s/deploy.yaml

portForward:
- resourceType: deployment
  resourceName: echo
  namespace: default
  port: 8000
  localPort: 8000

Steps to reproduce the behavior

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("hello world"))
	})

	fmt.Println("listenning on :8000")
	if err := http.ListenAndServe(":8000", nil); err != nil {
		log.Fatalln(err)
	}
}

2. skaffold build -v debug

[dfang]

after upgrade to 1.16, the same logs but add these lines.

Checking cache...
ERROR: (gcloud.config.config-helper)
This may be due to network connectivity issues. Please check your network settings, and the status of the service you are trying to reach.
<stucks here> .....

now i know it's network problem, so i set HTTP_PROXY, HTTPS_PROXY to http://127.0.0.1:8118 (my local shadowsocks proxy),

Checking cache...
ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: invalid_grant: Bad Request
Please run:

  $ gcloud auth login

to obtain new credentials, or if you have already logged in with a
different account:

  $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
DEBU[0007] Found dependencies for dockerfile: [{go.mod /src true} {main.go /src true}]
DEBU[0007] Could not import artifact from Docker, building instead (import of missing images disabled)
 - echo: Not found. Building
INFO[0007] Cache check complete in 7.115407394s
Found [docker-desktop] context, using local docker daemon.

clearly the gcloud auth helper still needs to be improved

[dfang]

after unset proxy, i got this error

 λ  skaffold build
Generating tags...
 - echo -> echo:e343c5a-dirty
Checking cache...



ERROR: (gcloud.config.config-helper) There was a problem refreshing your current auth tokens: invalid_grant: Bad Request
Please run:

  $ gcloud auth login

to obtain new credentials, or if you have already logged in with a
different account:

  $ gcloud config set account ACCOUNT

to select an already authenticated account to use.
 - echo: Error checking cache.
getting hash for artifact "echo": getting dependencies for "echo": parsing ONBUILD instructions: retrieving image "gcr.io/distroless/base": Get "https://gcr.io/v2/": dial tcp 74.125.203.82:443: i/o timeout

parsing ONBUILD instructions ? i don't have ONBUILD instructions in my Dockerfile

[tejal29]

@dfang Skaffold tries to parse "dockerfile" to find "OnBUILD" instruction to compute dependencies.
In this case, the error messaging isn't correct. The error is when fetching base image "gcr.io/distroless/base"

[dfang]

@tejal29 Yes, i debugged on that, my Dockerfile use multibuilds, the second FROM is 'gcr.io/distroless/base'.

i'm developing on local at the time, push set to false, it should not check auth

[briandealwis]

In this case, we're checking the base image to see if it has ONBUILD instructions. An image may have an ONBUILD that copies additional files into place, and so it's then necessary to factor those files into the cache-digest computation to see determine if any source files have changed.

You can disable artifact caching checks with --cache-artifacts=false.