Skip to content

Image pulls initiated by K8S are subject to a 2 minute timeout #6300

Open

Description

  • I have tried with the latest version of Docker Desktop
  • I have tried disabling enabled experimental features
  • I have uploaded Diagnostics
  • Diagnostics ID: 141CCEF5-12D2-44D4-AA0D-2A159BFBD826/20220502204139

Expected behavior

Image pulls initiated by K8S should succeed even if they take longer than 2 minutes.

Actual behavior

Image pulls initiated by K8S result in ImagePullBackoff if download does not complete within 2 minutes. The image pull is retried, but the pod will stay in this status forever if retries last longer than 2 minutes.

Information

  • Is it reproducible? Yes
  • Is the problem new? Possibly
  • Did the problem appear with an update? Only noticed in the past couple Docker Desktop versions
  • macOS Version: 10.15.7
  • Intel chip or Apple chip: Intel
  • Docker Desktop Version: 4.7.1 (77678)

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0031: does the Docker API work?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0001: is the application running?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0017: can a VM be started?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0007: is the backend responding?
[PASS] DD0014: are the backend processes running?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[FAIL] DD0012: is the VM networking working? network checks failed: failed to ping host: exit status 1
[2022-05-02T20:56:49.692594000Z][com.docker.diagnose][I] ipc.NewClient: a06a3fb2-diagnose-network -> <HOME>/Library/Containers/com.docker.docker/Data/diagnosticd.sock diagnosticsd
[common/pkg/diagkit/gather/diagnose.runIsVMNetworkingOK()
[	common/pkg/diagkit/gather/diagnose/network.go:34 +0xdd
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0x4d30320)
[	common/pkg/diagkit/gather/diagnose/test.go:46 +0x43
[common/pkg/diagkit/gather/diagnose.Run.func1(0x4d30320)
[	common/pkg/diagkit/gather/diagnose/run.go:17 +0x5a
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x2?, 0x4d30320)
[	common/pkg/diagkit/gather/diagnose/run.go:140 +0x77
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0x4d30320, 0xc000787730)
[	common/pkg/diagkit/gather/diagnose/run.go:146 +0x36
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0x4?, 0xc000787730)
[	common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkOnce(0x46eca00?, 0xc00035f890)
[	common/pkg/diagkit/gather/diagnose/run.go:135 +0xcc
[common/pkg/diagkit/gather/diagnose.Run(0x4d301a0, 0x46e6020?, {0xc00035fb18, 0x1, 0x1})
[	common/pkg/diagkit/gather/diagnose/run.go:16 +0x1cb
[main.checkCmd({0xc000032050?, 0x6?, 0x4?}, {0x0, 0x0})
[	common/cmd/com.docker.diagnose/main.go:131 +0x105
[main.main()
[	common/cmd/com.docker.diagnose/main.go:97 +0x2a8
[2022-05-02T20:56:49.692706000Z][com.docker.diagnose][I] (d8fd78f5) a06a3fb2-diagnose-network C->S diagnosticsd POST /check-network-connectivity: {"ips":["xxx.xxx.xxx.xxx","yyy.yyy.yyy.yyy"]}
[2022-05-02T20:56:50.208189000Z][com.docker.diagnose][E] (d8fd78f5) a06a3fb2-diagnose-network C<-S 95df0e1e-diagnosticsd POST /check-network-connectivity (515.501828ms): failed to ping host: exit status 1
[common/pkg/diagkit/gather/diagnose.runIsVMNetworkingOK()
[	common/pkg/diagkit/gather/diagnose/network.go:35 +0x15b
[common/pkg/diagkit/gather/diagnose.(*test).GetResult(0x4d30320)
[	common/pkg/diagkit/gather/diagnose/test.go:46 +0x43
[common/pkg/diagkit/gather/diagnose.Run.func1(0x4d30320)
[	common/pkg/diagkit/gather/diagnose/run.go:17 +0x5a
[common/pkg/diagkit/gather/diagnose.walkOnce.func1(0x2?, 0x4d30320)
[	common/pkg/diagkit/gather/diagnose/run.go:140 +0x77
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x1, 0x4d30320, 0xc000787730)
[	common/pkg/diagkit/gather/diagnose/run.go:146 +0x36
[common/pkg/diagkit/gather/diagnose.walkDepthFirst(0x0, 0x4?, 0xc000787730)
[	common/pkg/diagkit/gather/diagnose/run.go:149 +0x73
[common/pkg/diagkit/gather/diagnose.walkOnce(0x46eca00?, 0xc00035f890)
[	common/pkg/diagkit/gather/diagnose/run.go:135 +0xcc
[common/pkg/diagkit/gather/diagnose.Run(0x4d301a0, 0x46e6020?, {0xc00035fb18, 0x1, 0x1})
[	common/pkg/diagkit/gather/diagnose/run.go:16 +0x1cb
[main.checkCmd({0xc000032050?, 0x6?, 0x4?}, {0x0, 0x0})
[	common/cmd/com.docker.diagnose/main.go:131 +0x105
[main.main()
[	common/cmd/com.docker.diagnose/main.go:97 +0x2a8

[PASS] DD0032: do Docker networks overlap with host IPs?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0019: is the com.docker.vmnetd process responding?
[PASS] DD0033: does the host have Internet access?

Please investigate the following 1 issue:

1 : The test: is the VM networking working?
    Failed with: network checks failed: failed to ping host: exit status 1

VM seems to have a network connectivity issue. Please check your host firewall and anti-virus settings in case they are blocking the VM.

Steps to reproduce the behavior

This test pod uses a large image:

apiVersion: v1
kind: Pod
metadata:
  name: splunktest
spec:
  containers:
  - name: splunktest
    image: splunk/splunk
    env:
    - name: SPLUNK_START_ARGS
      value: --accept-license
    - name: SPLUNK_PASSWORD
      value: password
  1. Save the above yaml to a file such as test.yaml
  2. Run kubectl apply -f test.yaml
  3. If the connection is slow enough, the pod will enter ImagePullBackoff state after 2 minutes
  4. Notice that docker pull splunk/splunk will succeed, even if it takes longer than 2 minutes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions