Preload images into docker volume #6720

priyawadhwa · 2020-02-20T23:05:42Z

This PR adds support for tarring up docker images, storing them in GCS, and then creating a volume with them to mount into kic

To download the preloaded images run:

(60 s) minikube start --vm-driver docker --download-only
(41 s) minikube start --vm-driver docker
(5 s) minikube delete
(30 s) minikube start --vm-driver docker

TODO:

skip load images if using preloaded volume

… tar to gcs

… volume

k8s-ci-robot · 2020-02-20T23:06:06Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: priyawadhwa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [priyawadhwa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

medyagh

looks surprisingly easy ! some changes requested

pkg/drivers/kic/oci/oci.go

pkg/drivers/kic/oci/volumes.go

pkg/drivers/kic/preload/preload.go

pkg/drivers/kic/types.go

medyagh · 2020-02-20T23:45:55Z

on linux mount looks like this for refference.

root@minikube:/var/lib/minikube# mount | grep "var "
/dev/mapper/dhcp--100--106--128--167--vg-root on /var type ext4 (rw,relatime,errors=remount-ro,i_version)

pkg/drivers/kic/preload/preload.go

pkg/minikube/node/cache.go

medyagh · 2020-02-21T02:03:34Z

also This PR wont be complete without adding a Progress bar to the download. and letting user know this wait Worth it ! because it is one time !

medyagh · 2020-02-21T02:19:10Z

tested on my home mac (after running download-only first)

medmac@~/minikube (priyawadhwa-preload-images-into-docker-volume) $ make &&  ./out/minikube start --vm-driver=docker --download-only
make: `out/minikube' is up to date.
:smile:  minikube v1.7.3 on Darwin 10.13.6
:sparkles:  Using the docker (experimental) driver based on user configuration
:white_check_mark:  Download complete!
medmac@~/minikube (priyawadhwa-preload-images-into-docker-volume) $ make && time ./out/minikube start --vm-driver=docker
make: `out/minikube' is up to date.
:smile:  minikube v1.7.3 on Darwin 10.13.6
:sparkles:  Using the docker (experimental) driver based on user configuration
:fire:  Creating Kubernetes in docker container with (CPUs=2), Memory=2000MB (16384MB available) ...
[docker run --rm --entrypoint /usr/bin/tar -v /Users/medmac/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v1.17.3.tar.lz4:/preloaded.tar:ro -v k8s-v1.17.3:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.6@sha256:53725be5106d1d797dff4041d8c297383f32ab2edeff0a69fc3f50263cf17c79 -I lz4 -xvf /preloaded.tar -C /extractDir]
:whale:  Preparing Kubernetes v1.17.3 on Docker 19.03.2 ...
    :black_small_square: kubeadm.pod-network-cidr=10.244.0.0/16
:rocket:  Launching Kubernetes ...
:star2:  Enabling addons: default-storageclass, storage-provisioner
:hourglass:  Waiting for cluster to come online ...
:surfer:  Done! kubectl is now configured to use "minikube"
:warning:  /usr/local/bin/kubectl is version 1.15.1, and is incompatible with Kubernetes 1.17.3. You will need to update /usr/local/bin/kubectl or use 'minikube kubectl' to connect with this cluster

This PR

real	1m35.920s
user	0m4.348s
sys	0m2.437s

minikube 1.7.3 with normal docker driver (not preloaded)

real	1m45.611s
user	0m8.548s
sys	0m6.642s

medyagh · 2020-02-21T02:32:06Z

update:
after I restart docker the numbers change daramaticly better (that makes me think we need to make the user restart docker once in a while or if it is slow)

This PR on Mac

real	1m16.062s
user	0m4.373s
sys	0m2.355s

minikube v1.7.3 on mac

real	1m33.859s
user	0m8.660s
sys	0m6.671s

pkg/drivers/kic/preload/preload.go

minikube-pr-bot · 2020-02-21T06:43:04Z

All Times minikube: [ 94.411163 92.578566 92.994626]
All Times Minikube (PR 6720): [ 83.694567 80.446588 81.514605]

Average minikube: 93.328118
Average Minikube (PR 6720): 81.885254

Averages Time Per Log

+----------------------+-----------+--------------------+
|         LOG          | MINIKUBE  | MINIKUBE (PR 6720) |
+----------------------+-----------+--------------------+
| minikube v           |  0.130439 |           0.124516 |
| Creating kvm2        | 19.574173 |          19.684792 |
| Preparing Kubernetes | 50.478914 |           1.830890 |
| Pulling images       |           |                    |
| Launching Kubernetes | 20.983237 |          58.199285 |
| Waiting for cluster  |  0.216828 |           0.396553 |
+----------------------+-----------+--------------------+

codecov-io · 2020-03-02T22:04:48Z

Codecov Report

Merging #6720 into master will decrease coverage by 0.07%.
The diff coverage is 5.55%.

@@            Coverage Diff             @@
##           master    #6720      +/-   ##
==========================================
- Coverage    38.3%   38.23%   -0.08%     
==========================================
  Files         142      142              
  Lines        8740     8756      +16     
==========================================
  Hits         3348     3348              
- Misses       4971     4987      +16     
  Partials      421      421

Impacted Files	Coverage Δ
pkg/minikube/machine/cache_images.go	`0% <0%> (ø)`	⬆️
cmd/minikube/cmd/ssh.go	`8.69% <50%> (ø)`	⬆️

minikube-pr-bot · 2020-03-02T22:05:27Z

All Times Minikube (PR 6720): [ 89.696027 92.269080 91.165386]
All Times minikube: [ 93.393439 93.783989 93.086613]

Average minikube: 93.421347
Average Minikube (PR 6720): 91.043498

Averages Time Per Log

+----------------------+-----------+--------------------+
|         LOG          | MINIKUBE  | MINIKUBE (PR 6720) |
+----------------------+-----------+--------------------+
| minikube v           |  0.223245 |           0.225660 |
| Creating kvm2        | 20.622149 |          20.088689 |
| Preparing Kubernetes | 50.495944 |          49.684729 |
| Pulling images       |           |                    |
| Launching Kubernetes | 20.716543 |          19.839895 |
| Waiting for cluster  |  0.066176 |           0.049873 |
+----------------------+-----------+--------------------+

minikube-pr-bot · 2020-03-02T23:25:15Z

All Times minikube: [ 89.437817 91.676015 90.452250]
All Times Minikube (PR 6720): [ 91.985790 92.804147 93.489591]

Average minikube: 90.522028
Average Minikube (PR 6720): 92.759843

Averages Time Per Log

+----------------------+-----------+--------------------+
|         LOG          | MINIKUBE  | MINIKUBE (PR 6720) |
+----------------------+-----------+--------------------+
| minikube v           |  0.206848 |           0.204423 |
| Creating kvm2        | 20.139102 |          20.082953 |
| Preparing Kubernetes | 49.272713 |          50.219214 |
| Pulling images       |           |                    |
| Launching Kubernetes | 19.640481 |          20.653748 |
| Waiting for cluster  |  0.074660 |           0.061589 |
+----------------------+-----------+--------------------+

priyawadhwa · 2020-03-02T23:49:44Z

@medyagh PTAL whenever you have a chance. Thank you!

medyagh

lets make sure the integraiton tests download the file before all paralell tests run.

hack/preload-images/preload_images.go

…ster than go-containerregistry

minikube-pr-bot · 2020-03-03T01:45:23Z

All Times minikube: [ 91.814959 90.993627 89.940463]
All Times Minikube (PR 6720): [ 90.604335 91.289588 90.779862]

Average minikube: 90.916350
Average Minikube (PR 6720): 90.891262

Averages Time Per Log

+----------------------+-----------+--------------------+
|         LOG          | MINIKUBE  | MINIKUBE (PR 6720) |
+----------------------+-----------+--------------------+
| minikube v           |  0.211332 |           0.199944 |
| Creating kvm2        | 20.020063 |          19.957276 |
| Preparing Kubernetes | 49.072578 |          48.505356 |
| Pulling images       |           |                    |
| Launching Kubernetes | 20.351963 |          20.769722 |
| Waiting for cluster  |  0.046564 |           0.215646 |
+----------------------+-----------+--------------------+

medyagh · 2020-03-03T03:01:46Z

"TestDownloadOnlyDocker" fails on github actions:
https://github.com/kubernetes/minikube/suites/494573977/artifacts/2452817

=== RUN   TestDownloadOnlyDocker
--- FAIL: TestDownloadOnlyDocker (6.24s)
aaa_download_only_test.go:180: (dbg) Run:  ./minikube-linux-amd64 start --download-only -p download-docker-20200303T015709.338367775-2537 --force --alsologtostderr --vm-driver=docker
aaa_download_only_test.go:180: (dbg) Done: ./minikube-linux-amd64 start --download-only -p download-docker-20200303T015709.338367775-2537 --force --alsologtostderr --vm-driver=docker: (4.886875461s)
aaa_download_only_test.go:207: expected image does not exist in local daemon: Error response from daemon: reference does not exist
helpers.go:160: (dbg) Run:  ./minikube-linux-amd64 delete -p download-docker-20200303T015709.338367775-2537

medyagh

This is good work ! I am okay with merging this and solve the small issues in the following PRs.

I tested locally and it works in offline mode beautifully!
Thanks for the hard work on this.

please feel free to merge if you are ready.

medyagh · 2020-03-03T05:10:13Z

pkg/drivers/kic/kic.go

+	glog.Infof("Starting extracting preloaded images to volume")
+	// Extract preloaded images to container
+	if err := oci.ExtractTarballToVolume(preload.TarballFilepath(d.NodeConfig.KubernetesVersion), params.Name, BaseImage); err != nil {
+		glog.Infof("Unable to extract preloaded tarball to volume: %v", err)


optional nit: Warnf or Errorf so it doesn't get hidden in the logs.

medyagh · 2020-03-03T05:11:07Z

pkg/drivers/kic/oci/volumes.go

+	}
+	cmd := exec.Command(Docker, "run", "--rm", "--entrypoint", "/usr/bin/tar", "-v", fmt.Sprintf("%s:/preloaded.tar:ro", tarballPath), "-v", fmt.Sprintf("%s:/extractDir", volumeName), imageName, "-I", "lz4", "-xvf", "/preloaded.tar", "-C", "/extractDir")
+	if out, err := cmd.CombinedOutput(); err != nil {
+		return errors.Wrapf(err, "output %s", string(out))


optional nit:

Wrapf(err, "extract to volume: output %s", string(out))

Priya Wadhwa added 11 commits February 19, 2020 21:53

Download preloaded images tarball with --download-only flag

d506aa1

Create volume of preloaded images and mount it in

bb11bc6

Rebased on master

b8bc461

Name tarball by k8s version name

335637f

Add run to docker command

055e4bb

Delete container once it has finished executing

0e88697

Remove preloaded base image and add makefile rule to upload prelaoded…

97ea1b0

… tar to gcs

Compress with lz4

19fa296

Move all preloading code into preload package

b0f685c

skip transferring binaries and transferring imagse if using preloaded…

5ac8520

… volume

Debugging

55dd40e

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 20, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 20, 2020

k8s-ci-robot requested review from blueelvis and RA489 February 20, 2020 23:06

medyagh requested changes Feb 20, 2020

View reviewed changes

Rebased on master

86df9ff

medyagh reviewed Feb 21, 2020

View reviewed changes

pkg/drivers/kic/preload/preload.go Outdated Show resolved Hide resolved

medyagh reviewed Feb 21, 2020

View reviewed changes

pkg/minikube/node/cache.go Outdated Show resolved Hide resolved

medyagh reviewed Feb 21, 2020

View reviewed changes

pkg/drivers/kic/preload/preload.go Outdated Show resolved Hide resolved

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Feb 21, 2020

medyagh removed the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Feb 21, 2020

Add checksum verification

c99bf76

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 2, 2020

small fixes

095b4b4

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 2, 2020

Priya Wadhwa added 2 commits March 2, 2020 13:53

remove transfer binaries

742b3e4

remove unused functions

d6e94c1

Priya Wadhwa added 2 commits March 2, 2020 14:10

added overlay2 to preloaded images tarball name

481010d

check for preloaded images before loading again

4996b14

remove unnecessary PreloadedVolume

05a46f6

medyagh requested changes Mar 3, 2020

View reviewed changes

Priya Wadhwa added 4 commits March 2, 2020 16:54

Add integration test for docker download only

26de146

Include container runtime and tarball version in preloaded tarball name

5961513

Only add necessary directories to tarball

3d291fd

use docker command line to check if image exists, since it is much fa…

5d16281

…ster than go-containerregistry

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 3, 2020

priyawadhwa mentioned this pull request Mar 3, 2020

Build preloaded images tar from commit SHA #6865

Closed

medyagh approved these changes Mar 3, 2020

View reviewed changes

medyagh reviewed Mar 3, 2020

View reviewed changes

josedonizetti approved these changes Mar 3, 2020

View reviewed changes

priyawadhwa merged commit 22ca04a into kubernetes:master Mar 3, 2020

priyawadhwa deleted the preload-images-into-docker-volume branch March 3, 2020 18:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preload images into docker volume #6720

Preload images into docker volume #6720

priyawadhwa commented Feb 20, 2020

k8s-ci-robot commented Feb 20, 2020

medyagh left a comment

medyagh commented Feb 20, 2020

medyagh commented Feb 21, 2020

medyagh commented Feb 21, 2020 •

edited

Loading

medyagh commented Feb 21, 2020 •

edited

Loading

minikube-pr-bot commented Feb 21, 2020

codecov-io commented Mar 2, 2020 •

edited

Loading

minikube-pr-bot commented Mar 2, 2020

minikube-pr-bot commented Mar 2, 2020

priyawadhwa commented Mar 2, 2020

medyagh left a comment

minikube-pr-bot commented Mar 3, 2020

medyagh commented Mar 3, 2020

medyagh left a comment •

edited

Loading

medyagh Mar 3, 2020 •

edited

Loading

medyagh Mar 3, 2020 •

edited

Loading

Preload images into docker volume #6720

Preload images into docker volume #6720

Conversation

priyawadhwa commented Feb 20, 2020

k8s-ci-robot commented Feb 20, 2020

medyagh left a comment

Choose a reason for hiding this comment

medyagh commented Feb 20, 2020

medyagh commented Feb 21, 2020

medyagh commented Feb 21, 2020 • edited Loading

This PR

minikube 1.7.3 with normal docker driver (not preloaded)

medyagh commented Feb 21, 2020 • edited Loading

This PR on Mac

minikube v1.7.3 on mac

minikube-pr-bot commented Feb 21, 2020

codecov-io commented Mar 2, 2020 • edited Loading

Codecov Report

minikube-pr-bot commented Mar 2, 2020

minikube-pr-bot commented Mar 2, 2020

priyawadhwa commented Mar 2, 2020

medyagh left a comment

Choose a reason for hiding this comment

minikube-pr-bot commented Mar 3, 2020

medyagh commented Mar 3, 2020

medyagh left a comment • edited Loading

Choose a reason for hiding this comment

medyagh Mar 3, 2020 • edited Loading

Choose a reason for hiding this comment

medyagh Mar 3, 2020 • edited Loading

Choose a reason for hiding this comment

medyagh commented Feb 21, 2020 •

edited

Loading

medyagh commented Feb 21, 2020 •

edited

Loading

codecov-io commented Mar 2, 2020 •

edited

Loading

medyagh left a comment •

edited

Loading

medyagh Mar 3, 2020 •

edited

Loading

medyagh Mar 3, 2020 •

edited

Loading