Generate cri-o container runtime preload tarball #8581

afbjorklund · 2020-06-27T11:20:03Z

Added cri-o, to the list of preloaded tarballs:

A preloaded tarball for k8s version v1.18.3 - runtime "docker" already exists, skipping generation.
A preloaded tarball for k8s version v1.18.3 - runtime "containerd" already exists, skipping generation.
A preloaded tarball for k8s version v1.18.3 - runtime "cri-o" doesn't exist, generating now...
Image is up to date for k8s.gcr.io/kube-proxy@sha256:6a093c22e305039b7bd6c3f8eab8f202ad8238066ed210857b25524443aa8aff
Image is up to date for k8s.gcr.io/kube-scheduler@sha256:5381cd9680bf5fb16a5c8ac60141eaab242c1c4960f1c32a21807efcca3e765b
Image is up to date for k8s.gcr.io/kube-controller-manager@sha256:168e8f9276e5f947d68f93a9463243f6a525227d15f55fb40b427c8d6caa55c2
Image is up to date for k8s.gcr.io/kube-apiserver@sha256:b5bf4650d7f084d0c9f89ecae50c00ebf5b44edcdd487636d8c7fd400d27cd05
Image is up to date for k8s.gcr.io/coredns@sha256:2c8d61c46f484d881db43b34d13ca47a269336e576c81cf007ca740fa9ec0800
Image is up to date for k8s.gcr.io/etcd@sha256:4198ba6f82f642dfd18ecf840ee37afb9df4b596f06eef20e44d0aec4ea27216
Image is up to date for k8s.gcr.io/pause@sha256:4a1c4b21597c1b4415bdbecb28a3296c6b5e23ca4f9feeb599860a1dac6a0108
Image is up to date for gcr.io/k8s-minikube/storage-provisioner@sha256:088daa9fcbccf04c3f415d77d5a6360d2803922190b675cb7fc88a9d2d91985a
Image is up to date for docker.io/kubernetesui/dashboard@sha256:19207cca570f61bce2294595a767b51b4d7e1f223a003c7972c350cf8689f49e
Image is up to date for docker.io/kubernetesui/metrics-scraper@sha256:555981a24f184420f3be0c79d4efb6c948a85cfce84034f85a563f4151a81cbf
Image is up to date for docker.io/kindest/kindnetd@sha256:46e34ccb3e08557767b7c80e957741d9f2590968ff32646875632d40cf62adad

For #6934

codecov-commenter · 2020-06-27T11:28:32Z

Codecov Report

Merging #8581 into master will decrease coverage by 0.08%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #8581      +/-   ##
==========================================
- Coverage   32.14%   32.05%   -0.09%     
==========================================
  Files         161      161              
  Lines       10494    10523      +29     
==========================================
  Hits         3373     3373              
- Misses       6717     6746      +29     
  Partials      404      404

Impacted Files	Coverage Δ
pkg/minikube/cruntime/crio.go	`46.22% <0.00%> (-7.04%)`	⬇️
pkg/addons/addons.go	`40.95% <0.00%> (-3.16%)`	⬇️

medyagh

Could you plz test it manually ? You can generate preload tar and then put it in the cache folder and if it works successfully we will merge it and let the cron job create the tars

afbjorklund · 2020-06-27T17:19:52Z

It seems to auto-upload, so I guess I need to comment that out... And need to determine if that “cache” folder is required by the “storage”

So maybe need the preload part as well, to see whether it actually works ?

afbjorklund · 2020-06-27T18:33:53Z

I included everything for now, until I actually know what is in those databases...

./lib/containers/sigstore/
./lib/containers/storage/storage.lock
./lib/containers/storage/overlay-layers/*
./lib/containers/mounts/
./lib/containers/storage/overlay/*
./lib/containers/storage/overlay-images/*
./lib/containers/cache/blob-info-cache-v1.boltdb

Note that the "sigstore" and "mounts" are empty, so it's just about "cache".

Seems to have worked otherwise. Tested with just the default k8s version.

592M	out/preloaded-images-k8s-v4-v1.18.3-cri-o-overlay2-amd64.tar.lz4

Note: that this Step 1 was only about creating the tarball, not about using it

afbjorklund · 2020-06-27T18:42:14Z

The name "overlay2" seems to be hard-coded. It was supposed to be "overlay"...
It's the same thing, anyway. Docker just renamed theirs, since they had overlay1

// TarballName returns name of the tarball
func TarballName(k8sVersion, containerRuntime string) string {
        return fmt.Sprintf("preloaded-images-k8s-%s-%s-%s-overlay2-%s.tar.lz4", PreloadVersion, k8sVersion, containerRuntime, runtime.GOARCH)
}

~~But this name is actually implemented by "download", so it will be in Step 2: loading~~
~~It can also stay as overlay2 in the file name, I'm not sure if anyone actually cares ?~~

driver=""
container storage driver (default: "overlay")
Default Copy On Write (COW) container storage driver. Valid drivers are "overlay", "vfs", "devmapper", "aufs", "btrfs", and "zfs". Some drivers (for example, "zfs", "btrfs", and "aufs") may not work if your kernel lacks support for the filesystem.

Fixed the name from the start. The actual API is a bit stupid (uses strings), so still hard-coded.

       if containerRuntime == "cri-o" {
               storageDriver = "overlay"
       } else {
               storageDriver = "overlay2"
       }

afbjorklund · 2020-06-28T07:55:59Z

New name:

592M	out/preloaded-images-k8s-v4-v1.18.3-cri-o-overlay-amd64.tar.lz4

afbjorklund · 2020-06-28T12:41:49Z

@medyagh : trying to use preload with podman currently fails with the usual problem of overloading /var:

I0628 14:36:38.585784  176417 cli_runner.go:109] Run: sudo -n podman logs --timestamps crio
I0628 14:36:38.657245  176417 errors.go:60] Postmortem logs ("sudo -n podman logs --timestamps crio"): -- stdout --
2020-06-28T14:36:24.146928527+02:00 INFO: ensuring we can execute /bin/mount even with userns-remap
2020-06-28T14:36:24.164124815+02:00 INFO: remounting /sys read-only
2020-06-28T14:36:24.165968352+02:00 INFO: making mounts shared
2020-06-28T14:36:24.167145397+02:00 INFO: fix cgroup mounts for all subsystems
2020-06-28T14:36:24.169329664+02:00 INFO: clearing and regenerating /etc/machine-id
2020-06-28T14:36:24.174558308+02:00 Initializing machine ID from random generator.
2020-06-28T14:36:24.197085721+02:00 INFO: faking /sys/class/dmi/id/product_name to be "kind"
2020-06-28T14:36:24.199985019+02:00 INFO: faking /sys/class/dmi/id/product_uuid to be random
2020-06-28T14:36:24.201451595+02:00 INFO: faking /sys/devices/virtual/dmi/id/product_uuid as well
2020-06-28T14:36:24.215181859+02:00 INFO: setting iptables to detected mode: legacy
2020-06-28T14:36:24.215822257+02:00 update-alternatives: error: no alternatives for iptables

So we should fix that one (#8056 #8100 #8179) first, before enabling this feature. But we need these tarballs, first.

EDIT: This is only a problem for the KIC drivers, we can still use preloaded tarballs with the VM drivers

Pretty much the same as containerd, minus Restart (cherry picked from commit d9fb2e0)

hack/preload-images/generate.go

hack/preload-images/preload_images.go

k8s-ci-robot · 2020-06-30T16:58:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: afbjorklund

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [afbjorklund]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

afbjorklund · 2020-06-30T16:58:35Z

Tested locally, seems to work ok:

962M	out/preloaded-images-k8s-v4-v1.18.3-containerd-overlay2-amd64.tar.lz4
592M	out/preloaded-images-k8s-v4-v1.18.3-cri-o-overlay-amd64.tar.lz4
527M	out/preloaded-images-k8s-v4-v1.18.3-docker-overlay2-amd64.tar.lz4

TravisBuddy · 2020-06-30T17:03:52Z

Travis tests have failed

Hey @afbjorklund,
Please read the following log in order to understand the failure reason.
It'll be awesome if you fix what's wrong and commit the changes.

1st Build

View build log

make test

which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.13.9.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.0/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/...
gofmt -s -w pkg/minikube/assets/assets.go
which go-bindata || GO111MODULE=off GOBIN="/home/travis/gopath/bin" go get github.com/jteeuwen/go-bindata/...
/home/travis/gopath/bin/go-bindata
PATH="/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.13.9.linux.amd64/bin:/home/travis/bin:/home/travis/bin:/home/travis/.local/bin:/usr/local/lib/jvm/openjdk11/bin:/opt/pyenv/shims:/home/travis/.phpenv/shims:/home/travis/perl5/perlbrew/bin:/home/travis/.nvm/versions/node/v8.12.0/bin:/home/travis/.rvm/gems/ruby-2.5.3/bin:/home/travis/.rvm/gems/ruby-2.5.3@global/bin:/home/travis/.rvm/rubies/ruby-2.5.3/bin:/home/travis/gopath/bin:/home/travis/.gimme/versions/go1.11.1.linux.amd64/bin:/usr/local/maven-3.6.0/bin:/usr/local/cmake-3.12.4/bin:/usr/local/clang-7.0.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/home/travis/.rvm/bin:/home/travis/.phpenv/bin:/opt/pyenv/bin:/home/travis/.yarn/bin:/home/travis/gopath/bin" go-bindata -nomemcopy -o pkg/minikube/translate/translations.go -pkg translate translations/...
gofmt -s -w pkg/minikube/translate/translations.go
MINIKUBE_LDFLAGS="-X k8s.io/minikube/pkg/version.version=v1.12.0-beta.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.11.0 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="ad6e504fe38b7b4e4b7673242a8038dd66b5c0e1"" ./test.sh
= make lint =============================================================
golangci/golangci-lint info checking GitHub for tag 'v1.26.0'
golangci/golangci-lint info found version: 1.26.0 for v1.26.0/linux/amd64
golangci/golangci-lint info installed out/linters/golangci-lint
pkg/minikube/cruntime/cri.go:269: File is not `goimports`-ed (goimports)
        if !strings.Contains(imgName, ".io/") {
                return "docker.io/" + imgName
        } // else it already has repo name dont add anything
        return imgName
Makefile:390: recipe for target 'lint-ci' failed
make[1]: *** [lint-ci] Error 1
= go mod ================================================================
ok
= boilerplate ===========================================================
ok
Makefile:274: recipe for target 'test' failed
make: *** [test] Error 4

TravisBuddy Request Identifier: a9c15360-baf3-11ea-bd85-f15c2d376ac7

hack/preload-images/generate.go

medyagh

thank you

afbjorklund requested review from medyagh and priyawadhwa June 27, 2020 11:20

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 27, 2020

k8s-ci-robot requested a review from sharifelgamal June 27, 2020 11:20

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 27, 2020

medyagh reviewed Jun 27, 2020

View reviewed changes

Generate cri-o container runtime preload tarball

1bf268c

afbjorklund force-pushed the preload-crio branch from fdbab9f to 1bf268c Compare June 27, 2020 18:26

afbjorklund changed the title ~~Generate cri-o container runtime preload tarball~~ WIP: Generate cri-o container runtime preload tarball Jun 28, 2020

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 28, 2020

Use the actual storage driver name for cri-o

b87a837

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jun 28, 2020

afbjorklund changed the title ~~WIP: Generate cri-o container runtime preload tarball~~ Generate cri-o container runtime preload tarball Jun 28, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 28, 2020

Normalize name of the container runtime param

51b4d2e

afbjorklund mentioned this pull request Jun 28, 2020

Use preloaded tarball for cri-o container runtime #8588

Merged

Use preloaded tarball for cri-o container runtime

eaeb5f6

Pretty much the same as containerd, minus Restart (cherry picked from commit d9fb2e0)

medyagh requested changes Jun 30, 2020

View reviewed changes

hack/preload-images/generate.go Show resolved Hide resolved

medyagh requested changes Jun 30, 2020

View reviewed changes

hack/preload-images/preload_images.go Outdated Show resolved Hide resolved

Need to test storage driver of container runtime

b3e5e70

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 30, 2020

afbjorklund requested a review from medyagh June 30, 2020 16:58

Change indentation from gofmt to goimports

b0b3a1a

medyagh requested changes Jul 7, 2020

View reviewed changes

hack/preload-images/generate.go Show resolved Hide resolved

medyagh approved these changes Jul 20, 2020

View reviewed changes

medyagh merged commit 79088e6 into kubernetes:master Jul 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate cri-o container runtime preload tarball #8581

Generate cri-o container runtime preload tarball #8581

afbjorklund commented Jun 27, 2020 •

edited

Loading

codecov-commenter commented Jun 27, 2020 •

edited

Loading

medyagh left a comment

afbjorklund commented Jun 27, 2020

afbjorklund commented Jun 27, 2020 •

edited

Loading

afbjorklund commented Jun 27, 2020 •

edited

Loading

afbjorklund commented Jun 28, 2020

afbjorklund commented Jun 28, 2020 •

edited

Loading

k8s-ci-robot commented Jun 30, 2020

afbjorklund commented Jun 30, 2020

TravisBuddy commented Jun 30, 2020

medyagh left a comment

Generate cri-o container runtime preload tarball #8581

Generate cri-o container runtime preload tarball #8581

Conversation

afbjorklund commented Jun 27, 2020 • edited Loading

codecov-commenter commented Jun 27, 2020 • edited Loading

Codecov Report

medyagh left a comment

Choose a reason for hiding this comment

afbjorklund commented Jun 27, 2020

afbjorklund commented Jun 27, 2020 • edited Loading

afbjorklund commented Jun 27, 2020 • edited Loading

afbjorklund commented Jun 28, 2020

afbjorklund commented Jun 28, 2020 • edited Loading

k8s-ci-robot commented Jun 30, 2020

afbjorklund commented Jun 30, 2020

TravisBuddy commented Jun 30, 2020

Travis tests have failed

1st Build

TravisBuddy Request Identifier: a9c15360-baf3-11ea-bd85-f15c2d376ac7

medyagh left a comment

Choose a reason for hiding this comment

afbjorklund commented Jun 27, 2020 •

edited

Loading

codecov-commenter commented Jun 27, 2020 •

edited

Loading

afbjorklund commented Jun 27, 2020 •

edited

Loading

afbjorklund commented Jun 27, 2020 •

edited

Loading

afbjorklund commented Jun 28, 2020 •

edited

Loading