preload removes images built by user in docker-env #6981

medyagh · 2020-03-10T17:41:03Z

with docker-env on: build an image:

medya@~/Desktop $ docker build -t medya .
Sending build context to Docker daemon  23.29MB
Step 1/2 : FROM alpine
 ---> e7d92cdc71fe
Step 2/2 : RUN echo "HEllo world from medya" /hello.txt
 ---> Using cache
 ---> f4a736bf846b
Successfully built f4a736bf846b
Successfully tagged medya:latest

check the image is tehere

medya@~/Desktop $ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
medya                                     latest              f4a736bf846b        30 seconds ago      5.59MB
k8s.gcr.io/kube-proxy                     v1.17.3             ae853e93800d        3 weeks ago         116MB
k8s.gcr.io/kube-controller-manager        v1.17.3             b0f1517c1f4b        3 weeks ago         161MB
k8s.gcr.io/kube-apiserver                 v1.17.3             90d27391b780        3 weeks ago         171MB
k8s.gcr.io/kube-scheduler                 v1.17.3             d109c0821a2b        3 weeks ago         94.4MB
alpine                                    latest              e7d92cdc71fe        7 weeks ago         5.59MB
kubernetesui/dashboard                    v2.0.0-beta8        eb51a3597525        3 months ago        90.8MB
k8s.gcr.io/coredns                        1.6.5               70f311871ae1        4 months ago        41.6MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        4 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        4 months ago        288MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379        4 months ago        40.1MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca1        2 years ago         742kB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB

re-load minikube

medya@~/Desktop $ minikube start
😄  minikube v1.8.1 on Darwin 10.14.6
    ▪ MINIKUBE_ACTIVE_DOCKERD=minikube
✨  Using the hyperkit driver based on existing profile
💿  Downloading VM boot image ...
⌛  Reconfiguring existing host ...
🏃  Using the running hyperkit "minikube" VM ...
🐳  Preparing Kubernetes v1.17.3 on Docker 19.03.6 ...
🚀  Launching Kubernetes ... 
🌟  Enabling addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

check for images again, it is gone :(

medya@~/Desktop $ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
<none>                                    <none>              f4a736bf846b        3 minutes ago       5.59MB
k8s.gcr.io/kube-proxy                     v1.17.3             ae853e93800d        3 weeks ago         116MB
k8s.gcr.io/kube-apiserver                 v1.17.3             90d27391b780        3 weeks ago         171MB
k8s.gcr.io/kube-controller-manager        v1.17.3             b0f1517c1f4b        3 weeks ago         161MB
k8s.gcr.io/kube-scheduler                 v1.17.3             d109c0821a2b        3 weeks ago         94.4MB
kubernetesui/dashboard                    v2.0.0-beta8        eb51a3597525        3 months ago        90.8MB
k8s.gcr.io/coredns                        1.6.5               70f311871ae1        4 months ago        41.6MB
kindest/kindnetd                          0.5.3               aa67fec7d7ef        4 months ago        78.5MB
k8s.gcr.io/etcd                           3.4.3-0             303ce5db0e90        4 months ago        288MB
kubernetesui/metrics-scraper              v1.0.2              3b08661dc379        4 months ago        40.1MB
k8s.gcr.io/pause                          3.1                 da86e6ba6ca1        2 years ago         742kB
gcr.io/k8s-minikube/storage-provisioner   v1.8.1              4689081edb10        2 years ago         80.8MB

The text was updated successfully, but these errors were encountered:

afbjorklund · 2020-03-10T18:07:01Z

Maybe need to reindexed , if you are adding the storage layers directly ?

a-yiorgos · 2020-03-10T18:08:49Z

Hi!

I confirm the exact same behavior on:

Ubuntu 19.10 host
VirtualBox 6.0.14
minikube 1.8.1
kubectl 1.17.3

If I use --kubernetes-version=1.16.7 I get the expected behavior.

afbjorklund · 2020-03-10T18:56:20Z

@medyagh : and with "index", I meant the repositories.json

In the code, it is known as the refstore: https://github.com/moby/moby/blob/master/reference/store.go

afbjorklund · 2020-03-10T18:56:44Z

If I use --kubernetes-version=1.16.7 I get the expected behavior.

Probably only because there is no preload (yet?) for that version...

medyagh · 2020-03-10T19:03:58Z

since this behaviour only happesn on a minikube reload (start again) I feel like if we make minikube not to preload on a "restart" or "soft restart" then we should not see this problem.

Wdyt @priyawadhwa ?

medyagh · 2020-03-10T19:05:03Z

If I use --kubernetes-version=1.16.7 I get the expected behavior.

Probably only because there is no preload (yet?) for that version...

so you think the user's image is not over-written by our tar, it just needs re-index? how do we do re-index?

afbjorklund · 2020-03-10T19:06:32Z

Well, the layers themselves should be under the sha256 directories. It's the tags that need "merging"

My tarball has this: (jq < ./lib/docker/image/overlay2/repositories.json)

{
  "Repositories": {
    "gcr.io/k8s-minikube/storage-provisioner": {
      "gcr.io/k8s-minikube/storage-provisioner:v1.8.1": "sha256:4689081edb103a9e8174bf23a255bfbe0b2d9ed82edc907abab6989d1c60f02c",
      "gcr.io/k8s-minikube/storage-provisioner@sha256:088daa9fcbccf04c3f415d77d5a6360d2803922190b675cb7fc88a9d2d91985a": "sha256:4689081edb103a9e8174bf23a255bfbe0b2d9ed82edc907abab6989d1c60f02c"
    },
    "k8s.gcr.io/coredns": {
      "k8s.gcr.io/coredns:1.6.5": "sha256:70f311871ae12c14bd0e02028f249f933f925e4370744e4e35f706da773a8f61",
      "k8s.gcr.io/coredns@sha256:7ec975f167d815311a7136c32e70735f0d00b73781365df1befd46ed35bd4fe7": "sha256:70f311871ae12c14bd0e02028f249f933f925e4370744e4e35f706da773a8f61"
    },
    "k8s.gcr.io/etcd": {
      "k8s.gcr.io/etcd:3.4.3-0": "sha256:303ce5db0e90dab1c5728ec70d21091201a23cdf8aeca70ab54943bbaaf0833f",
      "k8s.gcr.io/etcd@sha256:4afb99b4690b418ffc2ceb67e1a17376457e441c1f09ab55447f0aaf992fa646": "sha256:303ce5db0e90dab1c5728ec70d21091201a23cdf8aeca70ab54943bbaaf0833f"
    },
    "k8s.gcr.io/kube-apiserver": {
      "k8s.gcr.io/kube-apiserver:v1.17.3": "sha256:90d27391b7808cde8d9a81cfa43b1e81de5c4912b4b52a7dccb19eb4fe3c236b",
      "k8s.gcr.io/kube-apiserver@sha256:33400ea29255bd20714b6b8092b22ebb045ae134030d6bf476bddfed9d33e900": "sha256:90d27391b7808cde8d9a81cfa43b1e81de5c4912b4b52a7dccb19eb4fe3c236b"
    },
    "k8s.gcr.io/kube-controller-manager": {
      "k8s.gcr.io/kube-controller-manager:v1.17.3": "sha256:b0f1517c1f4bb153597033d2efd81a9ac630e6a569307f993b2c0368afcf0302",
      "k8s.gcr.io/kube-controller-manager@sha256:2f0bf4d08e72a1fd6327c8eca3a72ad21af3a608283423bb3c10c98e68759844": "sha256:b0f1517c1f4bb153597033d2efd81a9ac630e6a569307f993b2c0368afcf0302"
    },
    "k8s.gcr.io/kube-proxy": {
      "k8s.gcr.io/kube-proxy:v1.17.3": "sha256:ae853e93800dc2572aeb425e5765cf9b25212bfc43695299e61dece06cffa4a1",
      "k8s.gcr.io/kube-proxy@sha256:3a70e2ab8d1d623680191a1a1f1dcb0bdbfd388784b1f153d5630a7397a63fd4": "sha256:ae853e93800dc2572aeb425e5765cf9b25212bfc43695299e61dece06cffa4a1"
    },
    "k8s.gcr.io/kube-scheduler": {
      "k8s.gcr.io/kube-scheduler:v1.17.3": "sha256:d109c0821a2b9225b69b99a95000df5cd1de5d606bc187b3620d730d7769c6ad",
      "k8s.gcr.io/kube-scheduler@sha256:b091f0db3bc61a3339fd3ba7ebb06c984c4ded32e1f2b1ef0fbdfab638e88462": "sha256:d109c0821a2b9225b69b99a95000df5cd1de5d606bc187b3620d730d7769c6ad"
    },
    "k8s.gcr.io/pause": {
      "k8s.gcr.io/pause:3.1": "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e",
      "k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea": "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e"
    },
    "kindest/kindnetd": {
      "kindest/kindnetd:0.5.3": "sha256:aa67fec7d7ef71445da9a84e9bc88afca2538e9a0aebcba6ef9509b7cf313d17",
      "kindest/kindnetd@sha256:bc1833b3da442bb639008dd5a62861a0419d3f64b58fce6fb38b749105232555": "sha256:aa67fec7d7ef71445da9a84e9bc88afca2538e9a0aebcba6ef9509b7cf313d17"
    },
    "kubernetesui/dashboard": {
      "kubernetesui/dashboard:v2.0.0-beta8": "sha256:eb51a359752560a66f314602e87155b75f428fb838bf951079ff1f9621958c0c",
      "kubernetesui/dashboard@sha256:fc90baec4fb62b809051a3227e71266c0427240685139bbd5673282715924ea7": "sha256:eb51a359752560a66f314602e87155b75f428fb838bf951079ff1f9621958c0c"
    },
    "kubernetesui/metrics-scraper": {
      "kubernetesui/metrics-scraper:v1.0.2": "sha256:3b08661dc379d9f80155be9d658f71578988640357ebae1aab287d6954c723d1",
      "kubernetesui/metrics-scraper@sha256:2026f9f7558d0f25cc6bab74ea201b4e9d5668fbc378ef64e13fddaea570efc0": "sha256:3b08661dc379d9f80155be9d658f71578988640357ebae1aab287d6954c723d1"
    }
  }
}

priyawadhwa · 2020-03-10T20:40:15Z

Yah, so the layers are fine but the repositories.json is being overwritten so the tags disappear. I think skipping preload on reload is probably the right move here, since all the needed images should be loaded into the daemon anyway.

h1771t · 2020-03-11T10:28:52Z

Exact same issue, ubuntu 18.04, minikube v1.8.1, kvm2 driver. Please fix and release.
Tried to roll back down to v1.8.0, but it even happens on v1.8.0.
Rolled back again to v1.7.3, seems fine on v1.7.3.

medyagh added co/preload kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Mar 10, 2020

afbjorklund changed the title ~~prelad removes dockers built by user in docker-env~~ preload removes images built by user in docker-env Mar 10, 2020

afbjorklund added the co/docker-env docker-env issues label Mar 10, 2020

priyawadhwa self-assigned this Mar 10, 2020

priyawadhwa mentioned this issue Mar 10, 2020

Merge repositories.json after extracting preloaded tarball so that reference store isn't lost #6985

Merged

medyagh closed this as completed in #6985 Mar 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

preload removes images built by user in docker-env #6981

preload removes images built by user in docker-env #6981

medyagh commented Mar 10, 2020 •

edited

Loading

afbjorklund commented Mar 10, 2020 •

edited

Loading

a-yiorgos commented Mar 10, 2020 •

edited

Loading

afbjorklund commented Mar 10, 2020 •

edited

Loading

afbjorklund commented Mar 10, 2020

medyagh commented Mar 10, 2020

medyagh commented Mar 10, 2020

afbjorklund commented Mar 10, 2020 •

edited

Loading

priyawadhwa commented Mar 10, 2020

h1771t commented Mar 11, 2020 •

edited

Loading

preload removes images built by user in docker-env #6981

preload removes images built by user in docker-env #6981

Comments

medyagh commented Mar 10, 2020 • edited Loading

with docker-env on: build an image:

check the image is tehere

re-load minikube

check for images again, it is gone :(

afbjorklund commented Mar 10, 2020 • edited Loading

a-yiorgos commented Mar 10, 2020 • edited Loading

afbjorklund commented Mar 10, 2020 • edited Loading

afbjorklund commented Mar 10, 2020

medyagh commented Mar 10, 2020

medyagh commented Mar 10, 2020

afbjorklund commented Mar 10, 2020 • edited Loading

priyawadhwa commented Mar 10, 2020

h1771t commented Mar 11, 2020 • edited Loading

medyagh commented Mar 10, 2020 •

edited

Loading

afbjorklund commented Mar 10, 2020 •

edited

Loading

a-yiorgos commented Mar 10, 2020 •

edited

Loading

afbjorklund commented Mar 10, 2020 •

edited

Loading

afbjorklund commented Mar 10, 2020 •

edited

Loading

h1771t commented Mar 11, 2020 •

edited

Loading