`cache reload` does not appear to be working properly

[0x-shanks]

Steps to reproduce the issue:

1. Pull the alpine:3.11 and alpine:3.12 image and tag alpine:3.11 as test

2. Add a cache with the following command
   minikube cache add alpine:test

3. Enter the minikube container and check if the image exists

4. Tag alpine:3.12 as test

5. Reload the cache
   minikube cache reload alpine:test

6. I check the image again, but it hasn't changed

Optional: Full output of minikube logs command:

user@user ~ % minikube cache reload alpine:test -v=6 --alsologtostderr
I1230 00:30:52.617760   11506 out.go:221] Setting OutFile to fd 1 ...
I1230 00:30:52.618211   11506 out.go:273] isatty.IsTerminal(1) = true
I1230 00:30:52.618219   11506 out.go:234] Setting ErrFile to fd 2...
I1230 00:30:52.618223   11506 out.go:273] isatty.IsTerminal(2) = true
I1230 00:30:52.618361   11506 root.go:280] Updating PATH: /Users/user/.minikube/bin
I1230 00:30:52.618678   11506 cache.go:92] acquiring lock: {Name:mk21f3e783644f4fb459dda56815a449b17b30ed Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I1230 00:30:52.618797   11506 cache.go:100] /Users/user/.minikube/cache/images/alpine_test exists
I1230 00:30:52.618810   11506 cache.go:81] cache image "alpine:test" -> "/Users/user/.minikube/cache/images/alpine_test" took 142.87µs
I1230 00:30:52.618817   11506 cache.go:66] save to tar file alpine:test -> /Users/user/.minikube/cache/images/alpine_test succeeded
I1230 00:30:52.618824   11506 cache.go:73] Successfully saved all images to host disk.
I1230 00:30:52.619292   11506 cli_runner.go:111] Run: docker ps -a --filter label=name.minikube.sigs.k8s.io --format {{.Names}}
I1230 00:30:52.768326   11506 cli_runner.go:111] Run: docker container inspect minikube --format={{.State.Status}}
I1230 00:30:52.903556   11506 ssh_runner.go:149] Run: systemctl --version
I1230 00:30:52.903697   11506 cli_runner.go:111] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I1230 00:30:53.036130   11506 sshutil.go:48] new ssh client: &{IP:127.0.0.1 Port:55007 SSHKeyPath:/Users/user/.minikube/machines/minikube/id_rsa Username:docker}
I1230 00:30:53.148720   11506 ssh_runner.go:149] Run: docker images --format {{.Repository}}:{{.Tag}}
I1230 00:30:53.207718   11506 docker.go:382] Got preloaded images: -- stdout --
alpine:test

[afbjorklund]

It only looks at the repo:tag, so it will only see that "alpine:test" is already there.

docker images --format {{.Repository}}:{{.Tag}}

There is no matching "AlwaysPullImages" option (yet?), it assumes unique tags

There is some old broken code to check id/digest

[0x-shanks]

I see, so even if the Image ID changes, if the repo:tag is the same, reloading won't help, and if you want to use a different image, you have to add a unique tag?

So when is reload used?

[afbjorklund]

There might be a bug (change of behaviour) in the "cache" command, after adding the preload...

It now checks the image list (repo:tag), even before comparing the Id and/or Digest whatsoever.

Images are preloaded, skipping loading

It would also add any missing images, or any that was added after the cluster was initially created

[afbjorklund]

Problem here is that an image on disk doesn't have any Id or Digest, that is added by the daemon/registry.

So you need to either load it locally (eew) or talk to the registry (eeew), whichever is worst (we do both of them).

There is also some compatibility issues with crane and the library behind it: google/go-containerregistry#890

At the moment, we can't even agree on the name of the container image. Much less on the actual contents.

• alpine
• alpine:latest
• docker.io/alpine
• docker.io/library/alpine:latest

Short-term fix will probably be to re-visit the preload code and maybe add an --always flag to minikube cache.

Assuming that the daemon/registry code for digest even works, I have some doubts about that:

• https://github.com/kubernetes/minikube/blob/v1.16.0/pkg/minikube/image/image.go#L51

• https://github.com/kubernetes/minikube/blob/v1.16.0/pkg/minikube/image/image.go#L65

[0x-shanks]

@afbjorklund
Thank you for your kindness.
I see that you have to add Id or Digest to the image list, not just repo:tags.

Short-term fix will probably be to re-visit the preload code and maybe add an --always flag to minikube cache.

As you said, it's short term, but I'm going to try to implement it this way, but I've never contributed to minikube, so I don't know the difficulty level.
Would it be better to wait quietly for the Id or Digest to be added?

[afbjorklund]

Sad thing is that neither Id nor Digest will work with all container runtimes and also offline (without a registry)...

I added a new feature to clean up the patched library, we can use this particular issue to go past the name only ?

[seperman]

Hello @afbjorklund
I'm in the same boat. Do you recommend any workarounds? I see the cached file in ~/.minikube/cache/images/localhost_5000
localhost_5000 is my local registry. When I run minikube cache delete localhost:5000/repo:tag it does remove that file. But if I do minikube cache add localhost:5000/repo:tag, it will still add the previous image even though the image is updated in the local registry. So it seems like it is caching the old image somewhere else too.

[afbjorklund]

If the image already exists in the docker daemon, it will not pull anything from the registry.

1. https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/daemon#Image

2. https://pkg.go.dev/github.com/google/go-containerregistry/pkg/v1/remote#Image

It actually always checks with dockerd, even if there no such thing running on the host.

# daemon.Image
retrieving image: busybox:latest
daemon lookup for busybox:latest: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
# remote.Image
cache image "busybox:latest" -> "/home/anders/.minikube/cache/images/busybox" took 2.80001642s
save to tar file busybox -> /home/anders/.minikube/cache/images/busybox succeeded

[afbjorklund]

So the current workaround would be docker pull localhost:5000/repo:tag

That will update the image in the daemon with the new one from the registry.

[seperman]

@afbjorklund Thanks for the response. Btw I see a new warning message on Minikube v1.18.1

"minikube cache" will be deprecated in upcoming versions, please switch to "minikube image load"

But the docs don't mention minikube image load: https://minikube.sigs.k8s.io/docs/handbook/pushing/

[afbjorklund]

@seperman : I'm actually not sure why @priyawadhwa wrote that "cache" would be deprecated e8945f2

They don't do exactly the same thing, even if they do share some code. The "cache" command has a memory

However, the "image load" and "image build" commands now have slightly better documentation (placeholders)