minikube cache doesn't work with crio runtime #8554
Comments
|
Can you confirm if the image was loaded OK, with |
|
Looks it is the usual quirk of podman/cri-o, that it loads them with a prefix: You can see here that it pulled the dashboard again, even though cached. |
|
Assuming that you don't want to rewrite all your images, this is upstream. minikube kubectl -- run --image localhost/my-local-image bug --image-pull-policy=IfNotPresent -- echo "hello world"
pod/bug created
minikube kubectl -- logs bug
hello worldThere was |
|
The workaround with prefixing the image name with |
btw skaffold doesnt use CRIO by default . I say we could add a solution message to the user to add the prefix or add it to the docs |
|
If this isn't fixed upstream, we need to workaround it for our cruntime loader... At first glance, it seems that only images from a registry have been "fixed". We add @@ -346,8 +346,9 @@
# List of registries to be used when pulling an unqualified image (e.g.,
# "alpine:latest"). By default, registries is set to "docker.io" for
# compatibility reasons. Depending on your workload and usecase you may add more
# registries (e.g., "quay.io", "registry.fedoraproject.org",
# "registry.opensuse.org", etc.).
-#registries = [
-# ]
+registries = [
+ "docker.io"
+]
The assumption here is that all the commands should render the same image.
Having users add a bogus "localhost/" in their manifest is an ugly workaround. |
Every tool should use the CRI (in the ideal world). Hardcoding stuff to docker doesn't help anyone. |
|
Here is the main difference: For the registry images, it is not the end of the world. It will just pull manifest again. But for the local images (like in this ticket), this prefix behaviour becomes an issue. |
|
Another interesting feature is the nasty warning you get when loading:
It also gets a different (localhost) digest, instead of the empty one you get when loading with docker. Which is a surprise, if you expect |
|
Workaround The problem is that we only have "foo.tar" (file) in the API, not "foo" (name) So need either scrape the output from load (eww), or add another API call... EDIT: As usual, also need to expand "foo" to "foo:latest" (if not having a tag) |
|
Still have a lot of unconditional calls to Docker, around the code base. DigestByDockerLib WriteImageToDaemon Usually these fail since we stop dockerd, but should be conditionalized. |
afbjorklund
added
kind/bug
|
@afbjorklund on From what I understand, it's nontrivial to get the name of the image that's been loaded in from a tarball, and this could be a workaround. |
|
I guess it needs some kind of horrible workaround like that, as long as the behaviour continues. Could file a bug about it upstream, but I'm not even sure where. Is it in podman, or is it cri-o ? |
|
@priyawadhwa : it seems like the only way out of this, is to add the "docker.io" prefix to the images in the cache. For the local images, I'm not sure it is possible. It seems to be adding the prefix anyway, even if re-tagging them. Upstream seems to be hacking the localhost/ prefix into crictl instead, so that it can be used always (in cri-o) cri-o/cri-o#3309 (1.19+) |
Images added to
minikubeviaminikube cache addaren't accessible to the nodes whencrioruntime is used.Steps to reproduce the issue:
minikube start --container-runtime=criodocker pull busybox && docker tag busybox my-local-imageminikube cache add my-local-imagekubectl run --image my-local-image bug --image-pull-policy=IfNotPresent -- echo "hello world"Full output of failed command:
As can be seen here, k8s tries to pull the image instead of just using it from the cache.
The expected behaviour is that the image will just be available instantly without pulling. It works as expected with
dockerandcontainerdruntimes, but not withcrio.Full output of
minikube startcommand used, if not already included:The text was updated successfully, but these errors were encountered: