Closed
Description
Images added to minikube via minikube cache add aren't accessible to the nodes when crio runtime is used.
Steps to reproduce the issue:
minikube start --container-runtime=criodocker pull busybox && docker tag busybox my-local-imageminikube cache add my-local-imagekubectl run --image my-local-image bug --image-pull-policy=IfNotPresent -- echo "hello world"
Full output of failed command:
$ kubectl describe pod bug
Name: bug
Namespace: default
Priority: 0
Node: minikube/172.17.0.2
Start Time: Thu, 25 Jun 2020 07:31:00 +0300
Labels: run=bug
Annotations: <none>
Status: Pending
IP: 10.1.0.6
IPs:
IP: 10.1.0.6
Containers:
bug:
Container ID:
Image: my-local-image
Image ID:
Port: <none>
Host Port: <none>
Args:
echo
hello world
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-c66kl (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-c66kl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-c66kl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8s default-scheduler Successfully assigned default/bug to minikube
Normal Pulling 7s kubelet, minikube Pulling image "my-local-image"
Warning Failed 1s kubelet, minikube Failed to pull image "my-local-image": rpc error: code = Unknown desc = Error reading manifest latest in quay.io/my-local-image: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again.</p>\n"
Warning Failed 1s kubelet, minikube Error: ErrImagePull
Normal BackOff 0s kubelet, minikube Back-off pulling image "my-local-image"
Warning Failed 0s kubelet, minikube Error: ImagePullBackOff
As can be seen here, k8s tries to pull the image instead of just using it from the cache.
The expected behaviour is that the image will just be available instantly without pulling. It works as expected with docker and containerd runtimes, but not with crio.
Full output of minikube start command used, if not already included:
😄 minikube v1.11.0 on Ubuntu 20.04
▪ KUBECONFIG=/home/mozgiii/.kube/config:/home/mozgiii/.kube/config.d/qwe
✨ Automatically selected the docker driver
👍 Starting control plane node minikube in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=16000MB) ...
🎁 Preparing Kubernetes v1.18.3 on CRI-O 1.17.3 ...
▪ kubeadm.pod-network-cidr=10.244.0.0/16
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube"