Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-tagging images not working (and image delete then fails) #2960

Closed
srfrnk opened this issue Oct 8, 2022 · 6 comments
Closed

Re-tagging images not working (and image delete then fails) #2960

srfrnk opened this issue Oct 8, 2022 · 6 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@srfrnk
Copy link

srfrnk commented Oct 8, 2022 •
edited
Loading

What happened:
My build script generates a unique build tag for every build.
It then "load" the image into kind node.
Then it deploys a manifest using the new tag.
While running this I found that f there are no changes to the image - thus load just re-tags it... the image will no longer be available within the node resulting in a ImagePullBackOff error.
The image cannot then be deleted (crictl rmi fails to delete) and the only solution is deleting the "kind cluster" and recreating.

What you expected to happen:
I would expect re-tagged images to be available to the node.
And also - "crictl rmi ..." should be able to remove images when run inside node.

How to reproduce it (as minimally and precisely as possible):
Create Makefile with:

bug:
	- kind delete cluster
	kind create cluster

	docker build -t bug:1 .
	docker build -t bug:2 .

	kind load docker-image bug:1
	kind load docker-image bug:2

	sleep 10

	kubectl run --image bug:1 bug-1
	kubectl run --image bug:2 bug-2

	sleep 10

	kubectl describe pod bug-1
	kubectl describe pod bug-2

Create Dockerfile with:

FROM ubuntu
CMD [ "sleep","infinity" ]

Run make bug (terminal bash in folder where files are created) - see output:

kind delete cluster
Deleting cluster "kind" ...
kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.25.2) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
docker build -t bug:1 .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM ubuntu
 ---> 216c552ea5ba
Step 2/2 : CMD [ "sleep","infinity" ]
 ---> Using cache
 ---> f315e0c74ef7
Successfully built f315e0c74ef7
Successfully tagged bug:1
docker build -t bug:2 .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM ubuntu
 ---> 216c552ea5ba
Step 2/2 : CMD [ "sleep","infinity" ]
 ---> Using cache
 ---> f315e0c74ef7
Successfully built f315e0c74ef7
Successfully tagged bug:2
kind load docker-image bug:1
Image: "bug:1" with ID "sha256:f315e0c74ef7ff890066a7d50ed4ad5459fab822f9e4d2e9416b11557ee47861" not yet present on node "kind-control-plane", loading...
kind load docker-image bug:2
Image with ID: sha256:f315e0c74ef7ff890066a7d50ed4ad5459fab822f9e4d2e9416b11557ee47861 already present on the node kind-control-plane but is missing the tag bug:2. re-tagging...
sleep 10
kubectl run --image bug:1 bug-1
pod/bug-1 created
kubectl run --image bug:2 bug-2
pod/bug-2 created
sleep 10
kubectl describe pod bug-1
Name:             bug-1
Namespace:        default
Priority:         0
Service Account:  default
Node:             kind-control-plane/172.18.0.2
Start Time:       Sat, 08 Oct 2022 16:50:54 +0100
Labels:           run=bug-1
Annotations:      <none>
Status:           Running
IP:               10.244.0.2
IPs:
  IP:  10.244.0.2
Containers:
  bug-1:
    Container ID:   containerd://36d8389ed31627c0fc32c107c5ac5e30aa1994497f95b247f5ba7f2898ae33e4
    Image:          bug:1
    Image ID:       docker.io/library/import-2022-10-08@sha256:a57c5cd324238cb98fbab7a8a7bdf6198bfda946a84cb2501616d7e590277bc9
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Sat, 08 Oct 2022 16:50:55 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-slnxs (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-slnxs:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  10s   default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled         4s    default-scheduler  Successfully assigned default/bug-1 to kind-control-plane
  Normal   Pulled            3s    kubelet            Container image "bug:1" already present on machine
  Normal   Created           3s    kubelet            Created container bug-1
  Normal   Started           3s    kubelet            Started container bug-1
kubectl describe pod bug-2
Name:             bug-2
Namespace:        default
Priority:         0
Service Account:  default
Node:             kind-control-plane/172.18.0.2
Start Time:       Sat, 08 Oct 2022 16:50:54 +0100
Labels:           run=bug-2
Annotations:      <none>
Status:           Pending
IP:               10.244.0.4
IPs:
  IP:  10.244.0.4
Containers:
  bug-2:
    Container ID:   
    Image:          bug:2
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ErrImagePull
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2cmm6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-2cmm6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  10s   default-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled         4s    default-scheduler  Successfully assigned default/bug-2 to kind-control-plane
  Normal   Pulling           3s    kubelet            Pulling image "bug:2"
  Warning  Failed            2s    kubelet            Failed to pull image "bug:2": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/bug:2": failed to resolve reference "docker.io/library/bug:2": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed            2s    kubelet            Error: ErrImagePull
  Normal   BackOff           2s    kubelet            Back-off pulling image "bug:2"
  Warning  Failed            2s    kubelet            Error: ImagePullBackOff

Further to that... when running docker exec -it kind-control-plane bash:

root@kind-control-plane:/# crictl img
IMAGE                                      TAG                  IMAGE ID            SIZE
docker.io/kindest/kindnetd                 v20220726-ed811e41   d921cee849482       25.8MB
docker.io/kindest/local-path-helper        v20220607-9a4d8d2a   d2f902e939cc3       2.86MB
docker.io/kindest/local-path-provisioner   v0.0.22-kind.0       4c1e997385b8f       17.4MB
docker.io/library/bug                      1                    f315e0c74ef7f       80.4MB
docker.io/library/bug                      2                    f315e0c74ef7f       80.4MB
registry.k8s.io/coredns/coredns            v1.9.3               5185b96f0becf       14.8MB
registry.k8s.io/etcd                       3.5.4-0              a8a176a5d5d69       102MB
registry.k8s.io/kube-apiserver             v1.25.2              9eebd178240fb       76.5MB
registry.k8s.io/kube-controller-manager    v1.25.2              d846cf6e13f87       64.5MB
registry.k8s.io/kube-proxy                 v1.25.2              817f51628b39c       63.3MB
registry.k8s.io/kube-scheduler             v1.25.2              bbff39abe40b4       51.9MB
registry.k8s.io/pause                      3.7                  221177c6082a8       311kB
root@kind-control-plane:/# 
root@kind-control-plane:/# crictl rmi docker.io/library/bug:2
ERRO[0000] no such image docker.io/library/bug:2        
FATA[0000] unable to remove the image(s)                
root@kind-control-plane:/# crictl rmi docker.io/library/bug:1
Deleted: docker.io/library/bug:1
Deleted: docker.io/library/bug:2
root@kind-control-plane:/# crictl img
IMAGE                                      TAG                  IMAGE ID            SIZE
docker.io/kindest/kindnetd                 v20220726-ed811e41   d921cee849482       25.8MB
docker.io/kindest/local-path-helper        v20220607-9a4d8d2a   d2f902e939cc3       2.86MB
docker.io/kindest/local-path-provisioner   v0.0.22-kind.0       4c1e997385b8f       17.4MB
docker.io/library/bug                      2                    f315e0c74ef7f       80.4MB
registry.k8s.io/coredns/coredns            v1.9.3               5185b96f0becf       14.8MB
registry.k8s.io/etcd                       3.5.4-0              a8a176a5d5d69       102MB
registry.k8s.io/kube-apiserver             v1.25.2              9eebd178240fb       76.5MB
registry.k8s.io/kube-controller-manager    v1.25.2              d846cf6e13f87       64.5MB
registry.k8s.io/kube-proxy                 v1.25.2              817f51628b39c       63.3MB
registry.k8s.io/kube-scheduler             v1.25.2              bbff39abe40b4       51.9MB
registry.k8s.io/pause                      3.7                  221177c6082a8       311kB
root@kind-control-plane:/# crictl rmi docker.io/library/bug:2
ERRO[0000] no such image docker.io/library/bug:2        
FATA[0000] unable to remove the image(s)                
root@kind-control-plane:/# crictl rmi docker.io/library/bug:1
ERRO[0000] no such image docker.io/library/bug:1        
FATA[0000] unable to remove the image(s)

Anything else we need to know?:

Environment:

  • kind version: (use kind version): kind v0.16.0 go1.18.1 linux/amd64
  • Kubernetes version: (use kubectl version): Client Version: v1.25.2
  • Docker version: (use docker info): 20.10.14
  • OS (e.g. from /etc/os-release): Ubuntu 22.04.1 LTS (Jammy Jellyfish)
@srfrnk srfrnk added the kind/bug Categorizes issue or PR as related to a bug. label Oct 8, 2022
@BenTheElder
Copy link
Member

Can you try from the latest kind sources?
This sounds like #2955

@srfrnk
Copy link
Author

srfrnk commented Oct 9, 2022

I can confirm issue is resolved entirely under latest sources.
Thanks. Will wait for next release.
Used the following to verify https://github.com/srfrnk/kind/tree/issue-2960/bug

@srfrnk srfrnk closed this as completed Oct 9, 2022
@BenTheElder
Copy link
Member

cc @aojea broken image loading behavior probably warrants a quicker follow-up release than previously planned

@aojea
Copy link
Contributor

aojea commented Oct 9, 2022

but this bug was always there, wasn't it?

@BenTheElder
Copy link
Member

BenTheElder commented Oct 9, 2022 •
edited
Loading

No, previously we would just load the image again using the normal loading path. It's a recent change that we attempt to re-tag instead when only the tag changed.

@aojea
Copy link
Contributor

aojea commented Oct 9, 2022

oh, then we should fix it

jmsanders added a commit to dagster-io/dagster that referenced this issue Oct 12, 2022
@jmsanders
Downgrade kind to 0.15.0
8718a0c 
Kind 0.16.0 breaks retagging images:

kubernetes-sigs/kind#2960

This downgrades us to 0.15.0 to attempt to get our k8s integration tests
running again.
jmsanders added a commit to dagster-io/dagster that referenced this issue Oct 13, 2022
@jmsanders
Downgrade kind to 0.15.0
5b5ade4 
Kind 0.16.0 breaks retagging images:

kubernetes-sigs/kind#2960

This downgrades us to 0.15.0 to attempt to get our k8s integration tests
running again.
jmsanders added a commit to dagster-io/dagster that referenced this issue Oct 18, 2022
@jmsanders
Downgrade kind to 0.15.0
d5402f5 
Kind 0.16.0 breaks retagging images:

kubernetes-sigs/kind#2960

This downgrades us to 0.15.0 to attempt to get our k8s integration tests
running again.
jmsanders added a commit to dagster-io/dagster that referenced this issue Oct 20, 2022
@jmsanders
Downgrade kind to 0.15.0
2e44026 
Kind 0.16.0 breaks retagging images:

kubernetes-sigs/kind#2960

This downgrades us to 0.15.0 to attempt to get our k8s integration tests
running again.
jmsanders added a commit to dagster-io/dagster that referenced this issue Oct 20, 2022
@jmsanders
Downgrade kind to 0.15.0
1811316 
Kind 0.16.0 breaks retagging images:

kubernetes-sigs/kind#2960

This downgrades us to 0.15.0 to attempt to get our k8s integration tests
running again.
jmsanders added a commit to dagster-io/dagster that referenced this issue Oct 20, 2022
@jmsanders
Downgrade kind to 0.15.0
774312d 
Kind 0.16.0 breaks retagging images:

kubernetes-sigs/kind#2960

This downgrades us to 0.15.0 to attempt to get our k8s integration tests
running again.
@lizardruss lizardruss mentioned this issue Oct 21, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@BenTheElder @srfrnk @aojea