Description
Summary
Image cannot pulled from the local registry
Reproduction Steps
When I disable the registry, destroy-storage and enable the registry it will work but after few upgrades it will failed again, can be after one upgrade and can be much more
Introspection Report
inspection-report-20240410_115206.tar.gz
Are you interested in contributing with a fix?
Yes if I can
The issue in detailed
I am using few microk8s 1.28 and 1.29 on Ubuntu 20.04 and 22.04, I got strange behavior from one of the pods, it is always the same pod (the admin) and we got 20 pods, sometime after system upgrade when I change the pod container version/tag it failed to start with status ImagePullBackOff.
When I run describe on that pod I see kubelet Back-off pulling image "localhost:32000/admin:8.3.1989"
Try to pull the image from the local registry will failed
The image exist on the server
When pushed it to the local registry it seems that all the layer are already exist in the registry
But it will not pull even after the push
Checking with curl and I can see the image and the tag in the registry
microk8s registry is set to 40G and the default-storage directory size is 24G
Create the registry using the command: microk8s enable registry:size=40Gi
Any help will be much appreciated
In detailed commands and output:
Try to pull the image from the local registry will failed
docker pull localhost:32000/admin:8.3.1989
Error response from daemon: manifest for localhost:32000/admin:8.3.1989 not found: manifest unknown: manifest unknown
It exist on the server
docker images | grep admin
localhost:32000/admin 8.3.1989 2890fcb56766 7 weeks ago 500MB
When push it it seems that all the layer are already exist
docker push localhost:32000/admin:8.3.1989
The push refers to repository [localhost:32000/admin]
94a14ce94eef: Layer already exists
b0aed591e1f6: Layer already exists
06855bd23f0b: Layer already exists
c2fc9fe8dc06: Layer already exists
6c804dddc7ac: Layer already exists
5bc86e618a6c: Layer already exists
425d004af7ba: Layer already exists
d101c9453715: Layer already exists
8.3.1989: digest: sha256:6df1160a2880c5c69af2d949a9c06df8fb6c7de4779427bef528a1ed9a825987 size: 1995
But it will not pull even after the push
docker pull localhost:32000/admin:8.3.1989
Error response from daemon: manifest for localhost:32000/admin:8.3.1989 not found: manifest unknown: manifest unknown
looking for the container tag it exist on the registry
curl -sSL "localhost:32000/v2/admin/tags/list"
{"name":"admin","tags":["8.3.1989","8.2.1617","8.4.5138","8.4.5160","8.4.5169","8.4.5178","8.4.5173","8.4.5144","8.4.5168"]}