[Bug]: Docker.io rancher-agent:v2.7.2 Tag Fails to Download all OS versions. manifest unknown output #1394
Comments
|
@bfreidank thanks for trying out zot. Noticed that you are using v1.4.3. Do you want to try our latest *-rc release v2.0.0-rc3? |
|
@rchincha Thank you for your suggestion and quick reply. However the issue persists in v2.0.0-rc3. The procedure followed and output while attempting to sync can be found below: Pulled down v2.0.0-rc3 and restarted the service: Confirmed the binary file version: Logged into server and attempted to pull the tag down again with the following commands: The pull failed in the same location with the following output: Getting image list signatures The output of zot.log includes the following:
Attempted to pull the tag down again with podman using the following command: The output of podman pull produces the same error output: The output of zot.log includes the following:
|
|
Indeed it is broken.
|
Currently, when pushing an image, validation is performed to check that a layer/blob in the manifest already exists. For non-distributable layers, that check needs to be skipped. Fixes issue project-zot#1394 Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
|
#1404 fixes this issue now. Our next rc release will include this patch |
Currently, when pushing an image, validation is performed to check that a layer/blob in the manifest already exists. For non-distributable layers, that check needs to be skipped. Fixes issue project-zot#1394 Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
Currently, when pushing an image, validation is performed to check that a layer/blob in the manifest already exists. For non-distributable layers, that check needs to be skipped. Fixes issue #1394 Signed-off-by: Ramkumar Chinchani <rchincha@cisco.com>
|
@bfreidank v2.0.0-rc4 has been released, pls try and let us know if it fixes your issue. |
|
Hello @rchincha Testing a pull with podman does still produce the manifest unknown error as above. Retesting with skopeo with the following command did successfully complete: Output of Skopeo copy: Retesting podman pull of image with the following command fails: Output of podman copy: Trying to pull server-fqdn:5000/docker/rancher/fleet-agent:v0.6.0... Conclusion: |
|
@bfreidank we would rather fix this issue. What version of podman are you using? |
|
Hello @rchincha The version of podman that I'm using is the following: For reference the sync section for the configuration file being used is the following: |
|
Pls see (for configuring docker as an upstream) Try your sync URLs as docker.io/library and docker.io/library/rancher |
|
Thank you for notifying of the changes needed in the sync registries section for docker as an upstream. When tested, I noted that the sync failed when the sync URLs were set to docker.io/library and docker.io/library/rancher. I adjusted the registries section to the following: Tested podman pull for rancher-agent:v2.7.2 using the following command fails: Output of podman pull Tested podman pull for ubuntu:latest using the following command completes: Output of podman pull Tested podman pull for grafana/grafana:latest using the following command completes: Output of podman pull Comments: |
|
Hello, I made a PR fixing this: #1421 |
|
@bfreidank we will cut another RC release shortly, but if you want to try this fix sooner, feel free to build/deploy zot (from top of |
|
Hello, I then built a new rancher cluster and set the registries section to point to the zot server and successfully deployed while pointed to the zot server. Thank you for working through this and getting it squared away! |
|
Closing issue. Fix currently merged into main branch. Changes expected to be implemented on next RC. |
zot version
v1.4.3
Describe the bug
Setup onDemand repository to point to https://docker.io/rancher which fails to sync all OS types/arch and produces the following output:
Trying to pull server-fqdn:5000/rancher/rancher-agent:v2.7.2... Error: initializing source docker://server-fqdn:5000/rancher/rancher-agent:v2.7.2: reading manifest v2.7.2 in server-fqdn:5000/rancher/rancher-agent: manifest unknown: manifest unknownThe output of zot.log includes the following:
2023-04-28 13:31:13 skipping syncing on demand rancher/rancher-agent from [https://docker.io] registry because it's filtered out by content config
2023-04-28 13:31:13 couldn't get upstream image docker.io/rancher/rancher-agent:v2.7.2 notary references
2023-04-28 13:31:13 failed to stat blob
2023-04-28 13:31:13 cache: not found
2023-04-28 13:31:13 cache: not found
2023-04-28 13:31:13 cache: not found
2023-04-28 13:31:13 cache: not found
2023-04-28 13:31:13 cache: not found
2023-04-28 13:31:13 failed to stat blob
2023-04-28 13:31:13 failed to stat blob
2023-04-28 13:31:13 cache: not found
2023-04-28 13:31:13 copying image docker.io/rancher/rancher-agent:v2.7.2 to /tmp/zot/rancher/rancher-agent/.sync/dae5b3da-dc51-4799-a530-7e81cafb26d5
2023-04-28 13:31:13 pushing synced local image /tmp/zot/rancher/rancher-agent/.sync/dae5b3da-dc51-4799-a530-7e81cafb26d5/rancher/rancher-agent:v2.7.2 to local registry
2023-04-28 13:31:13 sync routine: docker.io/rancher/rancher-agent:v2.7.2 exited
2023-04-28 13:31:20 HTTP API
2023-04-28 13:31:20 image not found, trying to get image rancher/rancher-agent:v2.7.2 by syncing on demand
2023-04-28 13:31:20 syncing image rancher/rancher-agent:v2.7.2
2023-04-28 13:31:20 skipping syncing on demand rancher/rancher-agent from [https://registry.k8s.io] registry because it's filtered out by content config
2023-04-28 13:31:20 skipping syncing on demand rancher/rancher-agent from [https://docker.io] registry because it's filtered out by content config
2023-04-28 13:31:20 syncing on demand with [https://docker.io/rancher]
2023-04-28 13:31:20 couldn't find any cosign signature from https://docker.io/rancher/v2/rancher-agent/manifests/sha256-64893bc04d326732e2d429abbd6d0f7f78ec1d34c65da011472174bbeff68359.sig, status code: 404 skipping
2023-04-28 13:31:20 couldn't get upstream image docker.io/rancher/rancher-agent:v2.7.2 cosign manifest
2023-04-28 13:31:20 couldn't find any notary signature from https://docker.io/rancher/oras/artifacts/v1/rancher-agent/manifests/sha256:64893bc04d326732e2d429abbd6d0f7f78ec1d34c65da011472174bbeff68359/referrers, status code: 404, skipping
2023-04-28 13:31:20 couldn't get upstream image docker.io/rancher/rancher-agent:v2.7.2 notary references
2023-04-28 13:31:20 copying image docker.io/rancher/rancher-agent:v2.7.2 to /tmp/zot/rancher/rancher-agent/.sync/db6bbc7a-b7d8-4512-8bc8-45c256d95ebb
2023-04-28 13:32:09 pushing synced local image /tmp/zot/rancher/rancher-agent/.sync/db6bbc7a-b7d8-4512-8bc8-45c256d95ebb/rancher/rancher-agent:v2.7.2 to local registry
2023-04-28 13:32:09 failed to stat blob
2023-04-28 13:32:09 cache: not found
2023-04-28 13:32:09 failed to stat blob
2023-04-28 13:32:09 couldn't read blob
2023-04-28 13:32:09 error while pushing synced cached image /tmp/zot/rancher/rancher-agent/.sync/db6bbc7a-b7d8-4512-8bc8-45c256d95ebb/rancher/rancher-agent:v2.7.2
2023-04-28 13:32:10 sync routine: starting routine to copy image docker.io/rancher/rancher-agent:v2.7.2, cause err: blob: not found
2023-04-28 13:32:10 HTTP API
2023-04-28 13:32:10 couldn't find any cosign signature from https://docker.io/rancher/v2/rancher-agent/manifests/sha256-64893bc04d326732e2d429abbd6d0f7f78ec1d34c65da011472174bbeff68359.sig, status code: 404 skipping
2023-04-28 13:32:10 couldn't get upstream image docker.io/rancher/rancher-agent:v2.7.2 cosign manifest
2023-04-28 13:32:10 couldn't find any notary signature from https://docker.io/rancher/oras/artifacts/v1/rancher-agent/manifests/sha256:64893bc04d326732e2d429abbd6d0f7f78ec1d34c65da011472174bbeff68359/referrers, status code: 404, skipping
2023-04-28 13:32:10 couldn't get upstream image docker.io/rancher/rancher-agent:v2.7.2 notary references
2023-04-28 13:32:10 copying image docker.io/rancher/rancher-agent:v2.7.2 to /tmp/zot/rancher/rancher-agent/.sync/c0e89636-16b0-4da0-a581-bb8572a5de19
2023-04-28 13:32:55 pushing synced local image /tmp/zot/rancher/rancher-agent/.sync/c0e89636-16b0-4da0-a581-bb8572a5de19/rancher/rancher-agent:v2.7.2 to local registry
2023-04-28 13:32:55 failed to stat blob
2023-04-28 13:32:55 cache: not found
2023-04-28 13:32:55 failed to stat blob
2023-04-28 13:32:55 couldn't read blob
2023-04-28 13:32:55 error while pushing synced cached image /tmp/zot/rancher/rancher-agent/.sync/c0e89636-16b0-4da0-a581-bb8572a5de19/rancher/rancher-agent:v2.7.2
2023-04-28 13:32:56 sync routine: error while copying image docker.io/rancher/rancher-agent:v2.7.2
2023-04-28 13:32:56 sync routine: docker.io/rancher/rancher-agent:v2.7.2 exited
Trying the sync the repo through skopeo:
skopeo --insecure-policy copy --src-tls-verify=false --multi-arch=all --format=oci docker://rancher/rancher-agent:v2.7.2 docker://server-fqdn:5000/rancher/rancher-agent:v2.7.2Output from command:
Getting image list signatures
Copying 5 of 5 images in list
Copying image sha256:cc4f4e6cdbafe7747f7819bab4f8ba8c360da05f5deb8d18946d8cb345cf6150 (1/5)
Getting image source signatures
Copying blob c1284e7e517e skipped: already exists
Copying blob 41fdb87eda23 skipped: already exists
Copying blob 6fb9de569c10 skipped: already exists
Copying blob ac65d1108f1e skipped: already exists
Copying blob 31a2e55fffbc skipped: already exists
Copying blob c482dda145e8 skipped: already exists
Copying blob 46e9ae0c71ef skipped: already exists
Copying config 312a19d49d done
Writing manifest to image destination
Storing signatures
Copying image sha256:e9619272120ab1b5217e03a518640a45c80d60220bc286fcbc04ca796d57423a (2/5)
Getting image source signatures
Copying blob 6b45531e87f6 skipped: already exists
Copying blob 4a4d2ddbaf32 skipped: already exists
Copying blob f675822dd2ec skipped: already exists
Copying blob 4e22d9981baf skipped: already exists
Copying blob 66c37be3dde5 skipped: already exists
Copying blob f9449bae4498 skipped: already exists
Copying blob bd3a8211e17b skipped: already exists
Copying config bf40fe8fd5 done
Writing manifest to image destination
Storing signatures
Copying image sha256:7567ce3e272b1592f142567c34e3d93abd1ed5197c30de6035da3738bbf62e4c (3/5)
Getting image source signatures
Copying blob f9aba08f6c1a done
Copying blob 4e2ae00553a0 done
Copying blob 052f2692335a done
Copying blob ec4391989485 done
Copying blob 1a0ea2886e06 done
Copying blob deaa6ebe0107 done
Copying blob 53e4b58dccb8 done
Copying blob 8750b0b24e63 done
Copying blob dd8dd609278c done
Copying blob 6c2d878be457 done
Copying blob 21b9ac665c20 done
Copying blob 698dca4a3067 done
Copying blob 9e664a8fbd13 done
Copying blob 1f3d3bd2466f done
Copying blob 9718c9a34a28 done
Copying blob b41414cefddf done
Copying blob 28ac86cb10e1 done
Copying config 4d85eb2247 done
Writing manifest to image destination
FATA[0009] copying image 3/5 from manifest list: writing manifest: uploading manifest sha256:0035a2729cb8ae9d1bbe189a62ceb7acec0e66e0a0b26b065d4b724451eefb9a to server-fqdn:5000/rancher/rancher-agent: blob unknown to registry
Syncing only amd64 linux agent completes successfully
skopeo --insecure-policy copy --src-tls-verify=false --format oci docker://rancher/rancher-agent:v2.7.2 docker://server-fqdn:5000/rancher/rancher-agent:v2.7.2Reviewing the size of each of the OS versions for the rancher-agent:v2.7.2 tag shows that there are two large windows versions. There is one large layer within the larger Windows version of the tag which fails to move properly.
This leads me to believe that the larger layer isn't being handled by Zot correctly and the manifest is also not being read properly by Zot for the converted registry/tag.
To reproduce
{ "urls": [ "https://docker.io/rancher" ], "onDemand": true, "tlsVerify": true, "maxRetries": 3, "retryDelay": 1, "content": [ { "prefix": "**", "destination": "/rancher" } ] }sudo podman pull server-fqdn:5000/rancher/rancher-agent:v2.7.2Error: initializing source docker://server-fqdn:5000/rancher/rancher-agent:v2.7.2: reading manifest v2.7.2 in server-fqdn:5000/rancher/rancher-agent: manifest unknown: manifest unknownExpected behavior
onDemand sync to work as expected
Screenshots
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: