[Bug]: Sync download blobs already available in the local repository

[AlbanBedel]

zot version

v2.1.1

Describe the bug

The sync code always download every blobs behind a tag even if they are already available locally.

To reproduce

1. Setup zot as pass through cache
2. Inspect an image index, for example using skopeo inspect docker://registry.example.com/repo/image:v1.2.3
3. Inspect one of image referenced in the index, for example using skopeo inspect docker://registry.example.com/repo/image:v1.2.3-arm64
4. Notice how slow it is although all blobs are already in storage, or check the repo/image/.sync/ directory to see that it download existing blobs again.

Expected behavior

Blobs from the same repository are not downloaded again when already available in local storage.

Screenshots

No response

Additional context

No response

[rchincha]

@AlbanBedel We take into account the case where images are being tagged with the same tag, meaning image with digest1 tagged with "tag1", oops!, pushed image with digest2 but retagged as same "tag1".

That said, we do make sure blobs/manifests don't get copied over if they already exist locally.

Is this not what you are seeing?

[AlbanBedel]

The case here is to pull an index tag that references blobs A,B,C and D. Then pull an image tag that reference only A and B. In this case A and B get downloaded again, with larger blobs this is very obvious alone from the time it takes.

[andaaron]

Hi @AlbanBedel, are those docker images or OCI images at docker://registry.example.com/repo/image? Or can you send us a link to the upstream image for which this issue reproduces?

For docker format we convert the manifest and config blobs locally to OCI. It does not however explain not reusing the blobs which contain the image file system.

[AlbanBedel]

No, they are OCI artifacts, so they have OCI media type along with custom artifactType. The images I'm testing with are in a private registry, I'll see if I can reproduce it with some public images.

[eusebiu-constantin-petu-dbk]

Yes you are right, we just compare the parent digest (index digest in this case), we don't check each child...