Proxy cache serves the stale digests for the latest tags

[hoptical]

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior:
When a new image is pushed on the same tag (e.g., latest), the harbor proxy cache doesn't update to the latest digest and still serves the stale one. This behavior persists even when all artifacts are removed from the harbor panel unless the artifact record is deleted from Redis.

It's expected that Harbor serves the most recent digest for the same tag.

Steps to reproduce the problem:

1. Create a project regarding the desired repository; ghcr here.
2. Pull the repository: ghcr.io/snapp-incubator/snappcloud-hub-catalog:latest
3. Push a new image with the latest tag
4. Pull the image, and you will see that the previous digest is pulled.

Versions:
Please specify the versions of following systems.

• harbor version: 2.7.0
• docker engine version: 20.10.17
• docker-compose version: 1.18.0

Additional context:

• Harbor config files: harbor.yaml:

  hostname: xxx
  
  http:
   port: 80
  https:
     port: 443
     private_key: xxx
     certificate: xxx
  harbor_admin_password: xxx
  database:
   password: xxx
   max_idle_conns: 100
   max_open_conns: 1000
  data_volume: /data
  jobservice:
   max_job_workers: 10
  notification:
   webhook_job_max_retry: 10
  chart:
   absolute_url: disabled
  log:
   level: info
   local:
     rotate_count: 50
     rotate_size: 200M
     location: /var/log/harbor
  _version: 2.7.0
  proxy:
   http_proxy: xxx
   https_proxy: xxx
   no_proxy: xxx
   components:
     - core
     - registry
  
  metric:
   enabled: true
   port: 9090
   path: /metrics
  
  cache:
   # not enabled by default
   enabled: true
   # keep cache for one day by default
   expire_hours: 24

• Log files: Attached; the desired digest is sha256:6920278b1beca46c117c655cdf7edd0932243f95620de35d0ea1e41d27c00335
  

[MinerYang]

Hi @hoptical
Could you check the log if the latest image fetched from the remote repo? Otherwise, if the digest has already presents locally, the newest artifact not been pushed and UI would not updated.
Additional, could you please provide more logs of your harbor?

[stonezdj]

Harbor always serves the client with the latest image. except:

1. The upstream registry is offline or has rate-limit setting
2. Is it an image index? for the image index, its digest could be different from the original server because Harbor always proxies the image partially (eg: pull for a specific platform), and the digest in the proxy server differs from the original digest.

You should review the Harbor core log to see what is the process logic.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[hoptical]

@MinerYang @stonezdj Thanks for the following up. I have attached a screenshot of the logs of a stale artifact which Harbor keeps pulling even by deleting the record from Harbor registry UI. The real latest artifact is not pulled unless the below records are deleted from Redis:

"cache:manifestlist:ghcr/snapp-incubator/snappcloud-hub-catalog:latest"
"cache:manifestlist:ghcr/snapp-incubator/snappcloud-hub-catalog:latest:contenttype"

@stonezdj We haven't set any rate limit on the ghcr registry and as far as I know it's not an image index.

[stonezdj]

Correct, it seems that the redis cache is not refreshed once the original server's image is changed.
"cache:manifestlist:ghcr/snapp-incubator/snappcloud-hub-catalog:latest"
"cache:manifestlist:ghcr/snapp-incubator/snappcloud-hub-catalog:latest:contenttype"

[stonezdj]

If pull with ctr, the error would like that

ERROR: failed to solve: xxx.xxx.xxx/dockerhub-proxy-cache/library/golang:1.21: 
failed to read expected number of bytes: unexpected EOF

[hoptical]

@stonezdj I didn't get it. Is it now solved by your commit?

[stonezdj]

@stonezdj I didn't get it. Is it now solved by your commit?

Yes, fixed

[hoptical]

@stonezdj I didn't get it. Is it now solved by your commit?

Yes, fixed

Thanks. Can we know about the release version In which the change is applied?