GC not working properly

[snowmanstark]

We recently upgraded to Harbor 2.10 hoping that the GC would be working fine. Since there was a huge backlog of artifacts not cleaned, to circumvent this we created a new Harbor instance and replicated all the data. This helped us bring down our S3 bucket size from 200TB to 20TB.

After we moved to the v2.10.0, initially the GC was working fine for a month and was clearing around 300Gb every week but last couple of weeks the GC is erroring out and no GC logs show up in the Harbor UI.

Our harbor infra consists of the following:

EKS(with EFS storage) + RDS + S3

[wy65701436]

can you execute a dry-run? You can provide the incremental log of job service pod on executing an GC.

[snowmanstark]

Well after some fiddling around, I found that the GC is ongoing but there are a few issues.

• Log file becomes too huge and it can't be viewed in the browser and you need to log into the pod to see whether it's running.
• Since the log file is huge, listing all the files, etc. takes a hit. I circumvented it by overwriting the file(with empty string) and temporarily resolving the issue.
• After a couple of days in the Job Service dashboard no workers seem to be allocated for GC but the GC keeps running.
• Also the GC is really slow and it would help if we incorporated this change by @hemanth132

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

[snowmanstark]

@wy65701436 Can you please reopen this issue.

[Vad1mo]

I'll keep that closed as it is related to #12948