Harbor full scan makes almost all images in pending state preventing pulls if pull configuration enabled

[hamdikh]

Hello, we have a few questions related to how harbor full scanning.
We have more than 57K images and the global scan is taking more than 12 hours to complete.
In our configuration, we've 450 job service workers and we're having 60 parallel scans in trivy adapter (updated SCANNER_JOB_QUEUE_WORKER_CONCURRENCY for 6 trivy instances).

PS: we started experiencing these problems after the v2.7.3 upgrade, we've moved from v2.6.4 to v2.7.3 to resolve database CPU consumption issues.
We're seeing a jobservice impact on how the queuing system works as the full scan puts the images in a pending state.
This makes the images that are being queued for scanning in a hostage state, as we've also configured the prevent images with vulns from being pulled this blocks all our image pulls.

How to reproduce:
1/
2/ Have around 7K-10K images
3/ Launch the Full Scan
4/ (Optionally) Push images to haror (they need to be scanned on push)

Once launched, you will start seeing images being in a hostage situation.

We would like to know if there are any optimizations that can be done from our side.
Any help will be much appreciated.

Harbor v2.7.3
Kubernetes version: 1.25.11

[hamdikh]

I provided a little bit more context to reproduce, I hope this helps

[jthin]

Hello,

To add additional information about Hamdi's issue:

• During a full scan, when an image is in the pending queue, the last scan report seems to be deleted, and the image is left waiting to be scanned.
• Since we are preventing project teams from pulling images with a severity level, the image must be scanned to obtain a Trivy scan result, even if the image has already been scanned during the initial push or has been manually scanned within the last few days/hours.
• If there are many images in the pending queue, project teams may have to wait for hours before they can pull the image.

At the moment, we do not have the option to perform a weekly full scan, and we hope that no project team will be affected by this issue.

[abenabid]

Hello,
I've noticed that when the scan task is added to the queue, the previous scan report is removed from the database.
IMO, the previous scan report is still legitimate, so it should bet kept in the database until the scanner finishes the scan successfully.
Can this solve the image pull issue ?

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[sizowie]

Please do not close.

[hamdikh]

Issue not stale please don't close

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[jthin]

Not stale