[Feat]: HEAD /v2/$tag/blobs/$digest could be faster

[the-sun-will-rise-tomorrow]

Is your feature request related to a problem? Please describe.

HEAD /v2/$tag/blobs/$digest requests take an unusual amount of time. Even performing repeated requests with the same $digest is slow, 200-500ms every time.

Describe the solution you'd like

I am not sure why exactly HEAD /v2/$tag/blobs/$digest requests are slow, but if it's not possible to make the underlying operation itself faster, then an in-memory cache of the results would at least speed up repeated queries.

Describe alternatives you've considered

#2589, perhaps.

Additional context

I am experimenting with efficient uploading of images with many layers (built using Nix). In my experiments, the vast majority of the time seems to be spent on Skopeo asking Zot if a layer blob is already in the registry.

[eusebiu-constantin-petu-dbk]

Hello @the-sun-will-rise-tomorrow

Do you use authorization, can you post the config you are using?

Thank you!

[the-sun-will-rise-tomorrow]

Hi!

Here is the config: #2589 (comment)

Authorization is not needed for these HEAD requests in our case (anonymous read access).

[eusebiu-constantin-petu-dbk]

Hei @the-sun-will-rise-tomorrow I investigated this issue, and I put the profiler and run our benchmark tool to see what piece of code takes so much time.

The issue is bcrypt.CompareHashAndPassword([]byte(passphraseHash), []byte(passphrase)) which seems to be by design.
Check here: https://stackoverflow.com/questions/49437359/why-bcrypt-library-comparehashandpassword-method-is-slow
Maybe try this: https://security.stackexchange.com/questions/17207/recommended-of-rounds-for-bcrypt/83382#83382

Thanks!

[the-sun-will-rise-tomorrow]

Good insight, thank you. Maybe that function could be memoized (with the cache flushed every minute to avoid indefinitely retaining sensitive information in memory)? That should fix the performance problem in theory.

[the-sun-will-rise-tomorrow]

Maybe try this: https://security.stackexchange.com/questions/17207/recommended-of-rounds-for-bcrypt/83382#83382

I am not sure how I would try this; the Cost parameter to GenerateFromPassword seems to be hard-coded in Zot: https://github.com/project-zot/zot/blob/513f2a0dc94c5f36d9d6e1455b562c192664a4c2/pkg/test/common/fs.go#L217C15-L217C42