Use HTTP cache mechanisms for efficient cache update

[nainformatique]

Hello,
First thanks a lot for your work on hugo!

I am using a lot of remote content for a hugo website and I ran into two cache issues:

• It seems that empty responses (4XX, 5XX or tcp failures?) are cached. A resource was unavailable temporary at build time (I don ̛t recall if it was 4XX, 5XX or tcp fail) and hugo cached the empty response.
• I can set an expiration date for my cache. But neverchanging files or frequently changing ones will be updated at the same rate.

For bandwidth efficiency and up to date HTTP resources, hugo could:

• Make a HEAD request to check if the online resource is newer than the cached one. (And update it if needed)
• Use the cached resource if the online is down or unfetchable (and do not overwrite it with empty answer)

If this check took place after the current cache expiry system, it may not break any expected behavior?
Do you think this would be a helpful feature to add? If not I could try to do it within a function or partial.

Thanks for your time!

First proposed on discourse

[bep]

It seems that empty responses (4XX, 5XX or tcp failures?) are cached. A resource was unavailable temporary at build time (I don ̛t recall if it was 4XX, 5XX or tcp fail) and hugo cached the empty response.

The only responses that gets cached are 200 and 404, which I think make sense in most situations.

[bep]

Make a HEAD request to check if the online resource is newer than the cached one. (And update it if needed)

The file cache we use are relatively simple and we use the timestamps on the file to determine if it has expired. This works for that purpose, but I don't think it works for the above purpose.

I suspect that an Etag based approach would be simpler to implement, but I'm not totally sure.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.