Discard cache based on file age

[allefeld]

I want to use bkt to cache preprocessor results, like the role lessfile / lesspipe plays for less. Important here is that the cache is renewed when the file has changed. --modtime helps in principle, but has the disadvantage that cache entries stick around even though they will never be used again (if I understand correctly how the option works). My workaround is to set --ttl to the file age.

A simple proof of concept, caching md5sum:

#!/bin/bash
file=$(realpath "$1")
mtime=$(stat -c %Y "$file")
now=$(date +%s)
age=$(( now - mtime ))
bkt --ttl "${age}s" -- md5sum "$file"

I'd be interested to know whether this indeed works as intended, or can be improved.

[dimo414]

Hey @allefeld thanks for the post! I think --modtime doesn't work quite how you're picturing it, but I might also be misunderstanding your requirements.

The example above, setting a TTL based on a file's "age", will cause the cache to be invalidated at some point in the future (e.g. if the file was last edited yesterday, you're effectively passing --ttl 24h) which sort of works as a heuristic for "how likely the file will remain unmodified going forward", but otherwise is rather arbitrary and I'm not sure what it affords you. Perhaps you can help me understand how you want to use this?

By contrast --modtime affects the cache key, not its validity duration. As soon as a file is modified every cached command associated with the prior --modtime becomes invalidated, and as long as the file is unchanged (regardless of its age) the cache remains valid.

There is definitely some room for me to improve cache cleanup, including what's discussed in #27. If this is a motivating concern for you, or if there's a different class of cache leak you're observing, I should be able to take a swing at it before the end of the year.