Store Gateway: index cache with local disk based backend

[yeya24]

Is your feature request related to a problem? Please describe.
Store Gateway Index Cache supports 4 backend types: inmemory, memcached, redis and multi level. Multi level is basically a combination of an inmemory cache and a remote cache such as memcached.

Inmemory cache is good for its performance and no additional bandwidth consumption. However, its has small capacity due to memory size.

Remote cache such as memcached are usually distributed and can be seen have much larger capacity. However, querying from remote cache consumes quite a lot of bandwidth.

A middle ground could be a local filesystem/disk based cache backend so that we can achieve a much bigger capacity than the inmemory cache, while keeping better performance and lower bandwidth comsumption than remote cache.

Describe the solution you'd like
Use a disk/filesystem based Storage such as KV store as cache. It should support sized based retention (eviction) strategy to control disk space

Additional context
There is a proposal in Thanos as well thanos-io/thanos#5374

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.