Consider removing `blocks-storage.bucket-store.max-chunk-pool-bytes` limits

[jdbaldry]

Is your feature request related to a problem? Please describe.
blocks-storage.bucket-store.max-chunk-pool-bytes is used to configure the maximum size of the chunk pool in the store-gateway bucket store. The chunk byte pool is used to reduce allocations but having the limit enforced can cause queries to fail when the pool is exhausted. I believe the purpose of the limit being enforced is to prevent store-gateway OOM kills but since you end up needing to set the limit to a much higher value to satisfy queries (e.g. 80% of the store-gateway requested memory) you don't really gain any extra protection.

Describe the solution you'd like
Removal of the blocks-storage.bucket-store.max-chunk-pool-bytes flag and associated configuration and using a chunk pool with no maximum.

[pracucci]

I'm also dubious about this limit at all. We use memory pools in other parts of the code base and they're never limited. Unless I'm missing something, this is the only pool we limit.

[pracucci]

The PR #3830 improved it a bit. Before the chunks pool was per-tenant, after it's global (shared across all tenants).

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

I keep having mixed feelings about this. Issue still valid.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.