[Feature Request] Optimise File Formats for Snapshots

[Bukhtawar]

Is your feature request related to a problem? Please describe

Snapshot uses XContent SMILE format to store snapshot metadata which bloats up due to the fact that there are repeatable keys. Though SMILE back-reference repeated keys it ends up allocating 1 byte for the reference per key which for high number of shards in the cluster ends up bloating memory.

For high number of shards we might need to paginate LIST snapshot calls which would work better if we could have the capability to perform block fetches so that we don't to buffer the entire respone in memory at the same time.

Describe the solution you'd like

Change the storage format to use position offsets that enables us to avoid serialising keys altogether and helps store data in a more compact format, stored in blocks with fixed/variable offsets enabling download of partial data to aid pagination requests

Related component

Storage:Snapshots

Describe alternatives you've considered

No response

Additional context

No response