[META] Generate large data corpora (1 to 10 TB) for the big5 workload

[gkamat]

Description

The big5 workload is available with two sizes of data corpora, 60 GB and a 100 GB. The latter features a more representative timestamp sequence. Larger data corpora would be appropriate for performance testing at scale. This issue is to track generation of such larger corpora.

Initially, a 1 TB corpus will be generated and tested out. OSB scaling and stability will also be relevant in this context. Once this size of corpus can be used effectively, larger corpora, up to 10 TB in size, perhaps with multiple indices will be tackled.

Task Breakdown

• Use pre-computed offset tables for indexing #519
• Streamline installation of parallel decompressors #525
• Verify stability of test runs with the big5 workload and its 1 TB data corpus opensearch-benchmark-workloads#296
• Permit large data corpora to be downloaded from the workloads repository #543
• Update big5 to use the full 1 TiB corpus #680
• Increase size of big5 data corpus to 10+ TB #585