support sub compaction to speed up large compaction

[bobotu]

Use sub compaction to avoid write stall due to large L1 to L2 compaction block L0 compaction, and speed up L0 to L1 compaction.

[coocood]

Can we only use the bottom level bounds to split the compaction?
And since the bottom level is usually 10x size of the top level, we can ignore the data size in top level, then we don't need to implement approximate size in range.

[bobotu]

@coocood This heuristic algorithm is adopted from RocksDB.

1. Select boundaries based on the natural boundary of input levels/files.
   • first and last key of L0 files
   • first and last key of non-0, non-last levels
   • first key of each SST file of the last level
2. Sort boundaries and unique.
3. Use ApproximateSize to estimate the data size in each boundary.
4. Merge boundaries to eliminate empty and smaller-than-average ranges.
   • find the average size in each range
   • starting from beginning greedily merge adjacent ranges until their total size exceeds the average

We should consider the size of each input file, so we cannot ignore the bounds of L0 files. Because each sst file has nearly the same size, each boundary added in step 1 represents like a data distribution in the whole input.

Supposing L0 inputs [a, d] [c, d] [e, j] and L1 inputs [a, f] [f, i] [i, k]. The result of step 2 is [a, c] [c, d] [d, e] [e, f] [f, i] [i, j] [j, k]. As you can see, the large range [a, f] is split into many smaller ranges.

Then we estimate the size of each small range, compute the estimated size of each sub compaction. Merge small ranges into a larger one.

This algorithm is not so determinate, because we cannot split the whole input equally without iterate over them, so we use some heuristic rules here, and it worked out well so far.

BTW, RocksDB disabled sub compaction for non-L0 levels, but I enabled this for L1 when it touches more than 10 sst, otherwise it may block the L0 -> L1 compaction.

[ngaut]

Ping @coocood

[coocood]

@bobotu
We can reserve the size estimation and remove the L0 bounds.

[coocood]

LGTM