Adaptive tiered storage

[v0y4g3r]

What type of enhancement is this?

Performance

What does the enhancement do?

Background

The memtable capacity must be reduced if we want to accommodate a large number of tables in GreptimeDB, which in turn results into too many small SST files in level 0. This not only degrades performance but also may reach the API rate limit when all SST files are written to OSS.

Tiered storage

We can set a threshold level, where all small SST levels under that threshold is stored on local disk, only SST files above level <threshold> will be uploaded to OSS.

That is to say, we reduce OSS API invocation by postponing the "mark WAL obsolete" from "after flush" to "compacting to <threshold>".

Benifits and deficiencies

Tiered storage proposal can help to reduce OSS API cost, also it makes large number of tables in single datanode possible.
A extra benefit would be compression since SST files reside in higher levels can be more adequately compressed.

But it also brings some issues. For example, it turns all SST files beneath level <threshold> volatile, the local storage is immediately reclaimed when pod is destroyed.
We need to either:

• mount an EBS to store SST files beneath <threshold>
• or postpone the "mark WAL obsolete" operation from "after flush" to "compacting to <threshold>"
  And both approach requires a larger local disk capacity.

Implementation challenges

The primary problem is, what kind of tables should enable tiered storage and which level should the threshold be?

More local SST files means more EBS cost but less OSS API invocation and higher compression rate. We can do a simple math and there must be a break-even point. For simplicity, we can maintain a stat of the size of each flush/compaction that happens in each level. If average output size of each flush/compaction is lower than some threshold, then this level remains in local disk.

[killme2008]

mount an EBS to store SST files beneath
or postpone the "mark WAL obsolete" operation from "after flush" to "compacting to "
And both approach requires a larger local disk capacity.

I prefer the second solution, it's much more friendly to dev operations.

[evenyag]

We could also consider using a hybrid strategy in the future:

• <threshold>
• periodically sync some SSTs to the object store (or SSTs older than N seconds)
• sync SSTs in level 1+