Tracking issue: streaming building/reading SST

[ShiKaiWi]

Describe This Problem

Current building/reading SST is not in a streaming way, that is to say, all the content will be buffered in the memory before dumping into the underlying storage for building procedure and all the content will be fetched into memory before feeding to the upside merge iterators. Such way to bulid/read sst may lead to high memory consumption.

Proposal

• Write SST in a stream way Build sst file in more resource friendly way #486
• Read SST in a stream way.

Additional Context

[jiacai2050]

Write is duplicated with #486

[jiacai2050]

The hard part of this new procedure is partial failure, how will you record partial failed tasks to avoid leave temp files?

[ShiKaiWi]

Write is duplicated with #486

Ok. #486 has been linked to this tracking issue as one sub task issue.

[ShiKaiWi]

The hard part of this new procedure is partial failure, how will you record partial failed tasks to avoid leave temp files?

Do clearing jobs, e.g. removing temp files or abort multipart upload, in the drop method. As for the leaked temp files, I guess we can just let it go for now.