[Proposal] Distributed write-ahead logging

[v0y4g3r]

What problem does the new feature solve?

Shared-data architecture binds dedicated local storage to every computing node, which has good performance for small IO operations like continous insertions. But, the replication and failover of shared-data style systems is complicated and error-prone. Shared-nothing architecture, on the other hand, benifits from the inherent replication and scalibility of cloud storages like S3, but the write performance may degrade when handling continous IO due the overhead of cloud storage operations.

Thus we need a distributed WAL to bridge the gap between shared-data and shared-nothing. Data inserted first go to this WAL to ensure durability and then apply to datanode. Insertions are buffered on datanode's local storage and write to cloud storage at a time to increase throughput. The data buffered on datanode is transient. Once datanode crashes, those data can be easily recovered from WAL.

What does the feature do?

Implement a distributed write-ahead logging component. Maybe the storage should be pluggable, allowing users to choose other streaming systems like Kafka as storage.

Implementation challenges

No response

[MichaelScofield]

Related to #2722. Will implement a kafka based remote WAL. With some generic design in mind. Our further custom remote WAL service will be placed in a seperate repo, so close this issue.