Use disk to buffer data before writing to sink

[liuzix]

Feature Request

Is your feature request related to a problem? Please describe:
When the downstream cannot handle the writes in time, usually due to a large throughput from the upstream, CDC will OOM because the sink buffers data only in memory.

Describe the feature you'd like:

• Use on-disk buffer in sink (basic solution).
• Make this on-disk buffer persistent along with relevant checkpointTs and resolvedTs, so that in the event of a crash or restart of the cdc process, the buffered data can be restored (optimization).

Describe alternatives you've considered:

• We could alternatively buffer data only at the sorter, but since the sorter caches unsorted data, it is not friendly to recording progress and resuming after crashes.

[zhangjinpeng87]

How to guarantee the durability of these data? How about ticdc node crash and data loss.

[overvenus]

How to guarantee the durability of these data? How about ticdc node crash and data loss.

TiCDC does not guarantee data durability that buffered in a disk. If a node crash, other nodes can pull data from TiKV again.

However, we do need to guarantee data integrity, maybe a checksum is enough just like RocksDB does.