sink to external storage (cdc) depends on local clock to generate file/dir name is not reliable for distributed system

[zhangjinpeng87]

Description

Sink to external storage will output DDL and DML events to files and then upload them to external storage like S3. TiCDC's path generator https://github.com/pingcap/tiflow/blob/master/pkg/sink/cloudstorage/path.go use a local monotonic clock's time as dir/file names which is not reliable under some cases, for example when a changfeed was handled by CDC node1, when CDC node1 crashed or restarted and this changfeed was re-scheduled to CDC node2, if there is clock drift between CDC node1 and CDC node2, from the consumer's perspective, there might be a time rewind issue which may cause the consumer missing some data.

|-------------|                               |-------------|
| CDC Node1   |                               | CDC Node2   |
|-------------|                               |-------------|
          |                                          |
          |        |---------------------------|     |
          |-->     |  External Storage like S3 |  <--|
                   |---------------------------|

file-20231227-084133-xxx(generated by cdc node1)
file-20231227-084132-xxx(generated by cdc node2 after cdc node1 crashed, consumer may skip this file since the consumer checkpoint may reached 20231227-084133 when cdc node1 crashed)

Enhancement

As a distributed system, TiCDC should use a reliable way like global monotonic timestamp to generate file/dir names. In this way, TiCDC can work as expected in case of cross region deployment or there is clock drift between different nodes, and other extreme cases.

Alternatives

Open NTP to keep the clock drift between nodes under some threshold like 500ms, and make sure the changefeed wait for a safe time range after it rescheduled to other nodes. This make TiCDC has a strong dependency with these preparations which is not a good design for a distributed system.

[zhangjinpeng87]

#10351 use pd clock as a short term fix.