stability (cdc) improve the stability of TiCDC

[zhangjinpeng87]

How to define stability of TiCDC?

TiCDC as a distributed system, it should continuously provide service with predictable replication lag under any reasonable situations like single TiCDC node failure, single upstream TiKV node failure, single upstream PD node failure, planned rolling upgrade/restart of TiCDC or upstream TiDB cluster, temporarily network partition between one CDC node and other CDC nodes, etc. TiCDC should recover the replication lag by itself quickly and tolerant different resilience cases.

Expected replication lag SLO under different cases

Category	Case Description	Expected Behavior
Planned Operations	Rolling upgrade/restart TiCDC	replication lag < 5s
	Scale-in/scale-out TiCDC	replication lag < 5s
	Rolling upgrade/restart upstream PD	replication lag < 5s
	Rolling upgrade/restart upstream TiKV	replication lag < 10s
	Scale-in/scale-out upstream TiDB	replication lag < 5s
	Rolling upgrade/restart downstream Kafka brokers	begin to sink ASAP kafka resumed
	Rolling upgrade/restart downstream MySQL/TiDB	begin to sink ASAP kafka resumed
Unplanned Failures	Single TiCDC node (random one) permanent failure	replication lag < 1min
	Single TiCDC node temporarily failure for 5 minutes	replication lag < 1min
	PD leader permanent failure or temporarily failure for 5 minutes	replication lag < 5s
	Network partition between one TiCDC node and PD leader for 5 minutes	replication lag < 5s
	Network partition between one TiCDC node and other TiCDC nodes	replication lag < 5s

Principle of prioritizing TiCDC stability issues

We deal with TiCDC stability issues as following priorities

• If this issue is related to data correctness or data completeness, it should be top priority(P0), we must fix them ASAP and cherry-pick them to other LTS versions.
• When this issue happens, the replication/changefeed can be stuck/failed and can't recover by TiCDC itself. We should treat this category as P1 priority. Because if such an issue happens, it means we/users should handle it manually.
• When this issue happens, the replication lag will increase unexpectedly a lot. This type of issue is not that critical as P1 issues, but will breach the SLO of TiCDC, we treat them as P2 issues.
• Enhancements like reducing resource usage, ... we treat them as P3 priority.

Tasks Tracking

• (P0) Data correctness
  • sink-to-mysql (CDC) missing explicit timezone session variable may cause wrong TIMESTAMP value #10393 @zhangjinpeng87
  • Kafka/Avro: truncated columns not handled correctly #8096 @asddongmen
• (P1) Changefeed might be stuck or fail
  • TiCDC changefeed's resolved-ts is stuck #10157 @sdojjy
  • puller(ticdc): fix resolvedTs get stuck when region split and merge #10488 @sdojjy
  • conflict detector(cdc): fix potential dead lock when a transaction generated multiple same hash #10335 @zhangjinpeng87
  • cdc gets wrong pd gc safe point and causes changefeed exceeds gc ttl wrongly #10463 @CharlesCheung96
  • TiCDC may incorrectly cancel gRPC streams to TiKV, causing the resolvedTs to become stuck #10239 @asddongmen
  • create table ddl event: 'create table otherSchema.tableA like tableB' raise changefeed closed #10545
• (P2) replication lag increase issues
  • sink to mysql (cdc) workload skew issue #10341 @zhangjinpeng87
  • [TiCDC MySQL DDLSink] reliable async add index when downstream is not TiDB #10267
  • Optimize memory usage of RowChangedEvent #10386 (potential OOM issue) @lidezhu
  • TiCDC incremental scans should be limited with required ranges tikv/tikv#16362 @hicqu
  • [RFC] large transactions block watermark and result in replication lag increasing #10525 @zhangjinpeng87
• (P3) enhancement
  • (TiCDC) back pressure mechanism #10298
  • sink to external storage (cdc) depends on local clock to generate file/dir name is not reliable for distributed system #10374 @CharlesCheung96
  • sink-to-cloudstorage (cdc) use slice after initialized #10350 @zhangjinpeng87
  • csv/canal encoder (cdc) never shrink the memory buffer #10416 @zhangjinpeng87
  • [RFC] Proposal of simplify CDC timezone handling #10423 @zhangjinpeng87
  • redo (cdc) remove inefficient redoCache to simply the code and reduce maintenance burden #10467 @zhangjinpeng87
  • mysql (ticdc): Improve the performance of the mysql sink by refining the transaction event batching logic #10466 @hongyunyan

[flowbehappy]

#10157 @zhangjinpeng1987 resolve ts gets stuck issue