changefeed lag reached 556s when inject network partition between one of ticdc and pd leader last for 10mins

[Lily2025]

What did you do?

1、create changefeed
2、run sysbench
3、inject network partition between one of ticdc and pd leader last for 10mins

What did you expect to see?

No response

What did you see instead?

changefeed lag reached 556s

Versions of the cluster

./cdc version
Release Version: v8.0.0
Git Commit Hash: ba58ef9
Git Branch: HEAD
UTC Build Time: 2024-03-19 08:07:03
Go Version: go version go

current status of DM cluster (execute query-status <task-name> in dmctl)

No response

[Lily2025]

/type enhancement

[Lily2025]

/remove-area dm
/area ticdc

[flowbehappy]

This issue is by designed. @asddongmen will see whether it can be addressed by etcd-io/etcd#17465 (comment). If not, then I suggest we address it in long term.

[fubinzh]

/severity moderate

[asddongmen]

Based on the existing design, the delay of cdc cannot be guaranteed in this situation.

[asddongmen]

TiCDC communicates with PD to obtain the latest region information from upstream TiKV in order to establish a data capture stream for the regions. If a cdc server experiences a network partition with the PD leader, it cannot receive this information, and as a result, it cannot establish the stream. This leads to an increase in changefeed lag.
It is nearly impossible to solve this problem based on the current architecture.

Wokraround
If network partition to PD leaser issue occurs in a CDC node, you can shut down the CDC node. The tables in this node will then be transferred to another CDC node for replication.

/cc @flowbehappy @fubinzh