checkpoint stucks for over 1.5h, results in changefeed failure

[fubinzh]

What did you do?

1. There are five changefeed (Kafka sink, simple protocol) running, each replicating a subset of the tables.
2. At first there are workload running for all the changefeed, cdc lag not stable
3. Later there is only workload running for the changefeed: 1k-odd, 1k-even
4. Wait and check lag status

What did you expect to see?

1. Lag for all changefeed should be normal, at least for changefeeds whose workload running the lag should be normal after incremental scan finishes.

What did you see instead?

changefeed xxx5k not restore to normal. changefeed point stucks for 1.5h +, and finally changefeed into failure state.

Versions of the cluster

Release Version: v8.2.0-alpha
Git Commit Hash: 90da67d
Git Branch: heads/refs/tags/v8.2.0-alpha
UTC Build Time: 2024-06-14 11:38:03
Go Version: go version go1.21.10 linux/amd64
Failpoint Build: false

[fubinzh]

/severity moderate
The throughtpu up to 300MB/s when the issue happens.