EtcdWorker ticks too often

[liuzix]

What did you do?

Create 32 changefeeds corresponding to 32 tables. Run the CDC cluster.

What did you expect to see?

The Etcd server (embedded in PD) would experience a reasonable amount of QPS pressure.

What did you see instead?

The total QPS to Etcd can be as high as 1000. It can be higher. The exact quantitative relation with the number of changefeeds is still under investigation.

Versions of the cluster

v5.1.x, v5.2.x

[asddongmen]

• Do more tests with different number of cdc nodes, changefeeds and different type of sink.
• Find possible workaround.
• Add a rate limiter to limit the ectdWorker tick reactor rate, which is a workaround to reduces etcd QPS. CDC/etcd_worker: add rate limiter to limit EtcdWorker tick frequency #3219
• Batch transaction sent from cdc node to etcd server, which further reduces etcd QPS. etcd_worker: batch etcd patch #3277

[overvenus]

This issue has been mitigated, change severity to major.

[asddongmen]

It seems that this bug has been solved by #3219 and #3217 , if it happends again, feel free to reopen this issue and @ me.