p2p may discard scheduling messages when synchronize 150k tables in single changefeed

[CharlesCheung96]

What did you do?

1. Create a changefeed which synchronize 150k tables.

What did you expect to see?

The changefeed lag is normal

What did you see instead?

The changefeed lag is up to 3 hours.

And we found the following log

[WARN] [transport.go:138] ["schedulerv3: message
 send failed since ignored, retry later"] [namespace=default] [changefeed=test] [to=018512f2-41c5-4260-9d84-a54b3d3
313cc] [ignoreCount=52] [totalMsg=1862847] [ignoreRate=0.00002791426241661285] [role=scheduler]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

(paste TiCDC version here)