cdc goroutines become crazy when initializing a capture with lots of regions and tikv.cdc.min-ts-interval = 200ms

[hicqu]

What did you do?

Create a capture on 16 MySQL tables, 131K regions totally with

• tikv.cdc.min-ts-interval = 200ms
• tikv.cdc.min-ts-interval = 1s

What did you expect to see?

Both 2 operations can be finished quickly. With tikv.cdc.min-ts-interval = 1s it's expected:

What did you see instead?

However with tikv.cdc.min-ts-interval = 200ms the capture creating can never finished, and cpu usage is very high:

And here is a 30s CPU profile:

We can see the CPU time is about 480s, which means all 16C are very busy on runtime.selectgo.

Versions of the cluster

All components are nightly.
Go version is 1.18.1.

Investigations

[nongfushanquan]

/label affect-5.4

[ti-chi-bot]

@nongfushanquan: The label(s) affect-5.4 cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, affects-4.0, affects-5.0, affects-5.1, affects-5.2, affects-5.3, affects-5.4, affects-6.0, affects-6.1, may-affects-4.0, may-affects-5.0, may-affects-5.1, may-affects-5.2, may-affects-5.3, may-affects-5.4, may-affects-6.0, may-affects-6.1, needs-cherry-pick-release-4.0, needs-cherry-pick-release-5.0, needs-cherry-pick-release-5.1, needs-cherry-pick-release-5.2, needs-cherry-pick-release-5.3, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.0, needs-cherry-pick-release-6.1, question, release-blocker, wontfix, require-LGT1, require-LGT3.

In response to this:

/label affect-5.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.