PR#55306 may caused 8.1% performance regression in benchmarksql

[owlsome2501]

Bug Report

1. Minimal reproduce step (Required)

1. Deploy a cluster with 3 TiDB and 3 Tikv.
2. Run benchmark benchmarksql (benchmarksql-1k).

2. What did you expect to see? (Required)

No performance regression.

3. What did you see instead (Required)

#55306 caused QPS regression 8.13% in benchmark benchmarksql (benchmarksql-1k) compared to baseline 60193.12.

description	tidb	tikv	pd	QPS
last success	29d52a7	tikv/tikv@0e38069	tikv/pd@f3602e3	61176.30
first regression	9a7e5cc	tikv/tikv@0e38069	tikv/pd@f3602e3	55297.83

4. What is your TiDB version? (Required)

9a7e5cc

[Yui-Song]

/type performance
/type regression
/severity critical
/assign @hawkingrei
/sig planner
/label affects-8.3

[Yui-Song]

The performance regression was caused by an intermittent hotspot issue that led to unbalanced TiKV CPU usage, rather than by PR#55306.

[Yui-Song]

/remove-severity critical
/severity major

[Yui-Song]

/remove-sig planner

[Yui-Song]

/label affects-7.5
/label affects-7.1
/label affects-8.1

[Yui-Song]

The same performance instability has been observed in both TPC-C and BenchmarkSQL. It may be necessary to enhance the PD scheduling strategy by incorporating TiKV CPU usage as a factor, in order to reduce the occurrence of similar instability issues.

[Yui-Song]

/assign @niubell
/unassign @hawkingrei