ddl: compared with v6.6.0, v7.0.0 has 10% - 13% performance regression

[Yui-Song]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. deploy a TiDB cluster and restore the data of ossinsight workload
2. add index alter table github_events add index test_idx(repo_id,type,action,closed_at,actor_login)

2. What did you expect to see? (Required)

No performance regression

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

v7.0.0

[lance6716]

The root cause is

after #40692 , I moved "generating job" logic into the single thread producer, and in each round, after all job is generated we send them to consumer

tidb/br/pkg/lightning/backend/local/local.go

Lines 1398 to 1426 in 4e30a64

	newJobs, err := local.prepareAndGenerateUnfinishedJob(
	ctx,
	engineUUID,
	lf,
	regionRanges,
	regionSplitSize,
	regionSplitKeys,
	)
	if err != nil {
	close(jobToWorkerCh)
	_ = workGroup.Wait()
	return err
	}
	pendingJobs = append(pendingJobs, newJobs...)
	if len(pendingJobs) == 0 {
	break
	}
	jobWg.Add(len(pendingJobs))
	for _, job := range pendingJobs {
	select {
	case <-workerCtx.Done():
	err2 := retryErr.Load()
	if err2 != nil {
	return errors.Trace(err2)
	}
	return errors.Trace(workGroup.Wait())
	case jobToWorkerCh <- job:

however, in old code generating job is scattered in multiple worker that runs concurrently

The CPU profile can locate this problem. in timestamp 1680237360

3 minutes later: