lightning: concurrency setting when write to tikv

[D3Hunter]

Enhancement

before 7.0.0, we can use range-concurrency to control concurrency(multiple by 2 actually) when write to tikv, but after 7.0.0, the write concurrency is table-concurrency * range-concurrency * 2 as we start range-concurrency * 2 workers for each engine, if we enlarge table-concurrency as before, the tikv write concurrency might be too large and overload tikv(io), it might increase io latency -> stuck raftstore -> leader change frequently -> cause region split fail even after retry(keeps reporting not leader or leader is none error)

tidb/br/pkg/lightning/backend/local/local.go

Lines 1420 to 1424 in 33d7a2c

	for i := 0; i < local.WorkerConcurrency; i++ {
	workGroup.Go(func() error {
	return local.startWorker(workerCtx, jobToWorkerCh, jobFromWorkerCh, &jobWg)
	})
	}

error="batch split regions failed: split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader:<region_id:751226 > : [BR:Restore:ErrRestoreSplitFailed]fail to split region; split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader:<region_id:751226 > : [BR:Restore:ErrRestoreSplitFailed]fail to split region; split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader:<region_id:751226 > : [BR:Restore:ErrRestoreSplitFailed]fail to split region; split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader:<region_id:751226 > : [BR:Restore:ErrRestoreSplitFailed]fail to split region"] [errorVerbose="the following errors occurred:
-  [BR:Restore:ErrRestoreSplitFailed]fail to split region
split region failed: err=message:\"peer is not leader for region 751226, leader may None\" not_leader:<region_id:751226 > 
github.com/pingcap/tidb/br/pkg/restore/split.sendSplitRegionRequest
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:370
github.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).sendSplitRegionRequest
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:315
github.com/pingcap/tidb/br/pkg/restore/split.(*pdClient).BatchSplitRegionsWithOrigin
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/client.go:413
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).BatchSplitRegions
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:400
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).SplitAndScatterRegionByRanges.func3
    /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/localhelper.go:276
golang.org/x/sync/errgroup.(*Group).Go.func1
    /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1650

[D3Hunter]

we can workaround this by make range-concurrency smaller