br: pitr restore failed. scan region return empty result

[AkiraXie]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2023/03/07 12:52:48.921 +00:00] [INFO] [collector.go:73] ["restore log failed summary"] [error="failed to restore kv files: scan region return empty result, startKey: 74800000000000F9FF675F72800000000EFF7AB9830000000000FA, endKey: 74800000000000F9FF675F72800000000EFF82: [BR:PD:ErrPDBatchScanRegion]batch scan region"] errorVerbose="[BR:PD:ErrPDBatchScanRegion]batch scan region\nscan region return empty result, startKey: 74800000000000F9FF675F72800000000EFF7AB9830000000000FA, endKey: 74800000000000F9FF675F72800000000EFF82[ngithub.com/pingcap/tidb/br/pkg/restore/split.CheckRegionConsistency\n\t/data2/lc/tidb/br/pkg/restore/split/split.go:49\ngithub.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n\t/data2/lc/tidb/br/pkg/restore/split/split.go:114\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/data2/lc/tidb/br/pkg/utils/retry.go:52\ngithub.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n\t/data2/lc/tidb/br/pkg/restore/split/split.go:91\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runOverRegions\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:141\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:183\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\ngithub.com/pingcap/tidb/br/pkg/restore.(*OverRegionsInRangeController).runInRegion\n\t/data2/lc/tidb/br/pkg/restore/import_retry.go:177\nfailed to restore kv files"]
Error: failed to restore kv files: scan region return empty result, startKey: 74800000000000F9FF675F72800000000EFF7AB9830000000000FA, endKey: 74800000000000F9FF675F72800000000EFF82: [BR:PD:ErrPDBatchScanRegion]batch scan region

4. What is your TiDB version? (Required)

release-6.5

[joccau]

/assigned

[tonyxuqqi]

Do we know the root cause of the slow splitting region?