dxf: add api to manually tune the calculation of resource related params (#63517)

[ti-chi-bot]

This is an automated cherry-pick of #63517

What problem does this PR solve?

Issue Number: ref #61702

Problem Summary:

What changed and how does it work?

add api to manually tune the calculation of resource related params

also consider the size of index when calculating params for import-into, in this pr we fix the size ratio per index to 1%, later we will get it by sampling.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

➜  tikv-worker-test curl localhost:10080/dxf/schedule/tune
This api only support SYSTEM keyspace, current keyspace is keyspace1%

➜  tikv-worker-test curl localhost:20080/dxf/schedule/tune
invalid or empty target keyspace %

➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=aa"
failed to load keyspace aa from PD: Load keyspace aa failed: type:ENTRY_NOT_FOUND message:"[PD:keyspace:ErrKeyspaceNotFound]keyspace does not exist" %

➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
{
 "amplify_factor": 1
}%

➜  tikv-worker-test curl -X PUT "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
This api only support GET and POST method%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=ss"
invalid ttl ss, error time: invalid duration "ss"%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=2h"
invalid amplify_factor , error strconv.ParseFloat: parsing "": invalid syntax%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=2h&amplify_factor=0.1"
amplify_factor 0.100000 is out of range [1.000000, 10.000000]%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=2h&amplify_factor=11"
amplify_factor 11.000000 is out of range [1.000000, 10.000000]%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=2h&amplify_factor=2"
{
 "ttl": 7200000000000,
 "expire_time": "2025-09-15T22:36:24.591155+08:00",
 "amplify_factor": 2
}%

➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
{
 "amplify_factor": 2
}%

after expire

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=10s&amplify_factor=2"
{
 "ttl": 10000000000,
 "expire_time": "2025-09-15T20:37:51.260874+08:00",
 "amplify_factor": 2
}%
➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
{
 "amplify_factor": 2
}%
➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
{
 "amplify_factor": 1
}%

test import-into, after set amplify factor=6

before
[2025/09/15 20:40:51.528 +08:00] [INFO] [import.go:1352] ["auto calculate resource related params"] [keyspaceName=keyspace1] [table=ks1] ["thread count"=1] ["max node count"=1] ["dist sql scan concurrency"=15] ["target node cpu count"=10] ["total file size"=10GiB] ["file count"=1] ["index size ratio"=0] ["amplify factor"=1]
after
[2025/09/15 20:42:20.528 +08:00] [INFO] [import.go:1352] ["auto calculate resource related params"] [keyspaceName=keyspace1] [table=ks1] ["thread count"=2] ["max node count"=1] ["dist sql scan concurrency"=30] ["target node cpu count"=10] ["total file size"=10GiB] ["file count"=1] ["index size ratio"=0] ["amplify factor"=6]

test add-index, after set amplify factor=6

before
[2025/09/15 20:40:56.986 +08:00] [INFO] [reorg_util.go:150] ["initialize reorg meta"] [keyspaceName=keyspace1] [category=ddl] [jobSchema=test] [jobTable=ks1] [jobType="add index"] [enableDistTask=true] [enableFastReorg=true] [targetScope=dxf_service] [maxNodeCount=1] [tableSizeInBytes=10GiB] [concurrency=1] [batchSize=256]
after
[2025/09/15 20:42:25.096 +08:00] [INFO] [reorg_util.go:150] ["initialize reorg meta"] [keyspaceName=keyspace1] [category=ddl] [jobSchema=test] [jobTable=ks1] [jobType="add index"] [enableDistTask=true] [enableFastReorg=true] [targetScope=dxf_service] [maxNodeCount=1] [tableSizeInBytes=10GiB] [concurrency=2] [batchSize=256]

• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[ti-chi-bot]

@D3Hunter This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign benjamin2037 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• pkg/ddl/OWNERS
• pkg/disttask/OWNERS
• pkg/executor/importer/OWNERS
• pkg/meta/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@ti-chi-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-integration-realcluster-test-next-gen	3b7e3a3	link	true	/test pull-integration-realcluster-test-next-gen
pull-mysql-client-test-next-gen	3b7e3a3	link	true	/test pull-mysql-client-test-next-gen
pull-build-next-gen	3b7e3a3	link	true	/test pull-build-next-gen

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[D3Hunter]

release-nextgen-20250815 is for internal usage, won't cp