dxf: add api to manually tune the calculation of resource related params by D3Hunter · Pull Request #63517 · pingcap/tidb

D3Hunter · 2025-09-15T12:31:49Z

What problem does this PR solve?

Issue Number: ref #61702

Problem Summary:

What changed and how does it work?

add api to manually tune the calculation of resource related params

also consider the size of index when calculating params for import-into, in this pr we fix the size ratio per index to 1%, later we will get it by sampling.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

➜  tikv-worker-test curl localhost:10080/dxf/schedule/tune
This api only support SYSTEM keyspace, current keyspace is keyspace1%

➜  tikv-worker-test curl localhost:20080/dxf/schedule/tune
invalid or empty target keyspace %

➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=aa"
failed to load keyspace aa from PD: Load keyspace aa failed: type:ENTRY_NOT_FOUND message:"[PD:keyspace:ErrKeyspaceNotFound]keyspace does not exist" %

➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
{
 "amplify_factor": 1
}%

➜  tikv-worker-test curl -X PUT "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
This api only support GET and POST method%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=ss"
invalid ttl ss, error time: invalid duration "ss"%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=2h"
invalid amplify_factor , error strconv.ParseFloat: parsing "": invalid syntax%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=2h&amplify_factor=0.1"
amplify_factor 0.100000 is out of range [1.000000, 10.000000]%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=2h&amplify_factor=11"
amplify_factor 11.000000 is out of range [1.000000, 10.000000]%

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=2h&amplify_factor=2"
{
 "ttl": 7200000000000,
 "expire_time": "2025-09-15T22:36:24.591155+08:00",
 "amplify_factor": 2
}%

➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
{
 "amplify_factor": 2
}%

after expire

➜  tikv-worker-test curl -X POST "localhost:20080/dxf/schedule/tune?keyspace=keyspace1&ttl=10s&amplify_factor=2"
{
 "ttl": 10000000000,
 "expire_time": "2025-09-15T20:37:51.260874+08:00",
 "amplify_factor": 2
}%
➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
{
 "amplify_factor": 2
}%
➜  tikv-worker-test curl "localhost:20080/dxf/schedule/tune?keyspace=keyspace1"
{
 "amplify_factor": 1
}%

test import-into, after set amplify factor=6

before
[2025/09/15 20:40:51.528 +08:00] [INFO] [import.go:1352] ["auto calculate resource related params"] [keyspaceName=keyspace1] [table=ks1] ["thread count"=1] ["max node count"=1] ["dist sql scan concurrency"=15] ["target node cpu count"=10] ["total file size"=10GiB] ["file count"=1] ["index size ratio"=0] ["amplify factor"=1]
after
[2025/09/15 20:42:20.528 +08:00] [INFO] [import.go:1352] ["auto calculate resource related params"] [keyspaceName=keyspace1] [table=ks1] ["thread count"=2] ["max node count"=1] ["dist sql scan concurrency"=30] ["target node cpu count"=10] ["total file size"=10GiB] ["file count"=1] ["index size ratio"=0] ["amplify factor"=6]

test add-index, after set amplify factor=6

before
[2025/09/15 20:40:56.986 +08:00] [INFO] [reorg_util.go:150] ["initialize reorg meta"] [keyspaceName=keyspace1] [category=ddl] [jobSchema=test] [jobTable=ks1] [jobType="add index"] [enableDistTask=true] [enableFastReorg=true] [targetScope=dxf_service] [maxNodeCount=1] [tableSizeInBytes=10GiB] [concurrency=1] [batchSize=256]
after
[2025/09/15 20:42:25.096 +08:00] [INFO] [reorg_util.go:150] ["initialize reorg meta"] [keyspaceName=keyspace1] [category=ddl] [jobSchema=test] [jobTable=ks1] [jobType="add index"] [enableDistTask=true] [enableFastReorg=true] [targetScope=dxf_service] [maxNodeCount=1] [tableSizeInBytes=10GiB] [concurrency=2] [batchSize=256]

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-09-15T12:32:05Z

Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-09-15T12:49:27Z

Codecov Report

❌ Patch coverage is 46.33028% with 117 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.8151%. Comparing base (8debad9) to head (cf01f95).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #63517        +/-   ##
================================================
+ Coverage   72.7876%   74.8151%   +2.0274%     
================================================
  Files          1840       1887        +47     
  Lines        497513     506125      +8612     
================================================
+ Hits         362128     378658     +16530     
+ Misses       113374     104079      -9295     
- Partials      22011      23388      +1377

Flag	Coverage Δ
integration	`48.7231% <2.2935%> (?)`
unit	`72.3981% <46.3302%> (+0.0704%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.8700% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.6274% <ø> (+17.1253%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

D3Hunter · 2025-09-16T05:42:11Z

/retest

tiprow · 2025-09-16T05:42:34Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-09-17T02:13:20Z

/retest

tiprow · 2025-09-17T02:13:42Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2025-09-18T03:01:47Z

+		factors, err := dxfhandle.GetScheduleTuneFactors(ctx, sctx.GetStore().GetKeyspace())
+		if err != nil {
+			return err
+		}


Maybe print this factor to below log?

ti-chi-bot · 2025-09-18T05:57:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: GMHDBJD, tangenta

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [GMHDBJD,tangenta]
~~pkg/ddl/OWNERS~~ [GMHDBJD,tangenta]
~~pkg/disttask/OWNERS~~ [GMHDBJD,tangenta]
~~pkg/executor/importer/OWNERS~~ [GMHDBJD]
~~pkg/meta/OWNERS~~ [GMHDBJD,tangenta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-09-18T05:57:59Z

[LGTM Timeline notifier]

Timeline:

2025-09-18 02:57:59.105243975 +0000 UTC m=+68640.583824622: ☑️ agreed by tangenta.
2025-09-18 05:57:58.337600732 +0000 UTC m=+79439.816181400: ☑️ agreed by GMHDBJD.

D3Hunter · 2025-09-18T06:32:15Z

/retest

tiprow · 2025-09-18T06:32:40Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-09-18T07:15:20Z

/retest

tiprow · 2025-09-18T07:15:42Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-09-18T12:57:05Z

/retest

tiprow · 2025-09-18T12:57:29Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

D3Hunter · 2025-09-18T16:54:15Z

/cherry-pick release-nextgen-20250815

ti-chi-bot · 2025-09-18T16:54:59Z

@D3Hunter: new pull request created to branch release-nextgen-20250815: #63606.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-nextgen-20250815

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

D3Hunter added 4 commits September 15, 2025 15:09

change

9a37575

change

ab7ac28

change

f06f0b4

change

fc9b741

ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 15, 2025

lint

d34e596

D3Hunter added 4 commits September 15, 2025 21:02

revert

a31ab8b

build

4b8a815

lint

b1799b1

change

89a11c7

D3Hunter changed the title ~~[wip]dxf: add api to manually tune the calculation of resource related params~~ dxf: add api to manually tune the calculation of resource related params Sep 16, 2025

ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 16, 2025

Merge remote-tracking branch 'upstream/master' into nextgen-tune-factors

25d5fc1

tangenta approved these changes Sep 18, 2025

View reviewed changes

ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 18, 2025

tangenta reviewed Sep 18, 2025

View reviewed changes

GMHDBJD approved these changes Sep 18, 2025

View reviewed changes

ti-chi-bot Bot added approved lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 18, 2025

Merge remote-tracking branch 'upstream/master' into nextgen-tune-factors

cf01f95

ti-chi-bot Bot merged commit 276ed0c into pingcap:master Sep 18, 2025
29 checks passed

D3Hunter deleted the nextgen-tune-factors branch September 18, 2025 16:54

ti-chi-bot mentioned this pull request Sep 18, 2025

dxf: add api to manually tune the calculation of resource related params (#63517) #63606

Closed

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Sep 18, 2025

This is an automated cherry-pick of pingcap#63517

3b7e3a3

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

This was referenced Sep 19, 2025

test: fix some tests in nextgen #63619

Merged

DXF service on nextgen kernel #61702

Open

Uh oh!

Conversation

D3Hunter commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow Bot commented Sep 15, 2025

Uh oh!

codecov Bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

D3Hunter commented Sep 16, 2025

Uh oh!

tiprow Bot commented Sep 16, 2025

Uh oh!

D3Hunter commented Sep 17, 2025

Uh oh!

tiprow Bot commented Sep 17, 2025

Uh oh!

tangenta Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

D3Hunter Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot Bot commented Sep 18, 2025

Uh oh!

ti-chi-bot Bot commented Sep 18, 2025

[LGTM Timeline notifier]

Uh oh!

D3Hunter commented Sep 18, 2025

Uh oh!

tiprow Bot commented Sep 18, 2025

Uh oh!

D3Hunter commented Sep 18, 2025

Uh oh!

tiprow Bot commented Sep 18, 2025

Uh oh!

D3Hunter commented Sep 18, 2025

Uh oh!

tiprow Bot commented Sep 18, 2025

Uh oh!

Uh oh!

D3Hunter commented Sep 18, 2025

Uh oh!

ti-chi-bot commented Sep 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

D3Hunter commented Sep 15, 2025 •

edited

Loading

codecov Bot commented Sep 15, 2025 •

edited

Loading

tangenta Sep 18, 2025 •

edited

Loading