dxf: calculate max_node_count and concurrency for next-gen automatically | tidb-test=pr/2584 by tangenta · Pull Request #62995 · pingcap/tidb

tangenta · 2025-08-14T09:34:11Z

What problem does this PR solve?

Issue Number: ref #61702

Problem Summary:

What changed and how does it work?

Forbid setting tidb_max_dist_task_nodes in next-gen.
Calculate the node count and concurrency automatically according to the data size for add index and import into.
Add table size estimation by scanning regions.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
TBD
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: tangenta <tangenta@126.com>

tiprow · 2025-08-14T09:34:31Z

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-08-14T09:53:32Z

Codecov Report

❌ Patch coverage is 45.63492% with 137 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.8622%. Comparing base (dd5f5ad) to head (cbfe9fe).
⚠️ Report is 43 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #62995        +/-   ##
================================================
+ Coverage   72.7739%   74.8622%   +2.0883%     
================================================
  Files          1810       1872        +62     
  Lines        493564     509565     +16001     
================================================
+ Hits         359186     381472     +22286     
+ Misses       112542     104602      -7940     
- Partials      21836      23491      +1655

Flag	Coverage Δ
integration	`48.5778% <28.5714%> (?)`
unit	`72.3388% <45.6349%> (+0.1143%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.8700% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.1914% <ø> (+16.8868%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

D3Hunter · 2025-08-14T10:08:31Z

other reorg DDLs which not run on DXF also need tidb_ddl_reorg_worker_cnt to set concurrency, such as lossy modify column, we need allow it changed by user

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter · 2025-08-15T11:11:30Z

+
+func estimateTableSizeByID(ctx context.Context, pdCli pdhttp.Client, pid int64) (int64, error) {
+	sk, ek := tablecodec.GetTableHandleKeyRange(pid)
+	sRegion, err := pdCli.GetRegionByKey(ctx, codec.EncodeBytes(nil, sk))


why we cannot use the previous sk, ek directly, and use the one get from PD region?

Not sure if there are specific reasons, this logic is copied from getRegionsInfoForSingleTable. Maybe I can ask the author later.

Signed-off-by: tangenta <tangenta@126.com>

tangenta · 2025-08-18T02:30:43Z

/retest

tiprow · 2025-08-18T02:31:06Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter · 2025-08-18T06:17:31Z

+		return 4
+	}
+	concurrency := size / (25 * units.GiB)
+	concurrency = min(concurrency, int64(coresPerNode-1))


we don't need to minus 1, the 1 is already removed when starting the pod, the runtime.GOMAXPROCS API will return the cores usable to the pod, say a 8c machine, the pod and the runtime.GOMAXPROCS will return 7 already

D3Hunter · 2025-08-18T06:33:40Z

and please finish the TBD in manual test part

Signed-off-by: tangenta <tangenta@126.com>

wuhuizuo · 2025-08-22T02:21:28Z

/test pull-mysql-client-test-next-gen pull-mysql-client-test

tiprow · 2025-08-22T02:21:50Z

@wuhuizuo: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

Details

In response to this:

/test pull-mysql-client-test-next-gen pull-mysql-client-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

wuhuizuo · 2025-08-22T02:23:00Z

/test pull-mysql-client-test

tiprow · 2025-08-22T02:23:22Z

@wuhuizuo: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

Details

In response to this:

/test pull-mysql-client-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter · 2025-08-22T02:51:13Z

/retest

D3Hunter · 2025-08-22T03:14:32Z

/retest

D3Hunter · 2025-08-22T05:45:51Z

/retest

Signed-off-by: tangenta <tangenta@126.com>

tangenta · 2025-08-22T06:42:26Z

/retest

D3Hunter · 2025-08-22T07:11:32Z

/retest

D3Hunter · 2025-08-22T08:12:14Z

/retest

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter · 2025-08-22T09:28:51Z

/retest

tangenta · 2025-08-22T09:56:03Z

/retest

D3Hunter · 2025-08-22T11:59:42Z

/retest

D3Hunter · 2025-08-22T12:44:32Z

/retest

D3Hunter · 2025-08-22T14:06:22Z

/retest

D3Hunter · 2025-08-22T14:28:06Z

/retest

D3Hunter · 2025-09-02T15:21:10Z

/cherry-pick release-nextgen-20250815

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2025-09-02T15:21:54Z

@D3Hunter: new pull request created to branch release-nextgen-20250815: #63347.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-nextgen-20250815

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

…lly (#62995) (#63347) ref #61702

tangenta added 2 commits August 14, 2025 17:29

ddl: calculate max_node_count and concurrency for next-gen automatically

a475a1e

Signed-off-by: tangenta <tangenta@126.com>

add test

209745a

ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Aug 14, 2025

tangenta added 4 commits August 14, 2025 19:18

don't forbid setting ddl_reorg_worker_cnt

62c9d44

Signed-off-by: tangenta <tangenta@126.com>

fix linter

2c097b5

Signed-off-by: tangenta <tangenta@126.com>

fix getTableSizeByID

50f18da

Signed-off-by: tangenta <tangenta@126.com>

add license header

8ccfe74

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter reviewed Aug 15, 2025

View reviewed changes

tangenta added 6 commits August 16, 2025 11:51

address comments

0d511fc

Signed-off-by: tangenta <tangenta@126.com>

Merge remote-tracking branch 'upstream/master' into next-gen-calc-auto

9963837

update bazel

8ac6c8e

Signed-off-by: tangenta <tangenta@126.com>

fix linter

10b60e3

Signed-off-by: tangenta <tangenta@126.com>

fix linter

709fab0

Signed-off-by: tangenta <tangenta@126.com>

fix linter

436b60c

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter reviewed Aug 18, 2025

View reviewed changes

Comment thread pkg/disttask/framework/storage/nodes.go

D3Hunter reviewed Aug 18, 2025

View reviewed changes

Comment thread pkg/disttask/framework/scheduler/autoscaler.go Outdated

Comment thread pkg/executor/importer/import.go Outdated

D3Hunter changed the title ~~ddl: calculate max_node_count and concurrency for next-gen automatically~~ dxf: calculate max_node_count and concurrency for next-gen automatically Aug 18, 2025

tangenta added 2 commits August 18, 2025 12:04

address comments

9f979b1

Signed-off-by: tangenta <tangenta@126.com>

fix build

f56477d

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter reviewed Aug 18, 2025

View reviewed changes

Comment thread pkg/disttask/framework/scheduler/autoscaler.go

D3Hunter reviewed Aug 18, 2025

View reviewed changes

Comment thread pkg/disttask/framework/scheduler/autoscaler.go Outdated

address comments

e3de730

Signed-off-by: tangenta <tangenta@126.com>

wuhuizuo changed the title ~~dxf: calculate max_node_count and concurrency for next-gen automatically | tidb-test=pull/2584~~ dxf: calculate max_node_count and concurrency for next-gen automatically | tidb-test=pr/2584 Aug 22, 2025

update bazel

1e57b36

Signed-off-by: tangenta <tangenta@126.com>

don't get table size / exec cpu node for classic kernel

89f1ad1

Signed-off-by: tangenta <tangenta@126.com>

fix typo

cbfe9fe

Signed-off-by: tangenta <tangenta@126.com>

ti-chi-bot Bot merged commit aa875ea into pingcap:master Aug 22, 2025
31 checks passed

This was referenced Aug 26, 2025

DXF service on nextgen kernel #61702

Open

next-gen: forbid setting tidb_max_dist_task_nodes in user keyspace #62687

Closed

JaySon-Huang mentioned this pull request Aug 26, 2025

next-gen integration ddl test failed pingcap/tiflash#10374

Closed

tangenta mentioned this pull request Aug 27, 2025

add-index: let adding index triggered by modify column backfills data by fast reorg #63179

Closed

13 tasks

ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this pull request Sep 2, 2025

This is an automated cherry-pick of pingcap#62995

48d12b0

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Sep 2, 2025

dxf: calculate max_node_count and concurrency for next-gen automatically (#62995) | tidb-test=pr/2595 #63347

Merged

13 tasks

ti-chi-bot Bot pushed a commit that referenced this pull request Sep 3, 2025

dxf: calculate max_node_count and concurrency for next-gen automatica…

73e525e

…lly (#62995) (#63347) ref #61702

Uh oh!

Conversation

tangenta commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow Bot commented Aug 14, 2025

Uh oh!

codecov Bot commented Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

D3Hunter commented Aug 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

D3Hunter Aug 15, 2025

Choose a reason for hiding this comment

Uh oh!

tangenta Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tangenta commented Aug 18, 2025

Uh oh!

tiprow Bot commented Aug 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

D3Hunter Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

D3Hunter commented Aug 18, 2025

Uh oh!

wuhuizuo commented Aug 22, 2025

Uh oh!

tiprow Bot commented Aug 22, 2025

Uh oh!

wuhuizuo commented Aug 22, 2025

Uh oh!

tiprow Bot commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

tangenta commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

tangenta commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

D3Hunter commented Aug 22, 2025

Uh oh!

Uh oh!

D3Hunter commented Sep 2, 2025

tangenta commented Aug 14, 2025 •

edited

Loading

codecov Bot commented Aug 14, 2025 •

edited

Loading