importinto: update thread count after initialized data files by tangenta · Pull Request #63200 · pingcap/tidb

tangenta · 2025-08-26T06:32:32Z

What problem does this PR solve?

Issue Number: ref #61702

Problem Summary:

Before this PR, the node count & thread auto calculation does not work because the data size is initialized after Plan.initOptions.

What changed and how does it work?

This PR moves the auto calculation logic after LoadDataController.InitDataFiles.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

mysql> import into sbtest1 from "s3://test/*.csv?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://127.0.0.1:9000&force-path-style=true" with SKIP_ROWS=1;
+--------+-----------+-------------------------------------------------------------------------------------------------------------------------+------------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+----------------------------+----------+-------------------------+---------------------+-----------------------+----------------+--------------+
| Job_ID | Group_Key | Data_Source                                                                                                             | Target_Table     | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By | Last_Update_Time           | Cur_Step | Cur_Step_Processed_Size | Cur_Step_Total_Size | Cur_Step_Progress_Pct | Cur_Step_Speed | Cur_Step_ETA |
+--------+-----------+-------------------------------------------------------------------------------------------------------------------------+------------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+----------------------------+----------+-------------------------+---------------------+-----------------------+----------------+--------------+
|      2 | NULL      | s3://test/*.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F127.0.0.1%3A9000&force-path-style=true&secret-access-key=xxxxxx | `test`.`sbtest1` |       10 |       | finished | 189.7MiB         |       1000000 |                | 2025-08-27 11:10:18.308084 | 2025-08-27 11:10:19.771820 | 2025-08-27 11:10:37.280906 | root@%     | 2025-08-27 11:10:37.280906 | NULL     | NULL                    | NULL                | NULL                  | NULL           | NULL         |
+--------+-----------+-------------------------------------------------------------------------------------------------------------------------+------------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+----------------------------+----------+-------------------------+---------------------+-----------------------+----------------+--------------+
1 row in set (19.26 sec)

[2025/08/27 11:10:19.787 +08:00] [INFO] [import.go:1319] ["set import thread count for nextgen kernel"] [keyspaceName=SYSTEM] [table=sbtest1] ["thread count"=1] ["max node count"=1] ["target node cpu count"=32] ["total file size"=189.7MiB]

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: tangenta <tangenta@126.com>

tiprow · 2025-08-26T06:32:48Z

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-08-26T06:52:41Z

Codecov Report

❌ Patch coverage is 30.00000% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.7851%. Comparing base (a3d6d9e) to head (1952a76).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #63200        +/-   ##
================================================
+ Coverage   72.7884%   74.7851%   +1.9966%     
================================================
  Files          1831       1877        +46     
  Lines        495404     503615      +8211     
================================================
+ Hits         360597     376629     +16032     
+ Misses       112870     103656      -9214     
- Partials      21937      23330      +1393

Flag	Coverage Δ
integration	`48.5438% <10.0000%> (?)`
unit	`72.3251% <30.0000%> (+0.0673%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.8700% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.2197% <ø> (+16.8789%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter · 2025-08-26T10:19:16Z

 	e.dataFiles = dataFiles
 	e.TotalFileSize = totalSize
+
+	if kerneltype.IsNextGen() {


add a integration test for it, we can enlarge the totalSize to be several times larger than original

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter · 2025-08-27T07:26:40Z

-		testfailpoint.Enable(s.T(), "github.com/pingcap/tidb/pkg/executor/importer/mockImportDataSize", fmt.Sprintf(`return(%d)`, tc.size))
+		testfailpoint.EnableCall(s.T(), "github.com/pingcap/tidb/pkg/executor/importer/mockImportDataSize", func(totalSize, targetNodeCPUCnt *int) {
+			*totalSize = tc.size
+			*targetNodeCPUCnt = 32


better not change this one, else the machine might not have enough slots to run the task. or you can mock the cpu count by mockNumCpu

ti-chi-bot · 2025-08-27T08:11:46Z

@tangenta: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-integration-realcluster-test-next-gen	`3a125f6`	link	true	`/test pull-integration-realcluster-test-next-gen`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter · 2025-08-27T16:06:02Z

/retest

tiprow · 2025-08-27T16:06:28Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2025-08-28T01:06:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CbcWestwolf, D3Hunter

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [CbcWestwolf,D3Hunter]
~~pkg/ddl/OWNERS~~ [D3Hunter]
~~pkg/disttask/OWNERS~~ [D3Hunter]
~~pkg/executor/importer/OWNERS~~ [D3Hunter]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2025-08-28T01:06:41Z

[LGTM Timeline notifier]

Timeline:

2025-08-27 15:47:33.016718207 +0000 UTC m=+368744.894549171: ☑️ agreed by D3Hunter.
2025-08-28 01:06:39.90423699 +0000 UTC m=+402291.782067934: ☑️ agreed by CbcWestwolf.

hawkingrei · 2025-08-28T01:47:55Z

/retest

tangenta · 2025-08-28T06:38:23Z

/retest

tiprow · 2025-08-28T06:38:46Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…rt-into-thread

D3Hunter · 2025-08-28T08:56:55Z

/retest

tiprow · 2025-08-28T08:57:18Z

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…#63200) ref pingcap#61702

importinto: update thread count after initialzed data files

baabb23

Signed-off-by: tangenta <tangenta@126.com>

ti-chi-bot Bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 26, 2025

fix import into max node count

3396cf3

Signed-off-by: tangenta <tangenta@126.com>

ti-chi-bot Bot removed the do-not-merge/needs-linked-issue label Aug 26, 2025

D3Hunter reviewed Aug 26, 2025

View reviewed changes

Comment thread pkg/executor/importer/import.go Outdated

address comment

601cbbe

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter reviewed Aug 26, 2025

View reviewed changes

D3Hunter mentioned this pull request Aug 26, 2025

DXF service on nextgen kernel #61702

Open

add tests and refine code

ad93d34

Signed-off-by: tangenta <tangenta@126.com>

ti-chi-bot Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 27, 2025

D3Hunter reviewed Aug 27, 2025

View reviewed changes

Comment thread tests/realtikvtest/importintotest/import_into_test.go

Comment thread pkg/executor/importer/import.go Outdated

address comment

3a125f6

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter reviewed Aug 27, 2025

View reviewed changes

D3Hunter mentioned this pull request Aug 27, 2025

test: fix part of unit tests for next-gen #63192

Merged

13 tasks

address comments

5730543

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter approved these changes Aug 27, 2025

View reviewed changes

ti-chi-bot Bot added approved needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 27, 2025

CbcWestwolf approved these changes Aug 28, 2025

View reviewed changes

ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Aug 28, 2025

Merge remote-tracking branch 'upstream/master' into next-gen-fix-impo…

1952a76

…rt-into-thread

ti-chi-bot Bot merged commit 8d1a00e into pingcap:master Aug 28, 2025
29 checks passed

tangenta added a commit to ti-chi-bot/tidb that referenced this pull request Sep 3, 2025

importinto: update thread count after initialized data files (pingcap…

571b580

…#63200) ref pingcap#61702

D3Hunter mentioned this pull request Sep 3, 2025

dxf: calculate max_node_count and concurrency for next-gen automatically (#62995) | tidb-test=pr/2595 #63347

Merged

13 tasks

Uh oh!

Conversation

tangenta commented Aug 26, 2025 • edited by D3Hunter Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow Bot commented Aug 26, 2025

Uh oh!

codecov Bot commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

D3Hunter Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

D3Hunter Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

ti-chi-bot Bot commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

D3Hunter commented Aug 27, 2025

Uh oh!

tiprow Bot commented Aug 27, 2025

Uh oh!

ti-chi-bot Bot commented Aug 28, 2025

Uh oh!

ti-chi-bot Bot commented Aug 28, 2025

[LGTM Timeline notifier]

Uh oh!

hawkingrei commented Aug 28, 2025

Uh oh!

tangenta commented Aug 28, 2025

Uh oh!

tiprow Bot commented Aug 28, 2025

Uh oh!

D3Hunter commented Aug 28, 2025

Uh oh!

tiprow Bot commented Aug 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tangenta commented Aug 26, 2025 •

edited by D3Hunter

Loading

codecov Bot commented Aug 26, 2025 •

edited

Loading

D3Hunter Aug 26, 2025 •

edited

Loading

ti-chi-bot Bot commented Aug 27, 2025 •

edited

Loading