*: collect subtask summary for `IMPORT INTO` tasks by joechenrh · Pull Request #61600 · pingcap/tidb

joechenrh · 2025-06-09T10:37:20Z

What problem does this PR solve?

Issue Number: ref #61088

Problem Summary:

What changed and how does it work?

What does this PR do:

add collector for each step executor of IMPORT INTO to collect summary.
store update time in summary. This value will be used to track the progress update time for each job.

Previously, imported row is stored in subtask meta, which is not easy to use. After this PR, we no longer need Result in subtask meta, row count can be calculated from all subtask summaries.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

This PR doesn't add new unit test, but extend existing tests to verfiy the functionality of each collector.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-06-09T10:37:37Z

Hi @joechenrh. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-06-09T10:41:54Z

-type JobImportResult struct {
-	Affected uint64
-	Warnings []contextutil.SQLWarn
-}
-


The only used field is Affected, so I removed this struct.

codecov · 2025-06-09T10:56:19Z

Codecov Report

Attention: Patch coverage is 39.60396% with 122 lines in your changes missing coverage. Please review.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #61600        +/-   ##
================================================
+ Coverage   72.8941%   75.6895%   +2.7954%     
================================================
  Files          1740       1794        +54     
  Lines        482921     499937     +17016     
================================================
+ Hits         352021     378400     +26379     
+ Misses       109294      98696     -10598     
- Partials      21606      22841      +1235

Flag	Coverage Δ
integration	`50.1205% <0.0000%> (?)`
unit	`73.0195% <40.0000%> (+0.8749%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7804% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.7431% <ø> (+17.4099%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

joechenrh · 2025-06-12T02:52:48Z

 	// updateSubtaskSummaryInterval is the interval for updating the subtask summary to
 	// subtask table.
-	updateSubtaskSummaryInterval = 3 * time.Second
+	updateSubtaskSummaryInterval = 5 * time.Second


I think we don't need to update the summary too frequently.

joechenrh · 2025-06-12T03:35:41Z

 	StateRestored       = "restored"      // source data bytes restored during restore engine
 	StateRestoreWritten = "written"       // kv bytes written during restore engine
 	StateImported       = "imported"      // kv bytes imported during import engine
+	StateMerged         = "merged"        // kv bytes merged during import engine


We can display merge speed in dashboard later

joechenrh · 2025-06-12T03:37:06Z

/cc @GMHDBJD @D3Hunter

D3Hunter

25/40

Co-authored-by: D3Hunter <jujj603@gmail.com>

joechenrh · 2025-07-03T03:15:33Z

/retest

tiprow · 2025-07-03T03:15:55Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-03T06:07:45Z

/retest

tiprow · 2025-07-03T06:08:08Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-03T08:46:02Z

/retest

tiprow · 2025-07-03T08:46:26Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-03T09:44:51Z

/retest

tiprow · 2025-07-03T09:45:13Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-03T10:21:45Z

/retest

tiprow · 2025-07-03T10:22:08Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-03T13:04:26Z

/retest

tiprow · 2025-07-03T13:04:50Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-03T15:15:13Z

/retest

tiprow · 2025-07-03T15:15:37Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-03T23:37:19Z

/retest

tiprow · 2025-07-03T23:37:40Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-04T00:49:58Z

/retest

tiprow · 2025-07-04T00:50:20Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-04T01:47:19Z

/retest

tiprow · 2025-07-04T01:47:41Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-04T04:04:36Z

/retest

tiprow · 2025-07-04T04:04:59Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

joechenrh · 2025-07-04T12:55:43Z

/retest

tiprow · 2025-07-04T12:56:05Z

@joechenrh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2025-07-04T13:53:01Z

@joechenrh: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
non-block-pull-unit-test-next-gen	`be955b1`	link	false	`/test pull-unit-test-next-gen`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

* origin/master: (79 commits) backend/local: add rate limiter for split region and ingest data (pingcap#61555) fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch (pingcap#61795) *: collect subtask summary for `IMPORT INTO` tasks (pingcap#61600) tests: update vector cluster test result (pingcap#62174) planner: change casetest instance-plan-cache test timeout. (pingcap#62195) planner: remove data race for instance plan cache (pingcap#62217) *: upgrade pd client to fix keyspace issue (pingcap#62189) resource_control: support collecting cross AZ traffic in ru consumption (pingcap#62003) test: wait for statistics owner to close on exit (pingcap#62198) planner: remove the selection above the table dual (pingcap#62193) telemetry: add telemetry related code back (pingcap#61753) planner: move physical limit related logic into physical op. (pingcap#62187) planner: remove useless code (pingcap#61968) retry: enhance retry for ingestor on nextgen (pingcap#62191) importinto: encode and ingest step use task keyspace in nextgen (pingcap#62138) planner: lift the can-push-to-cop check of aggregation/proj (pingcap#62068) ddl: cannot panic with failpoint panic in the DDLNotifier (pingcap#62151) planner: skip empty expression in the applyPredicateSimplification (pingcap#62178) codec: should also return the value even with error because the error may be ignored in the future (pingcap#62175) parser: also reduce literal list with charset (pingcap#61484) ...

ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jun 9, 2025

joechenrh commented Jun 9, 2025

View reviewed changes

joechenrh changed the title *: add summary to IMPORT INTO tasks *: add subtask summary to IMPORT INTO tasks Jun 10, 2025

ti-chi-bot Bot added component/dumpling This is related to Dumpling of TiDB. sig/planner SIG: Planner labels Jun 10, 2025

cp to new branch

213f68a

joechenrh force-pushed the import-summary branch 2 times, most recently from cb6cb2e to 2abb5a4 Compare June 10, 2025 01:39

joechenrh changed the title *: add subtask summary to IMPORT INTO tasks *: collect subtask summary for IMPORT INTO tasks Jun 10, 2025

remove some code

48c0bba

joechenrh force-pushed the import-summary branch from f1c194b to 48c0bba Compare June 10, 2025 03:32

joechenrh added 4 commits June 10, 2025 17:10

Add new column

9fe5c53

remove auto update for history table

86603ab

fix version

05f75e5

fix version

7594075

joechenrh mentioned this pull request Jun 11, 2025

Show more detailed progress in SHOW IMPORT JOBS #61088

Closed

1 task

joechenrh commented Jun 12, 2025

View reviewed changes

joechenrh added 2 commits June 12, 2025 11:00

update

6b26377

update

5ba5097

joechenrh commented Jun 12, 2025

View reviewed changes

ti-chi-bot Bot requested review from D3Hunter and GMHDBJD June 12, 2025 03:37

joechenrh added 2 commits June 12, 2025 11:41

fix comment

7e8c635

update comment

4e800bb

D3Hunter reviewed Jun 12, 2025

View reviewed changes

Comment thread pkg/lightning/metric/metric.go Outdated

Apply suggestions from code review

a9cde65

Co-authored-by: D3Hunter <jujj603@gmail.com>

ti-chi-bot Bot merged commit 0a704a4 into pingcap:master Jul 4, 2025
30 of 31 checks passed

joechenrh deleted the import-summary branch July 8, 2025 06:25

Uh oh!

Conversation

joechenrh commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow Bot commented Jun 9, 2025

Uh oh!

joechenrh Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

joechenrh Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

joechenrh Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joechenrh commented Jun 12, 2025

Uh oh!

D3Hunter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joechenrh commented Jul 3, 2025

Uh oh!

tiprow Bot commented Jul 3, 2025

Uh oh!

joechenrh commented Jul 3, 2025

Uh oh!

tiprow Bot commented Jul 3, 2025

Uh oh!

joechenrh commented Jul 3, 2025

Uh oh!

tiprow Bot commented Jul 3, 2025

Uh oh!

joechenrh commented Jul 3, 2025

Uh oh!

tiprow Bot commented Jul 3, 2025

Uh oh!

joechenrh commented Jul 3, 2025

Uh oh!

tiprow Bot commented Jul 3, 2025

Uh oh!

joechenrh commented Jul 3, 2025

Uh oh!

tiprow Bot commented Jul 3, 2025

Uh oh!

joechenrh commented Jul 3, 2025

Uh oh!

tiprow Bot commented Jul 3, 2025

Uh oh!

joechenrh commented Jul 3, 2025

Uh oh!

tiprow Bot commented Jul 3, 2025

Uh oh!

joechenrh commented Jul 4, 2025

Uh oh!

tiprow Bot commented Jul 4, 2025

Uh oh!

joechenrh commented Jul 4, 2025

Uh oh!

tiprow Bot commented Jul 4, 2025

Uh oh!

joechenrh commented Jul 4, 2025

Uh oh!

tiprow Bot commented Jul 4, 2025

Uh oh!

joechenrh commented Jul 4, 2025

Uh oh!

tiprow Bot commented Jul 4, 2025

Uh oh!

ti-chi-bot Bot commented Jul 4, 2025

joechenrh commented Jun 9, 2025 •

edited

Loading

codecov Bot commented Jun 9, 2025 •

edited

Loading

joechenrh Jun 12, 2025 •

edited

Loading