Skip to content

dxf: use correct store in add-index, checksum by copr in import-into, and enable DXF service#62224

Merged
ti-chi-bot[bot] merged 11 commits into
pingcap:masterfrom
D3Hunter:imp-checksum-task-ks-dxf-part
Jul 9, 2025
Merged

dxf: use correct store in add-index, checksum by copr in import-into, and enable DXF service#62224
ti-chi-bot[bot] merged 11 commits into
pingcap:masterfrom
D3Hunter:imp-checksum-task-ks-dxf-part

Conversation

@D3Hunter

@D3Hunter D3Hunter commented Jul 4, 2025

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: ref #61702

Problem Summary:

What changed and how does it work?

  • use correct store to get tableInfo and check table emptyness in add-index
  • for import into: pass Store related to the task ks, and use TiKVChecksumManager to do checksum for nextgen, and use task ks store to rebase auto ids.
  • add DXFSvcTaskMgr which is initialized in user keyspace to access DXF, previous TaskManager always point to current instance
  • refine log to print task-key
  • skip running DXF on user keyspace
  • for cross ks session, use a noop validator to avoid "Information schema is changed during the execution" error, as cross ks info schema is not synced right now.

Check List

Tests

  • Unit test, it's run manually, see TestPostProcessStepExecutor
  • Integration test
  • Manual test (add detailed scripts or steps below)

below tests are run in order

run on ks1
mysql> show config where name='keyspace-name';
+------+----------------------+---------------+-------+
| Type | Instance             | Name          | Value |
+------+----------------------+---------------+-------+
| tidb | 192.168.206.185:4000 | keyspace-name | ks1   |
+------+----------------------+---------------+-------+
1 row in set (0.06 sec)

mysql> create table ks1(ks1id int);
Query OK, 0 rows affected (0.04 sec)

mysql> select tidb_table_id, table_name from information_schema.tables where table_schema='test';
+---------------+------------+
| tidb_table_id | table_name |
+---------------+------------+
|           114 | ks1        |
+---------------+------------+
1 row in set (0.00 sec)

mysql> import into ks1 from 's3://mybucket/a.csv?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2f0.0.0.0%3a9000';
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                         | Target_Table | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
|      1 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`ks1` |      114 |       | finished | 3B               |             1 |                | 2025-07-04 17:30:44.514753 | 2025-07-04 17:30:47.766143 | 2025-07-04 17:30:49.281658 | root@%     |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
1 row in set (4.85 sec)

mysql> select * from ks1;
+-------+
| ks1id |
+-------+
|     1 |
+-------+
1 row in set (0.02 sec)

mysql> show import jobs;
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                         | Target_Table | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
|      1 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`ks1` |      114 |       | finished | 3B               |             1 |                | 2025-07-04 17:30:44.514753 | 2025-07-04 17:30:47.766143 | 2025-07-04 17:30:49.281658 | root@%     |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
1 row in set (0.00 sec)

mysql> alter table ks1 add index(ks1id);
Query OK, 0 rows affected (2.20 sec)

mysql> select * from ks1;
+-------+
| ks1id |
+-------+
|     1 |
+-------+
1 row in set (0.00 sec)

mysql> admin check table ks1;
Query OK, 0 rows affected (0.03 sec)

mysql> select id,state,step,task_key from mysql.tidb_global_task union select id,state,step,task_key from mysql.tidb_global_task_history;
Empty set (0.00 sec)
run on ks2, notice that the table id is different with ks1
mysql> show config where name='keyspace-name';
+------+----------------------+---------------+-------+
| Type | Instance             | Name          | Value |
+------+----------------------+---------------+-------+
| tidb | 192.168.206.185:6000 | keyspace-name | ks2   |
+------+----------------------+---------------+-------+
1 row in set (0.02 sec)

mysql> create table ks2(ks2id int);
Query OK, 0 rows affected (0.06 sec)

mysql> truncate table ks2;
Query OK, 0 rows affected (0.07 sec)

mysql> select tidb_table_id, table_name from information_schema.tables where table_schema='test';
+---------------+------------+
| tidb_table_id | table_name |
+---------------+------------+
|           116 | ks2        |
+---------------+------------+
1 row in set (0.01 sec)

mysql> import into ks2 from 's3://mybucket/a.csv?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2f0.0.0.0%3a9000';
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                         | Target_Table | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
|      1 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`ks2` |      116 |       | finished | 3B               |             1 |                | 2025-07-04 17:34:26.264263 | 2025-07-04 17:34:29.737029 | 2025-07-04 17:34:31.255906 | root@%     |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
1 row in set (5.16 sec)

mysql> select * from ks2;
+-------+
| ks2id |
+-------+
|     1 |
+-------+
1 row in set (0.01 sec)

mysql> show import jobs;
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                         | Target_Table | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
|      1 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`ks2` |      116 |       | finished | 3B               |             1 |                | 2025-07-04 17:34:26.264263 | 2025-07-04 17:34:29.737029 | 2025-07-04 17:34:31.255906 | root@%     |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
1 row in set (0.00 sec)

mysql> alter table ks2 add index(ks2id);
Query OK, 0 rows affected (2.12 sec)

mysql> select * from ks2;
+-------+
| ks2id |
+-------+
|     1 |
+-------+
1 row in set (0.00 sec)

mysql> admin check table ks2;
Query OK, 0 rows affected (0.02 sec)

mysql> select id,state,step,task_key from mysql.tidb_global_task union select id,state,step,task_key from mysql.tidb_global_task_history;
Empty set (0.00 sec)
run on SYSTEM, notice that the table is different with both ks1 and ks2
mysql> show config where name='keyspace-name';
+------+----------------------+---------------+--------+
| Type | Instance             | Name          | Value  |
+------+----------------------+---------------+--------+
| tidb | 192.168.206.185:5000 | keyspace-name | SYSTEM |
+------+----------------------+---------------+--------+
1 row in set (0.02 sec)

mysql> create table sys(sysid int);
Query OK, 0 rows affected (0.08 sec)

mysql> truncate table sys;truncate table sys;
Query OK, 0 rows affected (0.06 sec)

Query OK, 0 rows affected (0.04 sec)

mysql> select tidb_table_id, table_name from information_schema.tables where table_schema='test';
+---------------+------------+
| tidb_table_id | table_name |
+---------------+------------+
|           118 | sys        |
+---------------+------------+
1 row in set (0.00 sec)

mysql> import into sys from 's3://mybucket/a.csv?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2f0.0.0.0%3a9000';
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                         | Target_Table | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
|      1 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`sys` |      118 |       | finished | 3B               |             1 |                | 2025-07-04 17:36:06.009382 | 2025-07-04 17:36:06.532822 | 2025-07-04 17:36:10.055488 | root@%     |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
1 row in set (4.25 sec)

mysql> select * from sys;
+-------+
| sysid |
+-------+
|     1 |
+-------+
1 row in set (0.01 sec)

mysql> show import jobs;
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                         | Target_Table | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
|      1 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`sys` |      118 |       | finished | 3B               |             1 |                | 2025-07-04 17:36:06.009382 | 2025-07-04 17:36:06.532822 | 2025-07-04 17:36:10.055488 | root@%     |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
1 row in set (0.00 sec)

mysql> select * from sys;
+-------+
| sysid |
+-------+
|     1 |
+-------+
1 row in set (0.00 sec)

mysql> alter table sys add index(sysid);
Query OK, 0 rows affected (2.15 sec)

mysql> select * from sys;
+-------+
| sysid |
+-------+
|     1 |
+-------+
1 row in set (0.01 sec)

mysql> admin check table sys;
Query OK, 0 rows affected (0.02 sec)

mysql> select id,state,step,task_key from mysql.tidb_global_task union select id,state,step,task_key from mysql.tidb_global_task_history;
+-------+---------+------+-------------------------+
| id    | state   | step | task_key                |
+-------+---------+------+-------------------------+
| 30001 | succeed |   -2 | ks2/ImportInto/1        |
|     2 | succeed |   -2 | ks1/ddl/backfill/116    |
| 30002 | succeed |   -2 | ks2/ddl/backfill/118    |
|     1 | succeed |   -2 | ks1/ImportInto/1        |
| 60001 | succeed |   -2 | SYSTEM/ImportInto/1     |
| 60002 | succeed |   -2 | SYSTEM/ddl/backfill/120 |
+-------+---------+------+-------------------------+
6 rows in set (0.00 sec)
  • No need to test
    • I checked and no code files have been changed.

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

@ti-chi-bot ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 4, 2025
@tiprow

tiprow Bot commented Jul 4, 2025

Copy link
Copy Markdown

Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@codecov

codecov Bot commented Jul 4, 2025

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 46.70659% with 89 lines in your changes missing coverage. Please review.

Project coverage is 75.4146%. Comparing base (86ed1ca) to head (615a97b).
Report is 19 commits behind head on master.

Additional details and impacted files
@@               Coverage Diff                @@
##             master     #62224        +/-   ##
================================================
+ Coverage   72.9248%   75.4146%   +2.4898%     
================================================
  Files          1750       1797        +47     
  Lines        484467     498993     +14526     
================================================
+ Hits         353297     376314     +23017     
+ Misses       109575      99521     -10054     
- Partials      21595      23158      +1563     
Flag Coverage Δ
integration 49.4195% <7.9754%> (?)
unit 72.7045% <46.7065%> (+0.5370%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
dumpling 52.7804% <ø> (ø)
parser ∅ <ø> (∅)
br 63.2539% <ø> (+16.9241%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@D3Hunter

D3Hunter commented Jul 4, 2025

Copy link
Copy Markdown
Contributor Author

/retest

@tiprow

tiprow Bot commented Jul 4, 2025

Copy link
Copy Markdown

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

if err != nil {
return err
}
return err

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

redundant

@D3Hunter D3Hunter changed the title [WIP] dxf: use correct store in add-index, checksum by copr in import-into, and enable DXF service dxf: use correct store in add-index, checksum by copr in import-into, and enable DXF service Jul 4, 2025
@ti-chi-bot ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 4, 2025
@D3Hunter

D3Hunter commented Jul 7, 2025

Copy link
Copy Markdown
Contributor Author

/retest

@tiprow

tiprow Bot commented Jul 7, 2025

Copy link
Copy Markdown

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@D3Hunter

D3Hunter commented Jul 7, 2025

Copy link
Copy Markdown
Contributor Author

/retest

@tiprow

tiprow Bot commented Jul 7, 2025

Copy link
Copy Markdown

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 7, 2025
@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 7, 2025
@ti-chi-bot

ti-chi-bot Bot commented Jul 7, 2025

Copy link
Copy Markdown

[LGTM Timeline notifier]

Timeline:

  • 2025-07-07 04:51:36.289555637 +0000 UTC m=+1888949.012734620: ☑️ agreed by tangenta.
  • 2025-07-07 06:24:43.029043905 +0000 UTC m=+1894535.752222887: ☑️ agreed by GMHDBJD.

@D3Hunter

D3Hunter commented Jul 7, 2025

Copy link
Copy Markdown
Contributor Author

/retest

@tiprow

tiprow Bot commented Jul 7, 2025

Copy link
Copy Markdown

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot

ti-chi-bot Bot commented Jul 7, 2025

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: GMHDBJD, tangenta, XuHuaiyu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Jul 7, 2025
@D3Hunter

D3Hunter commented Jul 7, 2025

Copy link
Copy Markdown
Contributor Author

/hold

will wait for the dependent project ready, there are only 1 image tag, we cannot override it now

@ti-chi-bot ti-chi-bot Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 7, 2025
@D3Hunter

D3Hunter commented Jul 9, 2025

Copy link
Copy Markdown
Contributor Author

/unhold

@ti-chi-bot ti-chi-bot Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 9, 2025
@D3Hunter

D3Hunter commented Jul 9, 2025

Copy link
Copy Markdown
Contributor Author

/retest

@tiprow

tiprow Bot commented Jul 9, 2025

Copy link
Copy Markdown

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot Bot merged commit 5ec4f36 into pingcap:master Jul 9, 2025
31 checks passed
@D3Hunter D3Hunter deleted the imp-checksum-task-ks-dxf-part branch July 9, 2025 08:07
breezewish added a commit to breezewish/tidb that referenced this pull request Jul 9, 2025
…fix-scalar-func-init

* commit '5ec4f36c855574b64be98e4c9f6b4485ceb4143d': (48 commits)
  dxf: use correct store in add-index, checksum by copr in import-into, and enable DXF service (pingcap#62224)
  infoschema: Add new fields in tiflash system table (pingcap#62296)
  lightning: fix panic on checksum manager close when checksum is off (pingcap#62300)
  *: Add owners cfg file for pkg/metrics module (pingcap#62292)
  autoid: reserve a table ID range for downstream fork (pingcap#62157)
  planner: Fix expression rewriting and method signature mismatch in plan cache (pingcap#58506)
  planner: use prop based noCopPushDown mechanism to replace aggregation field. (pingcap#62249)
  fix: close issue 59457 by trim compare first (pingcap#61915)
  planner: add skew risk ratio for range pred (pingcap#62035)
  meta: unify definition of system or memory DB name (pingcap#62247)
  ddl: add retry for updateSelfVersion (pingcap#62190)
  *: upgrade client go to add config valid function (pingcap#62246)
  executor: fix the inappropriate RequiredRows set by `TopNExec` (pingcap#62154)
  planner: RegardNULLAsPoint should be true as default (pingcap#62194)
  planner: apply predicate simplification before extract condition (pingcap#62211)
  test: fix failed test caused by new version of `mc` (pingcap#61356)
  planner: deprecate the logical interface CanPushToCop and its implementation canPushToCopImpl. (pingcap#62235)
  fix: shut down test without error (pingcap#61921)
  planner: lift the canPushToCop check of logical join/window/selection (pingcap#62206)
  backend/local: add rate limiter for split region and ingest data (pingcap#61555)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants