importinto: use correct session to submit/update/query task in DXF service

[D3Hunter]

What problem does this PR solve?

Issue Number: ref #61702

Problem Summary:

What changed and how does it work?

Note: we have to uncomment GetTaskMgrToAccessDXFService to work in DXF service, as it's not ready now, such as the encode and checksum part, we still keep the old behavior

• use correct session to submit/update/query task in DXF service, i.e:
  • when running on user keyspace, use SYSTEM keyspace session to submit and query task. and use the task keyspace session to update import job status
• split into 2 transaction when submit task in user keyspace

Check List

Tests

• Unit test. it's run manually, as it only run in nextgen
• Integration test. it's run manually, as it only run in nextgen
• Manual test (add detailed scripts or steps below)

uncomment GetTaskMgrToAccessDXFService to submit task to DXF service, then start 2 TIDB, one SYSTEM ks, one ks1 ks

run import into on session of ks1, we can see the DXF task table is empty

mysql> show config where name='keyspace-name';
+------+------------------+---------------+-------+
| Type | Instance         | Name          | Value |
+------+------------------+---------------+-------+
| tidb | 192.168.2.4:4000 | keyspace-name | ks1   |
+------+------------------+---------------+-------+
1 row in set (0.07 sec)

mysql> create table t(id int);
Query OK, 0 rows affected (0.09 sec)

mysql> import into t from 's3://mybucket/a.csv?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2f0.0.0.0%3a9000' with checksum_table='off';
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                         | Target_Table | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
|      1 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`t`   |      114 |       | finished | 2B               |             0 |                | 2025-06-27 11:39:43.241895 | 2025-06-27 11:39:45.603677 | 2025-06-27 11:39:57.617254 | root@%     |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
1 row in set (14.47 sec)

mysql> import into t from 's3://mybucket/a.csv?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3a%2f%2f0.0.0.0%3a9000' with checksum_table='off';
ERROR 1105 (HY000): cancelled by user

mysql> select id,state,step,task_key from mysql.tidb_global_task union select id,state,step,task_key from mysql.tidb_global_task_history;
Empty set (0.01 sec)

and from the SYSTEM ks, we can see the DXF task table contains them

mysql> show config where name='keyspace-name';
+------+------------------+---------------+--------+
| Type | Instance         | Name          | Value  |
+------+------------------+---------------+--------+
| tidb | 192.168.2.4:5000 | keyspace-name | SYSTEM |
+------+------------------+---------------+--------+
1 row in set (0.08 sec)

mysql> show import jobs;
Empty set (0.01 sec)

mysql> select id,state,step,task_key from mysql.tidb_global_task union select id,state,step,task_key from mysql.tidb_global_task_history;
+----+----------+------+------------------+
| id | state    | step | task_key         |
+----+----------+------+------------------+
|  1 | succeed  |   -2 | ks1/ImportInto/1 |
|  2 | reverted |    2 | ks1/ImportInto/2 |
+----+----------+------+------------------+
2 rows in set (0.01 sec)

show state of a running import job and cancel the job in another session on ks1

mysql> show config where name='keyspace-name';
+------+------------------+---------------+-------+
| Type | Instance         | Name          | Value |
+------+------------------+---------------+-------+
| tidb | 192.168.2.4:4000 | keyspace-name | ks1   |
+------+------------------+---------------+-------+
1 row in set (0.02 sec)

mysql> show import jobs;
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+------------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                         | Target_Table | Table_ID | Phase      | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+------------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
|      1 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`t`   |      114 |            | finished | 2B               |             0 |                | 2025-06-27 11:39:43.241895 | 2025-06-27 11:39:45.603677 | 2025-06-27 11:39:57.617254 | root@%     |
|      2 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`t`   |      114 | validating | running  | 2B               |             0 |                | 2025-06-27 11:40:16.927224 | 2025-06-27 11:40:18.525710 | NULL                       | root@%     |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+------------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+
2 rows in set (0.03 sec)

mysql> cancel import job 2;
Query OK, 0 rows affected (0.63 sec)

mysql> show import jobs;
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+------------+-----------+------------------+---------------+-------------------+----------------------------+----------------------------+----------------------------+------------+
| Job_ID | Data_Source                                                                                         | Target_Table | Table_ID | Phase      | Status    | Source_File_Size | Imported_Rows | Result_Message    | Create_Time                | Start_Time                 | End_Time                   | Created_By |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+------------+-----------+------------------+---------------+-------------------+----------------------------+----------------------------+----------------------------+------------+
|      1 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`t`   |      114 |            | finished  | 2B               |             0 |                   | 2025-06-27 11:39:43.241895 | 2025-06-27 11:39:45.603677 | 2025-06-27 11:39:57.617254 | root@%     |
|      2 | s3://mybucket/a.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F0.0.0.0%3A9000&secret-access-key=xxxxxx | `test`.`t`   |      114 | validating | cancelled | 2B               |          NULL | cancelled by user | 2025-06-27 11:40:16.927224 | 2025-06-27 11:40:18.525710 | NULL                       | root@%     |
+--------+-----------------------------------------------------------------------------------------------------+--------------+----------+------------+-----------+------------------+---------------+-------------------+----------------------------+----------------------------+----------------------------+------------+
2 rows in set (0.00 sec)

• No need to test

  • I checked and no code files have been changed.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

[tiprow]

Hi @D3Hunter. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov]

Codecov Report

Attention: Patch coverage is 27.86885% with 88 lines in your changes missing coverage. Please review.

Project coverage is 73.4381%. Comparing base (3523946) to head (25a5d01).
Report is 6 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #62023        +/-   ##
================================================
+ Coverage   72.9309%   73.4381%   +0.5072%     
================================================
  Files          1736       1736                
  Lines        482403     483905      +1502     
================================================
+ Hits         351821     355371      +3550     
+ Misses       109000     106974      -2026     
+ Partials      21582      21560        -22     

Flag	Coverage Δ
integration	42.3558% <0.0000%> (?)
unit	72.3325% <27.8688%> (+0.1460%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	52.7804% <ø> (ø)
parser	∅ <ø> (∅)
br	46.3254% <ø> (-0.3053%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[D3Hunter]

will uncomment this part when we are ready to switch to DXF service, encode and checksum are not ready now

[tangenta]

If there is a plan to abandon the DXF service in user keyspace, I think we don't need to load the system DXF service every time? Simply initialize the only GetTaskManager to the DXF service one is enough.

[D3Hunter]

yes, we can do it later to avoid block the e2e test, right now, import-into only runs across keyspace

[D3Hunter]

/retest

[tiprow]

@D3Hunter: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tangenta, wjhuang2016

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [tangenta,wjhuang2016]
• pkg/disttask/OWNERS [tangenta,wjhuang2016]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2025-06-27 09:33:14.851939942 +0000 UTC m=+1041847.575118924: ☑️ agreed by wjhuang2016.
• 2025-06-28 05:23:01.648929495 +0000 UTC m=+1113234.372108514: ☑️ agreed by tangenta.