disttask: support metering on next-gen by tangenta · Pull Request #63397 · pingcap/tidb

tangenta · 2025-09-08T02:35:21Z

What problem does this PR solve?

Issue Number: ref #61702

Problem Summary:

Since DXF subtasks are executed by system keyspace TiDB scaled out automatically, we need to meter the resource usage for billing. This PR collects four metrics for DXF:

Table scan data network traffic.
Actual index data size.
S3 put request count.
S3 get request count.

What changed and how does it work?

Add metering system configuration and reporting goroutine.
Add unit test using local FS for metering data flush logic.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

mysql> show import job 30002;
+--------+-----------+-------------------------------------------------------------------------------------------------------------------------+------------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+----------------------------+----------+-------------------------+---------------------+-----------------------+----------------+--------------+
| Job_ID | Group_Key | Data_Source                                                                                                             | Target_Table     | Table_ID | Phase | Status   | Source_File_Size | Imported_Rows | Result_Message | Create_Time                | Start_Time                 | End_Time                   | Created_By | Last_Update_Time           | Cur_Step | Cur_Step_Processed_Size | Cur_Step_Total_Size | Cur_Step_Progress_Pct | Cur_Step_Speed | Cur_Step_ETA |
+--------+-----------+-------------------------------------------------------------------------------------------------------------------------+------------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+----------------------------+----------+-------------------------+---------------------+-----------------------+----------------+--------------+
|  30002 | NULL      | s3://test/*.csv?access-key=xxxxxx&endpoint=http%3A%2F%2F127.0.0.1%3A9000&force-path-style=true&secret-access-key=xxxxxx | `test`.`sbtest1` |       37 |       | finished | 189.7MiB         |       1000000 |                | 2025-09-17 13:16:37.205778 | 2025-09-17 13:16:37.729492 | 2025-09-17 13:16:50.737305 | root@%     | 2025-09-17 13:16:50.737305 | NULL     | NULL                    | NULL                | NULL                  | NULL           | NULL         |
+--------+-----------+-------------------------------------------------------------------------------------------------------------------------+------------------+----------+-------+----------+------------------+---------------+----------------+----------------------------+----------------------------+----------------------------+------------+----------------------------+----------+-------------------------+---------------------+-----------------------+----------------+--------------+
1 row in set (0.00 sec)

{
    "timestamp": 1758086220,
    "category": "dxf",
    "self_id": "546ef209_62d0_47c7_9cad_979edb401727",
    "part": 0,
    "data": [
        {
            "cluster_id": "SYSTEM",
            "get_external_requests": 40,
            "put_external_requests": 68,
            "read_data_bytes": {
                "value": 198895815,
                "unit": "bytes"
            },
            "source_name": "dxf",
            "task_id": 240003,
            "task_type": "import-into",
            "version": "1",
            "write_data_bytes": {
                "value": 254000000,
                "unit": "bytes"
            }
        }
    ]
}

mysql> admin show ddl jobs 1;
+--------+---------+------------+-----------+--------------+-----------+----------+-----------+----------------------------+----------------------------+----------------------------+--------+---------------------------------------------------------------------------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE  | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME                | START_TIME                 | END_TIME                   | STATE  | COMMENTS                                                                  |
+--------+---------+------------+-----------+--------------+-----------+----------+-----------+----------------------------+----------------------------+----------------------------+--------+---------------------------------------------------------------------------+
|     39 | test    | sbtest1    | add index | public       |         5 |       37 |   1000000 | 2025-09-17 13:17:05.838000 | 2025-09-17 13:17:05.888000 | 2025-09-17 13:17:14.089000 | synced | ingest, DXF, cloud, thread=1, service_scope=dxf_service, max_node_count=1 |
+--------+---------+------------+-----------+--------------+-----------+----------+-----------+----------------------------+----------------------------+----------------------------+--------+---------------------------------------------------------------------------+
1 row in set (0.01 sec)

{
    "timestamp": 1758086280,
    "category": "dxf",
    "self_id": "546ef209_62d0_47c7_9cad_979edb401727",
    "part": 0,
    "data": [
        {
            "cluster_id": "SYSTEM",
            "get_external_requests": 8,
            "put_external_requests": 17,
            "read_data_bytes": {
                "value": 16023636,
                "unit": "bytes"
            },
            "source_name": "dxf",
            "task_id": 240004,
            "task_type": "add-index",
            "version": "1",
            "write_data_bytes": {
                "value": 38000000,
                "unit": "bytes"
            }
        }
    ]
}

No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-09-08T02:35:36Z

Hi @tangenta. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2025-09-09T03:17:37Z

/retest

tiprow · 2025-09-09T03:18:00Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: tangenta <tangenta@126.com>

codecov · 2025-09-15T16:36:26Z

Codecov Report

❌ Patch coverage is 52.52525% with 94 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.0251%. Comparing base (eb25197) to head (7c0a162).
⚠️ Report is 22 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #63397        +/-   ##
================================================
+ Coverage   72.8045%   75.0251%   +2.2205%     
================================================
  Files          1840       1888        +48     
  Lines        497436     510889     +13453     
================================================
+ Hits         362156     383295     +21139     
+ Misses       113284     104149      -9135     
- Partials      21996      23445      +1449

Flag	Coverage Δ
integration	`48.7500% <0.0000%> (?)`
unit	`72.5092% <52.5252%> (+0.1567%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.8700% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.7005% <ø> (+17.1983%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: tangenta <tangenta@126.com>

tangenta · 2025-09-16T08:49:27Z

/retest

tiprow · 2025-09-16T08:49:52Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter

11/20

D3Hunter · 2025-09-16T12:24:01Z

+	m.logger.Warn("metering writer closed", zap.Error(err))
+}
+
+func (m *Meter) flush(ts int64, timeout time.Duration) {


print a log on success flush too with the time cost

D3Hunter · 2025-09-16T12:30:30Z

+}
+
+// RecordDXFReadDataBytes records the scan data bytes from user store for DXF.
+func RecordDXFReadDataBytes(store kv.Storage, taskType TaskType, taskID int64, size uint64) {


not called for import-into

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter · 2025-09-17T02:47:08Z

+	entry := meteringEntry{
+		keyspace: keyspace,
+		taskID:   taskID,
+		taskType: taskType,
+	}
+	orig := m.data[entry]
 	orig.merge(other)
-	m.data[userKeyspace] = orig
+	m.data[entry] = orig


maybe use taskID as key, it's unique already

BenchmarkMapAccess/composite_key,_copy_by_value BenchmarkMapAccess/composite_key,_copy_by_value-10 39376554 31.37 ns/op BenchmarkMapAccess/int_key,_copy_by_ref BenchmarkMapAccess/int_key,_copy_by_ref-10 793769133 1.514 ns/op PASS

type meteringEntry struct { keyspace string taskID int64 taskType string } type Data struct { putRequests uint64 getRequests uint64 readDataBytes uint64 writeDataBytes uint64 } type Data2 struct { meteringEntry putRequests uint64 getRequests uint64 readDataBytes uint64 writeDataBytes uint64 } func BenchmarkMapAccess(b *testing.B) { b.Run("composite key, copy by value", func(b *testing.B) { m := make(map[meteringEntry]Data) for i := 0; i < b.N; i++ { key := meteringEntry{keyspace: "ks1", taskID: 1, taskType: "import"} data := m[key] data.readDataBytes++ m[key] = data } }) b.Run("int key, copy by ref", func(b *testing.B) { m := make(map[int64]*Data2) for i := 0; i < b.N; i++ { if data, ok := m[1]; !ok { data = &Data2{meteringEntry: meteringEntry{keyspace: "ks1", taskID: 1, taskType: "import"}} m[1] = data } else { data.readDataBytes++ } } }) }

D3Hunter · 2025-09-17T02:49:32Z

+			zap.Duration("duration", time.Since(startTime)),
+			zap.Any("data", meteringData))
+	}
+


no log on success😓

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter

rest lgtm

Signed-off-by: tangenta <tangenta@126.com>

ti-chi-bot · 2025-09-18T02:32:45Z

[LGTM Timeline notifier]

Timeline:

2025-09-17 14:46:40.226827008 +0000 UTC m=+24761.705407656: ☑️ agreed by yudongusa.
2025-09-18 02:32:44.497886372 +0000 UTC m=+67125.976467019: ☑️ agreed by D3Hunter.

tiprow · 2025-09-18T02:51:45Z

@tangenta: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`03862ec`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Signed-off-by: tangenta <tangenta@126.com>

ti-chi-bot · 2025-09-18T05:53:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfzjywxk, D3Hunter, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [D3Hunter,cfzjywxk]
~~pkg/config/OWNERS~~ [yudongusa]
~~pkg/ddl/OWNERS~~ [D3Hunter]
~~pkg/disttask/OWNERS~~ [D3Hunter]
~~pkg/domain/OWNERS~~ [D3Hunter]
~~pkg/executor/importer/OWNERS~~ [D3Hunter]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tangenta · 2025-09-18T08:03:40Z

/retest

tiprow · 2025-09-18T08:04:05Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2025-09-18T09:00:41Z

/retest

tiprow · 2025-09-18T09:01:06Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2025-09-18T09:26:30Z

/retest

tiprow · 2025-09-18T09:26:52Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2025-09-18T11:08:34Z

/retest

tiprow · 2025-09-18T11:08:56Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2025-09-18T16:30:19Z

/retest

tiprow · 2025-09-18T16:30:42Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tangenta · 2025-09-19T00:46:21Z

/retest

tiprow · 2025-09-19T00:46:45Z

@tangenta: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot Bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 8, 2025

tangenta changed the title ~~disttask: support metering on next-gen~~ [WIP] disttask: support metering on next-gen Sep 8, 2025

ti-chi-bot Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. and removed do-not-merge/needs-linked-issue labels Sep 8, 2025

disttask: support metering on next-gen

99dec4d

Signed-off-by: tangenta <tangenta@126.com>

tangenta force-pushed the next-gen-metering branch from 1ce6cde to 99dec4d Compare September 15, 2025 16:18

tangenta changed the title ~~[WIP] disttask: support metering on next-gen~~ disttask: support metering on next-gen Sep 15, 2025

ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 15, 2025

tangenta added 5 commits September 16, 2025 10:26

attach task id and task type to metering data

4535738

Signed-off-by: tangenta <tangenta@126.com>

fix TestFlush

b278131

Signed-off-by: tangenta <tangenta@126.com>

fix TestMergeConfigItems

03b912e

Signed-off-by: tangenta <tangenta@126.com>

use metering_storage_uri to upload metering data

224d394

Signed-off-by: tangenta <tangenta@126.com>

fix linter and update bazel

759f281

Signed-off-by: tangenta <tangenta@126.com>

fix import order

2cf657b

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter reviewed Sep 16, 2025

View reviewed changes

tangenta added 2 commits September 16, 2025 21:41

address part of comments

98e6a6a

Signed-off-by: tangenta <tangenta@126.com>

unify record method

99bb74e

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter reviewed Sep 17, 2025

View reviewed changes

Comment thread pkg/disttask/framework/metering/metering.go Outdated

tangenta added 2 commits September 17, 2025 10:44

fix linter

d549ce9

Signed-off-by: tangenta <tangenta@126.com>

metering: only log on flush error

dc19721

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter reviewed Sep 17, 2025

View reviewed changes

tangenta added 2 commits September 17, 2025 12:44

fix unit test and index size recording

7314280

Signed-off-by: tangenta <tangenta@126.com>

fix TestDXFAddIndexRealtimeSummary

48309b1

Signed-off-by: tangenta <tangenta@126.com>

D3Hunter reviewed Sep 17, 2025

View reviewed changes

address comments

03862ec

Signed-off-by: tangenta <tangenta@126.com>

yudongusa approved these changes Sep 17, 2025

View reviewed changes

ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Sep 17, 2025

D3Hunter approved these changes Sep 18, 2025

View reviewed changes

ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Sep 18, 2025

tangenta added 2 commits September 18, 2025 12:19

upgrade metering_sdk

b066796

Signed-off-by: tangenta <tangenta@126.com>

update DEPS.bzl

7c0a162

Signed-off-by: tangenta <tangenta@126.com>

cfzjywxk approved these changes Sep 18, 2025

View reviewed changes

ti-chi-bot Bot added the approved label Sep 18, 2025

ti-chi-bot Bot merged commit b3011b6 into pingcap:master Sep 19, 2025
29 checks passed

D3Hunter mentioned this pull request Sep 22, 2025

DXF service on nextgen kernel #61702

Open

Uh oh!

Conversation

tangenta commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow Bot commented Sep 8, 2025

Uh oh!

tangenta commented Sep 9, 2025

Uh oh!

tiprow Bot commented Sep 9, 2025

Uh oh!

codecov Bot commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

tangenta commented Sep 16, 2025

Uh oh!

tiprow Bot commented Sep 16, 2025

Uh oh!

D3Hunter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

D3Hunter Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

D3Hunter Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

D3Hunter Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

D3Hunter Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

D3Hunter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ti-chi-bot Bot commented Sep 18, 2025

[LGTM Timeline notifier]

Uh oh!

tiprow Bot commented Sep 18, 2025

Uh oh!

ti-chi-bot Bot commented Sep 18, 2025

Uh oh!

tangenta commented Sep 18, 2025

Uh oh!

tiprow Bot commented Sep 18, 2025

Uh oh!

tangenta commented Sep 18, 2025

Uh oh!

tiprow Bot commented Sep 18, 2025

Uh oh!

tangenta commented Sep 18, 2025

Uh oh!

tiprow Bot commented Sep 18, 2025

Uh oh!

tangenta commented Sep 18, 2025

Uh oh!

tiprow Bot commented Sep 18, 2025

Uh oh!

tangenta commented Sep 18, 2025

Uh oh!

tangenta commented Sep 8, 2025 •

edited

Loading

codecov Bot commented Sep 15, 2025 •

edited

Loading