resource_control: support collecting cross AZ traffic in ru consumption by glorv · Pull Request #62003 · pingcap/tidb

glorv · 2025-06-25T09:33:44Z

What problem does this PR solve?

Issue Number: close #62001

Problem Summary:

What changed and how does it work?

Add cross AZ network traffic statistics in ru consumption and add related grafana panels.

This PR depends on tikv/client-go#1669

The new panels snapshot:

In comparison, the "coprocessor response size" metrics is as following:

NOTE:

there are 3 tikvs and 1 tidb instances, 2 tikvs are in different zone and 1 is in the same zone.
The cross AZ traffic is slightly higher the the total coprocessor response sizes(of 2 tikv instances) because the request size is also counted.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2025-06-25T09:33:59Z

Hi @glorv. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

glorv · 2025-06-26T02:29:37Z

/retest

tiprow · 2025-06-26T02:29:59Z

@glorv: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2025-06-26T04:10:22Z

Codecov Report

Attention: Patch coverage is 62.50000% with 6 lines in your changes missing coverage. Please review.

Project coverage is 75.5851%. Comparing base (4c9e2ea) to head (1de7652).
Report is 14 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #62003        +/-   ##
================================================
+ Coverage   72.9269%   75.5851%   +2.6581%     
================================================
  Files          1740       1794        +54     
  Lines        483048     498249     +15201     
================================================
+ Hits         352272     376602     +24330     
+ Misses       109207      98830     -10377     
- Partials      21569      22817      +1248

Flag	Coverage Δ
integration	`49.8390% <62.5000%> (?)`
unit	`72.9607% <62.5000%> (+0.7933%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`52.7804% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`63.0744% <ø> (+16.7420%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

glorv · 2025-06-26T06:35:18Z

@guo-shaoge @nolouch @JmPotato PTAL

JmPotato · 2025-06-26T06:55:23Z

 			ResourceGroupName:             sc.ResourceGroupName,
 			LoadBasedReplicaReadThreshold: vars.LoadBasedReplicaReadThreshold,
 			RunawayChecker:                sc.RunawayChecker,
+			RUConsumptionReporter:         s.GetDomain().(*domain.Domain).ResourceGroupsController(),


Just in case, I still want to ask if we won't encounter the same issue as in #61899 (comment) here, right?

No. The RUConsumptionReporter is always the global ResourceGroupsController which is not changed in the whole lifetime of this instance, so it is different from the pre-statement RunawayCheck.

guo-shaoge · 2025-07-02T07:19:40Z

+// GetInterZoneTrafficBytes returns the inter zone network traffic bytes involved
+// between tiflash instances.
+func (networkTraffic *TiFlashNetworkTrafficSummary) GetInterZoneTrafficBytes() uint64 {
+	return networkTraffic.interZoneSendBytes + networkTraffic.interZoneReceiveBytes


add comments to explain why send bytes is used

…z-traffic

glorv · 2025-07-03T08:11:12Z

@guo-shaoge @nolouch PTAL again, thanks~

ti-chi-bot · 2025-07-03T22:45:48Z

[LGTM Timeline notifier]

Timeline:

2025-06-26 07:15:42.653059944 +0000 UTC m=+947195.376238925: ☑️ agreed by JmPotato.
2025-07-03 22:45:47.547387481 +0000 UTC m=+1607800.270566463: ☑️ agreed by nolouch.

ti-chi-bot · 2025-07-04T03:31:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cfzjywxk, guo-shaoge, JmPotato, nolouch, yudongusa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [JmPotato,cfzjywxk,guo-shaoge,nolouch]
~~pkg/config/OWNERS~~ [yudongusa]
~~pkg/distsql/OWNERS~~ [cfzjywxk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

glorv · 2025-07-04T06:15:53Z

/retest

tiprow · 2025-07-04T06:16:24Z

@glorv: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

glorv · 2025-07-04T07:17:06Z

/retest

tiprow · 2025-07-04T07:17:28Z

@glorv: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

Details

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

* origin/master: (79 commits) backend/local: add rate limiter for split region and ingest data (pingcap#61555) fix(runaway): resolve the dead channel in UpdateNewAndDoneWatch (pingcap#61795) *: collect subtask summary for `IMPORT INTO` tasks (pingcap#61600) tests: update vector cluster test result (pingcap#62174) planner: change casetest instance-plan-cache test timeout. (pingcap#62195) planner: remove data race for instance plan cache (pingcap#62217) *: upgrade pd client to fix keyspace issue (pingcap#62189) resource_control: support collecting cross AZ traffic in ru consumption (pingcap#62003) test: wait for statistics owner to close on exit (pingcap#62198) planner: remove the selection above the table dual (pingcap#62193) telemetry: add telemetry related code back (pingcap#61753) planner: move physical limit related logic into physical op. (pingcap#62187) planner: remove useless code (pingcap#61968) retry: enhance retry for ingestor on nextgen (pingcap#62191) importinto: encode and ingest step use task keyspace in nextgen (pingcap#62138) planner: lift the can-push-to-cop check of aggregation/proj (pingcap#62068) ddl: cannot panic with failpoint panic in the DDLNotifier (pingcap#62151) planner: skip empty expression in the applyPredicateSimplification (pingcap#62178) codec: should also return the value even with error because the error may be ignored in the future (pingcap#62175) parser: also reduce literal list with charset (pingcap#61484) ...

glorv added 2 commits June 25, 2025 17:09

support collecting cross AZ network traffic

160dd20

update client-go

f100ba9

glorv requested review from Connor1996, JmPotato and nolouch June 25, 2025 09:33

ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 25, 2025

glorv requested a review from guo-shaoge June 25, 2025 09:34

glorv added 2 commits June 25, 2025 17:49

add new metrics panels

0b37d6e

update bzl

e97ed3c

ti-chi-bot Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 25, 2025

update client-go

39de203

glorv added 4 commits June 26, 2025 10:37

fix test

1e5b9c3

fix build

419b9b0

fix check

d9a6686

fix

0fc9c8d

JmPotato reviewed Jun 26, 2025

View reviewed changes

JmPotato approved these changes Jun 26, 2025

View reviewed changes

ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 26, 2025

guo-shaoge reviewed Jul 2, 2025

View reviewed changes

glorv added 2 commits July 3, 2025 11:21

Merge branch 'master' of https://github.com/pingcap/tidb into cross-a…

0dff3dc

…z-traffic

fix tiflash traffic

1de7652

nolouch approved these changes Jul 3, 2025

View reviewed changes

ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jul 3, 2025

guo-shaoge approved these changes Jul 4, 2025

View reviewed changes

cfzjywxk approved these changes Jul 4, 2025

View reviewed changes

yudongusa approved these changes Jul 4, 2025

View reviewed changes

ti-chi-bot Bot added the approved label Jul 4, 2025

ti-chi-bot Bot merged commit cc90d37 into pingcap:master Jul 4, 2025
24 checks passed

glorv deleted the cross-az-traffic branch July 4, 2025 11:57

Uh oh!

Conversation

glorv commented Jun 25, 2025

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

Uh oh!

tiprow Bot commented Jun 25, 2025

Uh oh!

glorv commented Jun 26, 2025

Uh oh!

tiprow Bot commented Jun 26, 2025

Uh oh!

codecov Bot commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

glorv commented Jun 26, 2025

Uh oh!

JmPotato Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

glorv Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

guo-shaoge Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

glorv commented Jul 3, 2025

Uh oh!

ti-chi-bot Bot commented Jul 3, 2025

[LGTM Timeline notifier]

Uh oh!

ti-chi-bot Bot commented Jul 4, 2025

Uh oh!

glorv commented Jul 4, 2025

Uh oh!

tiprow Bot commented Jul 4, 2025

Uh oh!

glorv commented Jul 4, 2025

Uh oh!

tiprow Bot commented Jul 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Jun 26, 2025 •

edited

Loading