domain: Introduce runaway manager #44339

Connor1996 · 2023-06-01T13:02:02Z

What problem does this PR solve?

Issue Number: Ref #43691

Problem Summary:

What is changed and how it works?

Introduce runaway manager to check query execution time and perform actions

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Introduce runaway query control based on resource group

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

ti-chi-bot · 2023-06-01T13:02:05Z

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

ti-chi-bot · 2023-06-01T13:02:05Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

executor/executor.go

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

executor/executor.go

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

glorv · 2023-06-15T04:19:02Z

/retest

glorv · 2023-06-15T04:50:34Z

/retest

nolouch

rest lgtm

domain/domain.go

domain/resourcegroup/runaway.go

nolouch · 2023-06-15T05:59:44Z

errno/errname.go

@@ -1138,6 +1138,7 @@ var MySQLErrName = map[uint16]*mysql.ErrMessage{
 	ErrResourceGroupSupportDisabled:   mysql.Message("Resource control feature is disabled. Run `SET GLOBAL tidb_enable_resource_control='on'` to enable the feature", nil),
 	ErrResourceGroupConfigUnavailable: mysql.Message("Resource group configuration is unavailable", nil),
 	ErrResourceGroupThrottled:         mysql.Message("Exceeded resource group quota limitation", nil),
+	ErrResourceGroupQueryRunaway:      mysql.Message("Query execution was interrupted, identified as runaway query", nil),


Mabye "Query execution was terminated, identified as a runaway query." is better

other places use interrupted

glorv · 2023-06-15T06:15:24Z

domain/resourcegroup/runaway.go

+
+// DeriveChecker derives a RunawayChecker from the given resource group
+func (rm *RunawayManager) DeriveChecker(resourceGroupName string, originalSQL string, planDigest string) *RunawayChecker {
+	group, err := rm.resourceGroupCtl.GetResourceGroup(resourceGroupName)


Using InfoSchema.ResourceGroupByName to get the group config is better here, not need to always fetch from pd.

resource group are cached in resouce manager client, and return the rmpb.ResourceGroup which is more convenient

But then you need to watch the resource group config changes in this client while info-schema is already the standard way to sync config info between all tidb servers.

We have considered info schema at first. As we already stored the resource group in both PD and TiDB meta, we just follow the original way to sync resource group by PD, otherwise, TiKV and Tiflash are hard to watch resource group changes.

Though I do think storing resource groups in both PD and TiDB info schema is a little messy

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Connor1996 · 2023-06-15T18:39:33Z

/retest

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

CabinfeverB · 2023-06-15T21:42:02Z

/test unit-test

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

ti-chi-bot · 2023-06-15T22:51:17Z

@CabinfeverB: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

HuSharp · 2023-06-16T00:01:21Z

domain/infosync/resource_manager_client.go

@@ -0,0 +1,147 @@
+// Copyright 2022 PingCAP, Inc.


Suggested change

// Copyright 2022 PingCAP, Inc.

// Copyright 2023 PingCAP, Inc.

It's a rename of an old file with some changes. Github regards it as a totally new file.

HuSharp

LGTM!

ti-chi-bot · 2023-06-16T00:16:20Z

@HuSharp: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

LGTM!

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

nolouch

lgtm

ti-chi-bot · 2023-06-16T02:48:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CabinfeverB, glorv, HuSharp, nolouch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [glorv]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2023-06-16T02:48:48Z

[LGTM Timeline notifier]

Timeline:

2023-06-14 07:27:06.691461943 +0000 UTC m=+157023.106066023: ☑️ agreed by glorv.
2023-06-16 02:48:47.854287561 +0000 UTC m=+313124.268891655: ☑️ agreed by nolouch.

glorv · 2023-06-16T03:20:20Z

/retest

Connor1996 added 2 commits June 1, 2023 14:10

introduce runaway manager

b38cbd3

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

refactor

0c29816

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

ti-chi-bot bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 1, 2023

update bazel

1ed89d4

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

ti-chi-bot bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 5, 2023

Connor1996 mentioned this pull request Jun 7, 2023

Runaway query control based on resource group #43691

Closed

28 tasks

fix build

7ce6412

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

ti-chi-bot bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 7, 2023

rename

dc2e0ee

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

CabinfeverB reviewed Jun 7, 2023

View reviewed changes

executor/executor.go Outdated Show resolved Hide resolved

Merge remote-tracking branch 'upstream/master' into runaway

1ca9233

ti-chi-bot bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 7, 2023

fix test

2b1dde6

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

CabinfeverB reviewed Jun 7, 2023

View reviewed changes

executor/executor.go Outdated Show resolved Hide resolved

fix test and init default resource group name

0dba575

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Connor1996 marked this pull request as ready for review June 7, 2023 12:37

Connor1996 requested a review from a team as a code owner June 7, 2023 12:37

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 7, 2023

address comment

5c8dadd

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Connor1996 force-pushed the runaway branch from d607b6a to 5c8dadd Compare June 7, 2023 12:55

Connor1996 added 2 commits June 7, 2023 21:48

fix resource group name

0ab9e11

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

set timeout

c3adcb0

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

ti-chi-bot bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Jun 8, 2023

fix test

ac060d3

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

update nogo ignore

032c9ec

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

nolouch reviewed Jun 15, 2023

View reviewed changes

glorv reviewed Jun 15, 2023

View reviewed changes

This was referenced Jun 15, 2023

Fix integration compilation failure of client-go #44591

Closed

mock: impl resource manager get default resource group tikv/client-go#839

Merged

Merge remote-tracking branch 'upstream/master' into runaway

5fe37ef

Connor1996 force-pushed the runaway branch from 814ec07 to d4d5cc8 Compare June 15, 2023 16:25

update client-go to fix test

608ad12

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Connor1996 force-pushed the runaway branch from d4d5cc8 to 608ad12 Compare June 15, 2023 16:26

Connor1996 added 2 commits June 15, 2023 10:47

address comment

5929766

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

update bazel

0f02a43

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

avoid data race

9382ffe

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

clean stale test

4a6f874

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

CabinfeverB approved these changes Jun 15, 2023

View reviewed changes

HuSharp reviewed Jun 16, 2023

View reviewed changes

HuSharp approved these changes Jun 16, 2023

View reviewed changes

nolouch approved these changes Jun 16, 2023

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 16, 2023

CabinfeverB mentioned this pull request Jun 16, 2023

executor: impl runaway watch check #44474

Merged

12 tasks

ti-chi-bot bot merged commit 7a29bec into pingcap:master Jun 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

domain: Introduce runaway manager #44339

domain: Introduce runaway manager #44339

Connor1996 commented Jun 1, 2023 •

edited

Loading

ti-chi-bot bot commented Jun 1, 2023

ti-chi-bot bot commented Jun 1, 2023

glorv commented Jun 15, 2023

glorv commented Jun 15, 2023

nolouch left a comment

nolouch Jun 15, 2023

Connor1996 Jun 15, 2023

glorv Jun 15, 2023

CabinfeverB Jun 15, 2023

glorv Jun 15, 2023

Connor1996 Jun 15, 2023

Connor1996 Jun 15, 2023

Connor1996 commented Jun 15, 2023

CabinfeverB commented Jun 15, 2023

ti-chi-bot bot commented Jun 15, 2023

HuSharp Jun 16, 2023

Connor1996 Jun 16, 2023

HuSharp left a comment

ti-chi-bot bot commented Jun 16, 2023

nolouch left a comment

ti-chi-bot bot commented Jun 16, 2023

ti-chi-bot bot commented Jun 16, 2023

glorv commented Jun 16, 2023

	// Copyright 2022 PingCAP, Inc.
	// Copyright 2023 PingCAP, Inc.

domain: Introduce runaway manager #44339

domain: Introduce runaway manager #44339

Conversation

Connor1996 commented Jun 1, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot bot commented Jun 1, 2023

ti-chi-bot bot commented Jun 1, 2023

glorv commented Jun 15, 2023

glorv commented Jun 15, 2023

nolouch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Connor1996 commented Jun 15, 2023

CabinfeverB commented Jun 15, 2023

ti-chi-bot bot commented Jun 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HuSharp left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Jun 16, 2023

nolouch left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Jun 16, 2023

ti-chi-bot bot commented Jun 16, 2023

[LGTM Timeline notifier]

glorv commented Jun 16, 2023

Connor1996 commented Jun 1, 2023 •

edited

Loading