domain: record runaway and quarantine query #44654

CabinfeverB · 2023-06-13T15:10:11Z

What problem does this PR solve?

Issue Number: ref #43691

Problem Summary:

What is changed and how it works?

Persist the runaway query records in KV data by the admin table

Once identified or watched quarantined are recorded in mysql.RUNAWAY_QUERIES. mysql.QUARANTINED_WATCH only active watch are kept

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

CabinfeverB · 2023-06-17T01:18:00Z

seems stale quarantined watch record is not cleaned

Can we keep the history for now? Users can use select * from mysql.quarantined_watch where end_time >= now(); to query the active rules.

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

CabinfeverB · 2023-06-17T01:20:52Z

cc @glorv @nolouch

Connor1996 · 2023-06-17T04:04:05Z

seems stale quarantined watch record is not cleaned

Can we keep the history for now? Users can use select * from mysql.quarantined_watch where end_time >= now(); to query the active rules.

Is it temporary? otherwise I need to change the doc.

Connor1996

LGTM

ti-chi-bot · 2023-06-17T04:04:21Z

@Connor1996: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

LGTM

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

glorv · 2023-06-17T04:33:05Z

/retest

CabinfeverB · 2023-06-17T07:57:50Z

IMO, it's not temporary. What do u think? @glorv

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

CabinfeverB · 2023-06-17T09:29:55Z

/retest-required

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

nolouch · 2023-06-17T11:06:29Z

session/bootstrap.go

@@ -604,6 +604,31 @@ const (
       KEY (create_time),
       KEY (create_user));`

+	// CreateRunawayTable stores the query which is identified as runaway or quarantined because of in watch list.
+	CreateRunawayTable = `CREATE TABLE IF NOT EXISTS mysql.runaway_queries (


Suggested change

CreateRunawayTable = `CREATE TABLE IF NOT EXISTS mysql.runaway_queries (

CreateRunawayTable = `CREATE TABLE IF NOT EXISTS mysql.tidb_runaway_queries (

I suggest using tidb_xxx as the name indicates tidb specific table.

nolouch · 2023-06-17T11:07:14Z

session/bootstrap.go

+	) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;`
+
+	// CreateQuarantineWatchTable stores the condition which is used to check whether query should be quarantined.
+	CreateQuarantineWatchTable = `CREATE TABLE IF NOT EXISTS mysql.quarantined_watch (


How about tidb_runaway_quarantined_watch

glorv · 2023-06-17T11:35:04Z

IMO, it's not temporary. What do u think? @glorv

As each quarantined query has a TLL, I think that means they should be GCed over time. We may treat them as logs, gc them after a period of time, e.g. after 30 days or so.

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

CabinfeverB · 2023-06-18T07:00:31Z

/test check-dev2

ti-chi-bot · 2023-06-18T08:02:49Z

[LGTM Timeline notifier]

Timeline:

2023-06-18 07:42:34.845022903 +0000 UTC m=+503551.259626982: ☑️ agreed by nolouch.
2023-06-18 08:02:48.089140563 +0000 UTC m=+504764.503744630: ☑️ agreed by glorv.

ti-chi-bot · 2023-06-18T15:32:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Connor1996, glorv, nolouch, qw4990

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [glorv,qw4990]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Connor1996 and others added 30 commits June 1, 2023 14:10

introduce runaway manager

b38cbd3

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

refactor

0c29816

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

update bazel

1ed89d4

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

runaway

ce9038c

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

update bazel

a12c48f

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

draft for watch

5da907c

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

draft for watch

1d430aa

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

fix draft

e530127

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

fix draft

a7751c7

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

differ error

17ad887

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

fix watch

0d2ce5d

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

merge master

e55f323

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

fix build

7ce6412

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

rename

dc2e0ee

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Merge remote-tracking branch 'upstream/master' into runaway

1ca9233

fix test

2b1dde6

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

fix test and init default resource group name

0dba575

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

address comment

5c8dadd

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

fix resource group name

0ab9e11

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

set timeout

c3adcb0

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

add test

3c5f58b

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

add test

8003741

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

fix test

ac060d3

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

update kvproto

bfec95f

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

fix lint

c9b2844

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

add more test cases

cc82ac9

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

update bazel

ec05ebc

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

refactor

fc8e927

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

avoid duplicated mark

5df043f

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

add mysql.quarantine_watch

d89aeb9

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

address comment

8732117

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

Connor1996 approved these changes Jun 17, 2023

View reviewed changes

address comment

259cb1f

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

fix test

0ce2208

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

nolouch reviewed Jun 17, 2023

View reviewed changes

CabinfeverB added 2 commits June 18, 2023 00:26

fix test

d77a972

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

fix test

c3b9d97

Signed-off-by: Cabinfever_B <cabinfeveroier@gmail.com>

nolouch approved these changes Jun 18, 2023

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 18, 2023

glorv approved these changes Jun 18, 2023

View reviewed changes

ti-chi-bot bot added lgtm approved and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 18, 2023

hawkingrei requested review from qw4990 and XuHuaiyu June 18, 2023 14:38

qw4990 approved these changes Jun 18, 2023

View reviewed changes

ti-chi-bot bot merged commit 0f48433 into pingcap:master Jun 18, 2023

CabinfeverB deleted the resource_manager/runaway_record branch June 19, 2023 06:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

domain: record runaway and quarantine query #44654

domain: record runaway and quarantine query #44654

CabinfeverB commented Jun 13, 2023 •

edited

Loading

CabinfeverB commented Jun 17, 2023 •

edited

Loading

CabinfeverB commented Jun 17, 2023

Connor1996 commented Jun 17, 2023

Connor1996 left a comment

ti-chi-bot bot commented Jun 17, 2023

glorv commented Jun 17, 2023

CabinfeverB commented Jun 17, 2023

CabinfeverB commented Jun 17, 2023

nolouch Jun 17, 2023 •

edited

Loading

glorv Jun 17, 2023

nolouch Jun 17, 2023

glorv commented Jun 17, 2023

CabinfeverB commented Jun 18, 2023

ti-chi-bot bot commented Jun 18, 2023

ti-chi-bot bot commented Jun 18, 2023

	CreateRunawayTable = `CREATE TABLE IF NOT EXISTS mysql.runaway_queries (
	CreateRunawayTable = `CREATE TABLE IF NOT EXISTS mysql.tidb_runaway_queries (

domain: record runaway and quarantine query #44654

domain: record runaway and quarantine query #44654

Conversation

CabinfeverB commented Jun 13, 2023 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

CabinfeverB commented Jun 17, 2023 • edited Loading

CabinfeverB commented Jun 17, 2023

Connor1996 commented Jun 17, 2023

Connor1996 left a comment

Choose a reason for hiding this comment

ti-chi-bot bot commented Jun 17, 2023

glorv commented Jun 17, 2023

CabinfeverB commented Jun 17, 2023

CabinfeverB commented Jun 17, 2023

nolouch Jun 17, 2023 • edited Loading

Choose a reason for hiding this comment

glorv Jun 17, 2023

Choose a reason for hiding this comment

nolouch Jun 17, 2023

Choose a reason for hiding this comment

glorv commented Jun 17, 2023

CabinfeverB commented Jun 18, 2023

ti-chi-bot bot commented Jun 18, 2023

[LGTM Timeline notifier]

ti-chi-bot bot commented Jun 18, 2023

CabinfeverB commented Jun 13, 2023 •

edited

Loading

CabinfeverB commented Jun 17, 2023 •

edited

Loading

nolouch Jun 17, 2023 •

edited

Loading