Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor: Add the HashAggExec runtime information #20577

Merged
merged 78 commits into from
Nov 25, 2020
Merged

executor: Add the HashAggExec runtime information #20577

merged 78 commits into from
Nov 25, 2020

Conversation

jyz0309
Copy link
Contributor

@jyz0309 jyz0309 commented Oct 22, 2020 •
edited
Loading

What problem does this PR solve?

Problem Summary:
Currently, the runtime stats of IndexMergeReaderExecutor is hard to understand, here is an example:

mysql> explain analyze SELECT /*+ HASH_AGG() */ count(*) FROM t1 WHERE a > 10;
+----------------------------+---------+---------+-----------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+--------------+------+
| id                         | estRows | actRows | task      | access object        | execution info                                                                                                                           | operator info                     | memory       | disk |
+----------------------------+---------+---------+-----------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+--------------+------+
| HashAgg_11                 | 1.00    | 1       | root      |                      | time:487.272µs, loops:2, test, PartialConcurrency:5, FinalConcurrency:5                                                                  | funcs:count(Column#5)->Column#4   | 3.6328125 KB | N/A  |
| └─IndexReader_12           | 1.00    | 1       | root      |                      | time:430.392µs, loops:2, cop_task: {num: 1, max:406.608µs, proc_keys: 0, rpc_num: 1, rpc_time: 378.445µs, copr_cache_hit_ratio: 1.00}    | index:HashAgg_6                   | 201 Bytes    | N/A  |
|   └─HashAgg_6              | 1.00    | 1       | cop[tikv] |                      | time:21ms, loops:10                                                                                                                      | funcs:count(1)->Column#5          | N/A          | N/A  |
|     └─IndexRangeScan_9     | 3333.33 | 9999    | cop[tikv] | table:t1, index:a(a) | time:21ms, loops:10                                                                                                                      | range:(10,+inf], keep order:false | N/A          | N/A  |
+----------------------------+---------+---------+-----------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+--------------+------+
4 rows in set (0.00 sec)

What is changed and how it works?

What's Changed:
Add runtime stats for HashAggExec, and add the String()、Merge()、Tp() for it.

Before this pr:

mysql> explain analyze SELECT /*+ HASH_AGG() */ count(*) FROM t1 WHERE a > 10;
+----------------------------+---------+---------+-----------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+--------------+------+
| id                         | estRows | actRows | task      | access object        | execution info                                                                                                                           | operator info                     | memory       | disk |
+----------------------------+---------+---------+-----------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+--------------+------+
| HashAgg_11                 | 1.00    | 1       | root      |                      | time:487.272µs, loops:2, test, PartialConcurrency:5, FinalConcurrency:5                                                                  | funcs:count(Column#5)->Column#4   | 3.6328125 KB | N/A  |
| └─IndexReader_12           | 1.00    | 1       | root      |                      | time:430.392µs, loops:2, cop_task: {num: 1, max:406.608µs, proc_keys: 0, rpc_num: 1, rpc_time: 378.445µs, copr_cache_hit_ratio: 1.00}    | index:HashAgg_6                   | 201 Bytes    | N/A  |
|   └─HashAgg_6              | 1.00    | 1       | cop[tikv] |                      | time:21ms, loops:10                                                                                                                      | funcs:count(1)->Column#5          | N/A          | N/A  |
|     └─IndexRangeScan_9     | 3333.33 | 9999    | cop[tikv] | table:t1, index:a(a) | time:21ms, loops:10                                                                                                                      | range:(10,+inf], keep order:false | N/A          | N/A  |
+----------------------------+---------+---------+-----------+----------------------+------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+--------------+------+
4 rows in set (0.00 sec)

After this pr

mysql> explain analyze SELECT /*+ HASH_AGG() */ sum(a) FROM t1 WHERE a > 10;
+----------------------------+----------+---------+-----------+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+---------------+------+
| id                         | estRows  | actRows | task      | access object          | execution info                                                                                                                                                                                                                                                                                           | operator info                     | memory        | disk |
+----------------------------+----------+---------+-----------+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+---------------+------+
| HashAgg_10                 | 1.00     | 1       | root      |                        | time:136.117588ms, loops:2, partial_worker:{wall_time:136.057357ms, concurrency:5, task_num:1, tot_wait:135.984667ms, tot_exec:6.27µs, tot_time:680.053624ms, max:136.026503ms, p95:136.026503ms}, final_worker:{wall_time:136.103916ms, concurrency:5, task_num:1, max:136.093494ms, p95:136.093494ms}  | funcs:sum(Column#7)->Column#6     | 13.9453125 KB | N/A  |
| └─IndexReader_11           | 1.00     | 1       | root      |                        | time:135.986737ms, loops:2, cop_task: {num: 1, max:135.92485ms, proc_keys: 0, rpc_num: 1, rpc_time: 135.846842ms, copr_cache_hit_ratio: 0.00}                                                                                                                                                            | index:HashAgg_6                   | 183 Bytes     | N/A  |
|   └─HashAgg_6              | 1.00     | 1       | cop[tikv] |                        | time:0ns, loops:0, tikv_task:{time:67.263469ms, loops:99981}                                                                                                                                                                                                                                             | funcs:sum(test.t1.a)->Column#7    | N/A           | N/A  |
|     └─IndexRangeScan_9     | 99987.80 | 99981   | cop[tikv] | table:t1, index:t1a(a) | time:0ns, loops:0, tikv_task:{time:24.81135ms, loops:99981}                                                                                                                                                                                                                                              | range:(10,+inf], keep order:false | N/A           | N/A  |
+----------------------------+----------+---------+-----------+------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------+---------------+------+
4 rows in set (0.14 sec)

Related changes

  • PR to update pingcap/docs/pingcap/docs-cn:
  • Need to cherry-pick to the release branch

Check List

Tests

  • Unit test
  • Manual test (add detailed scripts or steps below)

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM

Release note

  • executor: Add the HashAggExec runtime information

@jyz0309
init
cabdd3e 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
init
8acbb76 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
optimize the execution info
599c418 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
add the task num
c6a43f0 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309 jyz0309 requested a review from a team as a code owner October 22, 2020 02:55
@jyz0309 jyz0309 requested review from qw4990 and removed request for a team October 22, 2020 02:55
@github-actions github-actions bot added the sig/execution SIG execution label Oct 22, 2020
jyz0309 and others added 2 commits October 22, 2020 11:10
@jyz0309
go fmt
56d10d8 
Signed-off-by: jyz0309 <45495947@qq.com>
@qw4990
Merge branch 'master' into hashaggExec
fb2fd32
@qw4990 qw4990 requested a review from crazycs520 October 22, 2020 03:19
qw4990
qw4990 reviewed Oct 22, 2020
View reviewed changes
Copy link
Contributor

@qw4990 qw4990 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jyz0309 Please add some unit tests.

jyz0309 and others added 12 commits October 23, 2020 14:18
@jyz0309
add unit test
a4147ae 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
7bd75d4
@jyz0309
change the channel close location
4c18b39 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
test no close
a2a7c96 
Signed-off-by: jyz0309 <45495947@qq.com>
@crazycs520
Merge branch 'master' into hashaggExec
b2e71c2
@jyz0309
fix check-dev
49e93a3 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
78c73d3
@jyz0309
fix_check-dev
0b06e2c 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
fix the Merge() bug
ce36fe0 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
use slice instead of channel
933a5ff 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
fix bug
7f7bf09 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
remove the race condition
56c8a65 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
Copy link
Contributor Author

jyz0309 commented Oct 30, 2020

/run-check_dev_2

jyz0309 and others added 3 commits November 2, 2020 14:31
@jyz0309
add test
406001e 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
fix test
67b5da4 
Signed-off-by: jyz0309 <45495947@qq.com>
@jyz0309
Merge branch 'master' into hashaggExec
0e8c1b7
qw4990
qw4990 reviewed Nov 5, 2020
View reviewed changes
executor/aggregate.go Outdated Show resolved Hide resolved
qw4990
qw4990 reviewed Nov 5, 2020
View reviewed changes
executor/aggregate.go Outdated Show resolved Hide resolved
@crazycs520
refine code
2cf5bd1 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
Copy link
Contributor

/run-all-tests

@crazycs520
fix race
0969f2e 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
Copy link
Contributor

/run-all-tests

@crazycs520
Copy link
Contributor

/build

@crazycs520
try to fix race
7200a9a 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
Merge branch 'master' into hashaggExec
bb58664
@crazycs520
Copy link
Contributor

/run-unit-test

@crazycs520
try to fix race
2250aac 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
Merge branch 'master' into hashaggExec
5012135
@crazycs520
Copy link
Contributor

/run-unit-test

@crazycs520
try to fix race
1dab7a3 
Signed-off-by: crazycs520 <crazycs520@gmail.com>
@crazycs520
Copy link
Contributor

/run-unit-test

@crazycs520
Copy link
Contributor

/run-unit-test

@crazycs520
Copy link
Contributor

/run-unit-test

@crazycs520
Copy link
Contributor

/run-all-tests

@crazycs520
Copy link
Contributor

/run-all-tests

@crazycs520
Copy link
Contributor

/run-all-tests

@crazycs520
Copy link
Contributor

/run-all-tests

@crazycs520 crazycs520 merged commit ceefbf1 into pingcap:master Nov 25, 2020
ti-srebot pushed a commit to ti-srebot/tidb that referenced this pull request Nov 25, 2020
@jyz0309 @ti-srebot
cherry pick pingcap#20577 to release-4.0
970d3c0 
Signed-off-by: ti-srebot <ti-srebot@pingcap.com>
@ti-srebot ti-srebot mentioned this pull request Nov 25, 2020
Closed
@ti-srebot
Copy link
Contributor

cherry pick to release-4.0 in PR #21304

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qw4990 qw4990 qw4990 left review comments

@crazycs520 crazycs520 crazycs520 approved these changes

@ti-srebot ti-srebot ti-srebot left review comments

@XuHuaiyu XuHuaiyu Awaiting requested review from XuHuaiyu

Assignees
No one assigned
Labels
sig/execution SIG execution status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jyz0309 @ti-srebot @crazycs520 @qw4990 @XuHuaiyu