Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor: Add the HashAggExec runtime information #20577

Merged
merged 78 commits into from
Nov 25, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
cabdd3e
init
jyz0309 Oct 21, 2020
8acbb76
init
jyz0309 Oct 21, 2020
599c418
optimize the execution info
jyz0309 Oct 21, 2020
c6a43f0
add the task num
jyz0309 Oct 21, 2020
56d10d8
go fmt
jyz0309 Oct 22, 2020
fb2fd32
Merge branch 'master' into hashaggExec
qw4990 Oct 22, 2020
a4147ae
add unit test
jyz0309 Oct 23, 2020
7bd75d4
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
jyz0309 Oct 23, 2020
4c18b39
change the channel close location
jyz0309 Oct 23, 2020
a2a7c96
test no close
jyz0309 Oct 27, 2020
b2e71c2
Merge branch 'master' into hashaggExec
crazycs520 Oct 28, 2020
49e93a3
fix check-dev
jyz0309 Oct 29, 2020
78c73d3
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
jyz0309 Oct 29, 2020
0b06e2c
fix_check-dev
jyz0309 Oct 29, 2020
ce36fe0
fix the Merge() bug
jyz0309 Oct 29, 2020
933a5ff
use slice instead of channel
jyz0309 Oct 29, 2020
7f7bf09
fix bug
jyz0309 Oct 29, 2020
56c8a65
remove the race condition
jyz0309 Oct 30, 2020
406001e
add test
jyz0309 Nov 2, 2020
67b5da4
fix test
jyz0309 Nov 2, 2020
0e8c1b7
Merge branch 'master' into hashaggExec
jyz0309 Nov 2, 2020
798ba7f
just a backup
jyz0309 Nov 8, 2020
c8a3a7e
add test
jyz0309 Nov 9, 2020
86ed7db
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
jyz0309 Nov 9, 2020
79e95e6
fix test
jyz0309 Nov 9, 2020
02ebbba
fix test
jyz0309 Nov 9, 2020
65b13a3
fix
jyz0309 Nov 9, 2020
9d37226
Merge branch 'master' into hashaggExec
jyz0309 Nov 9, 2020
3201068
fix comment
jyz0309 Nov 11, 2020
b9c4e86
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
jyz0309 Nov 11, 2020
974e597
add aggWorkerStats in baseWorker
jyz0309 Nov 11, 2020
dc1974f
add tot_wait,tot_exec,tot_time
jyz0309 Nov 11, 2020
7aa1398
Merge branch 'master' into hashaggExec
jyz0309 Nov 11, 2020
b91806d
add Clone() Merge()
jyz0309 Nov 11, 2020
376bf0a
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
jyz0309 Nov 11, 2020
1c8a36a
add final worker
jyz0309 Nov 12, 2020
f0b5911
fix test
jyz0309 Nov 12, 2020
f75e925
add comment
jyz0309 Nov 12, 2020
95e7007
address comment
jyz0309 Nov 13, 2020
48ac38c
add comment
jyz0309 Nov 13, 2020
44a3aaf
add wait_time and exec_time timing in consumeintermdata
jyz0309 Nov 14, 2020
e995911
Merge branch 'master' into hashaggExec
crazycs520 Nov 16, 2020
d837b13
fix the data race condition
jyz0309 Nov 16, 2020
d92e53e
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
jyz0309 Nov 16, 2020
31caac6
Merge branch 'master' into hashaggExec
jyz0309 Nov 16, 2020
8a83f19
fix race
jyz0309 Nov 16, 2020
9f93581
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
jyz0309 Nov 16, 2020
1385791
fix test
jyz0309 Nov 16, 2020
08b789c
fix test
jyz0309 Nov 16, 2020
403bceb
test
jyz0309 Nov 16, 2020
d851336
test
jyz0309 Nov 16, 2020
3e2451b
make aggWorkerStats int64 and atomic
jyz0309 Nov 16, 2020
4406ea5
atomic merge() func
jyz0309 Nov 16, 2020
baee217
Merge branch 'master' into hashaggExec
crazycs520 Nov 16, 2020
870c775
add read slice
jyz0309 Nov 17, 2020
dafa8d0
Merge branch 'hashaggExec' of github.com:jyz0309/tidb into hashaggExec
jyz0309 Nov 17, 2020
4f9c5e5
test
jyz0309 Nov 17, 2020
4120181
fix race
jyz0309 Nov 20, 2020
cbfa2b2
Merge branch 'master' into hashaggExec
jyz0309 Nov 20, 2020
6aa48f6
fix
jyz0309 Nov 20, 2020
b44ba81
test check e.readPartialWorker nil
jyz0309 Nov 20, 2020
4e1e275
test
jyz0309 Nov 21, 2020
e98e112
Merge branch 'master' into hashaggExec
crazycs520 Nov 23, 2020
86fa981
refine code and fix race
crazycs520 Nov 23, 2020
09adbd2
refine record
crazycs520 Nov 23, 2020
2cf5bd1
refine code
crazycs520 Nov 23, 2020
0969f2e
fix race
crazycs520 Nov 23, 2020
7200a9a
try to fix race
crazycs520 Nov 23, 2020
bb58664
Merge branch 'master' into hashaggExec
crazycs520 Nov 23, 2020
2250aac
try to fix race
crazycs520 Nov 23, 2020
5012135
Merge branch 'master' into hashaggExec
crazycs520 Nov 23, 2020
1dab7a3
try to fix race
crazycs520 Nov 23, 2020
d2cebe6
Merge branch 'master' into hashaggExec
crazycs520 Nov 23, 2020
9d7f5f8
Merge branch 'master' into hashaggExec
crazycs520 Nov 23, 2020
e7f8be5
Merge branch 'master' into hashaggExec
crazycs520 Nov 24, 2020
9484e9a
Merge branch 'master' of https://github.com/pingcap/tidb into hashagg…
crazycs520 Nov 25, 2020
595fe51
Merge branch 'hashaggExec' of https://github.com/jyz0309/tidb into ha…
crazycs520 Nov 25, 2020
30b5a25
Merge branch 'master' into hashaggExec
crazycs520 Nov 25, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add test
Signed-off-by: jyz0309 <45495947@qq.com>
  • Loading branch information
jyz0309 committed Nov 2, 2020
commit 406001e2b79209a0941b8b9eb2f67f47d2031e16
5 changes: 4 additions & 1 deletion executor/aggregate.go
Original file line number Diff line number Diff line change
Expand Up @@ -923,8 +923,9 @@ func (e *HashAggRuntimeStats) Clone() execdetails.RuntimeStats {
FinalTask: e.FinalTask,
PartialNum: e.PartialNum,
FinalNum: e.FinalNum,
PartialTime: e.PartialTime,
FinalTime: e.FinalTime,
}

return newRs
}

Expand All @@ -940,6 +941,8 @@ func (e *HashAggRuntimeStats) Merge(other execdetails.RuntimeStats) {
e.FinalTaskNum += tmp.FinalTaskNum
e.PartialNum += tmp.PartialNum
e.FinalNum += tmp.FinalNum
e.FinalTime = append(e.FinalTime, tmp.FinalTime...)
e.PartialTime = append(e.PartialTime, tmp.PartialTime...)
}

// Tp implements the RuntimeStats interface.
Expand Down
6 changes: 4 additions & 2 deletions executor/aggregate_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1153,9 +1153,11 @@ func (s *testSuiteAgg) TestHashAggRuntimeStat(c *C) {
PartialTask: 1 * time.Second,
FinalTaskNum: 5,
FinalTask: 2 * time.Second,
PartialTime: []time.Duration{1 * time.Second, 1 * time.Second, 1 * time.Second, 2 * time.Second, 3 * time.Second},
FinalTime: []time.Duration{1 * time.Second, 1 * time.Second, 1 * time.Second, 2 * time.Second, 4 * time.Second},
}
c.Assert(stats.String(), Equals, "partial_worker:{used_time:1s, PartialConcurrency:5, task_num:5}, final_worker:{used_time:2s, FinalConcurrency:5, task_num:5}")
c.Assert(stats.String(), Equals, "partial_worker:{used_time:1s, PartialConcurrency:5, task_num:5, partial_max:3s, p95:3s}, final_worker:{used_time:2s, FinalConcurrency:5, task_num:5, partial_max:4s, p95:4s}")
c.Assert(stats.String(), Equals, stats.Clone().String())
stats.Merge(stats.Clone())
c.Assert(stats.String(), Equals, "partial_worker:{used_time:2s, PartialConcurrency:10, task_num:10}, final_worker:{used_time:4s, FinalConcurrency:10, task_num:10}")
c.Assert(stats.String(), Equals, "partial_worker:{used_time:2s, PartialConcurrency:10, task_num:10, partial_max:3s, p95:3s}, final_worker:{used_time:4s, FinalConcurrency:10, task_num:10, partial_max:4s, p95:4s}")
}