*: use iterator in show ddl jobs to avoid oom when there is too many history ddl jobs. #12472
Conversation
crazycs520
changed the title
Codecov Report
@@ Coverage Diff @@
## master #12472 +/- ##
===========================================
Coverage ? 79.8008%
===========================================
Files ? 460
Lines ? 102153
Branches ? 0
===========================================
Hits ? 81519
Misses ? 14746
Partials ? 5888 |
|
@crazycs520, please update your pull request. |
Co-Authored-By: reafans <30926443+reafans@users.noreply.github.com>
|
lgtm |
|
@crazycs520, please update your pull request. |
1 similar comment
|
@crazycs520, please update your pull request. |
| if len(jobs) < num { | ||
| jobs = make([]*model.Job, 0, num) | ||
| } | ||
| jobs = jobs[:0] |
There was a problem hiding this comment.
Warnning: This might be a cause of "memory leaks", see
https://stackoverflow.com/a/16972044 and https://github.com/golang/go/wiki/SliceTricks#delete-without-preserving-order
There was a problem hiding this comment.
No, If the variable jobs always been used, then it will cause the memory leaks.
The variable of jobs won't be always used, so it won't have the memory leaks problem.
executor/executor.go
Outdated
| m := meta.NewMeta(txn) | ||
| historyJobIter, err := m.GetLastHistoryDDLJobsIterator() | ||
|
|
||
| //historyJobs, err := admin.GetHistoryDDLJobs(txn, int(e.jobNumber)) |
There was a problem hiding this comment.
one question: I find the jobs returned is still the same, same slice size...
So the key to avoid oom are: 1.avoid sort & 2. kv iterator eliminate memory allocation?
There was a problem hiding this comment.
Yes, it mostly is eliminate the large memory allocation.
For example, before this PR, If we need to get 1 million history DDL jobs, we need to allocate a slice with 1 million size.
Signed-off-by: crazycs <crazycs520@gmail.com>
|
/merge |
sre-bot
added
the
status/can-merge
|
/run-all-tests |
…history ddl jobs. (pingcap#12472)
What problem does this PR solve?
admin show ddl jobs 100000oom when too many history DDL jobs.Test
Assume that we have 30,000 history ddl jobs.
Before this PR
This PR：
What is changed and how it works?
Use
iteratorto fetch history DDL jobs.Check List
Tests
Code changes
Side effects
Related changes
Release note
admin show ddl jobs 100000oom when there is too many history DDL jobs.