enhance: Tidy compactor and remove dup codes #32198
Conversation
mergify
bot
added
dco-passed
|
@XuanYang-cn Please associate the related issue to the body of your Pull Request. (eg. “issue: #”) |
|
before: |
|
@XuanYang-cn ut workflow job failed, comment |
|
@XuanYang-cn E2e jenkins job failed, comment |
XuanYang-cn
force-pushed
the
enhance/tidy-binlogio
branch
from
403d775 to
f12e76b
Compare
|
|
|
[2024/04/17 11:36:05.081 +08:00] [INFO] [compaction/mix_compactor.go:395] ["compact merge end"] [planID=449137855160647758] ["compactTo segment"=449137855160647760] ["remaining row count"=120000] ["expired entities"=0] ["binlog batch count"=6] ["download binlogs elapse"=704.603819ms] ["upload binlogs elapse"=1.524477025s] ["serWrite elapse"=1.146140723s] ["deRead elapse"=863.583372ms] ["total elapse"=4.238804939s] |
|
@XuanYang-cn E2e jenkins job failed, comment |
XuanYang-cn
force-pushed
the
enhance/tidy-binlogio
branch
2 times, most recently
from
fb4a9e6 to
140261e
Compare
|
see also: #32448 |
|
rerun ut |
XuanYang-cn
force-pushed
the
enhance/tidy-binlogio
branch
from
140261e to
02e9f9b
Compare
|
rerun ut |
|
@XuanYang-cn E2e jenkins job failed, comment |
|
/run-cpu-e2e |
XuanYang-cn
force-pushed
the
enhance/tidy-binlogio
branch
from
02e9f9b to
e2cc0d0
Compare
|
@XuanYang-cn E2e jenkins job failed, comment |
XuanYang-cn
force-pushed
the
enhance/tidy-binlogio
branch
from
e2cc0d0 to
e9ffa4e
Compare
|
@XuanYang-cn E2e jenkins job failed, comment |
|
/run-cpu-e2e |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #32198 +/- ##
==========================================
- Coverage 82.19% 82.14% -0.05%
==========================================
Files 1009 1009
Lines 128851 128673 -178
==========================================
- Hits 105912 105704 -208
- Misses 18949 18978 +29
- Partials 3990 3991 +1
|
|
@XuanYang-cn ut workflow job failed, comment |
1 similar comment
|
@XuanYang-cn ut workflow job failed, comment |
|
@XuanYang-cn E2e jenkins job failed, comment |
|
@XuanYang-cn ut workflow job failed, comment |
|
@XuanYang-cn E2e jenkins job failed, comment |
|
/run-cpu-e2e |
XuanYang-cn
force-pushed
the
enhance/tidy-binlogio
branch
from
cb22325 to
b37d7c0
Compare
|
@XuanYang-cn ut workflow job failed, comment |
XuanYang-cn
force-pushed
the
enhance/tidy-binlogio
branch
from
b37d7c0 to
58419f2
Compare
| } | ||
|
|
||
| func (w *SegmentWriter) IsFull() bool { | ||
| w.writer.Flush() |
There was a problem hiding this comment.
Why do we need to Flush each time we check IsFull?
There was a problem hiding this comment.
Written memory size is valid after Flush()
There was a problem hiding this comment.
So donot check isFull too frequently, its expensive
| @@ -34,10 +35,10 @@ const ( | |||
| ) | |||
|
|
|||
| type compactionExecutor struct { | |||
There was a problem hiding this comment.
also move compactionExecutor to datanode/compaction/ dir?
There was a problem hiding this comment.
Too many changes in one PR, maybe you could move it inside.
| @@ -50,7 +52,7 @@ import ( | |||
| ) | |||
|
|
|||
| type levelZeroCompactionTask struct { | |||
There was a problem hiding this comment.
also move l0_compactor.go to datanode/compaction/ dir?
There was a problem hiding this comment.
YES, planning to do so.
Add compaction package remove compactor.go and compactor_test.go Signed-off-by: yangxuan <xuan.yang@zilliz.com>
XuanYang-cn
force-pushed
the
enhance/tidy-binlogio
branch
from
58419f2 to
73acb1e
Compare
|
rerun ut |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: czs007, XuanYang-cn The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This PR consists of the following commits: - enhance: Tidy compactor and remove dup codes (milvus-io#32198) - fix: Fix l0 compactor may cause DN from OOM (milvus-io#33554) - enhance: Add deltaRowCount in l0 compaction (milvus-io#33997) - enhance: enable stream writer in compactions (milvus-io#32612) See also: milvus-io#32451, milvus-io#33547, milvus-io#33998, milvus-io#31679 pr: milvus-io#32198, milvus-io#33554, milvus-io#33997, milvus-io#32612 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
This PR consists of the following commits: - enhance: Tidy compactor and remove dup codes (#32198) - fix: Fix l0 compactor may cause DN from OOM (#33554) - enhance: Add deltaRowCount in l0 compaction (#33997) - enhance: enable stream writer in compactions (#32612) - fix: turn on compression on stream writers (#34067) - fix: adding blob memory size in binlog serde (#33324) See also: #32451, #33547, #33998, #31679 pr: #32198, #33554, #33997, #32612 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com> Signed-off-by: Ted Xu <ted.xu@zilliz.com> Co-authored-by: Ted Xu <ted.xu@zilliz.com>
See also: #32451