Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filter sealed segments #18065

Merged
merged 1 commit into from
Jul 7, 2022
Merged

Filter sealed segments #18065

merged 1 commit into from
Jul 7, 2022

Conversation

XuanYang-cn
Copy link
Contributor

  • Remove FilterThreshold in DataNode
  • Alter filter logic in flowgraph DDNode

See also: #17943

Signed-off-by: yangxuan xuan.yang@zilliz.com

/kind bug

@sre-ci-robot sre-ci-robot added the kind/bug Issues or changes related a bug label Jul 5, 2022
@sre-ci-robot sre-ci-robot requested review from cydrain and sunby July 5, 2022 07:42
@sre-ci-robot sre-ci-robot added the size/XL Denotes a PR that changes 500-999 lines. label Jul 5, 2022
@mergify mergify bot added the dco-passed DCO check passed. label Jul 5, 2022
@XuanYang-cn
Copy link
Contributor Author

/assign @congqixia @xiaofan-luan

@mergify
Copy link
Contributor

mergify bot commented Jul 5, 2022

@XuanYang-cn ut workflow job failed, comment rerun ut can trigger the job again.

@XuanYang-cn
Copy link
Contributor Author

rerun ut

@mergify
Copy link
Contributor

mergify bot commented Jul 5, 2022

@XuanYang-cn ut workflow job failed, comment rerun ut can trigger the job again.

@XuanYang-cn
Copy link
Contributor Author

rerun ut

blocked by #17871

@XuanYang-cn
Copy link
Contributor Author

rerun ut

@codecov
Copy link

codecov bot commented Jul 5, 2022 •
edited
Loading

Codecov Report

Merging #18065 (0d869e5) into master (8f1ba6a) will increase coverage by 0.14%.
The diff coverage is 88.17%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #18065      +/-   ##
==========================================
+ Coverage   81.68%   81.83%   +0.14%     
==========================================
  Files         472      475       +3     
  Lines       75285    75574     +289     
==========================================
+ Hits        61500    61849     +349     
+ Misses      11048    11005      -43     
+ Partials     2737     2720      -17
Impacted Files Coverage Δ
internal/datanode/data_node.go 82.03% <ø> (-0.05%) ⬇️
internal/datanode/data_sync_service.go 70.73% <82.22%> (+0.34%) ⬆️
internal/datanode/flow_graph_dd_node.go 89.86% <93.75%> (-1.80%) ⬇️
internal/indexnode/task_state.go 50.00% <0.00%> (-50.00%) ⬇️
internal/util/mock/grpc_indexnode_client.go 83.33% <0.00%> (-16.67%) ⬇️
internal/indexcoord/priority_queue.go 75.24% <0.00%> (-10.90%) ⬇️
internal/indexnode/indexnode_mock.go 80.43% <0.00%> (-2.48%) ⬇️
internal/indexcoord/util.go 97.67% <0.00%> (-2.33%) ⬇️
internal/indexnode/indexnode.go 86.59% <0.00%> (-2.01%) ⬇️
... and 27 more

@mergify mergify bot added the ci-passed label Jul 5, 2022
xiaofan-luan
xiaofan-luan reviewed Jul 6, 2022
View reviewed changes
internal/datanode/flow_graph_dd_node.go Outdated Show resolved Hide resolved
internal/datanode/flow_graph_dd_node.go Outdated Show resolved Hide resolved
internal/datanode/flow_graph_dd_node.go
fgMsg.insertMessages = append(fgMsg.insertMessages, imsg)

case commonpb.MsgType_Delete:
dmsg := msg.(*msgstream.DeleteMsg)
log.Debug("DDNode receive delete messages",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

handle dropped segment for delete?
though delete can be inserted multiple times, filter delete message before cp should still be a good optimization

@XuanYang-cn XuanYang-cn force-pushed the filter_sealed_segments branch from 6f514b4 to 96314b2 Compare July 7, 2022 03:05
@mergify mergify bot removed the ci-passed label Jul 7, 2022
@mergify
Copy link
Contributor

mergify bot commented Jul 7, 2022

@XuanYang-cn ut workflow job failed, comment rerun ut can trigger the job again.

@XuanYang-cn
Filter sealed segments
0d869e5 
- Remove FilterThreshold in DataNode
- Alter filter logic in flowgraph DDNode

See also: milvus-io#17943

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
@XuanYang-cn XuanYang-cn force-pushed the filter_sealed_segments branch from 96314b2 to 0d869e5 Compare July 7, 2022 06:45
@mergify
Copy link
Contributor

mergify bot commented Jul 7, 2022

@XuanYang-cn E2e jenkins job failed, comment /run-checks can trigger the job again.

@XuanYang-cn
Copy link
Contributor Author

rerun ut

@XuanYang-cn XuanYang-cn requested a review from xiaofan-luan July 7, 2022 08:27
@xiaofan-luan
Copy link
Collaborator

/lgtm
/approve

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xiaofan-luan, XuanYang-cn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@yanliang567 yanliang567 added ci-passed manual-pass manually set pass before ci-passed labeled labels Jul 7, 2022
@sre-ci-robot sre-ci-robot merged commit d9e8231 into milvus-io:master Jul 7, 2022
@XuanYang-cn XuanYang-cn deleted the filter_sealed_segments branch July 7, 2022 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cydrain cydrain Awaiting requested review from cydrain

@sunby sunby Awaiting requested review from sunby

@xiaofan-luan xiaofan-luan Awaiting requested review from xiaofan-luan

Assignees

@xiaofan-luan xiaofan-luan

@congqixia congqixia

Labels
approved ci-passed dco-passed DCO check passed. kind/bug Issues or changes related a bug lgtm manual-pass manually set pass before ci-passed labeled size/XL Denotes a PR that changes 500-999 lines.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@XuanYang-cn @xiaofan-luan @sre-ci-robot @yanliang567 @congqixia