Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance: reduce SyncTask AllocID call and refine code #29701

Merged
merged 2 commits into from
Jan 5, 2024

Conversation

congqixia
Copy link
Contributor

See also #27675

Allocator.Alloc and Allocator.AllocOne might be invoked multiple times if there were multiple blobs set in one sync task.

This PR add pre-fetch logic for all blobs and cache logIDs in sync task so that at most only one call of the allocator is needed.

See also milvus-io#27675

`Allocator.Alloc` and `Allocator.AllocOne` might be invoked multiple
times if there were multiple blobs set in one sync task.

This PR add pre-fetch logic for all blobs and cache logIDs in sync task
so that at most only one call of the allocator is needed.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@sre-ci-robot sre-ci-robot added approved size/L Denotes a PR that changes 100-499 lines. labels Jan 4, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Jan 4, 2024
Copy link
Contributor

mergify bot commented Jan 4, 2024

@congqixia ut workflow job failed, comment rerun ut can trigger the job again.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Copy link

codecov bot commented Jan 4, 2024

Codecov Report

Attention: 23 lines in your changes are missing coverage. Please review.

Comparison is base (99e0f1e) 83.72% compared to head (282c9a1) 80.80%.
Report is 14 commits behind head on master.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #29701      +/-   ##
==========================================
- Coverage   83.72%   80.80%   -2.93%     
==========================================
  Files         679      922     +243     
  Lines      108623   127406   +18783     
==========================================
+ Hits        90949   102948   +11999     
- Misses      14361    21149    +6788     
+ Partials     3313     3309       -4     
Files Coverage Δ
internal/datacoord/handler.go 79.85% <100.00%> (+0.29%) ⬆️
internal/datanode/syncmgr/storage_serializer.go 89.92% <100.00%> (+0.07%) ⬆️
internal/datanode/syncmgr/storage_v2_serializer.go 80.12% <100.00%> (+0.12%) ⬆️
internal/datanode/syncmgr/task.go 100.00% <100.00%> (+12.87%) ⬆️
internal/datanode/writebuffer/l0_write_buffer.go 70.37% <100.00%> (+2.37%) ⬆️
internal/datanode/writebuffer/write_buffer.go 92.07% <100.00%> (-1.33%) ⬇️
internal/proxy/meta_cache.go 89.64% <100.00%> (-0.18%) ⬇️
internal/proxy/task.go 85.48% <100.00%> (ø)
internal/proxy/task_delete.go 85.98% <100.00%> (ø)
internal/proxy/task_index.go 75.74% <100.00%> (ø)
... and 27 more

... and 253 files with indirect coverage changes

@mergify mergify bot added the ci-passed label Jan 4, 2024
@xiaofan-luan
Copy link
Collaborator

/lgtm
/approve

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot merged commit dc6a6a5 into milvus-io:master Jan 5, 2024
14 checks passed
@congqixia congqixia deleted the reduce_sync_task_allocid branch January 5, 2024 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement lgtm size/L Denotes a PR that changes 100-499 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants