enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

cqy123456 · 2025-01-12T16:46:32Z

some raw data status will change:
Intermin index has raw data:

sparse vector	growing segment	sealed segment
BM25	no	no
IP	yes	no

dense vector	growing segment	sealed segment
ivf flat cc	yes	yes
scann_dvr	no	no

sre-ci-robot · 2025-01-12T16:46:51Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cqy123456
To complete the pull request process, please assign liliu-z after the PR has been reviewed.
You can assign the PR to them by writing /assign @liliu-z in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2025-01-12T16:47:11Z

@cqy123456 Please associate the related pr of master to the body of your Pull Request. (eg. “pr: #”)

mergify · 2025-01-12T16:47:47Z

@cqy123456 Please associate the related issue to the body of your Pull Request. (eg. “issue: #”)

mergify · 2025-01-12T17:34:54Z

@cqy123456 go-sdk check failed, comment rerun go-sdk can trigger the job again.

codecov · 2025-01-12T18:32:01Z

Codecov Report

Attention: Patch coverage is 61.31148% with 118 lines in your changes missing coverage. Please review.

Project coverage is 69.37%. Comparing base (54036bc) to head (68d4c0e).
Report is 32 commits behind head on 2.5.

Files with missing lines	Patch %	Lines
...rnal/core/src/segcore/ChunkedSegmentSealedImpl.cpp	9.30%	39 Missing ⚠️
internal/core/src/segcore/SegmentSealedImpl.cpp	50.81%	30 Missing ⚠️
internal/core/src/segcore/segcore_init_c.cpp	5.00%	19 Missing ⚠️
internal/core/src/segcore/FieldIndexing.cpp	77.33%	17 Missing ⚠️
internal/core/src/storage/storage_c.cpp	0.00%	6 Missing ⚠️
internal/core/src/index/VectorMemIndex.cpp	72.72%	3 Missing ⚠️
internal/core/src/segcore/FieldIndexing.h	85.71%	2 Missing ⚠️
internal/core/src/segcore/SegcoreConfig.h	77.77%	2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (54036bc) and HEAD (68d4c0e). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (54036bc) HEAD (68d4c0e)

2 1

Additional details and impacted files

@@             Coverage Diff             @@
##              2.5   #39180       +/-   ##
===========================================
- Coverage   81.06%   69.37%   -11.69%     
===========================================
  Files        1399      300     -1099     
  Lines      196700    26958   -169742     
===========================================
- Hits       159448    18703   -140745     
+ Misses      31653     8255    -23398     
+ Partials     5599        0     -5599

Components	Coverage Δ
Client	`∅ <ø> (∅)`
Core	`69.37% <61.31%> (-0.16%)`	⬇️
Go	`∅ <ø> (∅)`

Files with missing lines	Coverage Δ
internal/core/src/index/VectorMemIndex.h	`100.00% <ø> (ø)`
internal/core/src/mmap/ChunkVector.h	`87.09% <ø> (+3.22%)`	⬆️
internal/core/src/query/SearchOnGrowing.cpp	`93.02% <100.00%> (ø)`
internal/core/src/segcore/ConcurrentVector.h	`96.68% <100.00%> (+0.85%)`	⬆️
internal/core/src/segcore/IndexConfigGenerator.cpp	`83.92% <100.00%> (+0.90%)`	⬆️
internal/core/src/segcore/InsertRecord.h	`89.55% <100.00%> (+0.63%)`	⬆️
internal/core/src/segcore/SegmentGrowingImpl.cpp	`82.66% <100.00%> (-0.14%)`	⬇️
internal/core/src/segcore/SegmentGrowingImpl.h	`68.29% <100.00%> (-0.76%)`	⬇️
internal/core/src/segcore/vector_index_c.cpp	`22.22% <ø> (ø)`
internal/core/src/storage/Types.h	`40.27% <100.00%> (+3.51%)`	⬆️
... and 8 more

... and 1129 files with indirect coverage changes

mergify · 2025-01-13T05:55:35Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2025-01-13T05:56:51Z

@cqy123456 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-13T05:56:52Z

@cqy123456 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-13T06:16:19Z

@cqy123456 go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2025-01-13T06:16:19Z

@cqy123456 cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2025-01-13T06:17:19Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

cqy123456 · 2025-01-13T07:57:24Z

configs/milvus.yaml

      nprobe: 16 # nprobe to search small index, based on your accuracy requirement, must smaller than nlist
+      subDim: 4 # interim index sub dim, recommend to (subDim % vector dim == 0)
+      refineRatio: 3.5 # interim index parameters, should set to be >= 1.0
+      withRawData: true # Whether to keep raw data inside the intermin index


replace withRawData by denseVectorIndexType

foxspy · 2025-01-13T07:57:02Z

configs/milvus.yaml

      nprobe: 16 # nprobe to search small index, based on your accuracy requirement, must smaller than nlist
+      subDim: 4 # interim index sub dim, recommend to (subDim % vector dim == 0)
+      refineRatio: 3.5 # interim index parameters, should set to be >= 1.0
+      withRawData: true # Whether to keep raw data inside the intermin index


This should be an index feature, not a configuration item.

foxspy · 2025-01-13T08:21:51Z

internal/core/src/segcore/SegmentGrowingImpl.cpp

@@ -102,7 +103,8 @@ SegmentGrowingImpl::Insert(int64_t reserved_offset,
        AssertInfo(field_id_to_offset.count(field_id),
                   fmt::format("can't find field {}", field_id.get()));
        auto data_offset = field_id_to_offset[field_id];
-        if (!indexing_record_.SyncDataWithIndex(field_id)) {
+        if (!indexing_record_.SyncDataWithIndex(field_id) ||
+            !indexing_record_.HasRawData(field_id)) {


This may seem difficult to understand; please change it to insert_raw_to_index = indexing_record_.SyncDataWithIndex(field_id) && indexing_record_.HasRawData(field_id); if (!insert_raw_to_index) xxx , which means that only the index holds the original data and the index data has been matched with the inserted data, and no more data will be written to the chunk.

mergify · 2025-01-13T09:14:49Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

cqy123456 · 2025-01-13T09:18:56Z

/run-cpu-e2e

…e fp16/bf16 Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>

cqy123456 · 2025-01-16T09:38:31Z

configs/milvus.yaml

@@ -404,8 +404,11 @@ queryNode:
      # Milvus will eventually seals and indexes all segments, but enabling this optimizes search performance for immediate queries following data insertion.
      # This defaults to true, indicating that Milvus creates temporary index for growing segments and the sealed segments that are not indexed upon searches.
      enableIndex: true
-      nlist: 128 # temp index nlist, recommend to set sqrt(chunkRows), must smaller than chunkRows/8
-      nprobe: 16 # nprobe to search small index, based on your accuracy requirement, must smaller than nlist
+      nlist: 128 # interim index nlist, recommend to set sqrt(chunkRows), must smaller than chunkRows/8


change these config before merge

mergify · 2025-01-16T11:19:44Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

cqy123456 · 2025-01-16T13:55:59Z

/run-cpu-e2e

mergify · 2025-01-16T15:08:04Z

@cqy123456 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

sre-ci-robot requested review from aoiasd and bigsheeper January 12, 2025 16:46

cqy123456 changed the title ~~enhance: intermin index support index without raw data and metric type fp16/bf16~~ enhance: [2.5]intermin index support index without raw data and metric type fp16/bf16 Jan 12, 2025

sre-ci-robot added area/compilation size/XL Denotes a PR that changes 500-999 lines. labels Jan 12, 2025

mergify bot added the dco-passed DCO check passed. label Jan 12, 2025

mergify bot added do-not-merge/missing-related-pr kind/enhancement Issues or changes related to enhancement labels Jan 12, 2025

mergify bot added the do-not-merge/missing-related-issue label Jan 12, 2025

cqy123456 force-pushed the new-intermin-index branch from 189ec69 to 476fb9c Compare January 13, 2025 05:50

cqy123456 changed the title ~~enhance: [2.5]intermin index support index without raw data and metric type fp16/bf16~~ enhance: [2.5]intermin index support different index type and more metric type(fp16/bf16) Jan 13, 2025

cqy123456 force-pushed the new-intermin-index branch from 476fb9c to eea69ac Compare January 13, 2025 06:10

cqy123456 commented Jan 13, 2025

View reviewed changes

cqy123456 force-pushed the new-intermin-index branch from eea69ac to 3d2a8d5 Compare January 13, 2025 08:02

cqy123456 changed the title ~~enhance: [2.5]intermin index support different index type and more metric type(fp16/bf16)~~ enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) Jan 13, 2025

foxspy reviewed Jan 13, 2025

View reviewed changes

cqy123456 force-pushed the new-intermin-index branch from 3d2a8d5 to 451f7bc Compare January 15, 2025 18:32

sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. and removed size/XL Denotes a PR that changes 500-999 lines. labels Jan 15, 2025

enhance: intermin index support index without raw data and metric typ…

68d4c0e

…e fp16/bf16 Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>

cqy123456 force-pushed the new-intermin-index branch from 451f7bc to 68d4c0e Compare January 16, 2025 08:39

cqy123456 commented Jan 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

cqy123456 commented Jan 12, 2025 •

edited

Loading

sre-ci-robot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

codecov bot commented Jan 12, 2025 •

edited

Loading

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

cqy123456 Jan 13, 2025

foxspy Jan 13, 2025

foxspy Jan 13, 2025

mergify bot commented Jan 13, 2025

cqy123456 commented Jan 13, 2025

cqy123456 Jan 16, 2025

mergify bot commented Jan 16, 2025

cqy123456 commented Jan 16, 2025

mergify bot commented Jan 16, 2025

enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

Are you sure you want to change the base?

enhance: [2.5]intermin index support different index type and more data type(fp16/bf16) #39180

Conversation

cqy123456 commented Jan 12, 2025 • edited Loading

sre-ci-robot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

mergify bot commented Jan 12, 2025

codecov bot commented Jan 12, 2025 • edited Loading

Codecov Report

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

mergify bot commented Jan 13, 2025

cqy123456 Jan 13, 2025

Choose a reason for hiding this comment

foxspy Jan 13, 2025

Choose a reason for hiding this comment

foxspy Jan 13, 2025

Choose a reason for hiding this comment

mergify bot commented Jan 13, 2025

cqy123456 commented Jan 13, 2025

cqy123456 Jan 16, 2025

Choose a reason for hiding this comment

mergify bot commented Jan 16, 2025

cqy123456 commented Jan 16, 2025

mergify bot commented Jan 16, 2025

cqy123456 commented Jan 12, 2025 •

edited

Loading

codecov bot commented Jan 12, 2025 •

edited

Loading