Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Support insert texts into Milvus directly and use BM25 to search #35853

Open
1 task done
zhengbuqian opened this issue Aug 30, 2024 · 1 comment
Open
1 task done

[Feature]: Support insert texts into Milvus directly and use BM25 to search #35853

zhengbuqian opened this issue Aug 30, 2024 · 1 comment
Assignees
@zhengbuqian @aoiasd
Labels
kind/feature Issues related to feature request from users

Comments

@zhengbuqian
Copy link
Collaborator

Is there an existing issue for this?

  • I have searched the existing issues

Is your feature request related to a problem? Please describe.

Currently in order to perform BM25 based text relevance search using vector ANN search, we have to :

  1. Gather the entire corpus to collect the data statistics, including term frequency and inverse document frequency, etc
  2. Compute the doc embeddings and insert those into Milvus as sparse embeddings
  3. Compute the query embeddings and search using IP metric

This approach is not good enough and hard to update when the corpus has been updated a lot.

We propose a new way of doing such: allowing inserting texts only and have Milvus to maintain the statistics and do the conversion at runtime.

Proposed approaches and APIs will be shared shortly.

This will be the umbrella issue for all related following issues and PRs.

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response
@zhengbuqian zhengbuqian added the kind/feature Issues related to feature request from users label Aug 30, 2024
@zhengbuqian zhengbuqian mentioned this issue Aug 30, 2024
Open
1 task
@xiaofan-luan
Copy link
Collaborator

excited about this!

This was referenced Sep 3, 2024
Closed
Merged
Merged
Merged
Merged
@zhengbuqian zhengbuqian mentioned this issue Sep 8, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Sep 12, 2024
@aoiasd
feat: Support create collection with functions (#35973)
da227ff 
relate: #35853
Support create collection with functions. Prepare for support bm25
function.

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Sep 19, 2024
@aoiasd
feat: support embedding bm25 sparse vector and flush bm25 stats log (#…
1397873 
…36036)

relate: #35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
This was referenced Oct 9, 2024
Merged
Merged
sre-ci-robot pushed a commit that referenced this issue Oct 11, 2024
@aoiasd
feat: support load and query with bm25 metric (#36071)
db34572 
relate: #35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
sre-ci-robot pushed a commit to milvus-io/pymilvus that referenced this issue Oct 11, 2024
@zhengbuqian
support new Function feature (#2257)
4ed5d3e 
currently only BM25 Function is supported.

issue: milvus-io/milvus#35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
@zhengbuqian zhengbuqian mentioned this issue Oct 11, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Oct 12, 2024
@zhengbuqian
feat: add bulk insert support for Functions (#36715)
82c5cf2 
issue: #35853 and
#35856

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
sre-ci-robot pushed a commit that referenced this issue Oct 13, 2024
@zhengbuqian
feat: Restful support for BM25 function (#36713)
16b533c 
issue: #35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
sre-ci-robot pushed a commit that referenced this issue Oct 13, 2024
@zhengbuqian
feat: added more checks for function creation check (#36766)
383350c 
issue: #35853

* BM25 Function now takes no params, k1, b should be passed via index
params
* support BM25 full text search when metric type is not present in
search request
* add more strict validation with functions at collection creation time

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
sre-ci-robot pushed a commit that referenced this issue Oct 14, 2024
@aoiasd
feat: support bm25 logs mixcompaction (#36072)
5ec4163 
relate: #35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@zhengbuqian zhengbuqian mentioned this issue Oct 14, 2024
Merged
@aoiasd aoiasd mentioned this issue Oct 15, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Oct 16, 2024
@zhengbuqian
fix: remove excessive logging (#36859)
9997c5d 
issue: #35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
@zhengbuqian zhengbuqian mentioned this issue Oct 16, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Oct 16, 2024
@aoiasd
fix: bm25 search failed when nq > 1 and remove idf oracle when no bm2…
72dc07b 
…5 field exist. (#36886)

relate: #35853

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@zhuwenxing zhuwenxing mentioned this issue Oct 16, 2024
Open
1 task
sre-ci-robot pushed a commit that referenced this issue Oct 16, 2024
@zhengbuqian
fix: add checks for bm25 k1 and b in sparse index checker (#36907)
51f13ba 
issue: #36883,
#35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
@aoiasd aoiasd mentioned this issue Oct 22, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 6, 2024
@aoiasd
feat: Tokenizer support build with params and clone for concurrency (#…
d67853f 
…37048)

relate: #35853
#36751

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@aoiasd aoiasd mentioned this issue Nov 6, 2024
Merged
@zhengbuqian zhengbuqian mentioned this issue Nov 6, 2024
Merged
@aoiasd aoiasd mentioned this issue Nov 7, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 7, 2024
@zhengbuqian
fix: add rescorer activation function for BM25 (#37481)
40b770c 
issue: #35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
sre-ci-robot pushed a commit that referenced this issue Nov 10, 2024
@aoiasd
enhance: rename tokenizer to analyzer and check analyzer params (#37478)
12951f0 
relate: #35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
congqixia added a commit to congqixia/milvus that referenced this issue Nov 11, 2024
@congqixia
enhance: [GoSDK] support Doc-in-doc-out APIs
af6a714 
Related to milvus-io#35853

This PR contains following changes:

- Add function and related proto and helper functions
- Remove the insert column missing check and leave it to server
- Add text as search input data
- Add some unit tests for logic above

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@congqixia congqixia mentioned this issue Nov 11, 2024
Merged
@zhengbuqian zhengbuqian mentioned this issue Nov 11, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 12, 2024
@congqixia
enhance: [GoSDK] support Doc-in-doc-out APIs (#37590)
24c6a4b 
Related to #35853

This PR contains following changes:

- Add function and related proto and helper functions
- Remove the insert column missing check and leave it to server
- Add text as search input data
- Add some unit tests for logic above

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
@aoiasd aoiasd mentioned this issue Nov 13, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Nov 14, 2024
@aoiasd
feat: Add chinese and english analyzer with refactor jieba tokenizer (#…
1c5b5e1 
…37494)

relate: #35853

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Nov 14, 2024
@zhengbuqian
fix: Sparse to use CC index as growing/temp index (#37591)
0565300 
relate: #35853

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
sre-ci-robot pushed a commit that referenced this issue Nov 14, 2024
@aoiasd
fix: brute force bm25 search lack avgdl param (#37650)
993051b 
relate: #35853

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhengbuqian zhengbuqian

@aoiasd aoiasd

Labels
kind/feature Issues related to feature request from users
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zhengbuqian @aoiasd @xiaofan-luan