Skip to content

feat: [2.5] Support run analyzer and more tokenizer. #41444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Apr 24, 2025
Merged

feat: [2.5] Support run analyzer and more tokenizer. #41444

merged 11 commits into from
Apr 24, 2025

Conversation

aoiasd
Copy link
Contributor

@aoiasd aoiasd commented Apr 22, 2025

aoiasd added 6 commits April 22, 2025 14:23
@aoiasd
enhance: pack analyzer code and support lindera tokenizer (milvus-io#…
cf819b7 
…39660)

relate: milvus-io#39659

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@aoiasd
enhance: support use jieba tokenizer with costum dictionary (milvus-i…
9915ccd 
…o#39854)

relate: milvus-io#40168

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@aoiasd
enhance: not build lindera features by default and support make milvu…
694ce8b 
…s with tantivy features (milvus-io#40813)

relate: milvus-io#39659

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@aoiasd
enhance: support run analyzer (milvus-io#39723)
7c778a5 
relate: milvus-io#39705

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@aoiasd
enhance: support run anayser return detaild token (milvus-io#40458)
ef00957 
relate: milvus-io#39705

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@aoiasd
enhance: support use lindera tag filter (milvus-io#40416)
385f965 
relate: milvus-io#39659

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@sre-ci-robot sre-ci-robot added size/XXL Denotes a PR that changes 1000+ lines. area/compilation area/dependency Pull requests that update a dependency file labels Apr 22, 2025
@mergify mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Apr 22, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 22, 2025

@aoiasd cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@aoiasd
feat: support icu and language identifier tokenizer (milvus-io#41214)
39a181c 
relate: milvus-io#41213

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 22, 2025

@aoiasd cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 22, 2025

@aoiasd go-sdk check failed, comment rerun go-sdk can trigger the job again.

@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 22, 2025

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

@aoiasd aoiasd changed the title feat: [2.5] Support run analyzer and linderatokenizer. feat: [2.5] Support run analyzer and more tokenizer. Apr 22, 2025
aoiasd and others added 3 commits April 23, 2025 11:44
@aoiasd
feat: support extend default dict for jieba tokenizer (milvus-io#41360)
6f2bc40 
relate: milvus-io#41213

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@aoiasd
feat: support more lauguage for build in stop words and add remove pu…
9bc61e9 
…nct, regex filter (milvus-io#41412)

relate: milvus-io#41213

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@aoiasd
update cargo lock
2c3da45 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@codecov Codecov
Copy link

codecov bot commented Apr 23, 2025 •
edited
Loading

Codecov Report

Attention: Patch coverage is 66.03774% with 18 lines in your changes missing coverage. Please review.

Project coverage is 80.88%. Comparing base (ca5199c) to head (8ca4427).
Report is 8 commits behind head on 2.5.

Files with missing lines Patch % Lines
internal/util/ctokenizer/c_token_stream.go 0.00% 10 Missing ⚠️
internal/proxy/impl.go 82.35% 3 Missing and 3 partials ⚠️
internal/distributed/proxy/service.go 0.00% 2 Missing ⚠️

❌ Your patch status has failed because the patch coverage (66.03%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##              2.5   #41444      +/-   ##
==========================================
- Coverage   80.90%   80.88%   -0.03%     
==========================================
  Files        1421     1421              
  Lines      203304   203379      +75     
==========================================
+ Hits       164491   164493       +2     
- Misses      33052    33106      +54     
- Partials     5761     5780      +19
Components Coverage Δ
Client 79.65% <ø> (ø)
Core 70.67% <100.00%> (-0.01%) ⬇️
Go 82.76% <60.86%> (-0.04%) ⬇️
Files with missing lines Coverage Δ
internal/core/src/segcore/token_stream_c.cpp 100.00% <100.00%> (ø)
internal/distributed/proxy/service.go 84.90% <0.00%> (-0.24%) ⬇️
internal/proxy/impl.go 83.70% <82.35%> (-0.01%) ⬇️
internal/util/ctokenizer/c_token_stream.go 54.54% <0.00%> (-45.46%) ⬇️

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@aoiasd
Copy link
Contributor Author

aoiasd commented Apr 23, 2025

rerun ut

@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 23, 2025

@aoiasd cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@aoiasd
Copy link
Contributor Author

aoiasd commented Apr 23, 2025

rerun cpp-unit-tes

@aoiasd
Copy link
Contributor Author

aoiasd commented Apr 23, 2025

rerun cpp-unit-test

@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 23, 2025

@aoiasd cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@aoiasd
Copy link
Contributor Author

aoiasd commented Apr 23, 2025

rerun cpp-unit-test

@mergify Mergify
Copy link
Contributor

mergify bot commented Apr 23, 2025

@aoiasd cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

@aoiasd
Copy link
Contributor Author

aoiasd commented Apr 23, 2025

rerun cpp-unit-test

@yanliang567 yanliang567 added this to the 2.5.11 milestone Apr 23, 2025
@aoiasd
format cpp
8ca4427 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
@mergify mergify bot added the ci-passed label Apr 23, 2025
@zhengbuqian
Copy link
Collaborator

/lgtm

@congqixia
Copy link
Contributor

/approve

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aoiasd, congqixia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot merged commit 8b3353c into milvus-io:2.5 Apr 24, 2025
18 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

@czs007 czs007 Awaiting requested review from czs007

Assignees

@zhengbuqian zhengbuqian

Labels
approved area/compilation area/dependency Pull requests that update a dependency file ci-passed dco-passed DCO check passed. kind/feature Issues related to feature request from users lgtm size/XXL Denotes a PR that changes 1000+ lines.
Projects
None yet
Milestone
2.5.11
Development

Successfully merging this pull request may close these issues.
5 participants
@aoiasd @zhengbuqian @congqixia @sre-ci-robot @yanliang567