feat: Add chinese and english analyzer with refactor jieba tokenizer #37494

aoiasd · 2024-11-07T03:42:42Z

relate: #35853

mergify · 2024-11-07T04:04:22Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-07T04:04:23Z

@aoiasd go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-07T04:06:43Z

@aoiasd cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-07T10:11:45Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-07T10:17:43Z

@aoiasd cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-07T10:18:59Z

@aoiasd go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-07T11:08:38Z

@aoiasd cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-11-07T11:13:49Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-11-07T11:25:05Z

@aoiasd go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-11-07T13:02:39Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

codecov · 2024-11-07T13:18:50Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.53%. Comparing base (2630717) to head (2053502).
Report is 14 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #37494       +/-   ##
===========================================
+ Coverage   68.15%   78.53%   +10.37%     
===========================================
  Files         290     1349     +1059     
  Lines       25392   189210   +163818     
===========================================
+ Hits        17306   148592   +131286     
- Misses       8086    35233    +27147     
- Partials        0     5385     +5385

Components	Coverage Δ
Client	`61.25% <ø> (∅)`
Core	`68.07% <ø> (-0.09%)`	⬇️
Go	`80.76% <ø> (∅)`

Files with missing lines	Coverage Δ
internal/util/function/bm25_function.go	`80.00% <ø> (ø)`

... and 1066 files with indirect coverage changes

aoiasd · 2024-11-08T01:34:22Z

/run-cpu-e2e

mergify · 2024-11-08T02:34:50Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

aoiasd · 2024-11-08T03:03:07Z

/run-cpu-e2e

mergify · 2024-11-08T03:56:21Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

aoiasd · 2024-11-08T06:00:55Z

/run-cpu-e2e

aoiasd · 2024-11-08T06:39:58Z

/run-cpu-e2e

mergify · 2024-11-08T06:40:46Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

aoiasd · 2024-11-08T07:06:39Z

/run-cpu-e2e

sre-ci-robot · 2024-11-11T07:07:23Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aoiasd
To complete the pull request process, please assign congqixia after the PR has been reviewed.
You can assign the PR to them by writing /assign @congqixia in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mergify · 2024-11-11T07:38:52Z

@aoiasd go-sdk check failed, comment rerun go-sdk can trigger the job again.

aoiasd · 2024-11-12T02:47:45Z

relate: #37419

zhengbuqian · 2024-11-12T03:56:59Z

/lgtm

thanks!

aoiasd · 2024-11-12T05:57:16Z

rerun ut

mergify · 2024-11-12T10:49:44Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

internal/core/thirdparty/tantivy/tantivy-binding/src/jieba_tokenizer.rs

chyezh · 2024-11-13T08:11:46Z

internal/core/thirdparty/tantivy/tantivy-binding/src/util.rs

+}
+
+pub(crate) fn get_stop_words_list(str_list:Vec<String>) -> Vec<String>{


use &[String] as parameter may be better?

Direct function param passing in rust means pass owner of struct to function, not memory copy, in this function means the passing struct will be free when get_stop_words_list finish.

chyezh · 2024-11-13T08:12:06Z

/lgtm

czs007 · 2024-11-13T08:38:30Z

internal/core/thirdparty/tantivy/tantivy-binding/src/tokenizer.rs

@@ -127,30 +157,34 @@ impl AnalyzerBuilder<'_>{
                    // build with filter if filter param exist
                    builder=self.build_filter(builder, value)?;
                },
-                "max_token_length" => {


why delete this?

tokenizer max token length in es means split long token not remove long token, but now we don't have same function, so remove this option.

internal/core/thirdparty/tantivy/tantivy-binding/src/tokenizer.rs

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

sre-ci-robot · 2024-11-13T10:14:08Z

New changes are detected. LGTM label has been removed.

mergify · 2024-11-13T11:40:57Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

aoiasd · 2024-11-13T11:52:47Z

/run-cpu-e2e

mergify · 2024-11-13T12:37:29Z

@aoiasd E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

aoiasd · 2024-11-13T13:50:10Z

/run-cpu-e2e

sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Nov 7, 2024

sre-ci-robot requested review from czs007 and godchen0212 November 7, 2024 03:43

mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Nov 7, 2024

aoiasd force-pushed the doc-in-tokenizer-3 branch from b6dcfc9 to 65cf5eb Compare November 7, 2024 09:53

aoiasd force-pushed the doc-in-tokenizer-3 branch from 65cf5eb to cae4b7d Compare November 7, 2024 10:47

aoiasd force-pushed the doc-in-tokenizer-3 branch from cae4b7d to e544b5b Compare November 7, 2024 12:05

aoiasd force-pushed the doc-in-tokenizer-3 branch from e544b5b to 5493bff Compare November 11, 2024 07:07

aoiasd force-pushed the doc-in-tokenizer-3 branch from 5493bff to 1c1817e Compare November 12, 2024 02:22

aoiasd force-pushed the doc-in-tokenizer-3 branch from 1c1817e to 4a062c2 Compare November 12, 2024 03:08

sre-ci-robot assigned zhengbuqian Nov 12, 2024

sre-ci-robot added the lgtm label Nov 12, 2024

aoiasd force-pushed the doc-in-tokenizer-3 branch from 4a062c2 to b5c1765 Compare November 12, 2024 06:05

sre-ci-robot removed the lgtm label Nov 12, 2024

aoiasd force-pushed the doc-in-tokenizer-3 branch 2 times, most recently from 9ce208a to 98cfda8 Compare November 12, 2024 09:08

sre-ci-robot added the area/dependency Pull requests that update a dependency file label Nov 12, 2024

aoiasd force-pushed the doc-in-tokenizer-3 branch from 98cfda8 to fce7377 Compare November 12, 2024 10:58

mergify bot added the ci-passed label Nov 13, 2024

chyezh reviewed Nov 13, 2024

View reviewed changes

sre-ci-robot assigned chyezh Nov 13, 2024

sre-ci-robot added the lgtm label Nov 13, 2024

czs007 reviewed Nov 13, 2024

View reviewed changes

internal/core/thirdparty/tantivy/tantivy-binding/src/tokenizer.rs Outdated Show resolved Hide resolved

Add chinese and english analyzer with refactor jieba tokenizer

2053502

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

aoiasd force-pushed the doc-in-tokenizer-3 branch from fce7377 to 2053502 Compare November 13, 2024 10:14

sre-ci-robot removed the lgtm label Nov 13, 2024

mergify bot removed the ci-passed label Nov 13, 2024

		}

		pub(crate) fn get_stop_words_list(str_list:Vec<String>) -> Vec<String>{

feat: Add chinese and english analyzer with refactor jieba tokenizer #37494

Are you sure you want to change the base?

feat: Add chinese and english analyzer with refactor jieba tokenizer #37494

Conversation

aoiasd commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

mergify bot commented Nov 7, 2024

codecov bot commented Nov 7, 2024 • edited Loading

Codecov Report

aoiasd commented Nov 8, 2024

mergify bot commented Nov 8, 2024

aoiasd commented Nov 8, 2024

mergify bot commented Nov 8, 2024

aoiasd commented Nov 8, 2024

aoiasd commented Nov 8, 2024

mergify bot commented Nov 8, 2024

aoiasd commented Nov 8, 2024

sre-ci-robot commented Nov 11, 2024

mergify bot commented Nov 11, 2024

aoiasd commented Nov 12, 2024

zhengbuqian commented Nov 12, 2024

aoiasd commented Nov 12, 2024

mergify bot commented Nov 12, 2024

chyezh Nov 13, 2024

Choose a reason for hiding this comment

aoiasd Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

chyezh commented Nov 13, 2024

czs007 Nov 13, 2024

Choose a reason for hiding this comment

aoiasd Nov 13, 2024

Choose a reason for hiding this comment

sre-ci-robot commented Nov 13, 2024

mergify bot commented Nov 13, 2024

aoiasd commented Nov 13, 2024

mergify bot commented Nov 13, 2024

aoiasd commented Nov 13, 2024

codecov bot commented Nov 7, 2024 •

edited

Loading

aoiasd Nov 13, 2024 •

edited

Loading