[Feature Request]: Use NLP word breaker in fulltext query.

[fengttt]

Is there an existing issue for the same feature request?

• I have checked the existing issues.

Is your feature request related to a problem?

At this moment we used a simple space/punctuation breaking for latin and 3-gram
for CJK in BOTH indexing time and query time.

Sometimes it is more desirable to use a NLP word breaker.

Describe the feature you'd like

Our word breaker is pluggable. We can use any word breaker in either indexing time or query time.

For Chinese, Chinese, for example, https://github.com/yanyiwu/gojieba

There are both advantages and disadvantages to use a NLP word breaker. I would suggest at this moment, we keep the simple 3-gram breaker at indexing time. For query time, esp when match in natural language mode, we should consider using NLP word breaker.

Describe implementation you've considered

No response

Documentation, Adoption, Use Case, Migration Strategy

Additional information

No response

[cpegeric]

I think we have to stick with one tokenizer in index time and search time. Otherwise, we cannot find the answer. e.g. 但是可以接入 3- gram will break down to "但是可", "是可以", "以接入", "接入", "入". but dictionary-based tokenizer will break down to "但是", "可以", "接入 ". If 3-gram in index table, search time with dict-based, It won't find anything, unless all search become prefix_eq() instead of ==.

[cpegeric]

there is a problem in gojieba library too. It doesn't support N-gram. e..g the search string "是可" cannot be found in index. I don't think gojieba library is suitable for our needs.

[cpegeric]

The only way to reduce the number of token is to use 2-GRAM.