Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Text match has issues with tokenizer in chinese , where punctuation marks and spaces are also treated as separate words #37419

Open
1 task done
zhuwenxing opened this issue Nov 4, 2024 · 2 comments
Open
1 task done

[Bug]: Text match has issues with tokenizer in chinese , where punctuation marks and spaces are also treated as separate words #37419

zhuwenxing opened this issue Nov 4, 2024 · 2 comments
Assignees
@zhuwenxing @aoiasd
Labels
2.5-features feature/text match kind/bug Issues or changes related a bug severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.5.0

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

img_v3_02ga_34a16fd7-63c2-4ebe-8f8f-872193fd84eg

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed ci:
https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20HA%20CI/detail/PR-37119/17/pipeline/

'公司 自己 数据 上海 电影 今年 朋友 增加 日期 一起' can match '有限电脑名称作者. 资源次数过程安全参加详细. 合作各种中国. 开始就是虽然表示. 留言游戏一直加入. 方式完成只有经济加入这是. 你好 生产当前完全留言日本支持决定.'

Anything else?

No response
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 4, 2024
@zhuwenxing
Copy link
Contributor Author

/assign @aoiasd

@yanliang567 yanliang567 added this to the 2.5.0 milestone Nov 5, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. 2.5-features and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 5, 2024
@yanliang567 yanliang567 removed their assignment Nov 5, 2024
@zhuwenxing zhuwenxing added the severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. label Nov 5, 2024
@aoiasd aoiasd mentioned this issue Nov 12, 2024
Merged
@liliu-z
Copy link
Member

liliu-z commented Nov 14, 2024

/assign @zhuwenxing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhuwenxing zhuwenxing

@aoiasd aoiasd

Labels
2.5-features feature/text match kind/bug Issues or changes related a bug severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.5.0
Development

No branches or pull requests
4 participants
@zhuwenxing @aoiasd @yanliang567 @liliu-z