[Bug]: search raises error `string field contains invalid UTF-8` when enabled queryNode.segcore.multipleChunkedEnable=true #37083

wangting0128 · 2024-10-23T11:44:41Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version:master-20241023-4746f472-amd64 
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.0rc102
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-e2e-update-sdk

Run all e2e tag L0L1L2 test cases

server:

NAME                                                              READY   STATUS      RESTARTS       AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-e2e-update-sdk-etcd-0                                     1/1     Running     0              3h4m    10.104.27.11    4am-node31   <none>           <none>
fouramf-e2e-update-sdk-etcd-1                                     1/1     Running     0              3h4m    10.104.24.70    4am-node29   <none>           <none>
fouramf-e2e-update-sdk-etcd-2                                     1/1     Running     0              3h4m    10.104.23.102   4am-node27   <none>           <none>
fouramf-e2e-update-sdk-milvus-datanode-6f9989ff6b-xqk4j           1/1     Running     1 (179m ago)   3h4m    10.104.1.103    4am-node10   <none>           <none>
fouramf-e2e-update-sdk-milvus-indexnode-7bd5766dd9-b9xn4          1/1     Running     1 (3h4m ago)   3h4m    10.104.13.107   4am-node16   <none>           <none>
fouramf-e2e-update-sdk-milvus-mixcoord-5f76977f4b-dzvxt           1/1     Running     1 (3h ago)     3h4m    10.104.6.104    4am-node13   <none>           <none>
fouramf-e2e-update-sdk-milvus-proxy-5c965cc696-sbhvn              1/1     Running     2 (3h ago)     3h4m    10.104.14.217   4am-node18   <none>           <none>
fouramf-e2e-update-sdk-milvus-querynode-6d454bcc67-jp5w6          1/1     Running     0              3h4m    10.104.6.103    4am-node13   <none>           <none>
fouramf-e2e-update-sdk-minio-0                                    1/1     Running     0              3h4m    10.104.23.99    4am-node27   <none>           <none>
fouramf-e2e-update-sdk-minio-1                                    1/1     Running     0              3h4m    10.104.27.13    4am-node31   <none>           <none>
fouramf-e2e-update-sdk-minio-2                                    1/1     Running     0              3h4m    10.104.24.73    4am-node29   <none>           <none>
fouramf-e2e-update-sdk-minio-3                                    1/1     Running     0              3h4m    10.104.17.228   4am-node23   <none>           <none>
fouramf-e2e-update-sdk-pulsar-bookie-0                            1/1     Running     0              3h4m    10.104.27.14    4am-node31   <none>           <none>
fouramf-e2e-update-sdk-pulsar-bookie-1                            1/1     Running     0              3h4m    10.104.24.74    4am-node29   <none>           <none>
fouramf-e2e-update-sdk-pulsar-bookie-2                            1/1     Running     0              3h4m    10.104.17.229   4am-node23   <none>           <none>
fouramf-e2e-update-sdk-pulsar-bookie-init-sb62t                   0/1     Completed   0              3h4m    10.104.9.74     4am-node14   <none>           <none>
fouramf-e2e-update-sdk-pulsar-broker-0                            1/1     Running     0              3h4m    10.104.9.76     4am-node14   <none>           <none>
fouramf-e2e-update-sdk-pulsar-proxy-0                             1/1     Running     0              3h4m    10.104.4.188    4am-node11   <none>           <none>
fouramf-e2e-update-sdk-pulsar-pulsar-init-h7q67                   0/1     Completed   0              3h4m    10.104.9.75     4am-node14   <none>           <none>
fouramf-e2e-update-sdk-pulsar-recovery-0                          1/1     Running     0              3h4m    10.104.14.218   4am-node18   <none>           <none>
fouramf-e2e-update-sdk-pulsar-zookeeper-0                         1/1     Running     0              3h4m    10.104.23.101   4am-node27   <none>           <none>
fouramf-e2e-update-sdk-pulsar-zookeeper-1                         1/1     Running     0              3h4m    10.104.16.184   4am-node21   <none>           <none>
fouramf-e2e-update-sdk-pulsar-zookeeper-2                         1/1     Running     0              3h3m    10.104.24.86    4am-node29   <none>           <none>

client log:
client.log

FAILED test_issues.py::TestIssues::test_issue_30607[True-int64] - assert 148 ...
FAILED test_issues.py::TestIssues::test_issue_30607[False-int64] - assert 133...
FAILED test_mix_scenes.py::TestMixScenes::test_bitmap_offset_cache_and_mmap
FAILED test_partition_key.py::TestPartitionKeyParams::test_partition_key_on_field_schema[int64]
FAILED test_search.py::TestSparseSearch::test_sparse_index_enable_mmap_search[SPARSE_WAND]
FAILED test_search.py::TestSparseSearch::test_sparse_index_enable_mmap_search[SPARSE_INVERTED_INDEX]
FAILED test_search.py::TestCollectionSearchNoneAndDefaultData::test_search_collection_with_non_default_data_after_release_load[200-False-0]
FAILED test_search.py::TestCollectionSearchNoneAndDefaultData::test_search_collection_with_non_default_data_after_release_load[200-True-0.5]
FAILED test_search.py::TestCollectionSearchNoneAndDefaultData::test_search_collection_with_non_default_data_after_release_load[200-False-0.5]
FAILED test_search.py::TestCollectionSearchNoneAndDefaultData::test_search_collection_with_non_default_data_after_release_load[200-True-0]
FAILED test_search.py::TestCollectionSearchNoneAndDefaultData::test_search_after_different_index_with_params_none_default_data[INVERTED-STL_SORT-0.5-False]
FAILED test_search.py::TestCollectionSearchNoneAndDefaultData::test_search_after_different_index_with_params_none_default_data[INVERTED-STL_SORT-0.5-True]
FAILED test_search.py::TestCollectionSearchNoneAndDefaultData::test_search_after_different_index_with_params_none_default_data[INVERTED-INVERTED-0.5-False]
FAILED test_search.py::TestCollectionSearchNoneAndDefaultData::test_search_after_different_index_with_params_none_default_data[INVERTED-INVERTED-0.5-True]
FAILED test_search.py::TestSearchWithTextMatchFilter::test_search_with_text_match_filter_normal[default-True-True]
FAILED test_search.py::TestSearchWithTextMatchFilter::test_search_with_text_match_filter_normal[jieba-True-False]
FAILED test_search.py::TestSearchWithTextMatchFilter::test_search_with_text_match_filter_normal[jieba-True-True]
FAILED test_search.py::TestSearchWithTextMatchFilter::test_search_with_text_match_filter_normal[jieba-False-False]
FAILED test_search.py::TestSearchWithTextMatchFilter::test_search_with_text_match_filter_normal[jieba-False-True]
FAILED test_search.py::TestSearchWithTextMatchFilter::test_search_with_text_match_filter_normal[default-False-True]

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

The text was updated successfully, but these errors were encountered:

xiaofan-luan · 2024-10-23T20:44:36Z

/assign @zhengbuqian
could you help on check this?

xiaofan-luan · 2024-10-23T20:45:48Z

maybe it's not related to queryNode.segcore.multipleChunkedEnable?

What' the exact error?

xiaofan-luan · 2024-10-23T23:28:26Z

I think this might not be related to mulitple chunk enabled.
but more like a issue related to charset you are using

wangting0128 · 2024-10-24T02:44:26Z

I think this might not be related to mulitple chunk enabled. but more like a issue related to charset you are using

The specific log is posted on the issue 👇

when queryNode.segcore.multipleChunkedEnable=false, the failed cases can be run passed

This is one of the failed cases👇

zhengbuqian · 2024-10-24T05:31:59Z

reproduced in my local env, the error happened when proxy tries to proto.Unmarshall search results from the delegaor, and was thrown from the proto library. still looking

sunby · 2024-10-24T07:34:03Z

fixed in #37104
it's because of saving arrow's data in string_views

zhengbuqian · 2024-10-24T08:08:43Z

/assign @sunby
/unassign

issue: #37083 We use vector of string_view to save data temporally but real string data will be released after record batch is deconstructed. Change it to vector of string to avoid memory corruption. --------- Signed-off-by: sunby <sunbingyi1992@gmail.com>

xiaofan-luan · 2024-10-28T22:28:23Z

/assign @wangting0128
could you help on reviewing it

wangting0128 · 2024-10-29T06:13:59Z

verification passed

argo task: fouramf-e2e-jz25z
image: 2.5-20241028-7134526d-amd64

wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 23, 2024

wangting0128 added this to the 2.5.0 milestone Oct 23, 2024

wangting0128 assigned sunby and yanliang567 Oct 23, 2024

wangting0128 mentioned this issue Oct 23, 2024

[Bug]: [benchmark] milvus panic SIGSEGV: segmentation violation when enabled queryNode.segcore.multipleChunkedEnable #36834

Closed

1 task

sre-ci-robot assigned zhengbuqian Oct 23, 2024

xiaofan-luan assigned zhengbuqian and unassigned zhengbuqian, yanliang567 and sunby Oct 23, 2024

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 24, 2024

sunby mentioned this issue Oct 24, 2024

fix: fix string field has invalid utf-8 #37104

Merged

sre-ci-robot assigned sunby and unassigned zhengbuqian Oct 24, 2024

sre-ci-robot assigned wangting0128 Oct 28, 2024

wangting0128 closed this as completed Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: search raises error `string field contains invalid UTF-8` when enabled queryNode.segcore.multipleChunkedEnable=true #37083

[Bug]: search raises error `string field contains invalid UTF-8` when enabled queryNode.segcore.multipleChunkedEnable=true #37083

wangting0128 commented Oct 23, 2024 •

edited

Loading

xiaofan-luan commented Oct 23, 2024

xiaofan-luan commented Oct 23, 2024

xiaofan-luan commented Oct 23, 2024

wangting0128 commented Oct 24, 2024

zhengbuqian commented Oct 24, 2024

sunby commented Oct 24, 2024 •

edited

Loading

zhengbuqian commented Oct 24, 2024

xiaofan-luan commented Oct 28, 2024

wangting0128 commented Oct 29, 2024

[Bug]: search raises error string field contains invalid UTF-8 when enabled queryNode.segcore.multipleChunkedEnable=true #37083

[Bug]: search raises error string field contains invalid UTF-8 when enabled queryNode.segcore.multipleChunkedEnable=true #37083

Comments

wangting0128 commented Oct 23, 2024 • edited Loading

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

xiaofan-luan commented Oct 23, 2024

xiaofan-luan commented Oct 23, 2024

xiaofan-luan commented Oct 23, 2024

wangting0128 commented Oct 24, 2024

zhengbuqian commented Oct 24, 2024

sunby commented Oct 24, 2024 • edited Loading

zhengbuqian commented Oct 24, 2024

xiaofan-luan commented Oct 28, 2024

wangting0128 commented Oct 29, 2024

verification passed

[Bug]: search raises error `string field contains invalid UTF-8` when enabled queryNode.segcore.multipleChunkedEnable=true #37083

[Bug]: search raises error `string field contains invalid UTF-8` when enabled queryNode.segcore.multipleChunkedEnable=true #37083

wangting0128 commented Oct 23, 2024 •

edited

Loading

sunby commented Oct 24, 2024 •

edited

Loading