[Bug]: After the standalone rolling upgrade is complete, search/query still continues to report errors.

[zhuwenxing]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:v2.4.10--> master-20241030-95521905-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2024-10-30T17:36:18.428Z] [2024-10-30 17:36:12 - DEBUG - ci_test]: (api_request)  : [Collection.search] args: [[[0.14402712490267577, 0.12490642546226964, 0.0018904240901919686, 0.03283683623105017, 0.12208735239607384, 0.07397166968439756, 0.041247542899978853, 0.09505378172530864, 0.03346715380466587, 0.1459959508015099, 0.132217605696564, 0.036814943401485715, 0.05930253946882387, 0.12677793400972412, 0......., kwargs: {} (api_request.py:62)

[2024-10-30T17:36:18.428Z] [2024-10-30 17:36:12 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=1, message=failed to search: loaded collection do not found any channel in target, may be in recovery: collection on recovering[collection=453589749382513574])>, <Time:{'RPC start': '2024-10-30 17:36:12.597417', 'RPC error': '2024-10-30 17:36:12.599468'}> (decorators.py:126)

[2024-10-30T17:36:18.428Z] [2024-10-30 17:36:12 - ERROR - ci_test]: Traceback (most recent call last):

[2024-10-30T17:36:18.428Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper

[2024-10-30T17:36:18.428Z]     res = func(*args, **_kwargs)

[2024-10-30T17:36:18.428Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request

[2024-10-30T17:36:18.428Z]     return func(*arg, **kwargs)

[2024-10-30T17:36:18.428Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 778, in search

[2024-10-30T17:36:18.428Z]     res = conn.search(

[2024-10-30T17:36:18.428Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 127, in handler

[2024-10-30T17:36:18.428Z]     raise e from e

[2024-10-30T17:36:18.428Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 123, in handler

[2024-10-30T17:36:18.428Z]     return func(*args, **kwargs)

[2024-10-30T17:36:18.428Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 162, in handler

[2024-10-30T17:36:18.428Z]     return func(self, *args, **kwargs)

[2024-10-30T17:36:18.428Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 102, in handler

[2024-10-30T17:36:18.428Z]     raise e from e

[2024-10-30T17:36:18.428Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 68, in handler

[2024-10-30T17:36:18.428Z]     return func(*args, **kwargs)

[2024-10-30T17:36:18.428Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 774, in search

[2024-10-30T17:36:18.428Z]     return self._execute_search_requests(

[2024-10-30T17:36:18.428Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 735, in _execute_search_requests

[2024-10-30T17:36:18.428Z]     raise pre_err from pre_err

[2024-10-30T17:36:18.428Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 726, in _execute_search_requests

[2024-10-30T17:36:18.428Z]     raise MilvusException(response.status.error_code, response.status.reason)

[2024-10-30T17:36:18.428Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=failed to search: loaded collection do not found any channel in target, may be in recovery: collection on recovering[collection=453589749382513574])>

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job:https://qa-jenkins.milvus.io/blue/organizations/jenkins/rolling_update_for_operator_test_simple/detail/rolling_update_for_operator_test_simple/5420/pipeline

log:
artifacts-kafka-standalone-5420-server-logs.tar.gz

cluster: 4am
ns: chaos-testing
pod info

[2024-10-30T17:10:35.828Z] + kubectl get pods -o wide

[2024-10-30T17:10:35.830Z] + grep kafka-standalone-5420

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-etcd-0                                     1/1     Running       0                11m     10.104.20.81    4am-node22   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-kafka-0                                    2/2     Running       0                11m     10.104.26.224   4am-node32   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-kafka-1                                    2/2     Running       0                11m     10.104.20.83    4am-node22   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-kafka-2                                    2/2     Running       0                11m     10.104.18.134   4am-node25   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-kafka-exporter-569d789fbb-8ctvn            1/1     Running       3 (11m ago)      11m     10.104.5.165    4am-node12   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-kafka-zookeeper-0                          1/1     Running       0                11m     10.104.18.133   4am-node25   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-kafka-zookeeper-1                          1/1     Running       0                11m     10.104.20.84    4am-node22   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-kafka-zookeeper-2                          1/1     Running       0                11m     10.104.16.171   4am-node21   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-milvus-standalone-6cdf45f4cf-n2dwc         1/1     Running       0                3m54s   10.104.26.241   4am-node32   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-minio-0                                    1/1     Running       0                11m     10.104.18.130   4am-node25   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-minio-1                                    1/1     Running       0                11m     10.104.20.82    4am-node22   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-minio-2                                    1/1     Running       0                11m     10.104.26.223   4am-node32   <none>           <none>

[2024-10-30T17:10:36.086Z] kafka-standalone-5420-minio-3                                    1/1     Running       0                11m     10.104.32.210   4am-node39   <none>           <none>

Anything else?

No response

[yanliang567]

/assign @liliu-z
/unassign

[liliu-z]

/assign @weiliu1031

[weiliu1031]

known issue, should be fixed by #36880
please verify this with the the commit after this pr

[weiliu1031]

/assign @zhuwenxing

[zhuwenxing]

verified and fixed in 2.1.4--> master-20241105-b83b376c-amd64