[Bug]: [Nightly] Search iterator failed reporting `service internal error: target version mismatch`

[NicoYuan1986]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 57bcb1b
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Search iterator failed reporting service internal error: target version mismatch.

2025-01-19T22:26:01Z {container="step-test"} [2025-01-19 22:23:07 - DEBUG - ci_test]: (api_request)  : [Collection.search_iterator] args: [[[0.5447480507751704, 0.8710746340126518, 0.7327436027221865, 0.8211554824066887, 0.008030292109812098, 0.7848156613185722, 0.04692184539084587, 0.8821729306844879, 0.7362091346901077, 0.15719823105772202, 0.8850523890626482, 0.6210358524967171, 0.734152907380209, 0.9637801301931097, 0.809115620891......, kwargs: {} (api_request.py:52)
2025-01-19T22:26:01Z {container="step-test"} [2025-01-19 22:23:29 - ERROR - ci_test]: Traceback (most recent call last):
2025-01-19T22:26:01Z {container="step-test"} File "/milvus/tests/python_client/utils/api_request.py", line 22, in inner_wrapper
2025-01-19T22:26:01Z {container="step-test"} res = func(*args, **_kwargs)
2025-01-19T22:26:01Z {container="step-test"} File "/milvus/tests/python_client/utils/api_request.py", line 53, in api_request
2025-01-19T22:26:01Z {container="step-test"} return func(*arg, **kwargs)
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 974, in search_iterator
2025-01-19T22:26:01Z {container="step-test"} return SearchIterator(
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/orm/iterator.py", line 514, in __init__
2025-01-19T22:26:01Z {container="step-test"} self.__init_search_iterator()
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/orm/iterator.py", line 517, in __init_search_iterator
2025-01-19T22:26:01Z {container="step-test"} init_page = self.__execute_next_search(self._param, self._expr, False)
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/orm/iterator.py", line 725, in __execute_next_search
2025-01-19T22:26:01Z {container="step-test"} res = self._conn.search(
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 141, in handler
2025-01-19T22:26:01Z {container="step-test"} raise e from e
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 137, in handler
2025-01-19T22:26:01Z {container="step-test"} return func(*args, **kwargs)
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 176, in handler
2025-01-19T22:26:01Z {container="step-test"} return func(self, *args, **kwargs)
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 116, in handler
2025-01-19T22:26:01Z {container="step-test"} raise e from e
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/decorators.py", line 86, in handler
2025-01-19T22:26:01Z {container="step-test"} return func(*args, **kwargs)
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 836, in search
2025-01-19T22:26:01Z {container="step-test"} return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 777, in _execute_search
2025-01-19T22:26:01Z {container="step-test"} raise e from e
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/client/grpc_handler.py", line 766, in _execute_search
2025-01-19T22:26:01Z {container="step-test"} check_status(response.status)
2025-01-19T22:26:01Z {container="step-test"} File "/usr/local/lib/python3.10/site-packages/pymilvus/client/utils.py", line 63, in check_status
2025-01-19T22:26:01Z {container="step-test"} raise MilvusException(status.code, status.reason, status.error_code)
2025-01-19T22:26:01Z {container="step-test"} pymilvus.exceptions.MilvusException: <MilvusException: (code=503, message=failed to search: service internal error: target version mismatch, collection: 455426373858493966, channel: by-dev-rootcoord-dml_2_455426373858493966v0,  current target version: 0, leader version: 0: channel not available[channel=by-dev-rootcoord-dml_2_455426373858493966v0])>
2025-01-19T22:26:01Z {container="step-test"} (api_request.py:35)
2025-01-19T22:26:01Z {container="step-test"} [2025-01-19 22:23:29 - ERROR - ci_test]: (api_response) : <MilvusException: (code=503, message=failed to search: service internal error: target version mismatch, collection: 455426373858493966, channel: by-dev-rootcoord-dml_2_455426373858493966v0,  current target version: 0, leader version: 0: channel not available[channel=by-dev-rootcoord-dml_2_4554263738...... (api_request.py:36)

Expected Behavior

pass

Steps To Reproduce

Milvus Log

https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI(new)/detail/2.5/53/pipeline/107

Anything else?

No response

[MrPresent-Han]

qn crushed: for
remove local directory:/var/lib/milvus/data/querynode/index_files/455426373857364128/1/ failed, error: Directory not empty at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:220

better to mark this issue as critical

[yanliang567]

verified fix on milvus v2.5.4