[Bug]: Failed to execute insert task in task scheduler: proxy TaskCondition context Done；message send timeout: TimeoutError

[TonyAnn]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.3.21
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar 
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

A single table write timeout caused the system to be blocked. The problem was fixed after restarting the abnormal proxy pod. Please help locate the problem.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

my-release-milvus-proxy-f6c8fd65b-rcgsv.log
my-release-milvus-proxy-f6c8fd65b-tl9h9.log

Anything else?

No response

[yanliang567]

@TonyAnn what do you mean "the system to be blocked"?

[TonyAnn]

@TonyAnn what do you mean "the system to be blocked"?
@yanliang567 Writing and deleting to this table are blocked until the pod is restarted, then normal writing and deleting can be resumed.

[yanliang567]

/assign @weiliu1031
/unassign

[weiliu1031]

@TonyAnn When a write operation encounters an exception, it seems that the proxy is in an abnormal state. Was there any operation performed on the proxy prior to this issue?

[TonyAnn]

At that point in time, when performing write and delete operations

[weiliu1031]

@TonyAnn Could you please confirm if a timeout is set for write operations? According to the log details, it appears that a large number of write requests have accumulated on the proxy, and these requests are failing due to timeouts.

[TonyAnn]

The client does not specify a timeout when writing

[weiliu1031]

The issue seems to be caused by the context (ctx) used for the insertion request timing out after 10 seconds, leading to the failure of the entire insertion operation. Upon reviewing the logic in Milvus Proxy and pymilvus, it has been confirmed that Milvus does not impose a 10-second timeout internally. Therefore, the potential causes of this issue might include:

1. A timeout was explicitly set via the timeout parameter when calling pymilvus.
2. There is a load balancer (LB) between pymilvus and the proxy, and the LB is configured with a connection timeout. We've encountered similar issues in the past.

Please verify these details, as they might help identify and resolve the problem.
/cc @TonyAnn