[Bug]: Flush hangs when running hello_milvus.py after pulsar recovered from pod kill chaos

[zhuwenxing]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20220610-36ad9895
- Deployment mode(standalone or cluster):cluster
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus==2.1.0.dev69
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The action has timed out when running hello_milvus.py
Outputting duplicate logs in proxy pod

[2022/06/12 18:57:51.851 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:52.355 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:52.356 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:52.861 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:52.866 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:53.370 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:53.370 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:53.873 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:53.877 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:54.379 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:54.381 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]
[2022/06/12 18:57:54.884 +00:00] [INFO] [impl.go:3573] ["received get flush state request"] [request="segmentIDs:433864038933069825 segmentIDs:433864038933069826 "]
[2022/06/12 18:57:54.885 +00:00] [INFO] [impl.go:3587] ["received get flush state response"] [response="status:<> "]

Expected Behavior

all test cases passed

Steps To Reproduce

see https://github.com/milvus-io/milvus/runs/6851761481?check_suite_focus=true

Milvus Log

failed job: https://github.com/milvus-io/milvus/runs/6851761481?check_suite_focus=true
log: https://github.com/milvus-io/milvus/suites/6898913792/artifacts/267696164

Anything else?

No response

[yanliang567]

@XuanYang-cn one more flush hang issue...
/assign @XuanYang-cn
/unassign

[XuanYang-cn]

may related to #17335, working on it

[XuanYang-cn]

/assign @zhuwenxing
Could you please have another try with the latest master commit?

[zhuwenxing]

It still happened in version master-20220613-e6225d92
Your PR is merged in the version

failed job: https://github.com/zhuwenxing/milvus/runs/6857375279?check_suite_focus=true
but the log is not collected due to the job is timeout, not the failure

I will try it again

[zhuwenxing]

failed job: https://github.com/zhuwenxing/milvus/runs/6858068872?check_suite_focus=true
log: https://github.com/zhuwenxing/milvus/suites/6904892315/artifacts/268116140

In GitHub action, this issue seems stable reproduced