Only one query node used in searching if 3 querynode replicas deployed

[del-zhenwu]

Please state your issue using the following template and, most importantly, in English.

Describe the bug
Only one query node used in searching if deployed 3 querynode replicas, the test client generates 100-user concurrent search request during this testing.

pod names:

NAME                                                   READY   STATUS    RESTARTS   AGE
benchmark-dm9h7-1-etcd-0                               1/1     Running   0          38s
benchmark-dm9h7-1-milvus-datacoord-7fccffb97d-g8zk5    1/1     Running   0          39s
benchmark-dm9h7-1-milvus-datanode-744b859975-2sjcq     1/1     Running   0          38s
benchmark-dm9h7-1-milvus-indexcoord-5744599f97-kcv4n   1/1     Running   0          38s
benchmark-dm9h7-1-milvus-indexnode-549d67b968-q4mqd    1/1     Running   0          38s
benchmark-dm9h7-1-milvus-proxy-858844cb57-7cmsz        1/1     Running   0          39s
benchmark-dm9h7-1-milvus-pulsar-758d7895db-t6j8v       1/1     Running   0          38s
benchmark-dm9h7-1-milvus-querycoord-59487bfb6-298xg    1/1     Running   0          39s
benchmark-dm9h7-1-milvus-querynode-d6ff86794-6t9qb     1/1     Running   0          39s
benchmark-dm9h7-1-milvus-querynode-d6ff86794-jfbcq     1/1     Running   0          39s
benchmark-dm9h7-1-milvus-querynode-d6ff86794-pxxsn     1/1     Running   0          39s
benchmark-dm9h7-1-milvus-rootcoord-5bff5c9fc8-rfj7s    1/1     Running   0          39s

logs:
https://kibana-dev.zilliz.cc/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2h,to:now))&_a=(columns:!(log),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:ad1e1c50-c9d9-11eb-9e1a-b3dcad9693fd,key:kubernetes.pod_name,negate:!f,params:!(benchmark-dm9h7-1-milvus-querynode-d6ff86794-6t9qb),type:phrases,value:benchmark-dm9h7-1-milvus-querynode-d6ff86794-6t9qb),query:(bool:(minimum_should_match:1,should:!((match_phrase:(kubernetes.pod_name:benchmark-dm9h7-1-milvus-querynode-d6ff86794-6t9qb))))))),hideChart:!t,index:ad1e1c50-c9d9-11eb-9e1a-b3dcad9693fd,interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))

Steps/Code to reproduce behavior
install cluster by helm, set querynode replicas 3.

dataset: sift-10m, use the default segment configuration

Expected behavior
A clear and concise description of what you expected to happen.

Method of installation

• Docker/cpu
• Docker/gpu
• Build from source

Environment details

• Hardware/Software conditions (OS, CPU, GPU, Memory)

• Milvus version (master or released version)

cluster version with 3 query nodes

Configuration file
Settings you made in server_config.yaml or milvus.yaml

paste-file-content-here

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[xige-16]

could you provide all the log file of the three query node? @del-zhenwu

[del-zhenwu]

could you provide all the log file of the three query node? @del-zhenwu

log uri uploaded.

[xige-16]

querynode7.log

[xige-16]

the nodeIDs are 4, 7 ,8. there are only two segment when loadCollection, and assigned to node4 and node7, and two dmChannels also are asisgned to node4 and node7, node7 connect etcd failed but keep alive， so there is only one querynode serving search.

[xige-16]

@bigsheeper querynode shouldn't keep alive after connecting etcd.

[xige-16]

/kind bug

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.