[Bug]: In stability testing, the latency of query/search is unstable and has increased by many times at certain moments.

[zhuwenxing]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:master-20241107-f813fb45-amd64
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When the delay in query/search increased, a lot of rate limits on insert/delete operations appeared.

At the point of increased latency, the CPU and memory usage of the query node are relatively stable.

Expected Behavior

The latency of search/query can be relatively stable.

Steps To Reproduce

No response

Milvus Log

cluster: 4am
ns: chaos-testing
pod info

❯ k get pod|grep fts-stable-test-11                                  
fts-stable-test-11-etcd-0                                         0/1     Pending                  0                 9h
fts-stable-test-11-etcd-1                                         1/1     Running                  0                 15h
fts-stable-test-11-etcd-2                                         1/1     Running                  0                 15h
fts-stable-test-11-kafka-0                                        0/2     Pending                  0                 9h
fts-stable-test-11-kafka-1                                        2/2     Running                  1 (15h ago)       15h
fts-stable-test-11-kafka-2                                        2/2     Running                  0                 15h
fts-stable-test-11-kafka-exporter-5d659487fb-v4fsz                1/1     Running                  4 (15h ago)       15h
fts-stable-test-11-milvus-datanode-6bd55bd8bd-2nggb               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-datanode-6bd55bd8bd-5d2t6               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-indexnode-964df8d55-4fmfv               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-indexnode-964df8d55-kqgmm               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-mixcoord-68b694d6df-xp6cl               1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-proxy-8458cfb765-5qbfv                  1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-querynode-56c6787d78-czx9t              1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-milvus-querynode-56c6787d78-d9zwd              1/1     Running                  3 (15h ago)       15h
fts-stable-test-11-milvus-querynode-56c6787d78-wl5jx              1/1     Running                  2 (15h ago)       15h
fts-stable-test-11-minio-0                                        1/1     Running                  0                 15h
fts-stable-test-11-minio-1                                        1/1     Running                  0                 15h
fts-stable-test-11-minio-2                                        0/1     Pending                  0                 9h
fts-stable-test-11-minio-3                                        1/1     Running                  0                 15h
fts-stable-test-11-zookeeper-0                                    1/1     Running                  0                 15h
fts-stable-test-11-zookeeper-1                                    1/1     Running                  0                 15h
fts-stable-test-11-zookeeper-2                                    0/1     Pending                  0                 9h

grafana link

Anything else?

Will the rate limit of insert/delete operations affect the latency of search/query? If so, it would contradict the design principle of Milvus's read-write separation.

[yanliang567]

/assign @liliu-z
/unassign

[liliu-z]

Noticed the host node CPU rising.

Pod CPU was reaching to its limit 4CPU

Some comments:

1. Plz paste the node number, since checking other pods' affect is the most common thing and we have to go through all nodes in 4am 1 by 1 to find out the match node.
2. Plz don't do any performance related test on 4am, especially long-period test.

[liliu-z]

/assign @zhuwenxing

[zhuwenxing]

/unassign

/assign @liliu-z

There's a strange point here: the time point at which the search/query delay increases coincides with the emergence of the rate limit on insert/delete operations. It's necessary to confirm whether these two events are related or independent.

detailed pod info including node name

[2024-11-07T11:08:50.570Z] + kubectl get pods -o wide

[2024-11-07T11:08:50.571Z] + grep fts-stable-test-11

[2024-11-07T11:08:51.131Z] fts-stable-test-11-etcd-0                                         1/1     Running            0                 2m58s   10.104.16.155   4am-node21   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-etcd-1                                         1/1     Running            0                 2m58s   10.104.30.158   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-etcd-2                                         1/1     Running            0                 2m58s   10.104.19.147   4am-node28   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-kafka-0                                        2/2     Running            0                 2m58s   10.104.16.157   4am-node21   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-kafka-1                                        2/2     Running            1 (2m26s ago)     2m58s   10.104.19.145   4am-node28   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-kafka-2                                        2/2     Running            0                 2m58s   10.104.30.159   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-kafka-exporter-5d659487fb-v4fsz                1/1     Running            4 (2m12s ago)     2m58s   10.104.30.150   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-datanode-6bd55bd8bd-2nggb               1/1     Running            2 (2m36s ago)     2m58s   10.104.18.224   4am-node25   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-datanode-6bd55bd8bd-5d2t6               1/1     Running            2 (2m35s ago)     2m58s   10.104.15.123   4am-node20   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-indexnode-964df8d55-4fmfv               1/1     Running            2 (2m36s ago)     2m58s   10.104.15.122   4am-node20   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-indexnode-964df8d55-kqgmm               1/1     Running            2 (2m36s ago)     2m58s   10.104.24.37    4am-node29   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-mixcoord-68b694d6df-xp6cl               1/1     Running            2 (2m36s ago)     2m58s   10.104.15.121   4am-node20   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-proxy-8458cfb765-5qbfv                  1/1     Running            2 (2m36s ago)     2m58s   10.104.30.149   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-querynode-56c6787d78-czx9t              1/1     Running            2 (2m35s ago)     2m58s   10.104.18.227   4am-node25   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-querynode-56c6787d78-d9zwd              1/1     Running            3 (2m10s ago)     2m58s   10.104.5.252    4am-node12   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-milvus-querynode-56c6787d78-wl5jx              1/1     Running            2 (2m35s ago)     2m58s   10.104.15.124   4am-node20   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-minio-0                                        1/1     Running            0                 2m58s   10.104.30.156   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-minio-1                                        1/1     Running            0                 2m58s   10.104.20.216   4am-node22   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-minio-2                                        1/1     Running            0                 2m58s   10.104.16.156   4am-node21   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-minio-3                                        1/1     Running            0                 2m58s   10.104.24.40    4am-node29   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-zookeeper-0                                    1/1     Running            0                 2m58s   10.104.20.215   4am-node22   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-zookeeper-1                                    1/1     Running            0                 2m58s   10.104.30.157   4am-node38   <none>           <none>

[2024-11-07T11:08:51.131Z] fts-stable-test-11-zookeeper-2                                    1/1     Running            0                 2m58s   10.104.16.159   4am-node21   <none>           <none>