Search time triples when nprobe value doubles

[del-zhenwu]

Describe the bug
log_path: /test/milvus/db_data_7/sift_1b_2048_128_l2_sq8_wal/logs/milvus-20-06-22-17:15-debug.log
search time triples when nprobe set 128 and 256

17:26:04  2020-06-22:09:25:53,615 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 1}}
17:26:04  2020-06-22:09:25:53,843 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:26:04  2020-06-22:09:26:04,823 INFO     [client.py:34] Milvus query run in 10.98s
17:26:05  2020-06-22:09:26:04,949 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 2}}
17:26:05  2020-06-22:09:26:05,107 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:26:13  2020-06-22:09:26:13,187 INFO     [client.py:34] Milvus query run in 8.08s
17:26:13  2020-06-22:09:26:13,322 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 4}}
17:26:13  2020-06-22:09:26:13,483 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:26:23  2020-06-22:09:26:22,744 INFO     [client.py:34] Milvus query run in 9.26s
17:26:23  2020-06-22:09:26:22,875 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 8}}
17:26:23  2020-06-22:09:26:23,8 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:26:33  2020-06-22:09:26:33,491 INFO     [client.py:34] Milvus query run in 10.48s
17:26:33  2020-06-22:09:26:33,618 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 16}}
17:26:33  2020-06-22:09:26:33,747 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:26:48  2020-06-22:09:26:47,604 INFO     [client.py:34] Milvus query run in 13.86s
17:26:48  2020-06-22:09:26:47,727 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 32}}
17:26:48  2020-06-22:09:26:47,853 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:27:10  2020-06-22:09:27:08,138 INFO     [client.py:34] Milvus query run in 20.29s
17:27:10  2020-06-22:09:27:08,251 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 64}}
17:27:10  2020-06-22:09:27:08,382 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:27:43  2020-06-22:09:27:41,171 INFO     [client.py:34] Milvus query run in 32.79s
17:27:43  2020-06-22:09:27:41,309 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 128}}
17:27:43  2020-06-22:09:27:41,489 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:29:04  2020-06-22:09:28:56,807 INFO     [client.py:34] Milvus query run in 75.32s
17:29:04  2020-06-22:09:28:56,953 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 256}}
17:29:04  2020-06-22:09:28:57,82 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:33:41  2020-06-22:09:33:32,688 INFO     [client.py:34] Milvus query run in 275.61s
17:33:41  2020-06-22:09:33:32,810 INFO     [k8s_runner.py:413] Query params: {"nq": 1000, "topk": 64, "search_param": {"nprobe": 512}}
17:33:41  2020-06-22:09:33:32,946 INFO     [runner.py:219] Start query, query params: top-k: 64, nq: 1000, actually length of vectors: 1000
17:41:18  2020-06-22:09:41:14,224 INFO     [client.py:34] Milvus query run in 461.28s

Steps/Code to reproduce behavior
based on sift-1b-sq8

Expected behavior
A clear and concise description of what you expected to happen.

Environment details
0.8.1
0.10.1 need to be test

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[cqy123456]

[version 0.10.1]
Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz * 12
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 12288K

index_type = IVF_SQ8, metric_type = L2
n = 4000000, d = 128, nlist = 16384
nq = 1000, tok = 64

(1) SSE
nprobe =1 : time: 58.39
nprobe =2 : time: 57.23
nprobe =4 : time: 60.84
nprobe =8 : time: 73.86
nprobe =16 : time: 91.33
nprobe =32 : time: 91.94
nprobe =64 : time: 112.97
nprobe =128 : time: 195.83
nprobe =256 : time: 514.84
nprobe =512 : time: 1049.85
nprobe =1024 : time: 1817.25
nprobe =2048 : time: 2569.01

(2) openblas
nprobe =1 : time: 74.95
nprobe =2 : time: 71.23
nprobe =4 : time: 64.61
nprobe =8 : time: 74.27
nprobe =16 : time: 62.77
nprobe =32 : time: 55.21
nprobe =64 : time: 58.43
nprobe =128 : time: 72.35
nprobe =256 : time: 81.79
nprobe =512 : time: 131.05
nprobe =1024 : time: 204.24
nprobe =2048 : time: 361.20

It is not reproduced on the local machine.

[cqy123456]

[version 0.10.1]
Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz* 48
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 30720K

index_type = IVF_SQ8, metric_type = L2
n = 4000000, d = 128, nlist = 16384
nq = 1000, tok = 64
(1) SSE
nprobe =1 : time: 31.29
nprobe =2 : time: 24.75
nprobe =4 : time: 26.02
nprobe =8 : time: 28.37
nprobe =16 : time: 33.81
nprobe =32 : time: 41.60
nprobe =64 : time: 63.49
nprobe =128 : time: 164.05
nprobe =256 : time: 952.78
nprobe =512 : time: 1337.55
nprobe =1024 : time: 1385.09
nprobe =2048 : time: 1946.31

(2)OpenBlas
nprobe =1 : time: 318.63
nprobe =2 : time: 451.08
nprobe =4 : time: 432.12
nprobe =8 : time: 441.69
nprobe =16 : time: 342.52
nprobe =32 : time: 505.20
nprobe =64 : time: 406.39
nprobe =128 : time: 399.81
nprobe =256 : time: 497.65
nprobe =512 : time: 478.39
nprobe =1024 : time: 499.43
nprobe =2048 : time: 597.84

When calculated with SSE on the server machine, it can be reproduced.

[cqy123456]

[version 0.10.1]
Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz* 48

index_type = FLAT, metric_type = L2
n = 16384, d = 128, nq = 1000

k 1 : time: 49.78
k 2 : time: 27.61
k 4 : time: 25.93
k 8 : time: 27.67
k 16 : time: 33.72
k 32 : time: 42.16
k 64 : time: 61.21
k 128 : time: 164.23
k 256 : time: 864.20
k 512 : time: 1262.92
k 1024 : time: 1397.52

Here is the most cost.

[cqy123456]

[version 0.10.2]
Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz* 48
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 30720K

index_type = IVF_SQ8, metric_type = L2
n = 4000000, d = 128, nlist = 16384
nq = 1000, tok = 64

(1)SSE
nprobe =1 : time: 49.11
nprobe =2 : time: 23.33
nprobe =4 : time: 23.54
nprobe =8 : time: 22.80
nprobe =16 : time: 23.49
nprobe =32 : time: 24.56
nprobe =64 : time: 26.42
nprobe =128 : time: 34.68
nprobe =256 : time: 49.55
nprobe =512 : time: 50.35
nprobe =1024 : time: 69.39
nprobe =2048 : time: 115.48

It is fixed.