[Bug]: [benchmark] milvus panic `SIGSEGV: segmentation violation` when enabled queryNode.segcore.multipleChunkedEnable #36834

wangting0128 · 2024-10-14T03:55:24Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version:master-20241013-d230b91b-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.5rc7
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: fouramf-concurrent-kd529
test case name: test_bitmap_locust_dql_dml_standalone

server:

NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouram-54-5077-etcd-0                                             1/1     Running            0               41m     10.104.17.251   4am-node23   <none>           <none>
fouram-54-5077-milvus-standalone-7d64cf9cf6-ftfbs                 0/1     CrashLoopBackOff   7 (68s ago)     41m     10.104.25.169   4am-node30   <none>           <none>
fouram-54-5077-minio-6888fdd5b4-4dhpx                             1/1     Running            0               41m     10.104.17.250   4am-node23   <none>           <none>

fouram-54-5077-milvus-standalone-7d64cf9cf6-ftfbs-panic.log

Expected Behavior

No response

Steps To Reproduce

concurrent test and calculation of RT and QPS

        :purpose:  `primary key: INT64 autoID`
            1. building `BITMAP` index on all supported 12 scalar fields
            2. 2 fields of different vector types
            3. verify DQL & DML requests

        :test steps:
            1. create collection with fields:
                'float_vector': 128dim
                'sparse_float_vector': sparse_range=[1, 100] <- the range of non-zero values of a sparse vector
                'id': primary key type is INT64

                all scalar fields: varchar max_length=100, array max_capacity=13
            2. build indexes:
                IVF_SQ8: 'float_vector'
                SPARSE_WAND: 'sparse_float_vector'
                BITMAP: all scalar fields
            3. insert 2 million data
            4. flush collection
            5. build indexes again using the same params
            6. load collection
            7. concurrent request:
                - search
                - query
                - hybrid_search
                - load
                - insert
                - delete: delete all inserted data
                - flush: ignore RateLimiter

Milvus Log

No response

Anything else?

server config:

{
     "standalone": {
          "resources": {
               "limits": {
                    "cpu": "16.0",
                    "memory": "64Gi"
               },
               "requests": {
                    "cpu": "9.0",
                    "memory": "33Gi"
               }
          }
     },
     "cluster": {
          "enabled": false
     },
     "etcd": {
          "replicaCount": 1,
          "metrics": {
               "enabled": true,
               "podMonitor": {
                    "enabled": true
               }
          }
     },
     "minio": {
          "mode": "standalone",
          "metrics": {
               "podMonitor": {
                    "enabled": true
               }
          }
     },
     "pulsar": {
          "enabled": false
     },
     "metrics": {
          "serviceMonitor": {
               "enabled": true
          }
     },
     "log": {
          "level": "debug"
     },
     "extraConfigFiles": {
          "user.yaml": "queryNode:\n  segcore:\n    multipleChunkedEnable: true\n"
     },
     "image": {
          "all": {
               "repository": "harbor.milvus.io/milvus/milvus",
               "tag": "master-20241013-d230b91b-amd64"
          }
     }
}

client config:

{
     "dataset_params": {
          "metric_type": "L2",
          "dim": 128,
          "max_length": 100,
          "scalars_index": {
               "int8_1": {
                    "index_type": "BITMAP"
               },
               "int16_1": {
                    "index_type": "BITMAP"
               },
               "int32_1": {
                    "index_type": "BITMAP"
               },
               "int64_1": {
                    "index_type": "BITMAP"
               },
               "varchar_1": {
                    "index_type": "BITMAP"
               },
               "bool_1": {
                    "index_type": "BITMAP"
               },
               "array_int8_1": {
                    "index_type": "BITMAP"
               },
               "array_int16_1": {
                    "index_type": "BITMAP"
               },
               "array_int32_1": {
                    "index_type": "BITMAP"
               },
               "array_int64_1": {
                    "index_type": "BITMAP"
               },
               "array_varchar_1": {
                    "index_type": "BITMAP"
               },
               "array_bool_1": {
                    "index_type": "BITMAP"
               }
          },
          "vectors_index": {
               "sparse_float_vector": {
                    "index_type": "SPARSE_INVERTED_INDEX",
                    "index_param": {
                         "drop_ratio_build": 0.2
                    },
                    "metric_type": "IP"
               }
          },
          "scalars_params": {
               "array_int8_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -128,
                                   128
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "array_int16_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -200,
                                   200
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "array_int32_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope",
                              "specify_range": [
                                   -300,
                                   300
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "array_int64_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "fixed_value_range",
                              "specify_range": [
                                   -400,
                                   432
                              ],
                              "batch": 50,
                              "max_capacity": 13
                         }
                    }
               },
               "array_varchar_1": {
                    "params": {
                         "max_capacity": 13
                    },
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -1500,
                                   1500
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "array_bool_1": {
                    "params": {
                         "max_capacity": 13
                    }
               },
               "int8_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -128,
                                   128
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "int16_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -200,
                                   200
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "int32_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "specify_scope",
                              "specify_range": [
                                   -300,
                                   300
                              ],
                              "max_capacity": 13
                         }
                    }
               },
               "int64_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "fixed_value_range",
                              "specify_range": [
                                   -400,
                                   432
                              ],
                              "batch": 50,
                              "max_capacity": 13
                         }
                    }
               },
               "varchar_1": {
                    "other_params": {
                         "dataset": "random_algorithm",
                         "algorithm_params": {
                              "algorithm_name": "random_range",
                              "specify_range": [
                                   -1500,
                                   1500
                              ],
                              "max_capacity": 13
                         }
                    }
               }
          },
          "dataset_name": "sift",
          "dataset_size": 2000000,
          "ni_per": 5000
     },
     "collection_params": {
          "other_fields": [
               "sparse_float_vector",
               "int8_1",
               "int16_1",
               "int32_1",
               "int64_1",
               "varchar_1",
               "bool_1",
               "array_int8_1",
               "array_int16_1",
               "array_int32_1",
               "array_int64_1",
               "array_varchar_1",
               "array_bool_1"
          ],
          "shards_num": 1,
          "auto_id": true
     },
     "resource_groups_params": {
          "reset": false
     },
     "database_user_params": {
          "reset_rbac": false,
          "reset_db": false
     },
     "index_params": {
          "index_type": "IVF_SQ8",
          "index_param": {
               "nlist": 1024
          }
     },
     "concurrent_params": {
          "concurrent_number": [
               20
          ],
          "during_time": "3h",
          "interval": 20
     },
     "concurrent_tasks": [
          {
               "type": "search",
               "weight": 1,
               "params": {
                    "nq": 10,
                    "top_k": 10,
                    "search_param": {
                         "nprobe": 16
                    },
                    "expr": "int8_1 == 100",
                    "guarantee_timestamp": null,
                    "partition_names": null,
                    "output_fields": [
                         "id",
                         "float_vector",
                         "int64_1"
                    ],
                    "ignore_growing": false,
                    "group_by_field": null,
                    "timeout": 60,
                    "random_data": true,
                    "check_task": "check_search_output",
                    "check_items": {
                         "nq": 10
                    }
               }
          },
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "ids": null,
                    "expr": "int64_1 > -1",
                    "output_fields": [
                         "*"
                    ],
                    "offset": null,
                    "limit": 10,
                    "ignore_growing": false,
                    "partition_names": null,
                    "timeout": 60,
                    "consistency_level": null,
                    "random_data": false,
                    "random_count": 0,
                    "random_range": [
                         0,
                         1
                    ],
                    "field_name": "id",
                    "field_type": "int64",
                    "check_task": "check_query_output",
                    "check_items": {
                         "expect_length": 10
                    }
               }
          },
          {
               "type": "hybrid_search",
               "weight": 1,
               "params": {
                    "nq": 10,
                    "top_k": 1,
                    "reqs": [
                         {
                              "search_param": {
                                   "nprobe": 128
                              },
                              "anns_field": "float_vector",
                              "expr": "(array_contains_any(array_int32_1, [0]) || array_contains(array_int64_1, 1)) || ((varchar_1 like \"1%\") and (bool_1 == True))",
                              "top_k": 100
                         },
                         {
                              "search_param": {
                                   "drop_ratio_search": 0.1
                              },
                              "anns_field": "sparse_float_vector",
                              "expr": "not (int16_1 == int8_1) && ARRAY_CONTAINS_ANY(array_int64_1, [-1, 0, 1])"
                         }
                    ],
                    "rerank": {
                         "RRFRanker": []
                    },
                    "output_fields": [
                         "*"
                    ],
                    "ignore_growing": false,
                    "guarantee_timestamp": null,
                    "partition_names": null,
                    "timeout": 120,
                    "random_data": true,
                    "check_task": "check_search_output",
                    "check_items": {
                         "output_fields": [
                              "sparse_float_vector",
                              "int8_1",
                              "int16_1",
                              "int32_1",
                              "int64_1",
                              "varchar_1",
                              "bool_1",
                              "array_int8_1",
                              "array_int16_1",
                              "array_int32_1",
                              "array_int64_1",
                              "array_varchar_1",
                              "array_bool_1",
                              "id",
                              "float_vector"
                         ],
                         "nq": 10
                    }
               }
          },
          {
               "type": "load",
               "weight": 1,
               "params": {
                    "replica_number": 1,
                    "timeout": 180,
                    "check_task": "check_response",
                    "check_items": null
               }
          },
          {
               "type": "insert",
               "weight": 1,
               "params": {
                    "nb": 10,
                    "timeout": 30,
                    "random_id": true,
                    "random_vector": true,
                    "varchar_filled": false,
                    "start_id": 2000000,
                    "shuffle_id": false,
                    "check_task": "check_response",
                    "check_items": null
               }
          },
          {
               "type": "delete",
               "weight": 1,
               "params": {
                    "expr": "",
                    "delete_length": 10,
                    "timeout": 30,
                    "check_task": "check_response",
                    "check_items": null
               }
          },
          {
               "type": "flush",
               "weight": 1,
               "params": {
                    "timeout": 600,
                    "check_task": "check_ignore_expected_errors",
                    "check_items": [
                         {
                              "message": "request is rejected by grpc RateLimiter middleware, please retry later"
                         },
                         {
                              "message": "wait for flush timeout"
                         }
                    ]
               }
          }
     ]
}

The text was updated successfully, but these errors were encountered:

wangting0128 · 2024-10-14T03:57:27Z

Running e2e test cases L0,L1,L2, has the same panic

argo task: fouramf-e2e-multi-chunked-enable

server:

NAME                                                              READY   STATUS             RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-e2e-multi-chunked-enable-etcd-0                           1/1     Running            0                43m     10.104.30.212   4am-node38   <none>           <none>
fouramf-e2e-multi-chunked-enable-etcd-1                           1/1     Running            0                43m     10.104.18.115   4am-node25   <none>           <none>
fouramf-e2e-multi-chunked-enable-etcd-2                           1/1     Running            0                43m     10.104.23.37    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-datanode-995b69b6bjkmxc   1/1     Running            2 (43m ago)      43m     10.104.34.7     4am-node37   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-indexnode-594b66d7vn2ml   1/1     Running            2 (43m ago)      43m     10.104.16.203   4am-node21   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-mixcoord-f4844558-5jkzn   1/1     Running            2 (43m ago)      43m     10.104.16.202   4am-node21   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-proxy-7f55b5ddcb-r5749    1/1     Running            2 (43m ago)      43m     10.104.32.153   4am-node39   <none>           <none>
fouramf-e2e-multi-chunked-enable-milvus-querynode-6bc48bdbgdrw9   0/1     Running            11 (5m12s ago)   43m     10.104.32.152   4am-node39   <none>           <none>
fouramf-e2e-multi-chunked-enable-minio-0                          1/1     Running            0                43m     10.104.23.34    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-minio-1                          1/1     Running            0                43m     10.104.30.211   4am-node38   <none>           <none>
fouramf-e2e-multi-chunked-enable-minio-2                          1/1     Running            0                43m     10.104.20.189   4am-node22   <none>           <none>
fouramf-e2e-multi-chunked-enable-minio-3                          1/1     Running            0                43m     10.104.18.119   4am-node25   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-bookie-0                  1/1     Running            0                43m     10.104.23.35    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-bookie-1                  1/1     Running            0                43m     10.104.30.213   4am-node38   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-bookie-2                  1/1     Running            0                43m     10.104.20.193   4am-node22   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-bookie-init-2k8gh         0/1     Completed          0                43m     10.104.23.27    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-broker-0                  1/1     Running            0                43m     10.104.6.198    4am-node13   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-proxy-0                   1/1     Running            0                43m     10.104.5.21     4am-node12   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-pulsar-init-zgkk5         0/1     Completed          0                43m     10.104.30.204   4am-node38   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-recovery-0                1/1     Running            0                43m     10.104.6.197    4am-node13   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-zookeeper-0               1/1     Running            0                43m     10.104.18.113   4am-node25   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-zookeeper-1               1/1     Running            0                43m     10.104.23.39    4am-node27   <none>           <none>
fouramf-e2e-multi-chunked-enable-pulsar-zookeeper-2               1/1     Running            0                42m     10.104.17.254   4am-node23   <none>           <none>

wangting0128 · 2024-10-14T06:25:05Z

different case, same panic

argo task: fouramf-concurrent-kd529
test case name: test_inverted_locust_hnsw_diskann_dml_dql_cluster
image: master-20241013-d230b91b-amd64

server:

NAME                                                              READY   STATUS             RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouram-10-2077-etcd-0                                             1/1     Running            0                3h12m   10.104.23.12    4am-node27   <none>           <none>
fouram-10-2077-etcd-1                                             1/1     Running            0                3h12m   10.104.18.99    4am-node25   <none>           <none>
fouram-10-2077-etcd-2                                             1/1     Running            0                3h12m   10.104.30.203   4am-node38   <none>           <none>
fouram-10-2077-milvus-datanode-74b5f89c87-jx7sn                   1/1     Running            2 (3h12m ago)    3h12m   10.104.14.33    4am-node18   <none>           <none>
fouram-10-2077-milvus-indexnode-846f6676b-64j52                   1/1     Running            2 (3h12m ago)    3h12m   10.104.19.163   4am-node28   <none>           <none>
fouram-10-2077-milvus-indexnode-846f6676b-k9d4t                   1/1     Running            2 (3h12m ago)    3h12m   10.104.1.125    4am-node10   <none>           <none>
fouram-10-2077-milvus-indexnode-846f6676b-k9rwr                   1/1     Running            2 (3h12m ago)    3h12m   10.104.17.252   4am-node23   <none>           <none>
fouram-10-2077-milvus-indexnode-846f6676b-n48bp                   1/1     Running            2 (3h12m ago)    3h12m   10.104.33.34    4am-node36   <none>           <none>
fouram-10-2077-milvus-mixcoord-5c988f665f-2s9n4                   1/1     Running            2 (3h12m ago)    3h12m   10.104.14.32    4am-node18   <none>           <none>
fouram-10-2077-milvus-proxy-74d98c7bff-wngqg                      1/1     Running            2 (3h12m ago)    3h12m   10.104.14.31    4am-node18   <none>           <none>
fouram-10-2077-milvus-querynode-6ccb56788d-sxb7d                  0/1     CrashLoopBackOff   28 (4m37s ago)   3h12m   10.104.24.234   4am-node29   <none>           <none>
fouram-10-2077-milvus-querynode-6ccb56788d-xdhsz                  0/1     CrashLoopBackOff   30 (3m1s ago)    3h12m   10.104.9.115    4am-node14   <none>           <none>
fouram-10-2077-minio-0                                            1/1     Running            0                3h12m   10.104.18.101   4am-node25   <none>           <none>
fouram-10-2077-minio-1                                            1/1     Running            0                3h12m   10.104.30.201   4am-node38   <none>           <none>
fouram-10-2077-minio-2                                            1/1     Running            0                3h12m   10.104.23.13    4am-node27   <none>           <none>
fouram-10-2077-minio-3                                            1/1     Running            0                3h12m   10.104.20.183   4am-node22   <none>           <none>
fouram-10-2077-pulsar-bookie-0                                    1/1     Running            0                3h12m   10.104.30.202   4am-node38   <none>           <none>
fouram-10-2077-pulsar-bookie-1                                    1/1     Running            0                3h12m   10.104.18.104   4am-node25   <none>           <none>
fouram-10-2077-pulsar-bookie-2                                    1/1     Running            0                3h12m   10.104.20.185   4am-node22   <none>           <none>
fouram-10-2077-pulsar-bookie-init-z659n                           0/1     Completed          0                3h12m   10.104.6.190    4am-node13   <none>           <none>
fouram-10-2077-pulsar-broker-0                                    1/1     Running            0                3h12m   10.104.4.220    4am-node11   <none>           <none>
fouram-10-2077-pulsar-proxy-0                                     1/1     Running            0                3h12m   10.104.4.219    4am-node11   <none>           <none>
fouram-10-2077-pulsar-pulsar-init-rq5tr                           0/1     Completed          0                3h12m   10.104.6.191    4am-node13   <none>           <none>
fouram-10-2077-pulsar-recovery-0                                  1/1     Running            0                3h12m   10.104.6.192    4am-node13   <none>           <none>
fouram-10-2077-pulsar-zookeeper-0                                 1/1     Running            0                3h12m   10.104.30.200   4am-node38   <none>           <none>
fouram-10-2077-pulsar-zookeeper-1                                 1/1     Running            0                3h11m   10.104.23.17    4am-node27   <none>           <none>
fouram-10-2077-pulsar-zookeeper-2                                 1/1     Running            0                3h11m   10.104.18.109   4am-node25   <none>           <none>

xiaofan-luan · 2024-10-14T06:51:28Z

/assign @sunby

let's enable this feature by default so we can find more bug

sunby · 2024-10-14T11:28:46Z

/assign @sunby

let's enable this feature by default so we can find more bug

yes, we plan to enable this after some stability tests

sunby · 2024-10-14T11:29:44Z

I will build a debug image and try to reproduce it.

Fix segmentation fault error and remove useless codes. #36834 Signed-off-by: sunby <sunbingyi1992@gmail.com>

wangting0128 · 2024-10-16T04:13:05Z

e2e test verification failed

argo task：fouramf-e2e-sby-fix
image: master-20241016-b2037c95-amd64

server:

NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-e2e-sby-fix-etcd-0                                        1/1     Running            0               23m     10.104.19.120   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-etcd-1                                        1/1     Running            0               23m     10.104.32.178   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-etcd-2                                        1/1     Running            0               23m     10.104.34.119   4am-node37   <none>           <none>
fouramf-e2e-sby-fix-milvus-datanode-66fcb7bd44-27csb              1/1     Running            3 (22m ago)     23m     10.104.32.169   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-milvus-indexnode-f79868cf7-gs8cx              1/1     Running            3 (22m ago)     23m     10.104.19.98    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-milvus-mixcoord-55df88fc46-w2vsf              1/1     Running            4 (18m ago)     23m     10.104.19.96    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-milvus-proxy-6894d45f48-778l6                 1/1     Running            4 (18m ago)     23m     10.104.19.100   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-milvus-querynode-76bc48c667-vdffr             0/1     CrashLoopBackOff   9 (3m16s ago)   23m     10.104.32.168   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-minio-0                                       1/1     Running            0               23m     10.104.19.122   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-minio-1                                       1/1     Running            0               23m     10.104.32.180   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-minio-2                                       1/1     Running            0               23m     10.104.34.120   4am-node37   <none>           <none>
fouramf-e2e-sby-fix-minio-3                                       1/1     Running            0               23m     10.104.20.105   4am-node22   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-0                               1/1     Running            0               23m     10.104.19.121   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-1                               1/1     Running            0               23m     10.104.32.179   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-2                               1/1     Running            0               23m     10.104.26.217   4am-node32   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-init-4f5hb                      0/1     Completed          0               23m     10.104.19.102   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-broker-0                               1/1     Running            0               23m     10.104.32.171   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-pulsar-proxy-0                                1/1     Running            0               23m     10.104.19.97    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-pulsar-init-k4bxx                      0/1     Completed          0               23m     10.104.19.99    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-recovery-0                             1/1     Running            0               23m     10.104.19.101   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-zookeeper-0                            1/1     Running            0               23m     10.104.19.119   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-zookeeper-1                            1/1     Running            0               22m     10.104.32.182   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-pulsar-zookeeper-2                            1/1     Running            0               21m     10.104.34.125   4am-node37   <none>           <none>

sunby · 2024-10-22T11:31:56Z

e2e test verification failed

argo task：fouramf-e2e-sby-fix image: master-20241016-b2037c95-amd64

server:

NAME                                                              READY   STATUS             RESTARTS        AGE     IP              NODE         NOMINATED NODE   READINESS GATES
fouramf-e2e-sby-fix-etcd-0                                        1/1     Running            0               23m     10.104.19.120   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-etcd-1                                        1/1     Running            0               23m     10.104.32.178   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-etcd-2                                        1/1     Running            0               23m     10.104.34.119   4am-node37   <none>           <none>
fouramf-e2e-sby-fix-milvus-datanode-66fcb7bd44-27csb              1/1     Running            3 (22m ago)     23m     10.104.32.169   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-milvus-indexnode-f79868cf7-gs8cx              1/1     Running            3 (22m ago)     23m     10.104.19.98    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-milvus-mixcoord-55df88fc46-w2vsf              1/1     Running            4 (18m ago)     23m     10.104.19.96    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-milvus-proxy-6894d45f48-778l6                 1/1     Running            4 (18m ago)     23m     10.104.19.100   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-milvus-querynode-76bc48c667-vdffr             0/1     CrashLoopBackOff   9 (3m16s ago)   23m     10.104.32.168   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-minio-0                                       1/1     Running            0               23m     10.104.19.122   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-minio-1                                       1/1     Running            0               23m     10.104.32.180   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-minio-2                                       1/1     Running            0               23m     10.104.34.120   4am-node37   <none>           <none>
fouramf-e2e-sby-fix-minio-3                                       1/1     Running            0               23m     10.104.20.105   4am-node22   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-0                               1/1     Running            0               23m     10.104.19.121   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-1                               1/1     Running            0               23m     10.104.32.179   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-2                               1/1     Running            0               23m     10.104.26.217   4am-node32   <none>           <none>
fouramf-e2e-sby-fix-pulsar-bookie-init-4f5hb                      0/1     Completed          0               23m     10.104.19.102   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-broker-0                               1/1     Running            0               23m     10.104.32.171   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-pulsar-proxy-0                                1/1     Running            0               23m     10.104.19.97    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-pulsar-init-k4bxx                      0/1     Completed          0               23m     10.104.19.99    4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-recovery-0                             1/1     Running            0               23m     10.104.19.101   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-zookeeper-0                            1/1     Running            0               23m     10.104.19.119   4am-node28   <none>           <none>
fouramf-e2e-sby-fix-pulsar-zookeeper-1                            1/1     Running            0               22m     10.104.32.182   4am-node39   <none>           <none>
fouramf-e2e-sby-fix-pulsar-zookeeper-2                            1/1     Running            0               21m     10.104.34.125   4am-node37   <none>           <none>

sorry, I made a stupid mistake. fixed in #37064

issue: #36834 Signed-off-by: sunby <sunbingyi1992@gmail.com>

xiaofan-luan · 2024-10-23T18:34:23Z

@wangting0128
please help to check on that

wangting0128 · 2024-10-23T19:53:48Z

@wangting0128

please help to check on that

Verification passed， milvus not panic, but raises new error：
#37083

wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels Oct 14, 2024

wangting0128 added this to the 2.5.0 milestone Oct 14, 2024

wangting0128 assigned sunby and yanliang567 Oct 14, 2024

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 14, 2024

yanliang567 removed their assignment Oct 14, 2024

sunby mentioned this issue Oct 15, 2024

fix: use chunk_row_nums to iterate #36882

Merged

sre-ci-robot pushed a commit that referenced this issue Oct 16, 2024

fix: use chunk_row_nums to iterate (#36882)

b2037c9

Fix segmentation fault error and remove useless codes. #36834 Signed-off-by: sunby <sunbingyi1992@gmail.com>

sunby mentioned this issue Oct 22, 2024

fix: fix missing return value in chunked column #37064

Merged

xiaofan-luan pushed a commit that referenced this issue Oct 22, 2024

fix: fix missing return value in chunked column (#37064)

90b3907

issue: #36834 Signed-off-by: sunby <sunbingyi1992@gmail.com>

xiaofan-luan closed this as completed Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: [benchmark] milvus panic `SIGSEGV: segmentation violation` when enabled queryNode.segcore.multipleChunkedEnable #36834

[Bug]: [benchmark] milvus panic `SIGSEGV: segmentation violation` when enabled queryNode.segcore.multipleChunkedEnable #36834

wangting0128 commented Oct 14, 2024

wangting0128 commented Oct 14, 2024

wangting0128 commented Oct 14, 2024

xiaofan-luan commented Oct 14, 2024

sunby commented Oct 14, 2024

sunby commented Oct 14, 2024 •

edited

Loading

wangting0128 commented Oct 16, 2024 •

edited

Loading

sunby commented Oct 22, 2024

e2e test verification failed

xiaofan-luan commented Oct 23, 2024

wangting0128 commented Oct 23, 2024

[Bug]: [benchmark] milvus panic SIGSEGV: segmentation violation when enabled queryNode.segcore.multipleChunkedEnable #36834

[Bug]: [benchmark] milvus panic SIGSEGV: segmentation violation when enabled queryNode.segcore.multipleChunkedEnable #36834

Comments

wangting0128 commented Oct 14, 2024

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

wangting0128 commented Oct 14, 2024

Running e2e test cases L0,L1,L2, has the same panic

wangting0128 commented Oct 14, 2024

different case, same panic

xiaofan-luan commented Oct 14, 2024

sunby commented Oct 14, 2024

sunby commented Oct 14, 2024 • edited Loading

wangting0128 commented Oct 16, 2024 • edited Loading

e2e test verification failed

sunby commented Oct 22, 2024

e2e test verification failed

xiaofan-luan commented Oct 23, 2024

wangting0128 commented Oct 23, 2024

[Bug]: [benchmark] milvus panic `SIGSEGV: segmentation violation` when enabled queryNode.segcore.multipleChunkedEnable #36834

[Bug]: [benchmark] milvus panic `SIGSEGV: segmentation violation` when enabled queryNode.segcore.multipleChunkedEnable #36834

sunby commented Oct 14, 2024 •

edited

Loading

wangting0128 commented Oct 16, 2024 •

edited

Loading