After IVF_SQ8H querying, an CUDA error will occur when Milvus exits #3760
Description
Please state your issue using the following template and, most importantly, in English.
Describe the bug
After IVF_SQ8H querying, an CUDA error will occur when Milvus exits
Steps/Code to reproduce behavior
(0) Set configuration GPU enable: true and gpu_search_threshold: 1000
(1) create a collection
(2) insert vectors into the collection and flush
(3) create index IVF_SQ8H
(4) query with nq = 5
(5) ctrl + 'C' (to exit Milvus server)
Expected behavior
Milvus exit with no error.
Method of installation
-
Docker/cpu
-
Docker/gpu
-
Build from source
-
Milvus version (master or released version)
0.10.2
Screenshots
......
Supported CPU instruction sets: avx2, sse4_2
Nvidia driver version: 435.21
CUDA Driver Version / Runtime Version : 10.1 / 10.0
FAISS hook AVX2
Milvus server started successfully!
^CMilvus server is going to shutdown ...
Milvus server exit...
Faiss assertion 'err__ == cudaSuccess' failed in faiss::gpu::DeviceTensor<T, Dim, InnerContig, IndexT, PtrTraits>::~DeviceTensor() [with T = float; int Dim = 1; bool InnerContig = true; IndexT = int; PtrTraits = faiss::gpu::traits::DefaultPtrTraits] at ./faiss/gpu/utils/DeviceTensor-inl.cuh:59; details: CUDA error 29 driver shutting down
Aborted (core dumped)
Additional context
Add any other context about the problem here.