Skip to content

After IVF_SQ8H querying, an CUDA error will occur when Milvus exits #3760

Closed
@shengjun1985

Description

Please state your issue using the following template and, most importantly, in English.

Describe the bug
After IVF_SQ8H querying, an CUDA error will occur when Milvus exits

Steps/Code to reproduce behavior
(0) Set configuration GPU enable: true and gpu_search_threshold: 1000
(1) create a collection
(2) insert vectors into the collection and flush
(3) create index IVF_SQ8H
(4) query with nq = 5
(5) ctrl + 'C' (to exit Milvus server)

Expected behavior
Milvus exit with no error.

Method of installation

  • Docker/cpu

  • Docker/gpu

  • Build from source

  • Milvus version (master or released version)
    0.10.2

Screenshots
......
Supported CPU instruction sets: avx2, sse4_2
Nvidia driver version: 435.21
CUDA Driver Version / Runtime Version : 10.1 / 10.0
FAISS hook AVX2
Milvus server started successfully!
^CMilvus server is going to shutdown ...
Milvus server exit...
Faiss assertion 'err__ == cudaSuccess' failed in faiss::gpu::DeviceTensor<T, Dim, InnerContig, IndexT, PtrTraits>::~DeviceTensor() [with T = float; int Dim = 1; bool InnerContig = true; IndexT = int; PtrTraits = faiss::gpu::traits::DefaultPtrTraits] at ./faiss/gpu/utils/DeviceTensor-inl.cuh:59; details: CUDA error 29 driver shutting down
Aborted (core dumped)

Additional context
Add any other context about the problem here.

Metadata

Assignees

Labels

kind/bugIssues or changes related a bug

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions