[Bug]: [laion1b-test] querynode terminated due to a SIGSEV error

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: cardinal-milvus-io-2.3-af54ce9-20231219
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):  pulsar 
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.3.3
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

1. milvus with config

  config:
    dataCoord:
      compaction:
        rpcTimeout: 600 
      enableActiveStandby: true
      segment:
        expansionRate: 1.15
        maxSize: 4096
        sealProportion: 0.08
    dataNode:
      dataSync:
        maxParallelSyncTaskNum: 64
    log:
      level: debug
    minio:
      accessKeyID: xxx
      bucketName: xx
      secretAccessKey: xxx
    queryCoord:
      enableActiveStandby: true
    rootCoord:
      dmlChannelNum: 16
      enableActiveStandby: true

2. concurrent search + query + insert_delete_flush -> querynode terminated with 139 error code
   

Loki log: https://grafana-4am.zilliz.cc/explore?orgId=1&left=%7B%22datasource%22:%22Loki%22,%22queries%22:%5B%7B%22refId%22:%22B%22,%22expr%22:%22%7Bcluster%3D%5C%224am%5C%22,namespace%3D%5C%22qa-milvus%5C%22,pod%3D%5C%22laion1b-test-2-milvus-querynode-d9f5478d6-v84kc%5C%22%7D%22,%22hide%22:false%7D%5D,%22range%22:%7B%22from%22:%221703183663000%22,%22to%22:%221703183700000%22%7D%7D

Expected Behavior

No response

Steps To Reproduce

argo: https://argo-workflows.zilliz.cc/workflows/qa/laion1b-test-cron-pr6vl?tab=workflow&nodeId=laion1b-test-cron-pr6vl&nodePanelView=inputs-outputs

Milvus Log

qn_v84kc.log

Anything else?

No response

[yanliang567]

/assign @chyezh
/unassign

[chyezh]

From the coredump file.

(gdb) print $_siginfo
$5 = {si_signo = 11, si_errno = 0, si_code = -6, _sifields = {_pad = {8,
      0 <repeats 27 times>}, _kill = {si_pid = 8, si_uid = 0}, _timer = {si_tid = 8,
      si_overrun = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _rt = {
      si_pid = 8, si_uid = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}},
    _sigchld = {si_pid = 8, si_uid = 0, si_status = 0, si_utime = 0, si_stime = 0},
    _sigfault = {si_addr = 0x8, _addr_lsb = 0, _addr_bnd = {_lower = 0x0,
        _upper = 0x0}}, _sigpoll = {si_band = 8, si_fd = 0}}}

(gdb) bt
#0  runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:154
#1  0x0000000001a3e0f0 in runtime.raisebadsignal (sig=11, c=0x7f54f21deb90)
    at /usr/local/go/src/runtime/signal_unix.go:947
#2  0x0000000001a3e48c in runtime.badsignal (sig=11, c=0x7f54f21deb90)
    at /usr/local/go/src/runtime/signal_unix.go:1064
#3  0x0000000001a3cce5 in runtime.sigtrampgo (sig=11, info=0x7f54f21ded30,
    ctx=0x7f54f21dec00) at /usr/local/go/src/runtime/signal_unix.go:461
#4  0x0000000001a61666 in runtime.sigtramp ()
    at /usr/local/go/src/runtime/sys_linux_amd64.s:354
#5  0x00007f57f98f93c0 in ?? () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x0000000000000007 in ?? ()
#7  0x0000000000000000 in ?? ()
(gdb) frame 3
#3  0x0000000001a3cce5 in runtime.sigtrampgo (sig=11, info=0x7f54f21ded30,
    ctx=0x7f54f21dec00) at /usr/local/go/src/runtime/signal_unix.go:461
461			badsignal(uintptr(sig), c)
(gdb) print *info
$6 = {siginfoFields = {si_signo = 11, si_errno = 0, si_code = 128, si_addr = 0},
  _ = '\000' <repeats 103 times>}

• si_signo=11 si_code = -6 is generated by tgkill.
• real signal is si_signo=11 si_code=128, but I cannot found where the si_code=128 came from.

[chyezh]

• In Milvus Builder environment, there's no si_code=128 defined for SIGSEGV.

root@24aa49a90d4d:/usr/include# grep -A 20 'SIGSEGV si_codes' asm-generic/siginfo.h
 * SIGSEGV si_codes
 */
#define SEGV_MAPERR	1	/* address not mapped to object */
#define SEGV_ACCERR	2	/* invalid permissions for mapped object */
#define SEGV_BNDERR	3	/* failed address bound checks */
#ifdef __ia64__
# define __SEGV_PSTKOVF	4	/* paragraph stack overflow */
#else
# define SEGV_PKUERR	4	/* failed protection key checks */
#endif
#define SEGV_ACCADI	5	/* ADI not enabled for mapped object */
#define SEGV_ADIDERR	6	/* Disrupting MCD error */
#define SEGV_ADIPERR	7	/* Precise MCD exception */
#define NSIGSEGV	7

• It seems that the signal is sent by kernel. Is there some systemcall crash?

#define SI_KERNEL	0x80		/* sent by the kernel from somewhere */

[xiaofan-luan]

is this only happened on cardinal?
@liliu-z

[liliu-z]

Checking

[liliu-z]

/assign @foxspy

[chyezh]

@foxspy coredump can be found at path in pod laion1b-test-2-milvus-querynode-89866ddf4-56vdc:/tmp/cores

dlv is installed at /root/go/bin/dlv

previous coredump analysis in based on the filecore-laion1b-test-2-milvus-querynode-89866ddf4-56vdc-cardinal_search-8-1703254037

[chyezh]

• In Milvus Builder environment, there's no si_code=128 defined for SIGSEGV.

root@24aa49a90d4d:/usr/include# grep -A 20 'SIGSEGV si_codes' asm-generic/siginfo.h
 * SIGSEGV si_codes
 */
#define SEGV_MAPERR	1	/* address not mapped to object */
#define SEGV_ACCERR	2	/* invalid permissions for mapped object */
#define SEGV_BNDERR	3	/* failed address bound checks */
#ifdef __ia64__
# define __SEGV_PSTKOVF	4	/* paragraph stack overflow */
#else
# define SEGV_PKUERR	4	/* failed protection key checks */
#endif
#define SEGV_ACCADI	5	/* ADI not enabled for mapped object */
#define SEGV_ADIDERR	6	/* Disrupting MCD error */
#define SEGV_ADIPERR	7	/* Precise MCD exception */
#define NSIGSEGV	7

• It seems that the signal is sent by kernel. Is there some systemcall crash?

#define SI_KERNEL	0x80		/* sent by the kernel from somewhere */

SI_KERNEL is found at #29339. So it may be not sent by system call.

[chyezh]

3 crash more is found at laion1b-test-2-milvus-querynode-f65bb9464-mpgfx.

(gdb) print *info
$6 = {siginfoFields = {si_signo = 11, si_errno = 0, si_code = 1, si_addr = 140661649626852}, _ = '\000' <repeats 103 times>}

(gdb) print *info
$2 = {siginfoFields = {si_signo = 11, si_errno = 0, si_code = 1, si_addr = 16}, _ = '\000' <repeats 103 times>}

(gdb) print *info
$1 = {siginfoFields = {si_signo = 11, si_errno = 0, si_code = 1, si_addr = 140051262240128}, _ = '\000' <repeats 103 times>}

si_code can be SEGV_MAPERR now.

[ThreadDao]

No reproduce cardinal-foxspy-fix_binlog_index_concurrency-1367e8e-20240114