[BUG]: gnn_fraud_detection_pipeline example intermittently segfaults on shutdown

[dagardner-nv]

Version

23.03

Which installation method(s) does this occur on?

Source

Describe the bug.

Seen in PR #758 happens roughly 1/10 runs I get a segfault on shutdown, and unfortunately I can't get it to generate a core dump.

Graph construction rate[Complete]: 265 messages [00:00, 2959.73 messages/s]
Inference rate[Complete]: 265 messages [00:03, 73.04 messages/s]
Add classification rate[Complete]: 265 messages [00:03, 72.50 messages/s]
Serialize rate[Complete]: 265 messages [00:03, 71.42 messages/s]
====Pipeline Complete====
*** Aborted at 1680026004 (unix time) try "date -d @1680026004" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 2987396 (TID 0x7f330cc6b740) from PID 0; stack trace: ***
    @     0x7f330bfd0197 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f330ca3daa0 (unknown)
    @     0x7f2fa920571c std::_Hashtable<>::find()
    @     0x7f2fa920e901 _ZNSt19_Sp_counted_deleterIPN4raft13interruptibleEZNS1_14get_token_implILb1EEESt10shared_ptrIS1_ENSt6thread2idEEUlT_E_SaIvELN9__gnu_cxx12_Lock_policyE2EE10_M_disposeEv  
    @     0x7f2fa91de380 std::shared_ptr<>::~shared_ptr()
    @     0x7f330b4faf2c (anonymous namespace)::run()
    @     0x7f330ca3ffb5 (unknown)
    @     0x7f330ca4012a exit
    @     0x55e28a690631 Py_Exit
    @     0x55e28a690580 handle_system_exit
    @     0x55e28a690253 _PyErr_PrintEx
    @     0x55e28a698b22 PyRun_SimpleFileExFlags
    @     0x55e28a68c173 Py_RunMain.cold
    @     0x55e28a7ccc49 Py_BytesMain
    @     0x7f330ca2920a (unknown)
    @     0x7f330ca292bc __libc_start_main
    @     0x55e28a7ccb55 (unknown)
Segmentation fault

Minimum reproducible example

mamba env update -n ${CONDA_DEFAULT_ENV} -f examples/gnn_fraud_detection_pipeline/requirements.yml
cd examples/gnn_fraud_detection_pipeline
python run.py

This is also reproducible using the morpheus CLI tool

Relevant log output

No response

Full env printout

No response

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[dagardner-nv]

I was able to produce a core dump, and the back-trace is:

#0  std::_Hash_bytes (ptr=<optimized out>, len=8, seed=<optimized out>) at ../../../../libstdc++-v3/libsupc++/hash_bytes.cc:44
44	../../../../libstdc++-v3/libsupc++/hash_bytes.cc: No such file or directory.
[Current thread is 1 (Thread 0x7f447e042740 (LWP 3014522))]
(gdb) bt
#0  std::_Hash_bytes (ptr=<optimized out>, len=8, seed=<optimized out>) at ../../../../libstdc++-v3/libsupc++/hash_bytes.cc:44
#1  0x00007f4119c0574d in std::_Hashtable<std::thread::id, std::pair<std::thread::id const, std::weak_ptr<raft::interruptible> >, std::allocator<std::pair<std::thread::id const, std::weak_ptr<raft::interruptible> > >, std::__detail::_Select1st, std::equal_to<std::thread::id>, std::hash<std::thread::id>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::find(std::thread::id const&) () from /home/dagardner/work/conda/envs/morpheus/lib/python3.8/site-packages/cuml/common/../../../../libcuml++.so
#2  0x00007f4119c0e901 in std::_Sp_counted_deleter<raft::interruptible*, raft::interruptible::get_token_impl<true>(std::thread::id)::{lambda(auto:1)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /home/dagardner/work/conda/envs/morpheus/lib/python3.8/site-packages/cuml/common/../../../../libcuml++.so
#3  0x00007f4119bde380 in std::shared_ptr<raft::interruptible>::~shared_ptr() () from /home/dagardner/work/conda/envs/morpheus/lib/python3.8/site-packages/cuml/common/../../../../libcuml++.so
#4  0x00007f447c8faf2c in (anonymous namespace)::run (p=<optimized out>) at ../../../../libstdc++-v3/libsupc++/atexit_thread.cc:80
#5  0x00007f447de3ffb5 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x00007f447de4012a in exit () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x000055a71b19d631 in Py_Exit (sts=0) at /usr/local/src/conda/python-3.8.15/Python/pylifecycle.c:2298
#8  0x000055a71b19d580 in handle_system_exit () at /usr/local/src/conda/python-3.8.15/Python/pythonrun.c:684
#9  0x000055a71b19d253 in _PyErr_PrintEx (tstate=0x55a71c272c30, set_sys_last_vars=set_sys_last_vars@entry=1) at /usr/local/src/conda/python-3.8.15/Python/pythonrun.c:694
#10 0x000055a71b19d231 in PyErr_PrintEx (set_sys_last_vars=set_sys_last_vars@entry=1) at /usr/local/src/conda/python-3.8.15/Python/pythonrun.c:789
#11 0x000055a71b19d226 in PyErr_Print () at /usr/local/src/conda/python-3.8.15/Python/pythonrun.c:795
#12 0x000055a71b1a5b22 in pyrun_simple_file (flags=0x7ffe8aa8e1a8, closeit=<optimized out>, filename=0x7f447d8b1a30, fp=0x55a71c270e30) at /usr/local/src/conda/python-3.8.15/Python/pythonrun.c:445
#13 PyRun_SimpleFileExFlags (fp=fp@entry=0x55a71c270e30, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7ffe8aa8e1a8) at /usr/local/src/conda/python-3.8.15/Python/pythonrun.c:472
#14 0x000055a71b1a5695 in PyRun_AnyFileExFlags (fp=fp@entry=0x55a71c270e30, filename=<optimized out>, closeit=closeit@entry=1, flags=flags@entry=0x7ffe8aa8e1a8) at /usr/local/src/conda/python-3.8.15/Python/pythonrun.c:90
#15 0x000055a71b199173 in pymain_run_file (cf=0x7ffe8aa8e1a8, config=0x55a71c271c00) at /usr/local/src/conda/python-3.8.15/Modules/main.c:389
#16 pymain_run_python (exitcode=0x7ffe8aa8e1a0) at /usr/local/src/conda/python-3.8.15/Modules/main.c:614
#17 Py_RunMain () at /usr/local/src/conda/python-3.8.15/Modules/main.c:693
#18 0x000055a71b2d9c49 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/local/src/conda/python-3.8.15/Modules/main.c:1113
#19 0x00007f447de2920a in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#20 0x00007f447de292bc in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#21 0x000055a71b2d9b55 in _start () at /usr/local/src/conda/python-3.8.15/Objects/typeobject.c:2283

[dagardner-nv]

I'm unable to reproduce this issue in 23.07