[CRASH] OpenSIPS crashes on start-up after receiving SIP message

[clifjones]

After updating to OpenSIPS 2.4 update to pick up a fix to a module, we noticed that OpenSIPS crashes sometimes at the end of start-up. It is difficult to produce this on demand and appears to be related to receiving a SIP message right at the end of the start-up sequence.

OpenSIPS version you are running

# opensips -V
version: opensips 2.4.6 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 5acb9ac7d
main.c compiled on 18:21:07 Jul 15 2019 with gcc 7

git_version: "5acb9ac7dac1baee4ec1fbf74582c43d7d00e667"

Crash Core Dump

Reading symbols from /usr/local/sbin/opensips...done.
[New LWP 6993]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /home/pcv-sip'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  fm_remove_free (n=0x7f29a65b6f98, qm=0x7f29a53a3000) at mem/f_malloc.c:209
209		*pf=n->u.nxt_free;
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-9.amzn2.0.1.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  fm_remove_free (n=0x7f29a65b6f98, qm=0x7f29a53a3000) at mem/f_malloc.c:209
#1  fm_malloc (qm=0x7f29a53a3000, size=40) at mem/f_malloc.c:438
#2  0x00000000004e0e6d in shm_malloc (size=<optimized out>)
    at evi/../mem/shm_mem.h:437
#3  new_avp (val=..., id=1, flags=<optimized out>) at usr_avp.c:117
#4  add_avp_last (flags=<optimized out>, name=1, val=...) at usr_avp.c:164
#5  0x00000000004fd966 in pv_set_avp (msg=<optimized out>,
    param=<optimized out>, op=52, val=<optimized out>) at pvar.c:2512
#6  0x00000000004fb452 in pv_set_value (msg=msg@entry=0x7f29e5438108,
    sp=sp@entry=0x7f29e53c3738, op=<optimized out>, value=<optimized out>)
    at pvar.c:4297
#7  0x000000000042dfe2 in do_assign (msg=msg@entry=0x7f29e5438108,
    a=a@entry=0x7f29e53c39a0) at action.c:290
#8  0x000000000042f626 in do_action (a=a@entry=0x7f29e53c39a0,
    msg=msg@entry=0x7f29e5438108) at action.c:1986
#9  0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=msg@entry=0x7f29e5438108) at action.c:172
#10 0x000000000043063c in do_action (a=a@entry=0x7f29e53c4858,
    msg=msg@entry=0x7f29e5438108) at action.c:1168
#11 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#12 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#13 0x0000000000433c61 in do_action (a=a@entry=0x7f29e5423770,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#14 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#15 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#16 0x0000000000433c61 in do_action (a=a@entry=0x7f29e541a598,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#17 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#18 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#19 0x0000000000433c61 in do_action (a=a@entry=0x7f29e5412be0,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#20 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#21 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#22 0x0000000000433c61 in do_action (a=a@entry=0x7f29e53f4068,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#23 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=msg@entry=0x7f29e5438108) at action.c:172
#24 0x0000000000434fc9 in do_action (a=a@entry=0x7f29e53f42c0,
    msg=msg@entry=0x7f29e5438108) at action.c:1124
#25 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#26 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#27 0x0000000000433c61 in do_action (a=a@entry=0x7f29e53cdd50,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#28 0x0000000000437bb6 in run_action_list (a=a@entry=0x7f29e53cce28,
    msg=msg@entry=0x7f29e5438108) at action.c:172
#29 0x000000000043806a in run_actions (msg=0x7f29e5438108, a=0x7f29e53cce28)
    at action.c:137
#30 run_top_route (a=0x7f29e53cce28, msg=msg@entry=0x7f29e5438108)
    at action.c:214
#31 0x000000000043ef6e in receive_msg (
    buf=0x937ec0 <buf> "ACK sip:12287281041@34.207.221.183:5060;transport=tcp SIP/2.0\r\nRoute: <sip:10.32.42.80:5060;ftag=228542;lr;r2=on>,<sip:52.20.161.184:5060;transport=tcp;ftag=228542;lr;r2=on>\r\nMax-Forwards:  8\r\nTo:  <s"...,
    len=<optimized out>, rcv_info=rcv_info@entry=0x7ffe798f4fd0,
    existing_context=existing_context@entry=0x0, flags=flags@entry=0)
    at receive.c:209
#32 0x0000000000609616 in udp_read_req (si=<optimized out>,
    bytes_read=<optimized out>) at net/proto_udp/proto_udp.c:182
#33 0x00000000005ec489 in handle_io (idx=0, event_type=2, fm=<optimized out>)
    at net/net_udp.c:261
#34 io_wait_loop_epoll (h=<optimized out>, t=<optimized out>,
    repeat=<optimized out>) at net/../io_wait_loop.h:284
#35 udp_start_processes (chd_rank=chd_rank@entry=0x924164 <chd_rank>,
    startup_done=startup_done@entry=0x7f29a57137e0) at net/net_udp.c:389
#36 0x000000000041c107 in main_loop () at main.c:782
#37 main (argc=<optimized out>, argv=<optimized out>) at main.c:1439
(gdb) quit

Describe the traffic that generated the bug

Receiving a UDP SIP ACK at the end of startup.

To Reproduce

Relevant System Logs

OS/environment information

• Operating System: Red Hat 7.3.1-5

[    0.000000] Linux version 4.14.128-112.105.amzn2.x86_64 (mockbuild@ip-10-0-1-
243) (gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)) #1 SMP Wed Jun 19 16:5
3:40 UTC 2019

• OpenSIPS installation: v2.4 built from source. (See commit ID above)
• other relevant information:
  Modules loaded:

loadmodule "proto_udp.so"
loadmodule "sl.so"
loadmodule "signaling.so"
loadmodule "uri.so"
loadmodule "maxfwd.so"
loadmodule "db_postgres.so"
loadmodule "db_virtual.so"
loadmodule "sipmsgops.so"
loadmodule "mi_datagram.so"
loadmodule "rr.so"
loadmodule "event_route.so"
loadmodule "cfgutils.so"
loadmodule "avpops.so"
loadmodule "tm.so"
loadmodule "dispatcher.so"
loadmodule "json.so"
loadmodule "statistics.so"
loadmodule "cachedb_local.so"
loadmodule "rest_client.so"
loadmodule "permissions.so"
loadmodule "nat_traversal.so"
loadmodule "proto_hep.so"
loadmodule "siptrace.so"
loadmodule "proto_tcp.so"
loadmodule "proto_tls.so"
loadmodule "tls_mgm.so"
loadmodule "exec.so"

Additional context

Every time that we see this crash, it is on start-up. It is not consistent. i.e. It does not consistently crash.

[clifjones]

Here are the logs around one of the crashes. There were a log of utimer_ticker warnings before what I'm showing here:

"Jul 16 14:51:47 [6969] WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 71860 ms (now 195760 ms), it may overlap.."
"Jul 16 14:51:47 [6969] WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 71860 ms (now 195760 ms), it may overlap.."
"Jul 16 14:51:47 [6969] WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 71860 ms (now 195760 ms), it may overlap.."
"Jul 16 14:51:47 [6969] WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 71860 ms (now 195760 ms), it may overlap.."
"Jul 16 14:51:47 [6969] WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 71860 ms (now 195760 ms), it may overlap.."
"Jul 16 14:51:47 [6969] WARNING:core:utimer_ticker: utimer task <tm-utimer> already scheduled for 71860 ms (now 195760 ms), it may overlap.."
"Jul 16 14:51:48 [6937] INFO:core:shutdown_opensips: process 3(6966) [MI Datagram] terminated, still waiting for 17 more"
"Jul 16 14:51:48 [6966] INFO:core:sig_usr: signal 15 received"
"Jul 16 14:51:48 [6937] INFO:core:shutdown_opensips: process 5(6968) [time_keeper] terminated, still waiting for 18 more"
"Jul 16 14:51:48 [6937] INFO:core:shutdown_opensips: process 6(6969) [timer] terminated, still waiting for 19 more"
"Jul 16 14:51:48 [6937] INFO:core:shutdown_opensips: process 4(6967) [MI Datagram] terminated, still waiting for 20 more"
"Jul 16 14:51:48 [6937] INFO:core:shutdown_opensips: process 2(6965) [MI Datagram] terminated, still waiting for 21 more"
"Jul 16 14:51:48 [6968] INFO:core:sig_usr: signal 15 received"
"Jul 16 14:51:48 [6937] INFO:core:shutdown_opensips: process 1(6964) [event-route handler] terminated, still waiting for 22 more"
"Jul 16 14:51:48 [6967] INFO:core:sig_usr: signal 15 received"
"Jul 16 14:51:48 [6969] INFO:core:sig_usr: signal 15 received"
"Jul 16 14:51:48 [6964] INFO:core:sig_usr: signal 15 received"
"Jul 16 14:51:48 [6965] INFO:core:sig_usr: signal 15 received"
"Jul 16 14:51:48 [6937] INFO:core:handle_sigs: terminating due to SIGCHLD"
"Jul 16 14:51:48 [6937] INFO:core:handle_sigs: core was generated"
"Jul 16 14:51:48 [6937] INFO:core:handle_sigs: child process 6978 exited by a signal 11"
"Jul 16 14:51:53 [6937] CRITICAL:core:fm_status: different free frag. count: 313!=61 for hash  75"
"Jul 16 14:51:53 [6937] NOTICE:db_virtual:destroy: destroying module..."
"Jul 16 14:51:53 [6937] NOTICE:event_route:destroy: destroy module ..."
"Jul 16 14:51:53 [6937] INFO:core:cleanup: cleanup"

[clifjones]

This load generated several core dumps. Adding additional traces.

[New LWP 6978]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /home/pcv-sip'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  fm_remove_free (n=0x7f688b8e3180, qm=0x7f688895c000) at mem/f_malloc.c:209
209		*pf=n->u.nxt_free;
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-9.amzn2.0.1.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  fm_remove_free (n=0x7f688b8e3180, qm=0x7f688895c000) at mem/f_malloc.c:209
#1  fm_malloc (qm=0x7f688895c000, size=40) at mem/f_malloc.c:438
#2  0x00000000004e07d1 in shm_malloc (size=<optimized out>)
    at evi/../mem/shm_mem.h:437
#3  new_avp (val=..., id=14, flags=0) at usr_avp.c:117
#4  add_avp (flags=flags@entry=2, name=14, val=...) at usr_avp.c:148
#5  0x00007f68831d4cf2 in push_ds_2_avps (partition=0x7f6888caf738,
    ds=<optimized out>) at dispatch.c:1458
#6  ds_select_dst (msg=msg@entry=0x7f68c89eb138,
    ds_select_ctl=ds_select_ctl@entry=0x7fff6e67aaa0,
    selected_dst=selected_dst@entry=0x7fff6e67aa50, ds_flags=<optimized out>)
    at dispatch.c:1814
#7  0x00007f68831b750c in w_ds_select (msg=0x7f68c89eb138,
    part_set=<optimized out>, alg=<optimized out>,
    max_results_flags=<optimized out>, mode=<optimized out>)
    at dispatcher.c:1140
#8  0x0000000000430467 in do_action (a=a@entry=0x7f68c89b79e8,
    msg=msg@entry=0x7f68c89eb138) at action.c:1866
#9  0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=msg@entry=0x7f68c89eb138) at action.c:172
#10 0x000000000047fa96 in eval_elem (e=0x7f68c89b7ac8, msg=0x7f68c89eb138,
    val=0x0) at route.c:1367
#11 0x000000000047f5ef in eval_expr (e=0x7f68c89b7ac8, msg=0x7f68c89eb138,
    val=0x0) at route.c:1691
#12 0x000000000047f62f in eval_expr (e=0x7f68c89b7b18, msg=0x7f68c89eb138,
    val=0x0) at route.c:1707
#13 0x000000000047f61b in eval_expr (e=0x7f68c89b7b68,
    msg=msg@entry=0x7f68c89eb138, val=val@entry=0x0) at route.c:1712
#14 0x00000000004308fe in do_action (a=a@entry=0x7f68c89b83f0,
    msg=msg@entry=0x7f68c89eb138) at action.c:1106
#15 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f68c89eb138) at action.c:172
#16 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#17 0x0000000000433c61 in do_action (a=a@entry=0x7f68c89ade70,
    msg=msg@entry=0x7f68c89eb138) at action.c:761
#18 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f68c89eb138) at action.c:172
#19 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#20 0x0000000000433c61 in do_action (a=a@entry=0x7f68c8986d50,
    msg=msg@entry=0x7f68c89eb138) at action.c:761
#21 0x0000000000437bb6 in run_action_list (a=a@entry=0x7f68c8985e28,
    msg=msg@entry=0x7f68c89eb138) at action.c:172
#22 0x000000000043806a in run_actions (msg=0x7f68c89eb138, a=0x7f68c8985e28)
    at action.c:137
#23 run_top_route (a=0x7f68c8985e28, msg=msg@entry=0x7f68c89eb138)
    at action.c:214
#24 0x000000000043ef6e in receive_msg (
    buf=0x937ec0 <buf> "INVITE sip:12287281036@testacdload36.mycompany.com:5060;transport=udp SIP/2.0\r\nTo:  <sip:12287281036@testacdload36.mycompany.com:5060>\r\nFrom:  <sip:+909560049534@testacdload36.mycompany.com:5"...,
    len=<optimized out>, rcv_info=rcv_info@entry=0x7fff6e67ca50,
    existing_context=existing_context@entry=0x0, flags=flags@entry=0)
    at receive.c:209
#25 0x0000000000609616 in udp_read_req (si=<optimized out>,
    bytes_read=<optimized out>) at net/proto_udp/proto_udp.c:182
#26 0x00000000005ec489 in handle_io (idx=0, event_type=2, fm=<optimized out>)
    at net/net_udp.c:261
#27 io_wait_loop_epoll (h=<optimized out>, t=<optimized out>,
    repeat=<optimized out>) at net/../io_wait_loop.h:284
#28 udp_start_processes (chd_rank=chd_rank@entry=0x924164 <chd_rank>,
    startup_done=startup_done@entry=0x7f6888ccc7e0) at net/net_udp.c:389
#29 0x000000000041c107 in main_loop () at main.c:782
#30 main (argc=<optimized out>, argv=<optimized out>) at main.c:1439
(gdb) quit

[New LWP 6982]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /home/pcv-sip'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  fm_remove_free (n=0x7f8b38b78da0, qm=0x7f8b38823000) at mem/f_malloc.c:209
209		*pf=n->u.nxt_free;
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-9.amzn2.0.1.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  fm_remove_free (n=0x7f8b38b78da0, qm=0x7f8b38823000) at mem/f_malloc.c:209
#1  fm_malloc (qm=0x7f8b38823000, size=size@entry=24) at mem/f_malloc.c:438
#2  0x00007f8b30773dbe in shm_malloc (size=24) at ../../mem/shm_mem.h:437
#3  os_malloc (size=24) at tls.h:89
#4  0x00007f8b37567478 in CRYPTO_malloc () from /lib64/libcrypto.so.10
#5  0x00007f8b3756a2b6 in OBJ_NAME_add () from /lib64/libcrypto.so.10
#6  0x00007f8b379a97c7 in SSL_library_init () from /lib64/libssl.so.10
#7  0x00007f8b320bf803 in Curl_ossl_init () from /lib64/libcurl.so.4
#8  0x00007f8b3208556d in global_init () from /lib64/libcurl.so.4
#9  0x00007f8b32085818 in curl_easy_init () from /lib64/libcurl.so.4
#10 0x00007f8b322e68c0 in init_sync_handle () at rest_methods.c:76
#11 0x00007f8b322ed5e2 in child_init (rank=<optimized out>)
    at rest_client.c:241
#12 0x00000000004e6bd2 in init_mod_child (m=0x7f8b78830230, rank=15,
    type=<optimized out>) at sr_module.c:579
#13 0x00000000004e6c38 in init_mod_child (type=0x655baa "CHILD", rank=15,
    m=0x7f8b7886d5a8) at sr_module.c:573
#14 init_mod_child (m=0x7f8b7886d8a0, rank=15, type=0x655baa "CHILD")
    at sr_module.c:573
#15 0x00000000004e6c38 in init_mod_child (type=0x655baa "CHILD", rank=15,
    m=0x7f8b7886dd08) at sr_module.c:573
#16 init_mod_child (m=0x7f8b7886e3d8, rank=15, type=0x655baa "CHILD")
    at sr_module.c:573
#17 0x00000000004e6c38 in init_mod_child (type=0x655baa "CHILD", rank=15,
    m=0x7f8b7886eb40) at sr_module.c:573
#18 init_mod_child (m=0x7f8b7886ec78, rank=15, type=0x655baa "CHILD")
    at sr_module.c:573
#19 0x00000000004e968d in init_mod_child (type=0x655baa "CHILD", rank=15,
    m=0x7f8b7886ee00) at sr_module.c:573
#20 init_mod_child (type=0x655baa "CHILD", rank=15, m=0x7f8b78872d90)
    at sr_module.c:573
#21 init_child (rank=15) at sr_module.c:622
#22 0x00000000006037d9 in tcp_start_processes (
    chd_rank=chd_rank@entry=0x924164 <chd_rank>,
    startup_done=startup_done@entry=0x7f8b38b937e0) at net/net_tcp.c:1881
#23 0x000000000041c201 in main_loop () at main.c:788
#24 main (argc=<optimized out>, argv=<optimized out>) at main.c:1439
(gdb) quit

[New LWP 6993]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /home/pcv-sip'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  fm_remove_free (n=0x7f29a65b6f98, qm=0x7f29a53a3000) at mem/f_malloc.c:209
209		*pf=n->u.nxt_free;
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-9.amzn2.0.1.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  fm_remove_free (n=0x7f29a65b6f98, qm=0x7f29a53a3000) at mem/f_malloc.c:209
#1  fm_malloc (qm=0x7f29a53a3000, size=40) at mem/f_malloc.c:438
#2  0x00000000004e0e6d in shm_malloc (size=<optimized out>)
    at evi/../mem/shm_mem.h:437
#3  new_avp (val=..., id=1, flags=<optimized out>) at usr_avp.c:117
#4  add_avp_last (flags=<optimized out>, name=1, val=...) at usr_avp.c:164
#5  0x00000000004fd966 in pv_set_avp (msg=<optimized out>,
    param=<optimized out>, op=52, val=<optimized out>) at pvar.c:2512
#6  0x00000000004fb452 in pv_set_value (msg=msg@entry=0x7f29e5438108,
    sp=sp@entry=0x7f29e53c3738, op=<optimized out>, value=<optimized out>)
    at pvar.c:4297
#7  0x000000000042dfe2 in do_assign (msg=msg@entry=0x7f29e5438108,
    a=a@entry=0x7f29e53c39a0) at action.c:290
#8  0x000000000042f626 in do_action (a=a@entry=0x7f29e53c39a0,
    msg=msg@entry=0x7f29e5438108) at action.c:1986
#9  0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=msg@entry=0x7f29e5438108) at action.c:172
#10 0x000000000043063c in do_action (a=a@entry=0x7f29e53c4858,
    msg=msg@entry=0x7f29e5438108) at action.c:1168
#11 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#12 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#13 0x0000000000433c61 in do_action (a=a@entry=0x7f29e5423770,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#14 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#15 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#16 0x0000000000433c61 in do_action (a=a@entry=0x7f29e541a598,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#17 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#18 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#19 0x0000000000433c61 in do_action (a=a@entry=0x7f29e5412be0,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#20 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#21 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#22 0x0000000000433c61 in do_action (a=a@entry=0x7f29e53f4068,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#23 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=msg@entry=0x7f29e5438108) at action.c:172
#24 0x0000000000434fc9 in do_action (a=a@entry=0x7f29e53f42c0,
    msg=msg@entry=0x7f29e5438108) at action.c:1124
#25 0x0000000000437bb6 in run_action_list (a=<optimized out>,
    msg=0x7f29e5438108) at action.c:172
#26 0x0000000000437ef7 in run_actions (a=<optimized out>, msg=<optimized out>)
    at action.c:137
#27 0x0000000000433c61 in do_action (a=a@entry=0x7f29e53cdd50,
    msg=msg@entry=0x7f29e5438108) at action.c:761
#28 0x0000000000437bb6 in run_action_list (a=a@entry=0x7f29e53cce28,
    msg=msg@entry=0x7f29e5438108) at action.c:172
#29 0x000000000043806a in run_actions (msg=0x7f29e5438108, a=0x7f29e53cce28)
    at action.c:137
#30 run_top_route (a=0x7f29e53cce28, msg=msg@entry=0x7f29e5438108)
    at action.c:214
#31 0x000000000043ef6e in receive_msg (
    buf=0x937ec0 <buf> "ACK sip:12287281041@34.207.221.183:5060;transport=tcp SIP/2.0\r\nRoute: <sip:10.32.42.80:5060;ftag=228542;lr;r2=on>,<sip:52.20.161.184:5060;transport=tcp;ftag=228542;lr;r2=on>\r\nMax-Forwards:  8\r\nTo:  <s"...,
    len=<optimized out>, rcv_info=rcv_info@entry=0x7ffe798f4fd0,
    existing_context=existing_context@entry=0x0, flags=flags@entry=0)
    at receive.c:209
#32 0x0000000000609616 in udp_read_req (si=<optimized out>,
    bytes_read=<optimized out>) at net/proto_udp/proto_udp.c:182
#33 0x00000000005ec489 in handle_io (idx=0, event_type=2, fm=<optimized out>)
    at net/net_udp.c:261
#34 io_wait_loop_epoll (h=<optimized out>, t=<optimized out>,
    repeat=<optimized out>) at net/../io_wait_loop.h:284
#35 udp_start_processes (chd_rank=chd_rank@entry=0x924164 <chd_rank>,
    startup_done=startup_done@entry=0x7f29a57137e0) at net/net_udp.c:389
#36 0x000000000041c107 in main_loop () at main.c:782
#37 main (argc=<optimized out>, argv=<optimized out>) at main.c:1439
(gdb) quit

[New LWP 8165]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /home/pcv-sip'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  fm_remove_free (n=0x7f70fdaf0bd0, qm=0x7f70fbec3000) at mem/f_malloc.c:209
209		*pf=n->u.nxt_free;
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-9.amzn2.0.1.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  fm_remove_free (n=0x7f70fdaf0bd0, qm=0x7f70fbec3000) at mem/f_malloc.c:209
#1  fm_malloc (qm=0x7f70fbec3000, size=size@entry=600) at mem/f_malloc.c:438
#2  0x00007f70f3e13dbe in shm_malloc (size=600) at ../../mem/shm_mem.h:437
#3  os_malloc (size=600) at tls.h:89
#4  0x00007f70fac07478 in CRYPTO_malloc () from /lib64/libcrypto.so.10
#5  0x00007f70facc1706 in ERR_get_state () from /lib64/libcrypto.so.10
#6  0x00007f70facc18ef in ERR_clear_error () from /lib64/libcrypto.so.10
#7  0x00007f70fb286552 in pqsecure_write () from /lib64/libpq.so.5
#8  0x00007f70fb27d4fe in pqSendSome () from /lib64/libpq.so.5
#9  0x00007f70fb27479f in closePGconn () from /lib64/libpq.so.5
#10 0x00007f70fb2747be in PQfinish () from /lib64/libpq.so.5
#11 0x00007f70fb4994c5 in db_postgres_free_connection (
    con=con@entry=0x7f713bf51f50) at pg_con.c:137
#12 0x00000000005baf49 in db_do_close (_h=0x7f713bf51e68,
    free_connection=0x7f70fb499451 <db_postgres_free_connection>)
    at db/db.c:357
#13 0x00007f70f7883df6 in db_virtual_close (_h=<optimized out>) at dbase.c:440
#14 0x00007f70f6735609 in ds_disconnect_db (
    partition=partition@entry=0x7f70fc216738) at dispatch.c:657
#15 0x00007f70f671edf2 in destroy () at dispatcher.c:970
#16 0x00000000004e94d3 in destroy_modules () at sr_module.c:554
#17 0x00000000004e51a9 in cleanup (show_status=show_status@entry=1)
    at main.c:334
#18 0x0000000000414627 in shutdown_opensips (status=status@entry=139)
    at main.c:490
#19 0x00000000004e5c90 in handle_sigs () at main.c:579
#20 0x000000000041c634 in main_loop () at main.c:841
#21 main (argc=<optimized out>, argv=<optimized out>) at main.c:1439
(gdb) quit

[New LWP 8179]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /home/pcv-sip'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  fm_remove_free (n=0x7f70fdaf0bd0, qm=0x7f70fbec3000) at mem/f_malloc.c:209
209		*pf=n->u.nxt_free;
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-9.amzn2.0.1.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  fm_remove_free (n=0x7f70fdaf0bd0, qm=0x7f70fbec3000) at mem/f_malloc.c:209
#1  fm_malloc (qm=0x7f70fbec3000, size=456, size@entry=451)
    at mem/f_malloc.c:438
#2  0x0000000000522ae4 in shm_malloc (size=451) at mem/shm_mem.h:437
#3  _shm_resize (p=<optimized out>, s=451) at mem/shm_mem.c:226
#4  0x00007f70f695cbe3 in relay_reply (t=<optimized out>,
    p_msg=<optimized out>, branch=<optimized out>, msg_status=<optimized out>,
    cancel_bitmap=<optimized out>) at t_reply.c:1279
#5  0x00007f70f696440d in reply_received (p_msg=0x7f713bf52300)
    at t_reply.c:1598
#6  0x0000000000464629 in forward_reply (msg=msg@entry=0x7f713bf52300)
    at forward.c:495
#7  0x000000000043ecd6 in receive_msg (
    buf=0x937ec0 <buf> "SIP/2.0 404 Not Found\r\nTo: \"Indianapolis IN\" <sip:13178221383@daoldcatset.mycompany.com>;tag=01a5-9b285f10232f78f06690630d122f1689\r\nFrom: \"testing8\" <sip:13175555555@52.205.203.245>;tag=XsyUi6E\r\nCa"..., len=<optimized out>, rcv_info=rcv_info@entry=0x7ffe86191ba0,
    existing_context=existing_context@entry=0x0, flags=flags@entry=0)
    at receive.c:258
#8  0x0000000000609616 in udp_read_req (si=<optimized out>,
    bytes_read=<optimized out>) at net/proto_udp/proto_udp.c:182
#9  0x00000000005ec489 in handle_io (idx=0, event_type=2, fm=<optimized out>)
    at net/net_udp.c:261
#10 io_wait_loop_epoll (h=<optimized out>, t=<optimized out>,
    repeat=<optimized out>) at net/../io_wait_loop.h:284
#11 udp_start_processes (chd_rank=chd_rank@entry=0x924164 <chd_rank>,
    startup_done=startup_done@entry=0x7f70fc246178) at net/net_udp.c:389
#12 0x000000000041c107 in main_loop () at main.c:782
#13 main (argc=<optimized out>, argv=<optimized out>) at main.c:1439
(gdb) quit

[bogdan-iancu]

@clifjones , thanks for the report - all the backraces you posted indicates some kind of memory corruption.
So, first update from GIT to latest 2.4 code. And have it compiled with QM_MALLOC + DGG_MALLOC - this will enable some self-debugging code in the memory manager. See https://opensips.org/Documentation/TroubleShooting-OutOfMem, the "How to handle it" chapter.
During a crash, more information (about its cause) will be printed.

[clifjones]

@bogdan-iancu I followed your instructions:
Pulled v2.4 HEAD:

commit ac4b46965d8559c2dfb61b1adb6d96986f893d89 (HEAD -> 2.4, origin/2.4)

I enabled QM_MALLOC, DBG_MALLOC, and set memdump=1.

I am attaching a collection of all the memdump reports that were produced. Some TLS-related failures did not produce the report. I did not attach core stack traces this time but can do so if it helps.
opensips_memdebug.log

[bogdan-iancu]

@clifjones , the mem dump is not needed. Have you seen any log line with "BUG", just before the crash?

[clifjones]

@bogdan-iancu I just searched the logs and unfortunately BUG was never logged for any of the 21 test runs.

[bogdan-iancu]

@clifjones , could you post the backtrace ? The logs show an abort there (signal 6) which may be triggered by the memory debugger.

[clifjones]

Here you go @bogdan-iancu . I see a common theme around the crypto libraries.

[New LWP 295]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /opensips/for'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f989f6f7760 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-11.amzn2.0.2.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  0x00007f989f6f7760 in raise () from /lib64/libc.so.6
#1  0x00007f989f6f8d41 in abort () from /lib64/libc.so.6
#2  0x00000000004bd594 in qm_malloc (qm=0x7f98936c2000, size=24,
    file=0x7f98936c2000 "\376<Y", func=0x7f9893ae4610 "\002", line=32664)
    at mem/q_malloc.c:455
#3  0x00007f988b67d55b in _shm_free_unsafe (ptr=0x7f9893ae4610,
    file=0x7f988b68fe83 "c lock\n",
    function=0x7f988b6944d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:350
#4  0x00007f988b67d5a3 in _shm_free (ptr=0x7f9893ae4610,
    file=0x7f988b68fe83 "c lock\n",
    function=0x7f988b6944d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:373
#5  0x00007f988b681b59 in os_free (ptr=0x7f9893ae4610) at tls.h:119
#6  0x00007f98924076bd in CRYPTO_free () from /lib64/libcrypto.so.10
#7  0x00007f989240a32d in OBJ_NAME_add () from /lib64/libcrypto.so.10
#8  0x00007f98924c91c5 in EVP_add_cipher () from /lib64/libcrypto.so.10
#9  0x00007f98928497fb in SSL_library_init () from /lib64/libssl.so.10
#10 0x00007f988cfac843 in Curl_ossl_init () from /lib64/libcurl.so.4
#11 0x00007f988cf7356d in global_init () from /lib64/libcurl.so.4
#12 0x00007f988cf73818 in curl_easy_init () from /lib64/libcurl.so.4
#13 0x00007f988d1d2e7e in init_sync_handle () at rest_methods.c:76
#14 0x00007f988d1da3ba in child_init (rank=2) at rest_client.c:241
#15 0x000000000049fe2c in start_module_procs () at sr_module.c:805
#16 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#17 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#18 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#19 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#20 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#21 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#22 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#23 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#24 0x000000000049ff71 in dp_time () at dprint.h:138
#25 0x000000000052a300 in udp_start_processes (
    chd_rank=0x80d13c <sa_sys_segv+124>, startup_done=0x7f9893ac1f20)
    at net/net_udp.c:359
#26 0x000000000049a932 in main (argc=0, argv=0x7f9893ac1f20) at main.c:894
(gdb) quit

[New LWP 297]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /opensips/for'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f1271164760 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-11.amzn2.0.2.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  0x00007f1271164760 in raise () from /lib64/libc.so.6
#1  0x00007f1271165d41 in abort () from /lib64/libc.so.6
#2  0x00000000004bd594 in qm_malloc (qm=0x7f126512f000, size=24,
    file=0x7f126512f000 "\376<Y", func=0x7f1265533688 "\002", line=32530)
    at mem/q_malloc.c:455
#3  0x00007f125d0ea55b in _shm_free_unsafe (ptr=0x7f1265533688,
    file=0x7f125d0fce83 "c lock\n",
    function=0x7f125d1014d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:350
#4  0x00007f125d0ea5a3 in _shm_free (ptr=0x7f1265533688,
    file=0x7f125d0fce83 "c lock\n",
    function=0x7f125d1014d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:373
#5  0x00007f125d0eeb59 in os_free (ptr=0x7f1265533688) at tls.h:119
#6  0x00007f1263e746bd in CRYPTO_free () from /lib64/libcrypto.so.10
#7  0x00007f1263e7732d in OBJ_NAME_add () from /lib64/libcrypto.so.10
#8  0x00007f12642b67ba in SSL_library_init () from /lib64/libssl.so.10
#9  0x00007f125ea19843 in Curl_ossl_init () from /lib64/libcurl.so.4
#10 0x00007f125e9e056d in global_init () from /lib64/libcurl.so.4
#11 0x00007f125e9e0818 in curl_easy_init () from /lib64/libcurl.so.4
#12 0x00007f125ec3fe7e in init_sync_handle () at rest_methods.c:76
#13 0x00007f125ec473ba in child_init (rank=3) at rest_client.c:241
#14 0x000000000049fe2c in start_module_procs () at sr_module.c:805
#15 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#16 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#17 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#18 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#19 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#20 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#21 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#22 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#23 0x000000000049ff71 in dp_time () at dprint.h:138
#24 0x000000000052a300 in udp_start_processes (
    chd_rank=0x80d13c <sa_sys_segv+124>, startup_done=0x7f126552ef40)
    at net/net_udp.c:359
#25 0x000000000049a932 in main (argc=0, argv=0x7f126552ef40) at main.c:894
(gdb) quit

[New LWP 305]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /opensips/for'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007faa7724d760 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-11.amzn2.0.2.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  0x00007faa7724d760 in raise () from /lib64/libc.so.6
#1  0x00007faa7724ed41 in abort () from /lib64/libc.so.6
#2  0x00000000004bd594 in qm_malloc (qm=0x7faa6b218000, size=24,
    file=0x7faa6b218000 "\376<Y", func=0x7faa6b61ed68 "\002", line=32682)
    at mem/q_malloc.c:455
#3  0x00007faa631d355b in _shm_free_unsafe (ptr=0x7faa6b61ed68,
    file=0x7faa631e5e83 "c lock\n",
    function=0x7faa631ea4d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:350
#4  0x00007faa631d35a3 in _shm_free (ptr=0x7faa6b61ed68,
    file=0x7faa631e5e83 "c lock\n",
    function=0x7faa631ea4d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:373
#5  0x00007faa631d7b59 in os_free (ptr=0x7faa6b61ed68) at tls.h:119
#6  0x00007faa69f5d6bd in CRYPTO_free () from /lib64/libcrypto.so.10
#7  0x00007faa69f6032d in OBJ_NAME_add () from /lib64/libcrypto.so.10
#8  0x00007faa6a01f1c5 in EVP_add_cipher () from /lib64/libcrypto.so.10
#9  0x00007faa6a024139 in OpenSSL_add_all_ciphers ()
   from /lib64/libcrypto.so.10
#10 0x00007faa6a02395e in OPENSSL_add_all_algorithms_noconf ()
   from /lib64/libcrypto.so.10
#11 0x00007faa64b02855 in Curl_ossl_init () from /lib64/libcurl.so.4
#12 0x00007faa64ac956d in global_init () from /lib64/libcurl.so.4
#13 0x00007faa64ac9818 in curl_easy_init () from /lib64/libcurl.so.4
#14 0x00007faa64d28e7e in init_sync_handle () at rest_methods.c:76
#15 0x00007faa64d303ba in child_init (rank=10) at rest_client.c:241
#16 0x000000000049fe2c in start_module_procs () at sr_module.c:805
#17 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#18 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#19 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#20 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#21 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#22 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#23 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#24 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#25 0x000000000049ff71 in dp_time () at dprint.h:138
#26 0x0000000000541fe0 in tcp_start_processes (
    chd_rank=0x80d13c <sa_sys_segv+124>, startup_done=0x7faa6b617f40)
    at net/net_tcp.c:1881
#27 0x000000000049a9b1 in main (argc=0, argv=0x7ffdedb66e50) at main.c:898
(gdb) quit

[New LWP 310]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /opensips/for'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fc3497bf760 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-11.amzn2.0.2.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  0x00007fc3497bf760 in raise () from /lib64/libc.so.6
#1  0x00007fc3497c0d41 in abort () from /lib64/libc.so.6
#2  0x00000000004bd594 in qm_malloc (qm=0x7fc33d78a000, size=24,
    file=0x7fc33d78a000 "\376<Y", func=0x7fc33dbaa858 "\002", line=32707)
    at mem/q_malloc.c:455
#3  0x00007fc33574555b in _shm_free_unsafe (ptr=0x7fc33dbaa858,
    file=0x7fc335757e83 "c lock\n",
    function=0x7fc33575c4d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:350
#4  0x00007fc3357455a3 in _shm_free (ptr=0x7fc33dbaa858,
    file=0x7fc335757e83 "c lock\n",
    function=0x7fc33575c4d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:373
#5  0x00007fc335749b59 in os_free (ptr=0x7fc33dbaa858) at tls.h:119
#6  0x00007fc33c4cf6bd in CRYPTO_free () from /lib64/libcrypto.so.10
#7  0x00007fc33c4d232d in OBJ_NAME_add () from /lib64/libcrypto.so.10
#8  0x00007fc33c9117ad in SSL_library_init () from /lib64/libssl.so.10
#9  0x00007fc337074843 in Curl_ossl_init () from /lib64/libcurl.so.4
#10 0x00007fc33703b56d in global_init () from /lib64/libcurl.so.4
#11 0x00007fc33703b818 in curl_easy_init () from /lib64/libcurl.so.4
#12 0x00007fc33729ae7e in init_sync_handle () at rest_methods.c:76
#13 0x00007fc3372a23ba in child_init (rank=15) at rest_client.c:241
#14 0x000000000049fe2c in start_module_procs () at sr_module.c:805
#15 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#16 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#17 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#18 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#19 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#20 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#21 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#22 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#23 0x000000000049ff71 in dp_time () at dprint.h:138
#24 0x0000000000541fe0 in tcp_start_processes (
    chd_rank=0x80d13c <sa_sys_segv+124>, startup_done=0x7fc33db89f60)
    at net/net_tcp.c:1881
#25 0x000000000049a9b1 in main (argc=0, argv=0x7fffd916f4b0) at main.c:898
(gdb) quit

[New LWP 311]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/local/sbin/opensips -P /var/run/inin/opensips-forward.pid -f /opensips/for'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f0881bfc760 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-11.amzn2.0.2.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-12.amzn2.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  0x00007f0881bfc760 in raise () from /lib64/libc.so.6
#1  0x00007f0881bfdd41 in abort () from /lib64/libc.so.6
#2  0x00000000004bd594 in qm_malloc (qm=0x7f0875bc7000, size=24,
    file=0x7f0875bc7000 "\376<Y", func=0x7f0875f0a8b0 "\002", line=32520)
    at mem/q_malloc.c:455
#3  0x00007f086db8255b in _shm_free_unsafe (ptr=0x7f0875f0a8b0,
    file=0x7f086db94e83 "c lock\n",
    function=0x7f086db994d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:350
#4  0x00007f086db825a3 in _shm_free (ptr=0x7f0875f0a8b0,
    file=0x7f086db94e83 "c lock\n",
    function=0x7f086db994d0 <__FUNCTION__.25015> "load_certificate_db",
    line=119) at ../../mem/shm_mem.h:373
#5  0x00007f086db86b59 in os_free (ptr=0x7f0875f0a8b0) at tls.h:119
#6  0x00007f087490c6bd in CRYPTO_free () from /lib64/libcrypto.so.10
#7  0x00007f087490f32d in OBJ_NAME_add () from /lib64/libcrypto.so.10
#8  0x00007f08749ce1c5 in EVP_add_cipher () from /lib64/libcrypto.so.10
#9  0x00007f08749d3139 in OpenSSL_add_all_ciphers ()
   from /lib64/libcrypto.so.10
#10 0x00007f08749d295e in OPENSSL_add_all_algorithms_noconf ()
   from /lib64/libcrypto.so.10
#11 0x00007f086f4b1855 in Curl_ossl_init () from /lib64/libcurl.so.4
#12 0x00007f086f47856d in global_init () from /lib64/libcurl.so.4
#13 0x00007f086f478818 in curl_easy_init () from /lib64/libcurl.so.4
#14 0x00007f086f6d7e7e in init_sync_handle () at rest_methods.c:76
#15 0x00007f086f6df3ba in child_init (rank=17) at rest_client.c:241
#16 0x000000000049fe2c in start_module_procs () at sr_module.c:805
#17 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#18 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#19 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#20 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#21 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#22 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#23 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#24 0x000000000049fd44 in start_module_procs () at sr_module.c:792
#25 0x000000000049ff71 in dp_time () at dprint.h:138
#26 0x000000000047cd59 in ip_addr2a (ip=0x80d13c <sa_sys_segv+124>)
    at parser/../ip_addr.h:398
#27 0x000000000049aa29 in main (argc=0, argv=0x7ffe9c2df530) at main.c:904
(gdb)

[bogdan-iancu]

@clifjones , the backtrace looks really weird (like the os_free() ends into a malloc operation). Are you 100% you recompiled the entire opensips (core and modules) after changing the MALLOC flags ?

[clifjones]

@bogdan-iancu 100% sure. The OpenSIPS proxies were tested in new Docker containers that cloned the OpenSIPS repo, then checked out by commit ID. This TLS issue is a high priority, so I will be happy to try anything else that you believe will help narrow this down.

[bogdan-iancu]

@clifjones , it is not about cloning the right revision or not, but I'm concerned about how the code was re-compiled after you changed the QM_MALLOC and DBG_MALLOC options - are you sure you recompiling the whole OpenSIPS code after that changed ? it suspect that the tls_mgm module was not, so the backtrace is completely broken. IF you check the sequence in the BT's, from frame #3 to frame #2, it makes no sense.....