openSIPS 2.2 crashes

[fedorna]

Hello, I'm getting core dumps once a week. I'm using 2.2.3 version.

/home/gc/local/opensips/sbin/opensips -V
version: opensips 2.2.3 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: 815a81a
main.c compiled on 23:13:22 Mar 22 2017 with gcc 5.4.0

I'm pasting 3 backtraces, I've only logs for 2nd and 3th core because I realized I hadn't set memdump in cfg file after the first crash.

1.- Process 11954, no log files.
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/gc/local/opensips/sbin/opensips...done.
[New LWP 11954]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/gc/local/opensips//sbin/opensips -P /home/gc/local/opensips/var/run/opens'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000004b3a05 in qm_debug_frag (qm=0x7fbe0431a000, f=0x7fbc04e4c588) at mem/q_malloc.c:138
138 if (f->check!=ST_CHECK_PATTERN){
(gdb) bt full
#0 0x00000000004b3a05 in qm_debug_frag (qm=0x7fbe0431a000, f=0x7fbc04e4c588) at mem/q_malloc.c:138
FUNCTION = "qm_debug_frag"
#1 0x00000000004b5856 in qm_free (qm=0x7fbe0431a000, p=0x7fbe04e4c640, file=0x7fbe03ceeadc "t_hooks.c", func=0x7fbe03ceef50 <FUNCTION.9622> "empty_tmcb_list", line=54)
at mem/q_malloc.c:521
f = 0x7fbe04e4c610
prev = 0x7fbc04e4c588
next = 0x7fbe04e4c698
size = 320
FUNCTION = "qm_free"
#2 0x00007fbe03cd2814 in empty_tmcb_list (head=0x7fbe064fafa8) at t_hooks.c:54
cbp = 0x7fbe04e4c5b8
cbp_tmp = 0x7fbe04e4c640
FUNCTION = "empty_tmcb_list"
#3 0x00007fbe03cb448f in free_cell (dead_cell=0x7fbe064faf38) at h_table.c:127
b = 0x7fbe044eee14 ""
i = 32767
rpl = 0x7fff1d05e080
tt = 0x7fbe03cb3f67 <_unlock+24>
foo = 0x70f600000000
p = 0x7fbe03cb43e7 <unlock_hash+57>
FUNCTION = "free_cell"
#4 0x00007fbe03cdb613 in delete_cell (p_cell=0x7fbe064faf38, unlock=1) at timer.c:239
FUNCTION = "delete_cell"
#5 0x00007fbe03cdbf47 in wait_handler (wait_tl=0x7fbe064fafb8) at timer.c:445
p_cell = 0x7fbe064faf38
FUNCTION = "wait_handler"
#6 0x00007fbe03cde012 in timer_routine (ticks=470376, set=0x0) at timer.c:1069
tl = 0x7fbe064fafb8
tmp_tl = 0x7fbe07185a80
id = 2
FUNCTION = "timer_routine"
#7 0x000000000046db4a in handle_timer_job () at timer.c:699
t = 0x7fbe04654cb8
l = 8
FUNCTION = "handle_timer_job"
#8 0x0000000000516e4d in handle_io (fm=0x7fbe123d9a00, idx=446, event_type=1) at net/net_tcp_proc.c:117
ret = 0
n = 5397410
con = 0x7fbe056245f8
s = 513
rw = 0
resp = 0
---Type to continue, or q to quit---
response = {140454116474656, 68719476737}
FUNCTION = "handle_io"
#9 0x000000000051630a in io_wait_loop_epoll (h=0x810e20 <_worker_io>, t=2, repeat=0) at net/../io_wait_loop.h:221
ret = 1
n = 1
r = 446
e = 0x7fbe123d9a00
FUNCTION = "io_wait_loop_epoll"
#10 0x0000000000517e84 in tcp_worker_proc (unix_sock=69) at net/net_tcp_proc.c:312
FUNCTION = "tcp_worker_proc"
#11 0x000000000052515c in tcp_start_processes (chd_rank=0x7dd168 <chd_rank>, startup_done=0x7fbe04747808) at net/net_tcp.c:1770
r = 5
n = 10
reader_fd = {67, 69}
pid = 0
si = 0x0
load_p = 0x7fbe0480a248
FUNCTION = "tcp_start_processes"
#12 0x0000000000495f6c in main_loop () at main.c:677
chd_rank = 38
startup_done = 0x7fbe04747808
FUNCTION = "main_loop"
#13 0x0000000000498a2e in main (argc=9, argv=0x7fff1d05e548) at main.c:1261
cfg_log_stderr = 1
cfg_stream = 0x14e6010
c = -1
r = 0
tmp = 0x7fff1d05e83e ""
tmp_len = 0
port = 0
proto = 0
protos_no = 3
options = 0x5645c8 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
ret = -1
seed = 528607206
rfd = 4
FUNCTION = "main"

---

2.- Process 1088, log files in 2 parts: https://pastebin.com/iULaChH8 and https://pastebin.com/xJwHW4me

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/gc/local/opensips/sbin/opensips...done.
[New LWP 1088]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/gc/local/opensips/sbin/opensips -P /home/gc/local/opensips/var/run/opensi'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000004b44f7 in qm_detach_free (qm=0x7fe5000f9000, frag=0x7fe500e22a00) at mem/q_malloc.c:295
295 prev->u.nxt_free=next;
(gdb) bt full
#0 0x00000000004b44f7 in qm_detach_free (qm=0x7fe5000f9000, frag=0x7fe500e22a00) at mem/q_malloc.c:295
prev = 0x6eb3c36963706563
next = 0x7fe700000000
#1 0x00000000004b5765 in qm_free (qm=0x7fe5000f9000, p=0x7fe500e229a0, file=0x7fe4ffac5494 "h_table.c", func=0x7fe4ffac54f8 <FUNCTION.9747> "free_cell", line=135)
at mem/q_malloc.c:508
f = 0x7fe500e22970
prev = 0x0
next = 0x7fe500e22a00
size = 48
FUNCTION = "qm_free"
#2 0x00007fe4ffa9352e in free_cell (dead_cell=0x7fe501d741e0) at h_table.c:135
b = 0x7fe500398394 ""
i = 32767
rpl = 0x7fffc53e1370
tt = 0x7fe4ffa92f67 <_unlock+24>
foo = 0xc1e600000000
p = 0x7fe4ffa933e7 <unlock_hash+57>
FUNCTION = "free_cell"
#3 0x00007fe4ffaba613 in delete_cell (p_cell=0x7fe501d741e0, unlock=1) at timer.c:239
FUNCTION = "delete_cell"
#4 0x00007fe4ffabaf47 in wait_handler (wait_tl=0x7fe501d74260) at timer.c:445
p_cell = 0x7fe501d741e0
FUNCTION = "wait_handler"
#5 0x00007fe4ffabd012 in timer_routine (ticks=1575, set=0x0) at timer.c:1069
tl = 0x7fe501d74260
tmp_tl = 0x7fe50589d978
id = 2
FUNCTION = "timer_routine"
#6 0x000000000046db4a in handle_timer_job () at timer.c:699
t = 0x7fe500433cb8
l = 8
FUNCTION = "handle_timer_job"
#7 0x00000000005122e5 in handle_io (fm=0x7fe50e1b8a00, idx=1, event_type=1) at net/net_udp.c:264
n = 0
read = -985787108
FUNCTION = "handle_io"
#8 0x0000000000510b17 in io_wait_loop_epoll (h=0x810e20 <_worker_io>, t=1, repeat=0) at net/../io_wait_loop.h:221
ret = 1
n = 1
r = 1
e = 0x7fe50e1b8a00
FUNCTION = "io_wait_loop_epoll"
---Type to continue, or q to quit---
#9 0x00000000005125de in udp_rcv_loop (si=0x7fe50e12ead8) at net/net_udp.c:308
FUNCTION = "udp_rcv_loop"
#10 0x0000000000512b99 in udp_start_processes (chd_rank=0x7dd168 <chd_rank>, startup_done=0x7fe500730d50) at net/net_udp.c:372
si = 0x7fe50e12ead8
load_p = 0x7fe500742088
pid = 0
i = 3
p = 1
FUNCTION = "udp_start_processes"
#11 0x0000000000495eed in main_loop () at main.c:671
chd_rank = 4
startup_done = 0x7fe500730d50
FUNCTION = "main_loop"
#12 0x0000000000498a2e in main (argc=9, argv=0x7fffc53e17c8) at main.c:1261
cfg_log_stderr = 1
cfg_stream = 0x22cf010
c = -1
r = 0
tmp = 0x7fffc53e2de9 ""
tmp_len = 0
port = 0
proto = 0
protos_no = 3
options = 0x5645c8 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
ret = -1
seed = 1614355773
rfd = 4
FUNCTION = "main"

---

3.- Process 19745, log files in 3 parts: https://pastebin.com/wDjJdAkt , https://pastebin.com/A0qCW3XL and https://pastebin.com/UE71QzMd

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/gc/local/opensips/sbin/opensips...done.
[New LWP 19745]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by /home/gc/local/opensips/sbin/opensips -P /home/gc/local/opensips/var/run/opensi'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f66d759a428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt full #0 0x00007f66d759a428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 resultvar = 0 pid = 19745 selftid = 19745 #1 0x00007f66d759c02a in __GI_abort () at abort.c:89 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x0, sa_sigaction = 0x0}, sa_mask = {__val = {0, 0, 0, 0, 0, 0, 0, 0, 140079677228800, 0, 140079680097104, 4294967295, 20626576, 0, 0, 0}}, sa_flags = -905803520, sa_restorer = 0x0} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00000000004b3e25 in qm_debug_frag (qm=0x7f66c8d64000, f=0x7f66ca610970) at mem/q_malloc.c:163 __FUNCTION__ = "qm_debug_frag" #3 0x00000000004b4b57 in qm_malloc (qm=0x7f66c8d64000, size=64, file=0x7f66c873ab3b "../../resolve.h", func=0x7f66c873b720 <__FUNCTION__.8904> "shm_clone_proxy", line=373) at mem/q_malloc.c:422 f = 0x7f66ca610970 hash = 16 list_cntr = 1 __FUNCTION__ = "qm_malloc" #4 0x00007f66c8728c67 in _shm_malloc (size=64, file=0x7f66c873ab3b "../../resolve.h", function=0x7f66c873b720 <__FUNCTION__.8904> "shm_clone_proxy", line=373) at ../../evi/../mem/shm_mem.h:241 p = 0x0 #5 0x00007f66c8728d3c in shm_clone_proxy (sp=0x7f66d6f43208, move_dn=1) at ../../resolve.h:373 dp = 0x7f66d6f3d448 __FUNCTION__ = "shm_clone_proxy" #6 0x00007f66c872a8df in add_uac (t=0x7f66ca8a9290, request=0x7f66d6f3d448, uri=0x7ffcba077130, next_hop=0x7ffcba077120, bflags=1, path=0x7f66d6f3d6c8, proxy=0x7f66d6f43208) at t_fwd.c:417 branch = 0 do_free_proxy = 1 ret = 0 __FUNCTION__ = "add_uac" #7 0x00007f66c872b969 in t_forward_nonack (t=0x7f66ca8a9290, p_msg=0x7f66d6f3d448, proxy=0x0) at t_fwd.c:696 reply_reason_487 = {s = 0x7f66c873b369 "Request Terminated", len = 18} backup_uri = {s = 0x7f66d6f43058 "sip:103@190.105.10.117:7361;transport=TCP", '\300' <repeats 15 times>, "\355\357ͫ\355\357ͫ ", len = 41} backup_dst = {s = 0x7f66d6f43178 "sip:103@190.105.10.117:7361;transport=TCP\001", len = 41} branch_ret = 32614 lowest_ret = -5 current_uri = {s = 0x7f66d6f43058 "sip:103@190.105.10.117:7361;transport=TCP", '\300' <repeats 15 times>, "\355\357ͫ\355\357ͫ ", len = 41} added_branches = 0 i = 0 q = 32764 t_invite = 0x7f66d6f3d448 success_branch = -1173917424 dst_uri = {s = 0x7f66c86fe3e7 <unlock_hash+57> "\220\311\303UH\211\345H\213\005\353\177%", len = 19729} ---Type <return> to continue, or q <return> to quit--- bk_sock = 0x7f66d6d9a708 br_flags = 3362775079 bk_bflags = 1 idx = 32764 path = {s = 0x7ffcba077190 "\360q\a\272\374\177", len = -932169314} bk_path = {s = 0x0, len = 0} __FUNCTION__ = "t_forward_nonack" #8 0x00007f66c86f659d in t_relay_to (p_msg=0x7f66d6f3d448, proxy=0x0, flags=0) at t_funcs.c:234 ret = 0 new_tran = 1 reply_ret = 0 t = 0x7f66ca8a9290 __FUNCTION__ = "t_relay_to" #9 0x00007f66c8714735 in w_t_relay (p_msg=0x7f66d6f3d448, proxy=0x0, flags=0x0) at tm.c:1215 p = 0x0 t = 0x0 ret = 32614 __FUNCTION__ = "w_t_relay" #10 0x0000000000421459 in do_action (a=0x7f66d6ded4c8, msg=0x7f66d6f3d448) at action.c:1845 increment = 32614 decrement = 0 val_number = 0 j = 0 cdb_res = 0xffffffff0000000a val_s = {s = 0x7f66c8725380 <_unlock+24> "\220\311\303UH\211\345H\211}\350\211u\344H\213E\350\213U\344Hc\322H\301\342\tH\001\320H\005\300\003", len = 32} cdb_reply = 0x6e aux = {s = 0x7f66c8725380 <_unlock+24> "\220\311\303UH\211\345H\211}\350\211u\344H\213E\350\213U\344Hc\322H\301\342\tH\001\320H\005\300\003", len = 32} i = 1 key_number = -1173916752 it = 0x7ffcba0774a1 avp_val = {n = -932031616, s = {s = 0x7f66c8725380 <_unlock+24> "\220\311\303UH\211\345H\211}\350\211u\344H\213E\350\213U\344Hc\322H\301\342\tH\001\320H\005\300\003", len = 32}} avp_name = {n = -1173914848, s = {s = 0x7ffcba077b20 "\277", len = -1173915040}} avp_type = 32614 ret = -5 v = -1 sec = 32614 usec = -1173916752 to = 0x3000000020 p = 0xffffffff00000000 tmp = 0x7f66d76f255e "%hu%n:%hu%n" new_uri = 0xffffffff0000000a <error: Cannot access memory at address 0xffffffff0000000a> ---Type <return> to continue, or q <return> to quit--- end = 0x7ffcba077ae0 "\230\377\377\377" crt = 0x0 len = -680581798 i = 32614 user = 0 expires = 0 vals = {{s = 0x7f6600000000 <error: Cannot access memory at address 0x7f6600000000>, len = 4955313}, {s = 0x71 <error: Cannot access memory at address 0x71>, len = 4955313}, {s = 0x1fa1 <error: Cannot access memory at address 0x1fa1>, len = -681674319}, {s = 0x2 <error: Cannot access memory at address 0x2>, len = 4}, { s = 0x7ffcba077440 "", len = -1173916448}} result = {s = 0x7f66c8725380 <_unlock+24> "\220\311\303UH\211\345H\211}\350\211u\344H\213E\350\213U\344Hc\322H\301\342\tH\001\320H\005\300\003", len = 32} uri = {user = {s = 0x2900554975 <error: Cannot access memory at address 0x2900554975>, len = -688639624}, passwd = { s = 0xffff800345f88bc1 <error: Cannot access memory at address 0xffff800345f88bc1>, len = -1173916609}, host = {s = 0x0, len = 2}, port = {s = 0x0, len = 0}, params = {s = 0x6e0000005b <error: Cannot access memory at address 0x6e0000005b>, len = 0}, headers = {s = 0x0, len = 119}, port_no = 0, proto = 0, type = ERROR_URI_T, transport = {s = 0x4b9cb1 <parse_uri+190> "\203\275T\377\377\377\004\017\216\331", <incomplete sequence \360>, len = 0}, ttl = { s = 0x7ffc00000000 <error: Cannot access memory at address 0x7ffc00000000>, len = 0}, user_param = {s = 0x7f66d7928b20 <main_arena> "", len = 4}, maddr = { s = 0x4 <error: Cannot access memory at address 0x4>, len = 4}, method = {s = 0x0, len = 0}, lr = { s = 0x7f66d763bf38 <re_string_reconstruct+1976> "\351\017\371\377\377\017\037", len = -1173946317}, r2 = {s = 0x0, len = 19961536}, gr = { s = 0x10 <error: Cannot access memory at address 0x10>, len = 0}, transport_val = {s = 0x0, len = 3}, ttl_val = { s = 0x7f66d75e898c <__GI___libc_free+76> "H\203\304([]A\\A]\303f\017\037\204", len = -688639620}, user_param_val = { s = 0x3 <error: Cannot access memory at address 0x3>, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 19961536}, lr_val = { s = 0x10 <error: Cannot access memory at address 0x10>, len = 0}, r2_val = {s = 0x7f66d764a329 <re_search_internal+649> "\213D$\b\353\006\220\270\001", len = 0}, gr_val = {s = 0x0, len = -1173916080}, u_name = {{s = 0x3 <error: Cannot access memory at address 0x3>, len = 13}, {s = 0x7f66d6dea318 "", len = 19603200}, { s = 0x3 <error: Cannot access memory at address 0x3>, len = 0}, {s = 0x7ffcba077620 "", len = 19602928}, {s = 0x7c3f60 <_tr_buffer> "TCP", len = -1173916148}, { s = 0x7ffcba077610 "", len = 3}, {s = 0x0, len = -1173916152}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, { s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = 1}}, u_val = {{s = 0x7ffcba077608 "", len = 1}, { s = 0x3 <error: Cannot access memory at address 0x3>, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = 8142688}, {s = 0x13a9c80 "\270\213\222\327f\177", len = 0}, {s = 0x0, len = 0}, {s = 0x300000000 <error: Cannot access memory at address 0x300000000>, len = 3}}, u_params_no = 3} next_hop = {user = {s = 0x300000003 <error: Cannot access memory at address 0x300000003>, len = 3}, passwd = {s = 0x0, len = 19603368}, host = { s = 0x1000001 <error: Cannot access memory at address 0x1000001>, len = 1}, port = {s = 0x12b1f00 "\360\037+\001", len = 0}, params = {s = 0x0, len = 0}, headers = { s = 0x0, len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 19603200}, maddr = {s = 0x7f66d6dea318 "", len = 8142688}, method = {s = 0x0, len = 0}, lr = {s = 0x7f66d764c9c3 <__regexec+179> "H\203\304 \211\302\203=p\r.", len = 0}, r2 = { s = 0x0, len = 0}, gr = {s = 0x7f66d764c94f <__regexec+63> "E1҉\302I\213,$\276\001", len = 0}, transport_val = { s = 0x4d11 <error: Cannot access memory at address 0x4d11>, len = -1173915648}, ttl_val = { s = 0x4114f0 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300@\374S", len = -1173909008}, user_param_val = {s = 0x0, len = 0}, maddr_val = { s = 0x45ffea <comp_s2s+1105> "\205\300\017\225\300\017\266\300\211E\334H\213E\300H\213\020H\213E\300\213@\bH\230H\001\302\017\266Eڈ\002\351\304\002", len = -1173915648}, method_val = {s = 0x7f66d6dea318 "", len = -1173915536}, lr_val = {s = 0x16ba077890 <error: Cannot access memory at address 0x16ba077890>, len = -690050848}, r2_val = {s = 0xd600d448 <error: Cannot access memory at address 0xd600d448>, len = 0}, gr_val = {s = 0x7f66d6f3d448 ",\017i", len = 19729}, u_name = {{s = 0x4d11 <error: Cannot access memory at address 0x4d11>, len = -1173915392}, {s = 0x4609e0 <comp_scriptvar+1108> "\351\005\003", len = -1173915552}, { s = 0x46f216 <run_transformations+186> "\211E\344\203", <incomplete sequence \344>, len = 0}, {s = 0x7ffcba077920 "H\324\363\326f\177", len = -690058584}, { s = 0x7f66d6f3d448 ",\017i", len = -690050248}, {s = 0x7f66d6deab20 "\023", len = 0}, {s = 0x7f66d6f3d448 ",\017i", len = -1173915472}, {s = 0x0, len = 8142688}, { s = 0x3 <error: Cannot access memory at address 0x3>, len = 0}, {s = 0x7f6600000000 <error: Cannot access memory at address 0x7f6600000000>, len = 8142688}}, ---Type <return> to continue, or q <return> to quit--- u_val = {{s = 0x3 <error: Cannot access memory at address 0x3>, len = 0}, {s = 0x7ffcba077920 "H\324\363\326f\177", len = -1173914608}, { s = 0x4624eb <eval_elem+6107> "\270\001", len = -1173914870}, {s = 0x25d0137bca028900 <error: Cannot access memory at address 0x25d0137bca028900>, len = -688663480}, {s = 0x4d11 <error: Cannot access memory at address 0x4d11>, len = 4265200}, {s = 0x7ffcba0791f0 "\t", len = 0}, {s = 0x0, len = -1173914528}, { s = 0x4627af <eval_elem+6815> "\211\205\330\374\377\377\353yH\213\005J\272\065", len = 19257915}, {s = 0x0, len = -688663480}, {s = 0x7f66d6deab18 "\002", len = 0}}, u_params_no = 0} u = 0xb00000001 port = 0 cmatch = -678259680 aitem = 0x0 adefault = 0x1 spec = 0x7f66d76f255e model = 0x7ffcba077a20 val = {rs = {s = 0x7ffcba077b20 "\277", len = -1173915040}, ri = -681769856, flags = 32614} pve = 0x7f66d6f3d448 name_s = {s = 0x7f66d75d0480 <__funlockfile> "H\213\227\210", len = -1173915424} start = {tv_sec = 140076063391744, tv_usec = 0} end_time = 32614 aux_counter = 110 __FUNCTION__ = "do_action" #11 0x00000000004191ff in run_action_list (a=0x7f66d6ded4c8, msg=0x7f66d6f3d448) at action.c:172 ret = -1 t = 0x7f66d6ded4c8 #12 0x0000000000461451 in eval_elem (e=0x7f66d6ded5f0, msg=0x7f66d6f3d448, val=0x0) at route.c:1618 uri = {user = {s = 0x1ba077b06 <error: Cannot access memory at address 0x1ba077b06>, len = -1173914868}, passwd = {s = 0x7ffcba077b08 "\220\274:\001", len = -1173914868}, host = {s = 0x7f66d6f3d448 ",\017i", len = -688639768}, port = {s = 0x7ffcba077aa0 " \213\222\327f\177", len = 0}, params = { s = 0x7f66d6f431d8 "@", len = -678261984}, headers = {s = 0x7f66d7928b20 <main_arena> "", len = 8}, port_no = 48284, proto = 314, type = ERROR_URI_T, transport = { s = 0x13abc80 "", len = -678261984}, ttl = {s = 0x17370 <error: Cannot access memory at address 0x17370>, len = 209}, user_param = {s = 0x13adc90 "", len = 8001}, maddr = {s = 0x7f66d75e79b1 <_int_realloc+705> "\351\334\375\377\377f.\017\037\204", len = 0}, method = {s = 0x13aa4e0 "x\213\222\327f\177", len = -104}, lr = { s = 0xb <error: Cannot access memory at address 0xb>, len = 5589365}, r2 = {s = 0x0, len = 21}, gr = {s = 0x13abc90 "", len = -1173914632}, transport_val = { s = 0x7f6600000001 <error: Cannot access memory at address 0x7f6600000001>, len = 191}, ttl_val = {s = 0x13abc80 "", len = 8208}, user_param_val = { s = 0x7f6600000001 <error: Cannot access memory at address 0x7f6600000001>, len = 19745}, maddr_val = {s = 0x13aa4e0 "x\213\222\327f\177", len = 0}, method_val = { s = 0xffffffffffffff98 <error: Cannot access memory at address 0xffffffffffffff98>, len = 11}, lr_val = {s = 0x13aa4e0 "x\213\222\327f\177", len = 21}, r2_val = { s = 0xffffffffffffff98 <error: Cannot access memory at address 0xffffffffffffff98>, len = 11}, gr_val = {s = 0x554975 "%.*s", len = 0}, u_name = {{ s = 0x7f66d75e898c <__GI___libc_free+76> "H\203\304([]A\\A]\303f\017\037\204", len = 0}, {s = 0x13aa4e0 "x\213\222\327f\177", len = 0}, { s = 0xffffffffffffff98 <error: Cannot access memory at address 0xffffffffffffff98>, len = 11}, {s = 0x13aa4e0 "x\213\222\327f\177", len = 21}, { s = 0xffffffffffffff98 <error: Cannot access memory at address 0xffffffffffffff98>, len = 11}, {s = 0x7f66d7664707 <__syslog+807> "H\201\304\001",
len = 1491229411}, {s = 0x25d0137bca028900 <error: Cannot access memory at address 0x25d0137bca028900>, len = 19729}, {
s = 0x4114f0 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300@\374S", len = -1173914528}, {s = 0x462988 <eval_expr+208> "\211E\354\351\064\002",
len = -1173914528}, {s = 0x0, len = -688663480}}, u_val = {{s = 0x7f66d6de8b38 "\002", len = 20626576}, {
s = 0x25d0137bca028900 <error: Cannot access memory at address 0x25d0137bca028900>, len = 19729}, {
s = 0x4114f0 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300@\374S", len = -1173914448}, {s = 0x462988 <eval_expr+208> "\211E\354\351\064\002",
len = 3}, {s = 0x0, len = -688663480}, {s = 0x7f66d6deab18 "\002", len = 19252640}, {s = 0xd6f3d448 <error: Cannot access memory at address 0xd6f3d448>,
---Type to continue, or q to quit---
len = 19729}, {s = 0x4114f0 <_start> "1\355I\211\321^H\211\342H\203\344\360PTI\307\300@\374S", len = -1173914368}, {
s = 0x462a14 <eval_expr+348> "\211E\354\351\250\001", len = 20618368}, {s = 0x0, len = -688663480}}, u_params_no = 43952}
ret = -5
ival = 32764
lval = {rs = {s = 0x125c5a0 "ART", len = -1173915072}, ri = -678261976, flags = 32614}
rval = {rs = {s = 0x7ffcba077ae0 "\230\377\377\377", len = -1173915104}, ri = -1173915057, flags = 32764}
p = 0x7ffcba077bdf "\377\v"
i = -678277280
n = 32614
FUNCTION = "eval_elem"
#13 0x0000000000462988 in eval_expr (e=0x7f66d6ded5f0, msg=0x7f66d6f3d448, val=0x0) at route.c:1968
rec_lev = 3
ret = 0
FUNCTION = "eval_expr"
#14 0x0000000000462a82 in eval_expr (e=0x7f66d6ded688, msg=0x7f66d6f3d448, val=0x0) at route.c:1984
rec_lev = 3
ret = 0
FUNCTION = "eval_expr"
#15 0x0000000000462abc in eval_expr (e=0x7f66d6ded720, msg=0x7f66d6f3d448, val=0x0) at route.c:1989
rec_lev = 3
ret = 0
FUNCTION = "eval_expr"

Thanks!

[fedorna]

Hello, I got another core today. Process 6067 previous logs:

2017-04-05T11:44:42.178143-03:00 lirio /home/gc/local/opensips/sbin/opensips[6067]: INFO:core:probe_max_sock_buff: using snd buffer of 416 kb
2017-04-05T11:44:42.178183-03:00 lirio /home/gc/local/opensips/sbin/opensips[6067]: INFO:core:init_sock_keepalive: TCP keepalive enabled on socket 7294

2017-04-05T11:44:42.282712-03:00 lirio /home/gc/local/opensips/sbin/opensips[6067]: INFO:core:probe_max_sock_buff: using snd buffer of 416 kb
2017-04-05T11:44:42.282743-03:00 lirio /home/gc/local/opensips/sbin/opensips[6067]: INFO:core:init_sock_keepalive: TCP keepalive enabled on socket 7301

2017-04-05T11:44:42.380295-03:00 lirio /home/gc/local/opensips/sbin/opensips[6067]: CRITICAL:core:send_fd: sendmsg failed on 69: Bad file descriptor
2017-04-05T11:44:42.380378-03:00 lirio /home/gc/local/opensips/sbin/opensips[6067]: ERROR:core:send2child: send_fd failed

2017-04-05T11:44:42.412414-03:00 lirio /home/gc/local/opensips/sbin/opensips[6067]: ERROR:core:handle_tcpconn_ev: no children available

and finally crashed:
2017-04-05T11:44:44.517143-03:00 lirio /home/gc/local/opensips/sbin/opensips[5973]: INFO:core:handle_sigs: child process 6067 exited by a signal 11
2017-04-05T11:44:44.517348-03:00 lirio /home/gc/local/opensips/sbin/opensips[5973]: INFO:core:handle_sigs: core was generated

backtrace:
root@lirio:/home/gc/local/opensips# gdb /home/gc/local/opensips/sbin/opensips core.6067.opensips.1491403482
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/gc/local/opensips/sbin/opensips...done.
[New LWP 6067]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/gc/local/opensips/sbin/opensips -P /home/gc/local/opensips/var/run/opensi'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000000000052aa5b in tcp_conn_clean (c=0x7f69dc583800) at net/proto_tcp/proto_tcp.c:230
230 for (r=0;rasync_chunks_no;r++) {
(gdb) bt full
#0 0x000000000052aa5b in tcp_conn_clean (c=0x7f69dc583800) at net/proto_tcp/proto_tcp.c:230
d = 0x0
r = 0
FUNCTION = "tcp_conn_clean"
#1 0x00000000005200be in _tcpconn_rm (c=0x7f69dc583800) at net/net_tcp.c:688
r = 1
FUNCTION = "_tcpconn_rm"
#2 0x0000000000521d9c in handle_tcpconn_ev (tcpconn=0x7f69dc583800, fd_i=2, event_type=2) at net/net_tcp.c:1098
fd = 7309
err = 5969
id = 117689
err_len = 0
FUNCTION = "handle_tcpconn_ev"
#3 0x00000000005234b7 in handle_io (fm=0x7f69e9e9ecc0, idx=2, event_type=2) at net/net_tcp.c:1396
ret = 16
FUNCTION = "handle_io"
#4 0x000000000051c9f4 in io_wait_loop_epoll (h=0x810e20 <_worker_io>, t=5, repeat=0) at net/../io_wait_loop.h:225
ret = 2
n = 1
r = 2
e = 0x7f69e9e9ecc0
FUNCTION = "io_wait_loop_epoll"
#5 0x0000000000523f1e in tcp_main_server () at net/net_tcp.c:1543
last_sec = 172274
flags = 2
si = 0x0
n = 16
FUNCTION = "tcp_main_server"
#6 0x0000000000525364 in tcp_start_listener () at net/net_tcp.c:1809
pid = 0
FUNCTION = "tcp_start_listener"
#7 0x0000000000496057 in main_loop () at main.c:689
chd_rank = 49
startup_done = 0x7f69dc47c9a8
FUNCTION = "main_loop"
#8 0x0000000000498a2e in main (argc=9, argv=0x7ffe983786a8) at main.c:1261
cfg_log_stderr = 1
cfg_stream = 0x1953010
c = -1
r = 0
tmp = 0x7ffe98379de9 ""
tmp_len = 0
---Type to continue, or q to quit---
port = 0
proto = 0
protos_no = 3
options = 0x5645c8 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
ret = -1
seed = 2330664900
rfd = 4
FUNCTION = "main"

Thanks

[bogdan-iancu]

Hi @fedorna - I see all the crashes are related to memory operations. So, could you please recompile with QM_MALLOC and DBG_MALLOC flags (see in Makefile.conf) - this will give us more hints on the real cause of the crashes

[fedorna]

Hi @bogdan-iancu , I had actually compiled with that options:
root@lirio:/home/gc/local/opensips# /home/gc/local/opensips/sbin/opensips -V
version: opensips 2.2.3 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
git revision: 815a81a
main.c compiled on 23:13:22 Mar 22 2017 with gcc 5.4.0
root@lirio:/home/gc/local/opensips#

is that what you mean or should I do something else?

thank you

[bogdan-iancu]

@fedorna , it there a way to get access to the core files and corresponding binaries + modules ? maybe an ssh login ? The crashes look quite complex, making the core inspection complex too.
Thanks, Bogdan

[fedorna]

Hi @bogdan-iancu
Sure, I'll send you the credentials in a private email. Sorry for the delay, I took some days off.
Thanks

[bogdan-iancu]

Thank you @fedorna , got your email, I will start digging tomorrow.

[fedorna]

Thanks @bogdan-iancu. It didn't happen again since 5-Apr and since you are busy with new release and the summit, we can keep this issue standby until next crash if you want. Anyway, feel free to look the core files, access is available.

[razvancrainea]

Hi, Federico!

Are you using an older version of 2.2.3? There have been some fixes at the end of March related to TCP connections, which I know you are using heavily :). Can you update to the latest git version of 2.2 and see if this continues to happen?

Best regards,
Răzvan

[fedorna]

Hi Răzvan, I'm using commit 815a81a (update changelog for 2.2.3, 23-Feb). I'll take your suggestion and will update to last commit.
Thanks!

[bogdan-iancu]

@fedorna , if you still get crashes with the latest code, please open a new report. I will close this one as the involved version is old and there were many fixes in the last 2 months.

[fedorna]

so far so good since last update, thanks for your help @bogdan-iancu and @razvancrainea.