[CRASH] - Opensips B2B Call-Center

[korithi]

OpenSIPS version you are running

version: opensips 3.1.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on with gcc 8

Crash Core Dump

Let me know if you need me to share the core dump file.

Describe the traffic that generated the bug

I have an call center configuration of opensips and I am facing random crashes.
There are calls that are coming to the queue and dispatched to agents.
I see nothing in the debug so I have generated a core file.

To Reproduce

Steps to reproduce the behavior:

1. Start OpenSIPS
2. Start traffic
3. Check OpenSIPS crashed

Relevant System Logs

OS/environment information
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian

Additional context

[bogdan-iancu]

@korithi , could you extract the backtrace from the core file (when OpenSIPS crashes) ?

[korithi]

Hi @bogdan-iancu,

Let me tell you that I am facing this crash and restart on 2 different machines with 2 types of b2b, cc and refer
In the syslog I always get this message on both machines:
"kernel: [770847.477945] opensips[89166]: segfault at 38 ip 00007f627c7cf2b4 sp 00007ffd69e565b0 error 4 in b2b_entities.so[7f627c7b4000+21000]"

Here you have the bt:
Thanks

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f7f2cf972b4 in b2b_tm_cback (t=0x7f7f2e120988, htable=0x7f7f2e0718b0, ps=0x7ffd673b97a0) at dlg.c:2622
2622 dlg.c: No such file or directory.
(gdb) bt full
#0 0x00007f7f2cf972b4 in b2b_tm_cback (t=0x7f7f2e120988, htable=0x7f7f2e0718b0, ps=0x7ffd673b97a0) at dlg.c:2622
msg = 0xffffffffffffffff
msg_body = {s = 0x6eaa0121 <error: Cannot access memory at address 0x6eaa0121>, len = 131}
b2b_key = 0x7f7f2e1116f0
hash_index = 280
local_index = 82
b2b_cback =
dlg =
previous_dlg =
aux_dlg =
new_dlg =
param = {s = 0x0, len = 0}
statuscode = 408
leg =
pto =
to_tag = {s = 0x1088060454008101 <error: Cannot access memory at address 0x1088060454008101>, len = 134251648}
callid = {s = 0xb600104410400002 <error: Cannot access memory at address 0xb600104410400002>, len = 550535170}
from_tag = {s = 0x2c0280804000174 <error: Cannot access memory at address 0x2c0280804000174>, len = 537592136}
extra_headers = {s = 0x0, len = 0}
body = {s = 0x0, len = 0}
hdr =
method_id = 1
cb = {error = 153354688, number = {s = 0x8094882a60b4308 <error: Cannot access memory at address 0x8094882a60b4308>, len = 205783816}, method = {
s = 0xcd9684610b8701c6 <error: Cannot access memory at address 0xcd9684610b8701c6>, len = 214286852}, method_id = 21}
cseq = {type = 820918452, name = {s = 0x7f7f30f1fa20 "", len = 0}, body = {s = 0x7f7f30f39292 <openaux+50> "H\211C [\303\017\037\204", len = 0}, len = 820325167,
parsed = 0x7ffd673b8da0, next = 0x7ffd673b8850, sibling = 0x7f7f30f39260 }
etype = B2B_SERVER
dlg_based_search = 0
callid_hdr = {type = 1731956128, name = {s = 0x7ffd673b8850 "\031", len = 248}, body = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 7},
len = 821304256, parsed = 0x1000100010001, next = 0x1000100010001, sibling = 0x1000100010001}
from_hdr = {type = -80, name = {s = 0x7ffd673b8520 "\260\377\377\377\377\377\377\377 \205;g\375\177", len = 0}, body = {s = 0x55f83d61cfb0 "\360I\365\060\177\177",
len = 13}, len = 91, parsed = 0x1, next = 0x8, sibling = 0x7ffd673b8510}
to_hdr = {type = 1629406994, name = {s = 0x7f7f30f3c2d4 <_dl_name_match_p+68> "\205\300u\350\272\001", len = 14}, body = {
s = 0x7f7f30d35b5e "ld-linux-x86-64.so.2", len = 0}, len = 821250377, parsed = 0x1673b88c0, next = 0x7f7f2acb2a05, sibling = 0x0}
to_hdr_parsed = {error = 2049, body = {
s = 0x7f7f30f32b02 <_dl_map_object_from_fd+3634> "L\213\225\340\376\377\377H\203\370\377\017\205\320\367\377\377L\215\rǗ\001", len = 0}, uri = {
s = 0x4000 <error: Cannot access memory at address 0x4000>, len = 14304}, display = {s = 0x37e0 <error: Cannot access memory at address 0x37e0>, len = 0},
tag_value = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 16384}, parsed_uri = {user = {s = 0x7ffd673b8870 "\020\216;g\375\177",
len = 1731955936}, passwd = {s = 0x0, len = 819157854}, host = {s = 0x7f7f30f1ff20 "", len = 0}, port = {
s = 0x7f7f30f356ec <check_match+316> "\205\300t\201\351o\377\377\377\017\037", len = 0}, params = {s = 0x10 <error: Cannot access memory at address 0x10>,
--Type for more, q to quit, c to continue without paging--
len = 821381616}, headers = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 821216232}, port_no = 23284, proto = 12531, type = 32639,
transport = {s = 0x10 <error: Cannot access memory at address 0x10>, len = 821217120}, ttl = {s = 0x7f7f30f549f0 <_rtld_global+2448> "", len = 1731954440},
user_param = {s = 0x7ffd673b8704 "", len = 821254238}, maddr = {s = 0xffffffff00000000 <error: Cannot access memory at address 0xffffffff00000000>,
len = 821217120}, method = {s = 0x7f7f30f2c430 "", len = 819142669}, lr = {s = 0x9c386b5 <error: Cannot access memory at address 0x9c386b5>,
len = 819077152}, r2 = {s = 0x270e1a <error: Cannot access memory at address 0x270e1a>, len = 1731954640}, gr = {s = 0x7ffd673b87c0 "\377\377\377\377",
len = 1731954440}, pn_provider = {s = 0x7f7f30f20508 "\220Q\365\060\177\177", len = 6}, pn_prid = {s = 0x2 <error: Cannot access memory at address 0x2>,
len = 0}, pn_param = {s = 0x7ffd673b8790 "\300\207;g\375\177", len = 163808949}, pn_purr = {s = 0x7f7f30f20280 "PT\365\060\177\177", len = 1731954792},
transport_val = {s = 0x7f7f30f1ff20 "", len = 0}, ttl_val = {s = 0x7f7f30f20da8 "\257\\323\060\177\177", len = 821257103}, user_param_val = {
s = 0x5 <error: Cannot access memory at address 0x5>, len = 821169576}, maddr_val = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 0},
method_val = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 821165856}, lr_val = {s = 0x7f7f30d3200d "_dl_exception_create", len = 2}, r2_val = {
s = 0x7ffd673b87c0 "\377\377\377\377", len = 1731954640}, gr_val = {s = 0x7f7f30f20280 "PT\365\060\177\177", len = 0}, pn_provider_val = {
s = 0x7ffd673b8d50 "\001", len = 1343}, pn_prid_val = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 83}, pn_param_val = {
s = 0x7f7f30f2c5b0 "S", len = 821381616}, pn_purr_val = {s = 0x7ffd673b9854 "", len = 1427393280}, u_name = {{
s = 0x7e5 <error: Cannot access memory at address 0x7e5>, len = 819565105}, {s = 0x7ffd673b9900 "\003", len = 1731958896}, {s = 0x0, len = 820875392}, {
s = 0x2 <error: Cannot access memory at address 0x2>, len = 1731956192}, {s = 0x7ffd673b8dd0 "\002", len = 5}, {s = 0x7ffd673b8f60 "", len = 819574920}, {
s = 0x19 <error: Cannot access memory at address 0x19>, len = 819388690}, {s = 0x8000080008 <error: Cannot access memory at address 0x8000080008>,
len = 1731956256}, {s = 0x7ffd673b8e10 "", len = 25}, {s = 0x7ffd673b8f60 "", len = 0}}, u_val = {{s = 0x7f7f30ed6420 <_IO_str_chk_jumps> "",
len = 819387153}, {s = 0x0, len = 1029579360}, {s = 0xd68 <error: Cannot access memory at address 0xd68>, len = 819570288}, {s = 0x0, len = 1029579360}, {
s = 0x0, len = 1731955664}, {s = 0x7ffd673b8fb0 "\v", len = 1029638815}, {s = 0x7ffd673b8bd8 "ip6-allrouters", len = 819560089}, {
s = 0xb <error: Cannot access memory at address 0xb>, len = 1731955664}, {s = 0xbf <error: Cannot access memory at address 0xbf>, len = 1731955632}, {
s = 0x2ff <error: Cannot access memory at address 0x2ff>, len = 1731955664}}, u_params_no = 51992}, param_lst = 0x0, last_param = 0x7ffd673b8bd0,
next = 0x17}
from_hdr_parsed = {error = 1731955672, body = {s = 0x7f7f2acb5332 <internal_getent+98> "I\211\306H\205\300\017\210\253\004", len = 12288}, uri = {
s = 0x7ffd673b8be8 "calhost", len = 1029579360}, display = {s = 0x3e0 <error: Cannot access memory at address 0x3e0>, len = 1731955264}, tag_value = {
s = 0x7ffd673b8fb0 "\v", len = 819055360}, parsed_uri = {user = {s = 0x7f7f30d1cb64 "\001", len = 0}, passwd = {
s = 0x2 <error: Cannot access memory at address 0x2>, len = 0}, host = {s = 0x7f7f2fd7d1d8 "", len = 1731958074}, port = {
s = 0x18 <error: Cannot access memory at address 0x18>, len = 1731955672}, params = {
s = 0x7f7f2acb5b9a <gethostbyname3_multi+538> "A\211\307XZA\203\377\376\017\204\227\002", len = 2}, headers = {
s = 0x7f7f2acb6099 <gethostbyname3_multi+1817> "H\213D$(L\213\004$\272\001", len = 2}, port_no = 51968, proto = 12497, type = 32639, transport = {
s = 0x7ffd673b8a60 "", len = 2}, ttl = {s = 0x55f83d5e2260 "", len = 1731957376}, user_param = {s = 0x7ffd673b8b00 "\002", len = 1731955616}, maddr = {
s = 0x7ffd673b9578 "@\263y<\370U", len = 1731959024}, method = {s = 0x7ffd673b8bd8 "ip6-allrouters", len = 1731955712}, lr = {
s = 0x400000002 <error: Cannot access memory at address 0x400000002>, len = 1731955648}, r2 = {s = 0x0, len = 16}, gr = {
s = 0x7ffd673b8a78 "\360\224;g\375\177", len = 1731958000}, pn_provider = {s = 0x7ffd673b9528 "s]\001", len = 10}, pn_prid = {
s = 0x20 <error: Cannot access memory at address 0x20>, len = 1731956584}, pn_param = {
s = 0x7ffd00000005 <error: Cannot access memory at address 0x7ffd00000005>, len = 5}, pn_purr = {s = 0x0, len = 820860768}, transport_val = {
s = 0xd68 <error: Cannot access memory at address 0xd68>, len = 759458653}, ttl_val = {s = 0x2 <error: Cannot access memory at address 0x2>,
len = 759458653}, user_param_val = {s = 0xd68 <error: Cannot access memory at address 0xd68>, len = 759458656}, maddr_val = {s = 0x0, len = 820860768},
method_val = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 16}, lr_val = {s = 0x7ffd673b8b18 ",\225;g\375\177", len = 1731958060}, r2_val = {
s = 0x7ffd673b953a "", len = 0}, gr_val = {s = 0x7ffd673b9140 " ", len = 820862880}, pn_provider_val = {s = 0x1 <error: Cannot access memory at address 0x1>,
len = 819524268}, pn_prid_val = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 820672608}, pn_param_val = {
--Type for more, q to quit, c to continue without paging--
s = 0x1 <error: Cannot access memory at address 0x1>, len = 0}, pn_purr_val = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 1427393280},
u_name = {{s = 0x1 <error: Cannot access memory at address 0x1>, len = 1731957056}, {s = 0x7ffd673b9130 "", len = 24}, {s = 0x7ffd673b8bb0 "\377\002",
len = 1024}, {s = 0x2ff <error: Cannot access memory at address 0x2ff>, len = 0}, {s = 0x7ffd673b8bb0 "\377\002", len = 0}, {
s = 0x323a3a32306600 <error: Cannot access memory at address 0x323a3a32306600>, len = 758542441}, {
s = 0x73726574756f <error: Cannot access memory at address 0x73726574756f>, len = 1751933283}, {
s = 0x706f6f6c2d367069 <error: Cannot access memory at address 0x706f6f6c2d367069>, len = 1801675106}, {s = 0x7ffd673b8be2 "ters", len = 1731955696}, {
s = 0x0, len = 821253661}}, u_val = {{s = 0x20 <error: Cannot access memory at address 0x20>, len = 105}, {s = 0x55f83d643b90 "", len = 0}, {
s = 0x7f7f2acb1550 "\247\377\264M\243\310>\200\313\006ڑ\027.|@ǯ\217\245'0!^a\242\222\006"u\224\037q3\314\316"[1=/Z\271*\021\271\255\340\347\003\265M%\260(\361\271&\254\365\061\244", <incomplete sequence \304>, len = 821254900}, {s = 0x69 <error: Cannot access memory at address 0x69>, len = 717955304}, {
s = 0x55f83d643b90 "", len = 1731955944}, {s = 0x7ffd673b8ce4 "\001", len = 820671935}, {s = 0x0, len = 717955304}, {s = 0x7f7f2acb1590 "",
len = 1731956784}, {s = 0x4db4ffa6 <error: Cannot access memory at address 0x4db4ffa6>, len = 0}, {
s = 0x136d3fe <error: Cannot access memory at address 0x136d3fe>, len = 1731956144}}, u_params_no = 36256}, param_lst = 0x0, last_param = 0x55f83d643b70,
next = 0x3}
dlg_state = 0
prev_state = 0
crd =
auth = 0x0
auth_nc_cnonce = {nc = 0x0, cnonce = 0x0}
response = "\377\377\377\377\377\377\377\377D\263y<\370U\000\000s]\001\000\000\000\000\000 8S-\177\177\000\000\210"
new_hdr =
status_buf = "0\000\000\000\060\000\000\000h\227;g\375\177\000\000\240\226;g\375\177"
sdp_ct = {s = 0x7f7f2cfa1bf8 "Content-Type: application/sdp\r\n", len = 31}
old_route_type =
storage = {buffer = {s = 0x19 <error: Cannot access memory at address 0x19>, len = 28},
front_pointer = 0x1f0000001e <error: Cannot access memory at address 0x1f0000001e>, size = 1731954800, type = 32765, next = 0x7ffd673b8ce0, src_id = 0}
b2b_ev = -1
ctx =
b2b_cb_flags = 0
FUNCTION = "b2b_tm_cback"
#1 0x00007f7f2d516bb9 in run_trans_callbacks (type=type@entry=256, trans=trans@entry=0x7f7f2e120988, req=, rpl=rpl@entry=0xffffffffffffffff,
code=code@entry=408) at t_hooks.c:209
params = {req = 0x7f7f2e1223f8, rpl = 0xffffffffffffffff, code = 408, param = 0x7f7f2e097ba0, extra1 = 0x0, extra2 = 0x0}
cbp = 0x7f7f2e097b90
backup = 0x55f83c7648a8 <global_avps>
trans_backup = 0x7f7f2e120988
FUNCTION = "run_trans_callbacks"
#2 0x00007f7f2d4e3bca in local_reply (t=t@entry=0x7f7f2e120988, p_msg=p_msg@entry=0xffffffffffffffff, branch=, msg_status=msg_status@entry=408,
cancel_bitmap=cancel_bitmap@entry=0x7ffd673b98e0) at t_reply.c:1469
local_store = 0
local_winner = 0
--Type for more, q to quit, c to continue without paging--
reply_status = RPS_COMPLETED
winning_msg = 0xffffffffffffffff
winning_code = 408
totag_retr =
FUNCTION = "local_reply"
#3 0x00007f7f2d521dbe in fake_reply (code=408, branch=, t=0x7f7f2e120988) at timer.c:251
cancel_bitmap = 0
reply_status =
cancel_bitmap =
reply_status =
#4 final_response_handler (fr_tl=0x7f7f2e120bd8) at timer.c:404
old_ctx = 0x0
r_buf = 0x7f7f2e120b60
t = 0x7f7f2e120988
cancel_bitmap =
my_ctx = 0x7f7f2fd7d1d8
old_ctx =
r_buf =
t =
cancel_bitmap =
FUNCTION = "final_response_handler"
__mptr =
#5 timer_routine (ticks=35, set=) at timer.c:1088
tl = 0x7f7f2e120bd8
tmp_tl = 0x0
id = 0
FUNCTION = "timer_routine"
#6 0x000055f83c46790a in handle_timer_job () at timer.c:856
t = 0x7f7f2e06cf60
l =
FUNCTION = "handle_timer_job"
#7 0x000055f83c565fd5 in handle_io (idx=, event_type=, fm=) at net/net_udp.c:276
n =
read = 32765
n =
read =
FUNCTION = "handle_io"
#8 io_wait_loop_epoll (repeat=0, t=1, h=) at net/../io_wait_loop.h:311
ret =
n =
--Type for more, q to quit, c to continue without paging--
r = 3
i =
e = 0x7f7f2fd62ec8
ep_event = {events = 1011432242, data = {ptr = 0x611ec6ee000055f8, fd = 22008, u32 = 22008, u64 = 6998249596484867576}}
fd =
curr_time = 35
FUNCTION = "io_wait_loop_epoll"
#9 0x000055f83c56b2e6 in udp_start_processes (chd_rank=chd_rank@entry=0x55f83c764988 <chd_rank>, startup_done=startup_done@entry=0x0) at net/net_udp.c:497
si =
p_id =
i =
p =
FUNCTION = "udp_start_processes"
#10 0x000055f83c400973 in main_loop () at main.c:802
startup_done = 0x0
last_check = 0
rc =
chd_rank = 27
startup_done =
last_check =
rc =
FUNCTION = "main_loop"
#11 main (argc=, argv=) at main.c:1491
c =
r =
tmp = 0x7ffd673b9bc6 ""
tmp_len =
port =
proto =
protos_no =
options = 0x55f83c5b6f60 "f:cCm:M:b:l:n:N:rRvdDFEVhw:t:u:g:p:P:G:W:o:a:k:s:"
seed = 1470252276
rfd =
FUNCTION = "main"

[korithi]

Hi @bogdan-iancu,

Did you find the time to look at this?

Thanks

[korithi]

Hi guys,

Can you please help?

Thank you,

[korithi]

Hello,

On the DBG messages I always get these logs before the crash:
DBG:b2b_entities:b2b_tm_cback: tm [0x7f9baf273bf8] notification cb for [408] reply
DBG:b2b_entities:b2b_parse_key: hash_index = [197] - local_index= [3232]
CRITICAL:core:sig_usr: segfault in process pid: 28660, id: 8

Thanks

[razvancrainea]

I believe this issue has been sorted out in 6c86451. Can you please update to the latest OpenSIPS 3.1 and check if this stills occurs?

[korithi]

Hi @razvancrainea

Thank you for your feedback.
As I said I had 2 machines, one b2b refer config version 3.1.1/ one b2b cc config version 3.1.4. On both I was facing this issue.
What I did yesterday was that changed the configuration of call flow I had, in order to avoid the time out from happening. This is a workaround because the timeout might happen on many cases.
Now I have upgraded both on version 3.1.5 and I will monitor from now on.
FYI I have installed them via repos.

Thanks and regards,

[korithi]

Hi @razvancrainea

I still face the issue after the upgrade. Now I have the version:
version: opensips 3.1.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on with gcc 8

There might be some other case scenario where the time out is still present because I get the same error.
DBG:b2b_entities:b2b_tm_cback: tm [0x7f070326eb28] notification cb for [408] reply
DBG:b2b_entities:b2b_parse_key: hash_index = [2457] - local_index= [2928]

Thank you,

[razvancrainea]

The issue we were talking about was a crash - are you still getting a crash? if so, please post the backtrace.

[korithi]

Hi @razvancrainea

Yes, I still got the crash and I still got the same logs as I posted above.
It is difficult to get a backtrace since now it happens rarely.
I will try to replicate with the same version in the lab in order to provide a backtrace, or maybe on the same machine.

Thank you,

[github-actions]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

[bogdan-iancu]

@korithi , whenever you get anohter crash (please use the latest 3.1 version!!!!), please update here with the backtrace and let's see