[CRASH] segfault in dialog.so when use event_route

[suharik71]

OpenSIPS version you are running

opensips -V
version: opensips 3.0.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 3d2bd5318
main.c compiled on 02:53:35 Nov  7 2019 with gcc 4.8.5

Crash Core Dump
https://drive.google.com/file/d/1lEC8vDvEY-qIQ9FqgrHKtnskc-cJ3eH2/view?usp=sharing

Describe the traffic that generated the bug

when the get_dialogs_by_profile function is executed in event_route for a dialog that no longer exists opensips falls into error

To Reproduce

1. Start OpenSIPS
2. Start traffic
3. Check OpenSIPS crashed

Relevant System Logs

segfault at 90 ip 00007fe212482d64 sp 00007ffef48c34b0 error 4 in dialog.so[7fe21243c000+77000]

OS/environment information

• Operating System: Centos 7
• OpenSIPS installation: opensips-yum-nightly repo
• other relevant information:

Additional context
an example of a config can be seen in the user email list
http://lists.opensips.org/pipermail/users/2019-November/041870.html

[bogdan-iancu]

Please, first try the latest GIT for 3.0 branch - there might be a fix from yesterday addressing a similar problem.
If still crashing, please provide the backrace, not the core file - see https://opensips.org/Documentation/TroubleShooting-Crash

[suharik71]

sorry.
https://pastebin.com/fG2xFznX

[bogdan-iancu]

What openSIPS version (opensips -V) corresponds to this last bracktrace ? Have you updgraded as indicated ?

[suharik71]

Updated to the latest build.

opensips -V
version: opensips 3.0.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: ec840cf9d
main.c compiled on 02:54:31 Nov 11 2019 with gcc 4.8.5

The problem persists.
in the log I get
segfault at 90 ip 00007fc56d3462a4 sp 00007ffd7f9e1860 error 4 in dialog.so[7fc56d300000+77000]

[bogdan-iancu]

OK, thank you. In frame 1, could you print the content of dlgplease:

f 1
p *dlg

[suharik71]

(gdb) bt full
#0 0x00007ff99b65c2a4 in dlg_get_json_out () from /usr/lib64/opensips/modules/dialog.so
No symbol table info available.
#1 0x00007ff99b65c93d in w_get_dlg_jsons_by_profile () from /usr/lib64/opensips/modules/dialog.so
No symbol table info available.
#2 0x000000000043011c in do_action ()
No symbol table info available.
#3 0x0000000000435670 in run_action_list ()
No symbol table info available.
#4 0x0000000000467057 in eval_expr ()
No symbol table info available.
#5 0x0000000000466dd5 in eval_expr ()
No symbol table info available.
#6 0x000000000042ff8a in do_action ()
No symbol table info available.
#7 0x0000000000435670 in run_action_list ()
No symbol table info available.
#8 0x0000000000433436 in do_action ()
No symbol table info available.
#9 0x0000000000435670 in run_action_list ()
No symbol table info available.
#10 0x0000000000435958 in run_top_route ()
No symbol table info available.
#11 0x00007ff997f28e79 in route_run () from /usr/lib64/opensips/modules/event_route.so
No symbol table info available.
#12 0x00007ff997f28ee9 in route_received () from /usr/lib64/opensips/modules/event_route.so
No symbol table info available.
#13 0x000000000045ba7b in ipc_handle_job ()
No symbol table info available.
#14 0x0000000000551c58 in io_wait_loop_epoll.constprop.6 ()
No symbol table info available.
#15 0x0000000000554884 in udp_start_processes ()
No symbol table info available.
#16 0x000000000041e9df in main ()
No symbol table info available.
(gdb) f 1
#1 0x00007ff99b65c93d in w_get_dlg_jsons_by_profile () from /usr/lib64/opensips/modules/dialog.so
(gdb) p *dlg
Cannot access memory at address 0x0

[bogdan-iancu]

Is this from the same corefile as this one ?
In your pastebin ( https://pastebin.com/fG2xFznX ), in frame 1 (line 4 of the paste), you have :

dlg_get_json_out (dlg=dlg@entry=0x7f4b4fac2bb8,......

I guess you mixed the corefiles a bit.....

[suharik71]

sorry. it's new corefile

this is old corefile

(gdb) bt full
#0 0x00007f4b49b16d64 in w_get_dlg_jsons_by_profile () from /usr/lib64/opensips/modules/dialog.so
No symbol table info available.
#1 0x00007f4b49b173ef in w_get_dlg_jsons_by_val () from /usr/lib64/opensips/modules/dialog.so
No symbol table info available.
#2 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) f 1
#1 0x00007f4b49b173ef in w_get_dlg_jsons_by_val () from /usr/lib64/opensips/modules/dialog.so
(gdb) p *dlg
Cannot access memory at address 0x0

[bogdan-iancu]

Please skip the old core and let's focus on the new core (with the latest vresion). Print the *dlg in frame 1 from the new core.

[suharik71]

(gdb) f 1
#1  dlg_get_json_out (dlg=dlg@entry=0x7ff9a157bf58, ctx=ctx@entry=1, out_len=out_len@entry=0x7fff0aba40d4) at dialog.c:1820
1820			i=snprintf(p,len,"{\"tag\":\"%.*s\",\"contact\":\"%.*s\",\"cseq\":\"%.*s\",\"route_set\":\"%.*s\",\"bind_addr\":\"%.*s\",\"sdp\":\"%.*s\"}",
(gdb) p *dlg
$1 = {ref = 2, next = 0x0, prev = 0x0, h_id = 1961910190, h_entry = 2123, state = 1, lifetime = 1800, lifetime_dirty = 0, locked_by = 0, start_ts = 0, flags = 30992, from_rr_nb = 0, user_flags = 0, mod_flags = 10, initial_t_hash_index = 10111, initial_t_label = 1871333066, replicated = 0, tl = {next = 0x0, prev = 0x0, timeout = 0}, pl = 0x0, reinvite_pl = 0x0,
  terminate_reason = {s = 0x0, len = 0}, callid = {s = 0x7ff9a157c058 "c97e1b68-710a-4a13-9cd6-39d0bcb2ce76sip:89746698953@10.23.100.40sip:+74852285555@185.128.105.38:\300\300\300\300\300\300\300\300\355\357ͫ\355\357ͫrecv\r\na=rtcp:349`\001", len = 36}, from_uri = {
    s = 0x7ff9a157c07c "sip:89746698953@10.23.100.40sip:+74852285555@185.128.105.38:\300\300\300\300\300\300\300\300\355\357ͫ\355\357ͫrecv\r\na=rtcp:349`\001", len = 28}, to_uri = {s = 0x7ff9a157c098 "sip:+74852285555@185.128.105.38:\300\300\300\300\300\300\300\300\355\357ͫ\355\357ͫrecv\r\na=rtcp:349`\001", len = 31}, legs = 0x7ff9a156f310,
  legs_no = "\002\002\000", cbs = {first = 0x7ff9a15c0d78, types = 304}, profile_links = 0x7ff9a1751a98, vals = 0x0, shtag = {s = 0x7ff9a16bcad0 "node_MSK_test", len = 13}}