[CRASH] usrloc module

[adigeo]

OpenSIPS version you are running

version: opensips 3.1.0-dev (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 7f16a1cdb
main.c compiled on  with gcc 9

Crash Core Dump

Core was generated by `/usr/sbin/opensips -w /run/opensips -P opensips.pid -a F_MALLOC_DBG -m 128'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fc0ed15f376 in wb_timer (_r=_r@entry=0x7fc0eeb5a870, ins_list=0x7fc0eeb4c168) at urecord.c:345
345 VAL_BIGINT(cid_vals+cid_len) = t->contact_id;
(gdb) bt
#0 0x00007fc0ed15f376 in wb_timer (_r=_r@entry=0x7fc0eeb5a870, ins_list=0x7fc0eeb4c168) at urecord.c:345
#1 0x00007fc0ed15f920 in timer_urecord (_r=_r@entry=0x7fc0eeb5a870, ins_list=ins_list@entry=0x7fc0eeb4c168) at urecord.c:439
#2 0x00007fc0ed14d232 in mem_timer_udomain (_d=) at udomain.c:1166
#3 0x00007fc0ed167489 in _synchronize_all_udomains () at dlist.c:1068
#4 0x00007fc0ed16d76c in destroy () at ul_mod.c:517
#5 0x0000557873eff15c in destroy_modules () at sr_module.c:544
#6 0x0000557873ef8126 in cleanup (show_status=1) at main.c:363
#7 0x0000557873ef88fc in shutdown_opensips (status=) at main.c:527
#8 0x0000557873ef904b in handle_sigs () at main.c:610
#9 0x0000557873e64ae8 in main_loop () at main.c:867
#10 main (argc=, argv=) at main.c:1488

Core file: http://download.ag-projects.com/cores/core.1789833.opensips.1590076653

Describe the traffic that generated the bug

No call traffic just Register.

To Reproduce

Relevant System Logs

May 21 17:57:32 node15 /usr/sbin/opensips[1789842]: CRITICAL:core:sig_usr: segfault in process pid: 1789842, id: 9
May 21 17:57:32 node15 kernel: opensips[1789842]: segfault at 10 ip 00007fc0ed15f376 sp 00007ffeb27289a0 error 6 in usrloc.so[7fc0ed147000+2d000]
May 21 17:57:33 node15 /usr/sbin/opensips[1789833]: INFO:core:handle_sigs: child process 1789842 exited by a signal 11
May 21 17:57:33 node15 /usr/sbin/opensips[1789833]: INFO:core:handle_sigs: core was generated
May 21 17:57:33 node15 /usr/sbin/opensips[1789833]: INFO:core:handle_sigs: terminating due to SIGCHLD
May 21 17:57:33 node15 /usr/sbin/opensips[1789834]: INFO:core:sig_usr: signal 15 received
May 21 17:57:33 node15 /usr/sbin/opensips[1789835]: INFO:core:sig_usr: signal 15 received
May 21 17:57:33 node15 /usr/sbin/opensips[1789836]: INFO:core:sig_usr: signal 15 received
May 21 17:57:33 node15 /usr/sbin/opensips[1789837]: INFO:core:sig_usr: signal 15 received

OS/environment information

• Operating System:
• OpenSIPS installation:
• other relevant information:

Additional context
Lastest git source

[adigeo]

With the applied fix, it is still crashing but not as often:

Core was generated by `/usr/sbin/opensips -w /run/opensips -P opensips.pid -a F_MALLOC_DBG -m 512'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f4d61212536 in timer_urecord ()
   from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
(gdb) bt
#0  0x00007f4d61212536 in timer_urecord ()
   from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
#1  0x00007f4d611fd88a in mem_timer_udomain ()
   from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
#2  0x00007f4d61219d89 in _synchronize_all_udomains ()
   from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
#3  0x00007f4d6122079c in ?? () from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
#4  0x000055a0cef81329 in destroy_modules ()
#5  0x000055a0cef7a364 in cleanup ()
#6  0x000055a0cef7ab97 in ?? ()
#7  0x000055a0cef7b4f6 in handle_sigs ()
#8  0x000055a0ceee3c0c in main ()
(gdb)

Core file
http://download.ag-projects.com/cores/core.12796.opensips.1590337474

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -w /run/opensips -P opensips.pid -a F_MALLOC_DBG -m 512'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f347892a536 in timer_urecord () from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
(gdb) bt
#0  0x00007f347892a536 in timer_urecord () from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
#1  0x00007f347891588a in mem_timer_udomain () from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
#2  0x00007f3478931d89 in _synchronize_all_udomains () from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
#3  0x00007f34789382d4 in ?? () from /usr/lib/x86_64-linux-gnu/opensips/modules/usrloc.so
#4  0x0000562b5132d5da in handle_timer_job ()
#5  0x0000562b51426135 in ?? ()
#6  0x0000562b5142b446 in udp_start_processes ()
#7  0x0000562b512c37fc in main ()
(gdb) 

[liviuchircu]

Right, this makes sense - the original bug is related to some scrambling (by myself) of the possible configurations of the module (e.g. with or w/o timer, with or w/o DB, etc.). Currently testing out the fix, and I expect to push it within a couple of hours.

[liviuchircu]

@adigeo could you paste your usrloc modparam settings, just to be sure I fixed the right issue? Thanks!

[adigeo]

modparam("usrloc", "use_domain", 1)
modparam("usrloc", "db_url", "mysql://opensips:xxx@localhost/opensips")
modparam("usrloc", "working_mode_preset", "single-instance-sql-write-through")
modparam("usrloc", "desc_time_order", 1)

[liviuchircu]

Thanks! While I haven't been able to reproduce your 2nd crash (after @rvlad-patrascu 's fix), I have pushed an update which makes the timer code more robust, so that type of crash cannot happen at all now.

If you still have issues, could you provide a gdb backtrace with debug-symbols enabled, similar to your initial one, so we can see at which line it crashes? I cannot work with a corefile unless you also provide the binaries to go with it -- and since 3.1-dev does not have any packaging, I'm having a hard time getting anything out of that corefile!

[stale]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.