[CRASH] B2BUA mode opensips restarts after any calls

[olegans1972]

OpenSIPS version you are running

version: opensips 3.0.2 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: eddee63
main.c compiled on 10:56:09 Jun 2 2020 with gcc 4.8.5

Crash Core Dump

[root@vms-1 tmp]# gdb core.22231
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-119.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
[New LWP 22231]
Missing separate debuginfo for the main executable file
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/21/0cdcd4bd28000a7456c81a281c115867d7f633
Core was generated by `/usr/sbin/opensips -P /run/opensips/opensips.pid -f /etc/opensips/opensips.cfg'.
Program terminated with signal 6, Aborted.
#0 0x00007f1db506c387 in ?? ()
"/tmp/core.22231" is a core file.
Please specify an executable to debug.
(gdb) bt full
#0 0x00007f1db506c387 in ?? ()
No symbol table info available.
#1 0x00007f1db506da78 in ?? ()
No symbol table info available.
#2 0x0000000000000020 in ?? ()
No symbol table info available.
#3 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)

Describe the traffic that generated the bug

Only one test call

To Reproduce

Relevant System Logs

Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: DBG:core:parse_headers: flags=ffffffffffffffff
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: DBG:core:prep_reassemble_body_parts: multi to multi part body reconstruction
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: DBG:core:prep_reassemble_body_parts: handing part with flags 0, mime application/sdp, dump function (nil)
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: DBG:core:prep_reassemble_body_parts: handing part with flags 0, mime application/isup;version=itu-t92+;base=itu-t92+, dump function (nil)
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: DBG:core:prep_reassemble_body_parts: resulting body len is 802
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: DBG:core:parse_headers: flags=ffffffffffffffff
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: DBG:core:reassemble_body_parts: multi to multi part body reconstruction
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: DBG:core:reassemble_body_parts: handing part with flags 0, mime application/sdp, dump function (nil)
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: DBG:core:reassemble_body_parts: handing part with flags 0, mime application/isup;version=itu-t92+;base=itu-t92+, dump function (nil)
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40182]: CRITICAL:core:fm_free: freeing already freed pkg pointer (0x7f1eeb7017b0), first free: #1: #3(0) - aborting!
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40173]: DBG:core:handle_sigs: OpenSIPS exit status = 6
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40173]: INFO:core:handle_sigs: child process 40182 exited by a signal 6
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40173]: INFO:core:handle_sigs: core was not generated
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40173]: INFO:core:handle_sigs: terminating due to SIGCHLD
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40173]: DBG:core:shutdown_opensips: Asking process 6 [SIP receiver udp:10.47.15.57:5060] to terminate
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40173]: DBG:core:shutdown_opensips: Asking process 8 [SIP receiver udp:10.47.15.57:5060] to terminate
Jul 20 20:36:14 vms-1 /usr/sbin/opensips[40173]: DBG:core:shutdown_opensips: Asking process 9 [SIP receiver udp:10.47.15.57:5060] to terminate

OS/environment information

• Operating System:
• OpenSIPS installation:
• other relevant information:
  Centos 7

Additional context

I am trying to use b2b scenario 'top hiding'. I use for it:

B2BUA

loadmodule "b2b_entities.so"
loadmodule "b2b_logic.so"
modparam("b2b_entities", "script_req_route", "b2b_request")
modparam("b2b_entities", "script_reply_route", "b2b_reply")
modparam("b2b_entities", "db_mode", 0)
modparam("b2b_logic", "db_mode", 0)
modparam("b2b_logic", "server_address", "sip:10.47.15.57:5060")

after call this function 'b2b_init_request("top hiding");' I get restart opensips

I use only db_text. As I understatnd right I can use b2b without any DB

[rvlad-patrascu]

Hi @olegans1972,

As I could not reproduce this myself, please post the backtrace for an opensips installation with debug symbols, because this one contains no usable information. Also, judging from the logs it seems that you are using the sip_i module. Can you give some more details about this?

[olegans1972]

HI @rvlad-patrascu !

My opensips has a connection to MSC on one side and freeswitch on the other side. To work with MSC, I use the sip_i module. The business logic is as follows: opensips gets INVITE from MSC, parses the ISUP body and sends it via the rest client asynchronously. Depending on the response INVITE will go either to freeswitch or back to MSC. Any time when opensips is trying to send INVITE to MSC I see restart.
loading_opensips.txt

[rvlad-patrascu]

I see that you have attached some extra logs but I was referring to a proper backtrace from the core file produced by the crash.

[olegans1972]

I'm sorry I did coredump as I see here https://www.opensips.org/Documentation/TroubleShooting-Crash and I got next:

[opodguyko@dev-opensips-1 tmp]$ sudo gdb /usr/local/sbin/opensips core.7114
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/local/sbin/opensips...done.
[New LWP 7114]
Core was generated by `/usr/sbin/opensips -P /run/opensips/opensips.pid -f /etc/opensips/opensips.cfg'.
Program terminated with signal 6, Aborted.
#0 0x00007fbbc41f7387 in ?? ()
(gdb) bt full
#0 0x00007fbbc41f7387 in ?? ()
No symbol table info available.
#1 0x00007fbbc41f8a78 in ?? ()
No symbol table info available.
#2 0x0000000000000020 in ?? ()
No symbol table info available.
#3 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)

This is a new core

[olegans1972]

HI @rvlad-patrascu !
If you need to do something differently, tell me how

[rvlad-patrascu]

If you installed opensips from packages you also have to install the corresponding debuginfo packages (opensips-debuginfo, opensips-debugsource, opensips-b2bua-module-debuginfo etc.).

[olegans1972]

I have found why my opensips restarted. I use next flow INVITE--->Opensips---> Opensips-Rest-Client-Async-Request--->Get-Response--->Opensips-select-dest-via-dispatcher--->Opensips-b2b_init_request---Crach.

I remove "Opensips-Rest-Client-Async-Request--->Get-Response"

And now I have:

INVITE--->Opensips--->Opensips-select-dest-via-dispatcher--->Opensips-b2b_init_request--->INVITE sent

This chain "Opensips-Rest-Client-Async-Request--->Get-Response" results in crash in this script.
It remains to understand why.

[olegans1972]

If you installed opensips from packages you also have to install the corresponding debuginfo packages (opensips-debuginfo, opensips-debugsource, opensips-b2bua-module-debuginfo etc.).

[opodguyko@dev-opensips-1 tmp]$ sudo rpm -qa | grep debug
gcc-base-debuginfo-4.8.5-39.el7.x86_64
yum-plugin-auto-update-debug-info-1.1.31-54.el7_8.noarch
glibc-debuginfo-common-2.17-307.el7.1.x86_64
gcc-debuginfo-4.8.5-39.el7.x86_64
opensips-debuginfo-3.0.2-2.el7.x86_64
glibc-debuginfo-2.17-307.el7.1.x86_64
ncurses-debuginfo-5.9-14.20130511.el7_4.x86_64

[rvlad-patrascu]

I think you also need to install the module specific debuginfo packages (eg. opensips-b2bua-module-debuginfo) if you still get only lines like this in the backtrace:
#0 0x00007fbbc41f7387 in ?? ()
No symbol table info available.

[stale]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

[stale]

Marking as closed due to lack of progress for more than 30 days. If this issue is still relevant, please re-open it with additional details.