[CRASH] OpenSIPS 2.4.7 is crashing at random times [possible invalid memory access?]

[abreualexandre]

OpenSIPs version 2.4.7

[root@ log]# opensips -V
version: opensips 2.4.7 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 10:38:18 Jan 29 2020 with gcc 4.8.5

Crash Core Dump
Link to backtrace full: https://pastebin.com/Mq6AFGHK

Describe the traffic that generated the bug
It seems that the crash is generated by module drouting. The role of this module is just to build and send SIP Options at a given interval according to these parameters:

loadmodule "drouting.so"
modparam("drouting", "db_url","mysql://opensips:opensipsrw@localhost/opensips")
modparam("drouting", "use_domain", 0)
modparam("drouting", "probing_interval", 60)
modparam("drouting", "probing_from", "sip:options@mydomain.com")
modparam("drouting", "probing_method", "OPTIONS")
modparam("drouting", "probing_reply_codes", "501, 403, 404, 200")

To Reproduce
It is not reproduceable: The crash happens in a random way. It can crash one time or two in a week. Just start the OpenSIPS and let the monit tells you about the restart.

Relevant System Logs
This link shows the syslog lines before the crash
https://pastebin.com/5T3cge99

OS/environment information
[root@ log]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)

Packages installed:
opensips-proto_tls-2.4.7-1.el7.x86_64
opensips-debuginfo-2.4.7-1.el7.x86_64
opensips-2.4.7-1.el7.x86_64
opensips-db_mysql-2.4.7-1.el7.x86_64
opensips-tls_mgm-2.4.7-1.el7.x86_64
opensips-yum-releases-2.4-5.el7.noarch

Kernel:
Linux 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Memory INFO:
KiB Mem : 8173968 total, 1340416 free, 1082008 used, 5751544 buff/cache
KiB Swap: 4194300 total, 4105212 free, 89088 used. 6338176 avail Mem

Processor:
Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz

Additional context
The OpenSIPS was started with:
[root@ log]# /usr/sbin/opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 128 -M 8

The load and concurrent calls is low.

[bogdan-iancu]

@abreualexandre , thank you for your report.
I see the corefile was generated

INFO:core:handle_sigs: core was generated

Could you please extract the backtrace and share it with us ( see "how-to" instructions here )

[abreualexandre]

Hi Bogdan, thanks for the answer.
The provided BT FULL didn't help?

Link to backtrace full:
https://pastebin.com/Mq6AFGHK

[bogdan-iancu]

sorry @abreualexandre , my bad, I missed the link in the original post. Let me check.

[bogdan-iancu]

@abreualexandre it may be a microsoft specific virus :P ...

I see the crash happens in some pretty straight forward code, where copy the hostent structure returned by the DNS resolver. But I suspect the memory may be corrupted from a different part of the code and what we see here is just a side effect, not the real cause.
Could you recompile your opensips with the DBG_MALLOC + QM_MALLOC to see if the memory allocator spots some mem issues ? See https://opensips.org/Documentation/TroubleShooting-OutOfMem for instructions.

[abreualexandre]

Hi Bogdan, let's get rid of the virus then :-)

Another information that may be useful:
I have this very same crash using the same logic (DR probing) and version (2.4.7) on different hardwares.

One thing I considered at least strange is the fact that "buff" in frame #7, the server URI ends with ".comm" and the host is indeed ".com". I see this happening on those scenarios crashing too.

That said, should I go ahead and jump from RPMs to the source code version?

[bogdan-iancu]

That .comm is a false alarm, as that buffer has the proper len:

uri = {s = 0x7f7b9dd976a0 <buff.12883> "sip:sip.pstnhub.microsoft.comm", len = 29}

I guess you need to get the code and compile it. Or upgrade to 3.0 where the mem allocator is runtime option ;)

[abreualexandre]

Hi Bogdan, FIY the 2.4.8 is already running with QM_MALLOC and DBG_MALLOC stuff. I'll update the issue when the crashes happens again.

[bogdan-iancu]

thanks for update @abreualexandre

[abreualexandre]

Updating the issue: I just moved from RPM (2.4.7) to the source code version 2.4.8 with memory debug and then the crash stopped.
Then I moved the same box to the updated RPM version 2.4.8 and still no crashes. Today, I updated another box from source code 2.4.8 (w/ debug) to RPM version 2.4.8, just to make sure it's really OK.

If it doesn't happen anymore, I'll ask you to close this issue.

Thanks!

[abreualexandre]

The issue does not happen in version 2.4.8. It is safe to close this issue. Thanks to OpenSIPS team.