[CRASH] core dump in next_branches --> search_next_avp in 2.4.8 (nightly build)

[rrevels-bw]

OpenSIPS version you are running

version: opensips 2.4.8 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 02:33:06 Oct 15 2020 with gcc 4.8.5

Crash Core Dump

Our current configuration doesnt give us the symbol files so I will have to update this tracker as we make progress fixing that. Here is the function tree that I do have access to:
(gdb) bt full
#0 0x00000000004f3ef1 in search_next_avp ()
No symbol table info available.
#1 0x00000000004973ef in next_branches ()
No symbol table info available.
#2 0x00000000004a253e in do_action ()
No symbol table info available.
#3 0x00000000004aacf1 in run_action_list ()
[...]

Describe the traffic that generated the bug

This appears to be memory corruption that is found when attempting to fail over to the next branch (serial) that was gotten from a 300 earlier. We can go weeks between crashes but do seem to see the problem on all proxies in the network sooner or later.

To Reproduce

Start traffic that will get a 300 redirect and then have the first attempt fail. At some point the pointer to the current contact will get corrupted and cause a general protection error when accessed.

Relevant System Logs

We see no warning what-so-ever leading up to the crash.

OS/environment information

• Operating System: Cent OS 7
• OpenSIPS installation: opensips compile from source (and rpm build) from nightly build opensips-2.4.8-09142020.tar.gz
• other relevant information:

Additional context

[liviuchircu]

@rrevels-bw this looks like a bug that was fixed in df68e3b, on Aug 26th. I'm not sure what "nightly build" you have from that "opensips-2.4.8-09142020.tar.gz" tarball, but it definitely looks outdated, since OpenSIPS 2.4 is up to the 2.4.9 minor release now.

Closing for now and marking as "outdated". However, if the problem persists even with 2.4.9, please re-open the ticket and I can take a closer look.

[rrevels-bw]

I am, hmm, 75% sure the source files we are using came from a Sept nightly build. I will try to determine the best way to verify that. Maybe based on the file content from this first commit below but here is the output from git log in the source directory I am fairly sure we built from:

commit 83d0ca9 (HEAD -> 2.4, origin/2.4)
Author: Ovidiu Sas osas@voipembedded.com
Date: Mon Sep 7 21:14:51 2020 -0400

dialog: fix/reformat some logs

(cherry picked from commit 2efaeee24c69841e9776e360960bc8d644081ff9)

[liviuchircu]

I missed that idea, @rrevels-bw - indeed, "Sep 7" would be a pretty decent build and it would include Răzvan's patch for the contact parsing. However, do note your output above:

version: opensips 2.4.8 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 02:33:06 Oct 15 2020 with gcc 4.8.5

Notice how there is no "git revision:" string! Which means that we're talking about a 2.4.8 release package, which dates all the way back to Jan 2020! Are you able to confirm 100% that the corefile was produced by the custom built opensips from Sep 7th and not by the Jan 2020 release package of 2.4.8? Thanks!

[rrevels-bw]

I understand your confusion with that output as i experienced it as well. The only thing I can think of is I may have downloaded the tar file via wget as i didnt want the git info in the file we would be putting into our own repo for creating the rpm. (would that suppress the git line?)

However, I did go look at the commit you referenced and the changes from it are indeed in our source files. For example:

#include "../../parser/parse_to.h"
#include "../../parser/parse_cseq.h"
#include "../../parser/contact/contact.h"
#include "../../parser/contact/parse_contact.h"

We have tried retracing our history with this issue and its just been so long that its difficult to say for certain but we THINK that we had a crash problem in this same general area (search_first_avp) with 2.4.6 . Once we stepped up to 2.4.8 nightly build the crashes seemed to have been less frequent and are always in next_branches where that was not the case with the 2.4.6 crashes. I should be able to get a decent backtrace in the not too distant future.