core dump with xmlrpc in v1.11.3

[telematico]

It seems to be a problem with xmlrpc module.

With a population of 4000 users, when I tried to get the registered AORs form XMLRPC it tooks about 30 seconds and 7MB of PKG mem. doing it via fifo it tooks no more than 2 seconds.

After some hours collecting statistics using XMLRPC (for Nagios and Cacti) opensips starts to crash with signal 11.

I have some cores, I will publish backtraces

Thanks for your help,
Carlos Oliva

[telematico]

Here is one bt full:

For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/opensips...Reading symbols from /usr/lib/debug/usr/sbin/opensips...done.
done.
[New LWP 30701]
[New LWP 2137]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 128 -M 16 -u opensips -'.
Program terminated with signal 11, Segmentation fault.
#0  add_next (flags=3, value_len=1, value=<optimized out>, name_len=23, name=<optimized out>, brother=0x6172747369676572) at mi/tree.c:184
184             brother->last->next = new;
(gdb) bt full
#0  add_next (flags=3, value_len=1, value=<optimized out>, name_len=23, name=<optimized out>, brother=0x6172747369676572) at mi/tree.c:184
        new = 0x7f0decfbb830
#1  add_mi_node_child (parent=parent@entry=0x7f0decfbbe58, flags=flags@entry=3, name=name@entry=0x7f0decfbac20 "registrar:rejected_regseess", name_len=name_len@entry=23, 
    value=<optimized out>, value_len=value_len@entry=1) at mi/tree.c:219
No locals.
#2  0x00000000004fa603 in addf_mi_node_child (parent=parent@entry=0x7f0decfbbe58, flags=flags@entry=1, name=name@entry=0x7f0decfbac20 "registrar:rejected_regseess", name_len=23, 
    fmt_val=fmt_val@entry=0x537367 "%lu") at mi/tree.c:239
        ap = {{gp_offset = 48, fp_offset = 48, overflow_arg_area = 0x7f0def1545c0, reg_save_area = 0x7f0def1544e0}}
        p = 0x72 <Address 0x72 out of bounds>
        len = 1
#3  0x0000000000484696 in mi_print_stat (rpl=rpl@entry=0x7f0decfbbe58, mod=mod@entry=0x7f0ddd51ce80, stat=stat@entry=0x7f0ddd199fa0, val=1) at statistics.c:708
        tmp_buf = {s = 0x7f0decfbac20 "registrar:rejected_regseess", len = 23}
        tmp = 0x7f0decfbac20 "registrar:rejected_regseess"
        __FUNCTION__ = "mi_print_stat"
#4  0x0000000000484c06 in mi_add_module_stats (mods=0x7f0ddd51ce80, rpl=0x7f0decfbbe58) at statistics.c:735
        stat = 0x7f0ddd199f98
        ret = <optimized out>
#5  mi_get_stats (cmd=<optimized out>, param=<optimized out>) at statistics.c:784
        rpl_tree = 0x7f0decfbbe38
        rpl = 0x7f0decfbbe58
        arg = 0x7f0decfbad28
        stat = <optimized out>
        val = {s = 0xa00b00 "registrar:", len = 9}
        i = <optimized out>
#6  0x00007f0de63818f7 in run_mi_cmd (param=0x7f0def154790, f=<optimized out>, t=0x7f0decfbaa80, cmd=<optimized out>) at ../../mi/mi.h:109
        ret = <optimized out>
#7  default_method (env=0x7f0def154790, host=<optimized out>, methodName=0xa050f0 "get_statistics", paramArray=0xa00690, serverInfo=<optimized out>) at xr_server.c:223
        ret = 0x0
        mi_cmd = 0x7f0decfbaa80
        mi_rpl = 0x0
        hdl = 0x0
        f = 0x7f0decf9fe80
        response = 0x0
        is_shm = 0
        __FUNCTION__ = "default_method"
#8  0x00007f0de5f73f5b in xmlrpc_dispatchCall () from /usr/lib/libxmlrpc_server.so.3
No symbol table info available.
#9  0x00007f0de5f7407a in xmlrpc_registry_process_call2 () from /usr/lib/libxmlrpc_server.so.3
No symbol table info available.
#10 0x00007f0de617aea8 in ?? () from /usr/lib/libxmlrpc_server_abyss.so.3
No symbol table info available.
#11 0x00007f0de5d69ac4 in ?? () from /usr/lib/libxmlrpc_abyss.so.3
No symbol table info available.
#12 0x00007f0de5d64977 in ?? () from /usr/lib/libxmlrpc_abyss.so.3
No symbol table info available.
#13 0x00007f0de5d6c444 in ?? () from /usr/lib/libxmlrpc_abyss.so.3
No symbol table info available.
#14 0x00007f0dec5b1b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#15 0x00007f0dee2257bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#16 0x0000000000000000 in ?? ()
No symbol table info available.

[telematico]

and another core backtrace:

Reading symbols from /usr/sbin/opensips...Reading symbols from /usr/lib/debug/usr/sbin/opensips...done.
done.
[New LWP 17044]
[New LWP 32091]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/opensips -P /var/run/opensips/opensips.pid -m 128 -M 16 -u opensips -'.
Program terminated with signal 11, Segmentation fault.
#0  0x00000000004b7410 in fm_remove_free (n=0x7f37c1f97298, qm=0x7f37c1f0e010) at mem/f_malloc.c:172
172             *pf=n->u.nxt_free;
(gdb) bt full
#0  0x00000000004b7410 in fm_remove_free (n=0x7f37c1f97298, qm=0x7f37c1f0e010) at mem/f_malloc.c:172
        pf = 0x0
        hash = 9
#1  fm_malloc (qm=0x7f37c1f0e010, size=size@entry=72) at mem/f_malloc.c:386
        frag = 0x7f37c1f97298
        n = <optimized out>
        hash = <optimized out>
        __FUNCTION__ = "fm_malloc"
#2  0x00000000004f9ecf in create_mi_node (flags=0, value_len=3, value=<optimized out>, name_len=0, name=0x0) at mi/tree.c:139
        name_pos = <optimized out>
        new = <optimized out>
        size_mem = <optimized out>
        value_pos = 0
#3  add_next (flags=0, value_len=3, value=<optimized out>, name_len=0, name=<optimized out>, brother=0x7f37c1f97310) at mi/tree.c:180
        new = 0xa
#4  add_mi_node_child (parent=parent@entry=0x7f37c1f9e6b8, flags=flags@entry=0, name=name@entry=0x0, name_len=name_len@entry=0, value=<optimized out>, value_len=3)
    at mi/tree.c:219
No locals.
#5  0x00007f37bb35e1c5 in xr_parse_tree (env=env@entry=0x7f37c4131790, paramArray=paramArray@entry=0x1ac1100) at xr_parser.c:173
        mi_root = 0x7f37c1f9e698
        size = 10
        i = <optimized out>
        intValue = 1
        boolValue = 0
        doubleValue = 1.3861402994067678e-316
        stringValue = 0x1ac3ea0 "tm:"
        byteStringValue = 0x0
        item = 0x1ac31c0
        __FUNCTION__ = "xr_parse_tree"
#6  0x00007f37bb35ee86 in default_method (env=0x7f37c4131790, host=<optimized out>, methodName=0x1ac29c0 "get_statistics", paramArray=0x1ac1100, serverInfo=<optimized out>)
    at xr_server.c:210
        ret = 0x0
        mi_cmd = 0x0
        mi_rpl = 0x0
        hdl = 0x0
        f = 0x7f37c1f7ce80
        response = 0x0
        is_shm = 0
        __FUNCTION__ = "default_method"
#7  0x00007f37baf50f5b in xmlrpc_dispatchCall () from /usr/lib/libxmlrpc_server.so.3
No symbol table info available.
#8  0x00007f37baf5107a in xmlrpc_registry_process_call2 () from /usr/lib/libxmlrpc_server.so.3
No symbol table info available.
#9  0x00007f37bb157ea8 in ?? () from /usr/lib/libxmlrpc_server_abyss.so.3
No symbol table info available.
#10 0x00007f37bad46ac4 in ?? () from /usr/lib/libxmlrpc_abyss.so.3
No symbol table info available.
#11 0x00007f37bad41977 in ?? () from /usr/lib/libxmlrpc_abyss.so.3
No symbol table info available.
#12 0x00007f37bad49444 in ?? () from /usr/lib/libxmlrpc_abyss.so.3
No symbol table info available.
#13 0x00007f37c158eb50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#14 0x00007f37c32027bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#15 0x0000000000000000 in ?? ()
No symbol table info available

[bogdan-iancu]

HI @telematico , I suspect you have a libxmlrpc-c3 library which is compiled with threads supports (and not with multi-processes support) - this is incompatible with OpenSIPS and leads to memory corruption. Because of such problems, we are currently migrating to mi_xmlrpc_ng - please try using that module and let me know if you still have the problem.

[telematico]

Hi @bogdan-iancu , I'm using oficial packages of debian 7.7 (x86_64), but i have no idea if libxmlrpc-core-c3 is compiled with multi-process support or not, I can't find the info in debian docs.

I'm testing now mi_xmlrpc_ng as you suggested and things seems to be right in my test machine. I will make more tests and send you feedback.

Thanks a lot

[bogdan-iancu]

AFAIK, libxmlrpc-c3 is by default compiled with threads support - we had a long struggle with that.
Please run tests with mi_xmlrpc_ng and report if you find any issues (open other tickets for that, I will close this one).
In 2.1, the mi_xlmrpc will become obsolete and removed.