opensips-2.2-beta: async: incompatible with rest_client

[chrismanivong]

We observed this in 2.1.2 already. Previous versions have not been tested.

async(rest_post("https://URL/push/unregister", "$var(jrq)", "application/json", "$var(body)", "$var(ct)", "$var(rcode)"), resume_route);

The resume_route will never be called. When you call this once, the first worker will hang, if you call this a second time in a second worker process, it will throw a

kernel: opensips[12330]: segfault at 0 ip 00007fb09bfedafe sp 00007fffc9b265d0 error 4 in libcurl.so.4.3.0[7fb09bfab000+71000]

[bogdan-iancu]

@krys1976 , try to run opensips in debug level, and post the logs somewhere, this will give us some idea about what is going on.

Also, on the crash, see my other reply in how to get a backtrace from the core file.

[chrismanivong]

[root@sip01-dev-vsc /]# gdb opensips core.16346
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/sbin/opensips...Reading symbols from /usr/lib/debug/usr/sbin/opensips.debug...done.
done.
[New LWP 16346]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /lib64/libmysqlclient.so.18
Try: yum --enablerepo='debug' install /usr/lib/debug/.build-id/df/ef2833c397fd7ebea0ba670b3d5b07776a26e7.debug
Core was generated by `opensips -P /var/run/opensips.pid -f /etc/opensips/opensips.cfg -m 32 -M 2048'.
Program terminated with signal 11, Segmentation fault.
#0 Curl_raw_nequal (first=0x0, second=second@entry=0x7f985d731f21 "User-Agent:", max=max@entry=11) at rawstr.c:115
115 while(*first && *second && max) {

[chrismanivong]

bt_16607.txt

[bogdan-iancu]

@krys1976 , I guess the backtraces are swapped with #846 :) .

Anyhow, as I see it in both cases, the issues may be around function for mem management. Similar to openssl, we register mem management functions to curl too. I suspect it may be same problem as for openssl.

Have you tried with another, maybe older libcurl version ?

[chrismanivong]

@bogdan-iancu , yes sorry for confusion. My colleague Yrjö told me, after he had a first look onto the code, that it might be possible, that the shared allocation is dropped from the first, meanwhile the second is trying to allocate it again or something like that. I tried different curl versions from 7.29 up to 7.43. Same behavior everywhere.
The async version of the rest_client call will be executed in the curl library, we can see the request at the https server. if we slow down the client, which normaly asks as second time, we can see the first task hanging in opensips. The call back or resume route wouldn't be called.

[bogdan-iancu]

You can compile the memory debugging (see here how to do it http://www.opensips.org/Documentation/TroubleShooting-OutOfMem). Hopefully some additional info may pop up during the crash.
Let me know if you managed to compiled the mem debugger and to get the debug logs at runtime.

[bogdan-iancu]

Hi @krys1976, @liviuchircu run several tests with rest_client and async support (including memory debugging support) and everything seems to be working fine.
I would say the problems are strongly depending of your particular setup - what OS and version are you using ? what;s the rest query you are running ? I'm looking to any information that might help us in reproducing your setup.

[liviuchircu]

Fixed by ed8984c. Resume routes will be always run now.