segfault on fm_malloc

[flaviogoncalves]

Hi,

I had tens of segfaults after updating to the latest opensips 1.8. At the same time I have also increased the query_buffer_size to 500. So one of them is causing the issues. I already decreased to 200, the previous value. The backtrace follows:

http://pastebin.com/CcQTP9VJ
http://pastebin.com/C0m26WJh
http://pastebin.com/ZynRBGAK

[flaviogoncalves]

I found this error too in the log. Seems to be related to the failure:

ERROR:db_mysql:db_mysql_val2str: destination buffer too short
Jul 7 08:19:53 sip /sbin/opensips[11279]: ERROR:core:db_print_values: Error while converting value to string
Jul 7 08:19:53 sip /sbin/opensips[11279]: ERROR:core:db_do_insert: error while preparing insert operation
Jul 7 08:19:53 sip /sbin/opensips[11279]: ERROR:core:ql_timer_routine: failed to insert rows to DB
Jul 7 08:20:03 sip /sbin/opensips[11181]: ERROR:core:db_do_insert: error while submitting query
Jul 7 08:20:03 sip /sbin/opensips[11181]: ERROR:core:flush_query_list: failed to insert into DB

08:20:03 sip /sbin/opensips[11181]: CRITICAL:db_mysql:wrapper_single_mysql_real_query: driver error (1048): Column 'method' cannot be null
Jul 7 08:21:03 sip /sbin/opensips[11181]: CRITICAL:core:sig_alarm_abort: BUG - shutdown timeout triggered, dying...

[vladpaiu]

Hello,

If you keep the current version of 1.8, but reduce the size of the query buffering, do you still get crashes ?

From the looks of the error logs you pasted, it seems that setting query_buffer_size to 500 leads to the OpenSIPS internal buffer for one query ( which is statically allocated to 65536 ) getting filled up since it is not enough to fit that many rows.
Still, it should not crash - probably there is a miss-treated error case somewhere in the query buffering part, which lead to a memory corruption.

As a first step, I'd recommend either reducing the query_buffer_size, or if reducing the value is not an option, edit db/db_ut.h and change SQL_BUF_LEN to a higher value and re-compile and re-install your OpenSIPS.
That should ( hopefully ) stop OpenSIPS from crashing.

In order to actually trace the bug, would it be possible to ( privately ) give us access to the core files, so we can look more into how the memory got corrupted ?

Best Regards,
Vlad

[flaviogoncalves]

Hi Vlad,

I've reduced to 200 yesterday, but we still had a failure later in the
night (with an small load, maybe a process or report running together).
Today I've reduced even more to 100. We have a billing process that grabs
the acc files to another database everyday and it is important to run 500
lines at the same time because we can avoid using all opensips children
processes during the day (load: 100).

** It is a production server, please be careful **

IP Address and password will be sent in your email

Corefiles from yesterday are in /home/corefiles. The ones I sent you are in
/tmp. I had to delete all of the other because of disk space.

Thanks for your help,

Flavio E. Goncalves
CEO - V.Office Redes e Telecomunicações Ltda.
Phone: +55 48-3025-8590, +1 248-688-0960
VoIP: SIP:flavio@opensips.org,Skype:flaviogoncalves1
Linkedin: www.linkedin.com/in/flavioegoncalves
Twitter: www.twitter.com/asteriskguide

2013/7/8 vladpaiu notifications@github.com

Hello,

If you keep the current version of 1.8, but reduce the size of the query
buffering, do you still get crashes ?

From the looks of the error logs you pasted, it seems that setting
query_buffer_size to 500 leads to the OpenSIPS internal buffer for one
query ( which is statically allocated to 65536 ) getting filled up since it
is not enough to fit that many rows.
Still, it should not crash - probably there is a miss-treated error case
somewhere in the query buffering part, which lead to a memory corruption.

As a first step, I'd recommend either reducing the query_buffer_size, or
if reducing the value is not an option, edit db/db_ut.h and change
SQL_BUF_LEN to a higher value and re-compile and re-install your OpenSIPS.
That should ( hopefully ) stop OpenSIPS from crashing.

In order to actually trace the bug, would it be possible to ( privately )
give us access to the core files, so we can look more into how the memory
got corrupted ?

Best Regards,
Vlad

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9#issuecomment-20616383
.

[flaviogoncalves]

Hi Vlad,

I had the same issue with 100 in the query buffer size. It seems to get more stable if I decrease the query buffer, but still happens.

Flavio

[flaviogoncalves]

Additional Information. We had new crash with query_buffer_size=100 and it seems to be related to a siptrace. When a customer activates a SIPTRACE with high volume. I think a 100 siptrace requests depending on the size of the request can overflow the buffer.