Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Memory leak in OpenSIPS 3.4.0 - apparently related to dialog variables #3135

Closed
StellaTeam opened this issue Jul 27, 2023 · 9 comments
Closed

Comments

@StellaTeam
Copy link

Hello

Since we upgraded from 3.1.3 to 3.4.0 we experience memory leaks.

Details follows.

Please let me know how I can help diagnosing the problem.

Thank you!

OpenSIPS version you are running

version: opensips 3.4.0 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: f3e0d5333
main.c compiled on  with gcc 8

Describe the bug
Since upgrade from 3.1.3 to 3.4.0, there is a memory leak on some threads.

To Reproduce
As it is a leak, there is only to wait for memory usage to increase and free memory to decrease

Expected behavior
No memory leak

Relevant System Logs
Once memory is saturated we get all kinds of errors we can imagine before crashing.
Messages follows

 Jul 27 10:24:50 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (352 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:24:50 [25054] ERROR:core:fm_malloc: not enough free pkg memory (352 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:24:51 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (312 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:24:51 [25054] ERROR:core:fm_malloc: not enough free pkg memory (312 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:24:51 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (320 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:24:51 [25054] ERROR:core:fm_malloc: not enough free pkg memory (320 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:24:56 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (224 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:24:56 [25054] ERROR:core:fm_malloc: not enough free pkg memory (224 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:24:57 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (344 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:24:57 [25054] ERROR:core:fm_malloc: not enough free pkg memory (344 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:01 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1216 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:01 [25054] ERROR:core:fm_malloc: not enough free pkg memory (1216 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:03 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1512 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:03 [25054] ERROR:core:fm_malloc: not enough free pkg memory (1512 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:04 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (824 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:04 [25054] ERROR:core:fm_malloc: not enough free pkg memory (824 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:07 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1160 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:07 [25054] ERROR:core:fm_malloc: not enough free pkg memory (1160 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:07 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (2224 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:07 [25054] ERROR:core:fm_malloc: not enough free pkg memory (2224 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:07 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1400 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:07 [25054] ERROR:core:fm_malloc: not enough free pkg memory (1400 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:09 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (144 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:09 [25054] ERROR:core:fm_malloc: not enough free pkg memory (144 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:09 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (464 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:09 [25054] ERROR:core:fm_malloc: not enough free pkg memory (464 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:10 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1600 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:10 [25054] ERROR:core:fm_malloc: not enough free pkg memory (1600 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:11 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (840 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:11 [25054] ERROR:core:fm_malloc: not enough free pkg memory (840 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:11 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (608 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:11 [25054] ERROR:core:fm_malloc: not enough free pkg memory (608 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:13 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (736 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:13 [25054] ERROR:core:fm_malloc: not enough free pkg memory (736 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:14 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (520 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:14 [25054] ERROR:core:fm_malloc: not enough free pkg memory (520 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:14 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (16 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:14 [25054] ERROR:core:fm_malloc: not enough free pkg memory (16 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:15 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (144 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:15 [25054] ERROR:core:fm_malloc: not enough free pkg memory (144 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:25:17 [25054] WARNING:core:fm_malloc: not enough contiguous free pkg memory (320 bytes left, need 1000), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:25:17 [25054] ERROR:core:fm_malloc: not enough free pkg memory (320 bytes left, need 1000), please increase the "-M" command line parameter!
 Jul 27 10:51:51 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1192 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:51:51 [25058] ERROR:core:fm_malloc: not enough free pkg memory (1192 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:05 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1088 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:05 [25058] ERROR:core:fm_malloc: not enough free pkg memory (1088 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:06 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (2880 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:06 [25058] ERROR:core:fm_malloc: not enough free pkg memory (2880 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:11 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (2056 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:11 [25058] ERROR:core:fm_malloc: not enough free pkg memory (2056 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:11 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1480 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:11 [25058] ERROR:core:fm_malloc: not enough free pkg memory (1480 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:11 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1288 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:11 [25058] ERROR:core:fm_malloc: not enough free pkg memory (1288 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:13 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1856 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:13 [25058] ERROR:core:fm_malloc: not enough free pkg memory (1856 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:16 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (736 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:16 [25058] ERROR:core:fm_malloc: not enough free pkg memory (736 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:17 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1048 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:17 [25058] ERROR:core:fm_malloc: not enough free pkg memory (1048 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:18 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (464 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:18 [25058] ERROR:core:fm_malloc: not enough free pkg memory (464 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:18 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (672 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:18 [25058] ERROR:core:fm_malloc: not enough free pkg memory (672 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:18 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (672 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:18 [25058] ERROR:core:fm_malloc: not enough free pkg memory (672 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:19 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (136 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:19 [25058] ERROR:core:fm_malloc: not enough free pkg memory (136 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:20 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (136 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:20 [25058] ERROR:core:fm_malloc: not enough free pkg memory (136 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:20 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (296 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:20 [25058] ERROR:core:fm_malloc: not enough free pkg memory (296 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:21 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (440 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:21 [25058] ERROR:core:fm_malloc: not enough free pkg memory (440 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:21 [25058] ERROR:rtpengine:parse_flags: error processing flag `': no more memory
 Jul 27 10:52:21 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1032 bytes left, need 1136), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:21 [25058] ERROR:core:fm_malloc: not enough free pkg memory (1032 bytes left, need 1136), please increase the "-M" command line parameter!
 Jul 27 10:52:21 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (2464 bytes left, need 2472), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:21 [25058] ERROR:core:fm_malloc: not enough free pkg memory (2464 bytes left, need 2472), please increase the "-M" command line parameter!
 Jul 27 10:52:21 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1336 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:21 [25058] ERROR:core:fm_malloc: not enough free pkg memory (1336 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (504 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (504 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (600 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (600 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:rtpengine:rtpe_function_call: out of memory - bencode failed
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (64 bytes left, need 784), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (64 bytes left, need 784), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (360 bytes left, need 2472), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (360 bytes left, need 2472), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (200 bytes left, need 160), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (200 bytes left, need 160), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:parse_sip_body: Unable to allocate memory
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (200 bytes left, need 168), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (200 bytes left, need 168), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (856 bytes left, need 1024), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (856 bytes left, need 1024), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:rewrite_ruri: memory allocation failure
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (208 bytes left, need 352), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (208 bytes left, need 352), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (856 bytes left, need 2472), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (856 bytes left, need 2472), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (616 bytes left, need 1008), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (616 bytes left, need 1008), please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (520 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:22 [25058] ERROR:core:fm_malloc: not enough free pkg memory (520 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 56), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 56), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:insert_new_lump_before: out of pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 32), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 32), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (16 bytes left, need 160), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (16 bytes left, need 160), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:parse_sip_body: Unable to allocate memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (16 bytes left, need 56), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (16 bytes left, need 56), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:parse_params: no pkg memory left
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (16 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (16 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (16 bytes left, need 24), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (16 bytes left, need 24), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (16 bytes left, need 24), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (16 bytes left, need 24), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (16 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (16 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (856 bytes left, need 1024), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (856 bytes left, need 1024), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:rewrite_ruri: memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (856 bytes left, need 1024), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (856 bytes left, need 1024), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:rewrite_ruri: memory allocation failure
 Jul 27 10:52:23 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 16), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 16), please increase the "-M" command line parameter!
 Jul 27 10:52:23 [25058] CRITICAL:core:hostent_cpy: pkg memory allocation failure
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (264 bytes left, need 600), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (264 bytes left, need 600), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (48 bytes left, need 56), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (48 bytes left, need 56), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:insert_subst_lump_after: out of pkg memory
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 24), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 24), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:24 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:24 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 8), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 8), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] CRITICAL:core:comp_s2s: pkg memory allocation failure
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 16), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 16), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (32 bytes left, need 48), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (32 bytes left, need 48), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:parse_content_type_hdr: Unable to allocate memory
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (32 bytes left, need 56), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (32 bytes left, need 56), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:parse_params: no pkg memory left
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (32 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (32 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 56), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 56), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:insert_new_lump_before: out of pkg memory
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 56), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 56), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:insert_new_lump_before: out of pkg memory
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (32 bytes left, need 56), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (32 bytes left, need 56), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:eval_elem: no more pkg memory
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (56 bytes left, need 1024), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (56 bytes left, need 1024), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:rewrite_ruri: memory allocation failure
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (56 bytes left, need 1024), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (56 bytes left, need 1024), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:rewrite_ruri: memory allocation failure
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (56 bytes left, need 352), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (56 bytes left, need 352), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 350
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (56 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (56 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] CRITICAL:core:mk_proxy: pkg memory allocation failure
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (56 bytes left, need 416), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (56 bytes left, need 416), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 410
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (56 bytes left, need 416), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (56 bytes left, need 416), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:build_res_buf_from_sip_req: out of pkg memory; needs 410
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (440 bytes left, need 536), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (440 bytes left, need 536), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:rtpengine:rtpe_function_call: out of memory - bencode failed
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (136 bytes left, need 560), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (136 bytes left, need 560), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (320 bytes left, need 360), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (320 bytes left, need 360), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (1032 bytes left, need 2472), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (1032 bytes left, need 2472), please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (144 bytes left, need 552), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:25 [25058] ERROR:core:fm_malloc: not enough free pkg memory (144 bytes left, need 552), please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 56), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 56), please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:insert_new_lump_after: out of pkg memory
 Jul 27 10:52:26 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 24), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 24), please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (0 bytes left, need 64), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:fm_malloc: not enough free pkg memory (0 bytes left, need 64), please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:parse_via: no pkg memory left
 Jul 27 10:52:26 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (208 bytes left, need 160), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:fm_malloc: not enough free pkg memory (208 bytes left, need 160), please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:parse_sip_body: Unable to allocate memory
 Jul 27 10:52:26 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (32 bytes left, need 56), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:fm_malloc: not enough free pkg memory (32 bytes left, need 56), please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:insert_new_lump_before: out of pkg memory
 Jul 27 10:52:26 [25058] WARNING:core:fm_malloc: not enough contiguous free pkg memory (2168 bytes left, need 872), attempting defragmentation... please increase the "-M" command line parameter!
 Jul 27 10:52:26 [25058] ERROR:core:fm_malloc: not enough free pkg memory (2168 bytes left, need 872), please increase the "-M" command line parameter!
 Jul 27 10:52:38 [2356] WARNING:core:init_reactor_size: shrinking reactor size from 262144 (autodetected via rlimit) to 83886 (limited by memory of 10% from 32Mb)
 Jul 27 10:52:38 [2356] WARNING:core:init_reactor_size: use 'open_files_limit' to enforce other limit or increase pkg memory

OS/environment information
DEBIAN buster 4.19.0-17 or DEBIAN bullseye 5.10.0-8

Additional context

The problem occurs on several machines.
The leak wasn't there on OpenSIPS 3.1.3.
Config is identical (except syslog parameters naming changes)

It occurs with default memory allocator (F_MALLOC if i'm right) but also with Q_MALLOC_DBG.

Not all threads seems to be affected by the leak the same way as, on 25 threads, 2 have greater leaks than the others.
I don't know why but, on both servers, the most affected threads are 12 and 16.

PID	fragments	free_size	max_used_size	real_used_size	total_size	used_size
global	7252		1067946472		10225064		5795352		1073741824		4841632
1	6088		27464680		6089752		6089752		33554432		5286848
2	6091		27468312		6086232		6086120		33554432		5282928
5	22346		25263448		8349976		8290984		33554432		5924528
6	12901		26544264		7051440		7010168		33554432		5550240
7	48079		21794848		11789752		11759584		33554432		6922664
8	93858		15634816		18012408		17919616		33554432		8688008
9	16252		26130776		7481248		7423656		33554432		5641840
10	33837		23826312		9791888		9728120		33554432		6258144
11	74708		18514208		15095536		15040224		33554432		7646824
12	155532		7943504		25665664		25610928		33554432		10457656
13	17583		25992904		7613656		7561528		33554432		5652320
14	37309		23468688		10142376		10085744		33554432		6282552
15	74702		18642712		15001496		14911720		33554432		7518224
16	159220		7790016		25817768		25764416		33554432		10258440
17	12272		26663384		6950408		6891048		33554432		5491312
18	22230		25381056		8234192		8173376		33554432		5817864
19	44683		22480760		11129576		11073672		33554432		6562384
20	86731		17099320		16524656		16455112		33554432		7907504
21	6094		27464752		6137400		6089680		33554432		5285912
22	6108		27460256		6141640		6094176		33554432		5288968
23	6096		27461672		6137360		6092760		33554432		5288704
24	6102		27466096		6097160		6088336		33554432		5283512
25	6104		27462848		6100264		6091584		33554432		5286472

recently crashed server - values are lower but difference is already visible, and difference between 12/16 and the other threads is clear

PID	fragments	free_size	max_used_size	real_used_size	total_size	used_size
global	5711		1068683448		9159536		5058376		1073741824		4674952
1	6090		27896504		5657928		5657928		33554432		5283128
2	6087		27900928		5654032		5653504		33554432		5278872
5	12510		27302320		6300192		6252112		33554432		5516056
6	17110		26877136		6738040		6677296		33554432		5683976
7	22605		26379960		7213976		7174472		33554432		5873040
8	34898		25256424		8388600		8298008		33554432		6308616
9	16225		27008576		6644008		6545856		33554432		5601592
10	22343		26476456		7168168		7077976		33554432		5791216
11	33995		25459536		8182248		8094896		33554432		6155680
12	51370		23943592		9710864		9610840		33554432		6698400
13	14701		27180984		6444824		6373448		33554432		5514976
14	21132		26632152		6993904		6922280		33554432		5703336
15	32637		25665160		7937544		7889272		33554432		6025992
16	53657		23903400		9705656		9651032		33554432		6610968
17	11151		27479264		6134848		6075168		33554432		5415160
18	14593		27194568		6432992		6359864		33554432		5507160
19	30829		25867016		7765104		7687416		33554432		5925384
20	20486		26711088		6919040		6843344		33554432		5660632
21	6084		27901256		5656696		5653176		33554432		5278656
22	6084		27901256		5658712		5653176		33554432		5278656
23	6084		27901256		5659056		5653176		33554432		5278656
24	6084		27901256		5659056		5653176		33554432		5278656
25	6084		27901256		5659056		5653176		33554432		5278656

According to memory dumps (attached to this issue), it seems the memory is lost in

WARNING:    4739600 : 136948 x [dlg_vals.c: fetch_dlg_value, line 191]

mem2.txt
mem1.txt

@StellaTeam
Copy link
Author

StellaTeam commented Jul 27, 2023

Hello,

I made more investigations about the issue as the crashes are coming fast.
I try to find differences with previous changes (and there are a lot).

I also checked again memory allocation debug and, except dlg_vals, the more memory consuming module is io_wait.c.
So, as I found a new commit ad6aef0
that is relative to locking and deadlocks.

As i have monitoring using APIs, i wonder if the crashes i experience are not relative to a combination of:

  • memory allocations
  • memory locking
  • memory not freed because of locks

I wonder if both problems might not be linked.

Thank you for helping

mem1bb.txt

@StellaTeam
Copy link
Author

Hello,

I made two things:

  • downgrade to 3.3.6 on one of the servers + increasing PKG (to delay the crashs if still present)
  • increasing PKG to delay the crashs on the other one, staying on 3.4

On 3.3.6, the leak is no more there, memory stats are very clean.
On 3.4, when compared, that's really dirty, as, both servers have the same configuration.

Below are there respective stats, that's obvious :-)

PID	fragments	free_size	max_used_size	real_used_size	total_size	used_size
global	19803		1064699976		10265560		9041848		1073741824		6831584
1	6081		521517576		15353328		15353336		536870912		14551104
2	6078		521522120		15349480		15348792		536870912		14546848
5	6132		521505936		15505032		15364976		536870912		14555640
6	6132		521505528		15490992		15365384		536870912		14555664
7	6132		521505352		15545440		15365560		536870912		14555744
8	6131		521505568		15535480		15365344		536870912		14555912
9	6132		521505696		15546104		15365216		536870912		14555496
10	6130		521505640		15533016		15365272		536870912		14555168
11	6130		521505936		15534808		15364976		536870912		14555352
12	6132		521505864		15553624		15365048		536870912		14555232
13	6129		521505800		15523784		15365112		536870912		14555488
14	6131		521506032		15559544		15364880		536870912		14555352
15	6131		521505392		15509536		15365520		536870912		14555512
16	6130		521505864		15521696		15365048		536870912		14555424
17	6112		521509472		15478216		15361440		536870912		14554408
18	6114		521508808		15504480		15362104		536870912		14554496
19	6115		521508984		15513608		15361928		536870912		14554608
20	6115		521508784		15506776		15362128		536870912		14554424
21	6075		521522528		15355344		15348384		536870912		14546632
22	6075		521522528		15355184		15348384		536870912		14546632
23	6075		521522528		15355344		15348384		536870912		14546632
24	6075		521522528		15355440		15348384		536870912		14546632
25	6088		521514656		15400240		15356256		536870912		14552968
PID	fragments	free_size	max_used_size	real_used_size	total_size	used_size
global	21212		1064008848		10311328		9732976		1073741824		7376984
1	6088		521511744		15359168		15359168		536870912		14556264
2	6091		521515376		15355648		15355536		536870912		14552344
5	28772		518455824		18464072		18415088		536870912		15431352
6	10425		520924072		16007336		15946840		536870912		14724800
7	16998		520038896		16890568		16832016		536870912		14979160
8	61956		513974728		22953216		22896184		536870912		16726976
9	12560		520658264		16287544		16212648		536870912		14785648
10	22651		519344944		17586920		17525968		536870912		15129944
11	47369		516118720		20805680		20752192		536870912		15982856
12	97009		509642376		27295256		27228536		536870912		17693472
13	13197		520593440		16331568		16277472		536870912		14789224
14	98207		509669792		27256456		27201120		536870912		17552488
15	25242		519059568		17874192		17811344		536870912		15166776
16	48522		516064952		20861896		20805960		536870912		15926608
17	10209		520976888		15947856		15894024		536870912		14692528
18	16164		520206736		16739520		16664176		536870912		14891000
19	31457		518234856		18704808		18636056		536870912		15394560
20	56756		514994040		21960024		21876872		536870912		16206576
21	6096		521508728		15409808		15362184		536870912		14558128
22	6086		521513248		15361976		15357664		536870912		14554760
23	6083		521516576		15361392		15354336		536870912		14551816
24	6116		521505920		15412424		15364992		536870912		14558920
25	6102		521507976		15410528		15362936		536870912		14558304

Thank you

@liviuchircu liviuchircu added this to the 3.4.1 milestone Jul 28, 2023
@liviuchircu
Copy link
Member

Hello, @StellaTeam!

Thank you for the report! I managed to reproduce the memory leak fairly easily -- I suspect the issue is rooted down in the $dlg_val changes from a while back ago, but it could also be something else.

Let me see what a fix would look like and come back to you.

@guestus
Copy link

guestus commented Aug 29, 2023

There is the same problem.
On 3.4 some processes have memory leaks.
Next 3 requests in 10 hours with less than 95% free memory:

Tue Aug 29 12:52:50 2023
SIP receiver udp:192.168.xxx.yyy:5060[17]
free=89.617%
..total_size: 2147483648, used_size: 76946424, real_used_size: 222974816, max_used_size: 223088472, free_size: 1924508832, fragments: 2607001

Tue Aug 29 13:55:53 2023
SIP receiver udp:192.168.xxx.yyy:5060[17]
free=88.045%
..total_size: 2147483648, used_size: 87139824, real_used_size: 256733968, max_used_size: 256824416, free_size: 1890749680, fragments: 3027816

Tue Aug 29 14:50:16  2023
SIP receiver udp:192.168.xxx.yyy:5060[16]
free=89.288%
..total_size: 2147483648, used_size: 79075656, real_used_size: 230039160, max_used_size: 230150712, free_size: 1917444488, fragments: 2695130
SIP receiver udp:192.168.xxx.yyy:5060[17]
free=86.765%
..total_size: 2147483648, used_size: 95410768, real_used_size: 284215040, max_used_size: 284317752, free_size: 1863268608, fragments: 3370853

increasing the P_MEMORY prolongs the life of the process before a crash.

@guestus
Copy link

guestus commented Sep 7, 2023

mem_pkg_dump :

Memory status (pkg):
fm_status (0x7fd320960010):
 heap size= 2147483648
 used= 184914224, used+overhead=566213288, free=1962569424
 max used (+overhead)= 566301264
dumping free list:
hash =   1 fragments no.:     1, unused:     0#012#011#011 bucket size:         8 -         8 (first         8)
hash =   4 fragments no.:     1, unused:     0#012#011#011 bucket size:        32 -        32 (first        32)
hash =   6 fragments no.:     4, unused:     0#012#011#011 bucket size:        48 -        48 (first        48)
hash =   7 fragments no.:     1, unused:     0#012#011#011 bucket size:        56 -        56 (first        56)
hash =   8 fragments no.:     2, unused:     0#012#011#011 bucket size:        64 -        64 (first        64)
hash =   9 fragments no.:     8, unused:     0#012#011#011 bucket size:        72 -        72 (first        72)
hash =  10 fragments no.:     1, unused:     0#012#011#011 bucket size:        80 -        80 (first        80)
hash =  11 fragments no.:     5, unused:     0#012#011#011 bucket size:        88 -        88 (first        88)
hash =  15 fragments no.:     1, unused:     0#012#011#011 bucket size:       120 -       120 (first       120)
hash =  24 fragments no.:     1, unused:     0#012#011#011 bucket size:       192 -       192 (first       192)
hash =  30 fragments no.:     1, unused:     0#012#011#011 bucket size:       240 -       240 (first       240)
hash =  32 fragments no.:     1, unused:     0#012#011#011 bucket size:       256 -       256 (first       256)
hash =  33 fragments no.:     1, unused:     0#012#011#011 bucket size:       264 -       264 (first       264)
hash =  61 fragments no.:     1, unused:     0#012#011#011 bucket size:       488 -       488 (first       488)
hash =  67 fragments no.:     1, unused:     0#012#011#011 bucket size:       536 -       536 (first       536)
hash = 114 fragments no.:     1, unused:     0#012#011#011 bucket size:       912 -       912 (first       912)
hash = 115 fragments no.:     1, unused:     0#012#011#011 bucket size:       920 -       920 (first       920)
hash = 119 fragments no.:     1, unused:     0#012#011#011 bucket size:       952 -       952 (first       952)
hash = 132 fragments no.:     1, unused:     0#012#011#011 bucket size:      1056 -      1056 (first      1056)
hash = 179 fragments no.:     1, unused:     0#012#011#011 bucket size:      1432 -      1432 (first      1432)
hash = 309 fragments no.:     1, unused:     0#012#011#011 bucket size:      2472 -      2472 (first      2472)
hash = 2065 fragments no.:     1, unused:     0#012#011#011 bucket size: 1073741824 - 2147483648 (first 1581259008)
TOTAL:     37 free fragments = 1581270360 free bytes
TOTAL: 56 overhead
-----------------------------

@guestus
Copy link

guestus commented Sep 8, 2023

sorry for the dump without info.
for a time, after memory debug was on:

< INFO: used= 14033656, used+overhead=14458704, free=2133024944
---
> INFO: used= 14033936, used+overhead=14460040, free=2133023608

< INFO:       1568 : 82 x [dlg_vals.c: fetch_dlg_value, line 191]
---
> INFO:       1848 : 91 x [dlg_vals.c: fetch_dlg_value, line 191]

used memory was grown (example from 15 m)

@bogdan-iancu
Copy link
Member

The issue was found, @liviuchircu is actively working on a fix here.

@fedorna
Copy link

fedorna commented Sep 14, 2023

Just as a workarround in the cfg file, using the function fetch_dlg_value:

fetch_dlg_value("test_dlg",$var(test))

instead of using the pseudo var:

$var(test) = $dlg_val(test_dlg);

avoids de pkg leak

liviuchircu added a commit that referenced this issue Sep 15, 2023
Fixes #3135

(cherry picked from commit e2bfeda)
@guestus
Copy link

guestus commented Sep 18, 2023

Confirmed. No memory leak now.

sobomax pushed a commit to sippy/opensips that referenced this issue Dec 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants