[BUG] Possible deadlock on opensips cgrates module

[abisaim]

OpenSIPS version you are running

version: opensips 3.2.2 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
main.c compiled on 17:29:54 Aug 19 2021 with gcc 8

Describe the bug

To Reproduce

Expected behavior

Relevant System Logs

OS/environment information

• Operating System: <!-cent os 8 ) -->
• OpenSIPS installation:
• other relevant information:

Additional context

[razvancrainea]

Please provide more information about the issue.

[abisaim]

Opensips 3.2.2 is running with opensips cgrates module communicate with cgrates.
On every invite opensips calls below block
if (!async(cgrates_auth("$fU", "$rU"), resume_cgr_auth)) {
sl_send_reply(503, "Service Unavailable");
exit;
}

....
route [resume_cgr_auth] {
......
......

cgrates_acc("cdr", "$fU", "$rU");

}

When opensips gets more user invites, CPU shoots per each worker and doesn't return to normal after users stop calling

the trap per the 100% processes have below :

--start 71528 (opensips 71528 71475 14 06:27 ? 00:02:08 /usr/sbin/opensips -P /run/opensips/opensips.pid -f /etc/opensips/opensips.cfg -m 64 -M 4)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f21509db9b2 in read () from /lib64/libpthread.so.0
#0 0x00007f21509db9b2 in read () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007f2143df5506 in cgrc_async_read () from /usr/lib64/opensips/modules/cgrates.so
No symbol table info available.
#2 0x00007f2143df5d2e in cgr_handle_cmd () from /usr/lib64/opensips/modules/cgrates.so
No symbol table info available.
#3 0x00007f2143de092e in cgr_dlg_callback () from /usr/lib64/opensips/modules/cgrates.so
No symbol table info available.
#4 0x00007f214aeca55d in run_dlg_callbacks () from /usr/lib64/opensips/modules/dialog.so
No symbol table info available.
#5 0x00007f214aec1828 in dual_bye_event () from /usr/lib64/opensips/modules/dialog.so
No symbol table info available.
#6 0x00007f214aec2e15 in dlg_send_dual_bye () from /usr/lib64/opensips/modules/dialog.so
No symbol table info available.
#7 0x00007f214aec3efe in dlg_end_dlg () from /usr/lib64/opensips/modules/dialog.so
No symbol table info available.
#8 0x00007f214aef2d6c in terminate_dlg () from /usr/lib64/opensips/modules/dialog.so
No symbol table info available.
#9 0x00007f2143de7fa8 in cgr_acc_terminate () from /usr/lib64/opensips/modules/cgrates.so
No symbol table info available.
#10 0x00007f2143df4c4d in cgrates_process () from /usr/lib64/opensips/modules/cgrates.so
No symbol table info available.
#11 0x00007f2143df55de in cgrc_async_read () from /usr/lib64/opensips/modules/cgrates.so
No symbol table info available.
#12 0x00007f2143df63d4 in cgrates_async_resume_req () from /usr/lib64/opensips/modules/cgrates.so
No symbol table info available.
#13 0x000000000045d783 in async_fd_resume ()
No symbol table info available.
#14 0x000000000059bcdb in handle_io ()
No symbol table info available.
#15 0x000000000059e201 in tcp_worker_proc_loop ()
No symbol table info available.
#16 0x00000000005ad68b in tcp_start_processes ()
No symbol table info available.
#17 0x000000000041bc29 in main ()
No symbol table info available.
[Inferior 1 (process 71528) detached]

Only after restarting opensips does cpu usage for the opensips tcpworkers goes down

[abisaim]

Hello All ,

Trying to get more information as requested by @razvancrainea

It it correct to pick high cpu reason usage on a pid by using gdb -p and then type bt.

How do I gather information on why CPU remains high even after traffic is no longer coming. in debug i see DBG:dialog:destroy_dlg: dlg expired or not in list and on messages I see a bye returning a not found and wonder if related

currently gdb - p bt gives below

(gdb) bt
#0 0x00007fb0439e79b2 in read () from /lib64/libpthread.so.0
#1 0x00007fb037427506 in cgrc_async_read () from /usr/lib64/opensips/modules/cgrates.so
#2 0x00007fb037427d2e in cgr_handle_cmd () from /usr/lib64/opensips/modules/cgrates.so
#3 0x00007fb03741292e in cgr_dlg_callback () from /usr/lib64/opensips/modules/cgrates.so
#4 0x00007fb03ded655d in run_dlg_callbacks () from /usr/lib64/opensips/modules/dialog.so
#5 0x00007fb03decd828 in dual_bye_event () from /usr/lib64/opensips/modules/dialog.so
#6 0x00007fb03decee15 in dlg_send_dual_bye () from /usr/lib64/opensips/modules/dialog.so
#7 0x00007fb03decfefe in dlg_end_dlg () from /usr/lib64/opensips/modules/dialog.so
#8 0x00007fb03defed6c in terminate_dlg () from /usr/lib64/opensips/modules/dialog.so
#9 0x00007fb037419fa8 in cgr_acc_terminate () from /usr/lib64/opensips/modules/cgrates.so
#10 0x00007fb037426c4d in cgrates_process () from /usr/lib64/opensips/modules/cgrates.so
#11 0x00007fb0374275de in cgrc_async_read () from /usr/lib64/opensips/modules/cgrates.so
#12 0x00007fb0374283d4 in cgrates_async_resume_req () from /usr/lib64/opensips/modules/cgrates.so
#13 0x000000000045d783 in async_fd_resume ()
#14 0x000000000058efed in io_wait_loop_epoll.constprop ()
#15 0x0000000000594322 in udp_start_processes ()
#16 0x000000000041bc12 in main ()

[abisaim]

Please disregard this was a configuration loop.