[BUG] auto-scaling failure

[cc3283]

OpenSIPS version you are running

/usr/sbin/opensips -V

version: opensips 3.1.1 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 229ec07
main.c compiled on 19:29:24 Jan 8 2021 with gcc 9

Describe the bug

Hello,

Ran across this in my syslogs, so far I have not been able to determine if it’s actually affecting calls as our service is not based on live calls. Here is the full message:

Jul 16 08:25:49 opensips[553787]: CRITICAL:core:fork_dynamic_udp_process: cannot fork UDP process
Jul 16 08:25:49 opensips[553787]: ERROR:core:do_workers_auto_scaling: failed to fork new process for group 1 (current 5 procs)
Jul 16 08:37:26 opensips[553787]: CRITICAL:core:internal_fork: #12>>> no free process slot found while trying to fork again#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues#012

My profiling config:

auto_scaling_profile = PROFILE_A
scale up to 30 on 30% for 4 cycles within 5
scale down to 5 on 18% for 10 cycles

a low percentage of these messages a day, about ~200 out of ~5200000 Subscribe / Notify messages a day per site. Seeing these at both sites.

To Reproduce

Cause undetermined, and unable to reproduce at will.
Expected behavior

Relevant System Logs

Jul 16 08:25:49 opensips[553787]: CRITICAL:core:fork_dynamic_udp_process: cannot fork UDP process
Jul 16 08:25:49 opensips[553787]: ERROR:core:do_workers_auto_scaling: failed to fork new process for group 1 (current 5 procs)
Jul 16 08:37:26 opensips[553787]: CRITICAL:core:internal_fork: #12>>> no free process slot found while trying to fork again#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues#012

OS/environment information

• Operating System:
• OpenSIPS installation:
• other relevant information:
  Distributor ID: Ubuntu
  Description: Ubuntu 20.04.2 LTS
  Release: 20.04
  Codename: focal

5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Additional context

[bogdan-iancu]

Hi @cc3283, what do you think is more acceptable for you : (a) I provide a small patch to trigger a stop + coredump when the BUG happens, so I can investigate the core file (I should have all the info I need there) or (b) I provide a patch to try to print (without crashing opensips) as much info as possible when the BUG happens (but we will be limited to the printed info, so multiple iterations may be required here) ?

[cc3283]

Hello Bogdan, thanks for picking this up. Because I can't produce this in the LAB I believe option 'b' in production would be the best option for us right now.

[github-actions]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

[bogdan-iancu]

Hi @cc3283 , here is the patch:

diff --git a/pt.c b/pt.c
index 6af76a9f8..fe32fae9d 100644
--- a/pt.c
+++ b/pt.c
@@ -182,6 +182,7 @@ void set_proc_attrs( char *fmt, ...)
 
        /* for sure the process is running */
        pt[process_no].flags |= OSS_PROC_IS_RUNNING;
+       LM_NOTICE("AUTOSCALING: process %d is running now\n",process_no);
 }
 
 
@@ -203,6 +204,7 @@ void reset_process_slot( int p_id )
        pt[p_id].pg_filter = NULL;
        pt[p_id].desc[0] = 0;
        pt[p_id].flags = 0;
+       LM_NOTICE("AUTOSCALING: process %d not running anymore\n",p_id);
 
        pt[p_id].ipc_pipe[0] = pt[p_id].ipc_pipe[1] = -1;
        pt[p_id].ipc_sync_pipe[0] = pt[p_id].ipc_sync_pipe[1] = -1;
@@ -247,7 +249,11 @@ int internal_fork(char *proc_desc, unsigned int flags,
        for( ; new_idx<counted_max_processes ; new_idx++)
                if ( (pt[new_idx].flags&OSS_PROC_IS_RUNNING)==0 ) break;
        if (new_idx==counted_max_processes) {
-               LM_BUG("no free process slot found while trying to fork again\n");
+               LM_BUG("no free process slot found while trying to fork again, "
+                       "all %d running\n",counted_max_processes);
+               for( new_idx=1 ; new_idx<counted_max_processes ; new_idx++)
+                       LM_NOTICE("AUTOSCALING: process %d, pid %d, type %d, desc <%s>, flags %d\n",
+                               new_idx, pt[new_idx].pid, pt[new_idx].type, pt[new_idx].desc, pt[new_idx].flags );
                return -1;
        }
 
@@ -281,6 +287,7 @@ int internal_fork(char *proc_desc, unsigned int flags,
 
        pt[new_idx].pid = 0;
        pt[new_idx].flags = OSS_PROC_IS_RUNNING;
+       LM_NOTICE("AUTOSCALING: process %d is set as running\n",new_idx);
 
        if ( (pid=fork())<0 ){
                LM_CRIT("cannot fork \"%s\" process (%d: %s)\n",proc_desc,

The logs specific to the issue are tagged with "AUTOSCALING" and printed on "NOTICE" level (so be sure you log level is at least 2.
So, with the patch, get your opensips running and ideally try to preserve all the "AUTOSCALING" logs from the beginning to the point of the error/bug.
For any clarifications, just let me know.

[cc3283]

HI bogdan-iancu, I am still working on getting this patch put in.

[github-actions]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

[cc3283]

Not yet!

…

On Sat, Sep 11, 2021 at 2:39 AM github-actions[bot] < ***@***.***> wrote: Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2575 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4OTADQMN7YQ7TTFN2RMLLUBL2RTANCNFSM5AQFVOVA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.