Child worker process should send WORKER_UP message within 2 seconds

[anti-social]

In our project we have global data which loaded in Loader.on_worker_process_init method. Sometimes it takes more than 2 seconds. In that case when child process dies AsynPool starts new one and then kills it.

There are many messages in logs:

Timed out waiting for UP message from <Worker(Worker-6, started daemon)>

Moving data loading into worker_process_init signal handler doesn't help.
Now I moved loading data into Loader.on_task_init.

Maybe there should be option or setting which could set AsynPool._proc_alive_timeout attribute?
Or perhaps add information in docs that worker_process_init signal and Loader.on_worker_process_init should be executed within 2 seconds.

[ask]

There could be a worker_process_ready signal I guess, but optimally none of the signal handlers should be blocking for long.

[oppianmatt]

I'm getting this error but we don't have any signal handlers on init.

We get this if a task has exceeded it's time limit.

[oppianmatt]

Did some stracing, and it was just because we had so many python files that it was taking to long to startup. Managed to fix it with a monkey path in the settings file:

# try monkey patch startup timeout since we take longer than 4.0 seconds to startup
from celery.concurrency import asynpool
asynpool.PROC_ALIVE_TIMEOUT = 60.0

[bryanhelmig]

We hit this too @ask - it might be pretty handy to have this as a configuration option. Lots of head scratching as a result...

[kotrakrishna]

@oppianmatt Thanks for the monkey patch script.

[jpays]

In case someone is still having that kind of problem, you can upgrade to 4.4.2 and use the new configuration option "worker_proc_alive_timeout" to increase init timeout.

[michaelosthege]

It looks like worker_proc_alive_timeout does not affect the AsynPool? maybe I'm missing something, but according to the search results there's no code linking proc_alive_timeout to the PROC_ALIVE_TIMEOUT constant that was monkeypatched in the comment above.

This was not our actual issue (in the context of an Apache Airflow Celery setup), but I think my comment is still valid.