Airflow 3.0.6 - apache-airflow-providers-celery 3.12.2 - Process 'ForkPoolWorker-' exited with 'signal 9 (SIGKILL)

[hss-rmatthews]

Hi All,

We are running Airflow version 3.0.6 in MWAA and have a task failing without any logs showing in Airflow.

The task log shows:
An error occurred (ResourceNotFoundException) when calling the GetLogEvents operation: The specified log stream does not exist.

However, when investigating the worker logs we are finding the following entries:

[error] Timed out waiting for UP message from <ForkProcess(ForkPoolWorker-245, started daemon)> [celery.concurrency.asynpool]
[ERROR/MainProcess] Process 'ForkPoolWorker-245' pid: exited with 'signal 9 (SIGKILL)'

This particular task is using the S3ToRedshiftOperator and I have verified that the command is actually present and in our Redshift logs and it has succeeded.

The spec of the environment in MWAA is: mw1.medium with a maximum worker count of 20. We currently only have a single DAG running once a day (as we are in the process of migrating from version 2.10.3).

Thanks

Rob