Skip to content

Conversation

@arkadiuszbach
Copy link
Contributor

@arkadiuszbach arkadiuszbach commented Nov 26, 2025

What
Derive celery sync_parallelism from the scheduler's resources.limits.cpu when it is defined.

Why
The default value of sync_parallelism in Airflow is 0, which causes it to fall back to multiprocessing.cpu_count:

self._sync_parallelism = max(1, cpu_count() - 1)

In containerized environments, cpu_count() returns the number of host machine cores, rather than the number of cores actually allocated to the container - see: python/cpython#80235

This leads to incorrect behavior. For example:
If the scheduler container has 500m CPU allocated but is running on a 16 vCPU node, Airflow will incorrectly spawn 15 (as there is minus 1) processes instead of 1 when sending Celery tasks via apply_async and when there are >= 15 tasks in queued state, leading to unnecessary resource contention and overhead.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Copy link
Contributor

@jscheffl jscheffl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Besides one sanity question about the parameter you are adding ( I assume you want to address worker_concurrency!) it would be good to add some more documentation as well.

@jscheffl jscheffl added this to the Airflow Helm Chart 1.19.0 milestone Dec 7, 2025
@jscheffl jscheffl merged commit 31e4381 into apache:main Dec 7, 2025
86 checks passed
amoghrajesh pushed a commit to astronomer/airflow that referenced this pull request Dec 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:helm-chart Airflow Helm Chart

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants