Leak in Kubernetes Executor Running Tasks Slot Count

[dirrao]

Apache Airflow version

main (development)

What happened

Schedulers are racing for pod adoption when there is a delay in schedulers' heartbeats. However, the schedulers are alive but not dead their heartbeat is delayed due to network timeout or heavy processing, etc. This leads to a leak in the executor.running_tasks slots. Eventually, the schedulers are not able to launch the pods due to executor.running_tasks=parallelism.

What you think should happen instead

We should remove the entry from the Kubernetes executor running queue when we worker pod deleted / moved to another scheduler.

How to reproduce

Reduce the scheduler_health_check_threshold=5 and orphaned_tasks_check_interval=10 values in the airflow config file
Launch the airflow with two schedulers and try to schedule multiple DAGs with backfill for every 1/5 mins.

Operating System

CentOS 6

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes=7.9.0

Deployment

Other Docker-based deployment

Deployment details

Terraform

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[c-thiel]

We had the same problem and only noticed it from Airflow 2.7.3, but it was there before.
Maybe this helps for debugging:

On the left of the red line we had Airflow 2.6.X
On the right side is the update to 2.7.3.

Each line is one scheduler.

Before the update, scheduler open slots continuously went down to even below zero sometimes, but only for some executors.
After the update this is much more distributed between all the executors.

Our Workaround
What helped us is limit the parallelism via pools, and just set AIRFLOW__CORE__PARALLELISM to an arbitrary high number. Also regular deployment restarts helps.

[dirrao]

@c-thiel
Yes. This workaround works and we are following the same. I have identified the leaks and am going to open the MR soon.