Enable Airflow DAG processor horizontally scalable

[dirrao]

Description

Right now, the airflow DAG processor running in standalone or non standalone mode. Running multiple instances of the DAG processor does duplicate work. It's good idea to run the DAG processor in distributed mode to share the parsing workload.

Use case/motivation

Running the Airflow at a large scale requires running all the components of the airflow in distributed mode. Right now, the webserver and scheduler can able to scale horizontally. Right now, the DAG processor is vertically scalable but not horizontally scalable. This feature request is to make the DAG processor horizontally scalable.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Yep. It's an intresting feature. It would be great if someone takes it and implement it. Nothing against such feature, it just needs someone to do it.

[pegasas]

Hi, @potiuk , @dirrao

I am also following and interested in some related work which is similar to this item.

I am willing to submit a PR of this feature.

Count me in if you don't mind.

[potiuk]

Sure @pegasas !

[raphaelauv]

You can manually horizontally scale the DAG processor by using the subdir option https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#dag-processor.

I agree that an automatic distribution of the files to parse (pondered by the number of dags present in each file) to every up and running dag processes would be nice

[dirrao]

Yes. We can do that for the time being. However, we have to work on multiple external things (manual segregation of DAGs, HA, etc.). It would be nice to handle this out of the box.

[pegasas]

Hi, @dirrao , @raphaelauv , @potiuk

I have one performance concern, maybe it is naive:

Our architecture only allows one scheduler to access meta database,
Will horizontal DAG processors be worsening DB lock contention?
Especially currently we use "select ... for update"?

Indeed, DAG Processing is a CPU-Bound job and Python parallelism is limited by GIL.
Is there any other evidence showing that multiple singletons are our bottleneck of Airflow?

Correct me if I am wrong.

[potiuk]

No. Dag FileProcessor generally does not manipulate Dag Runs and Task instance - and those are the ones that locks are mainly used for. Creating DagRuns and queuing tasks is all happening in schduler.

We are also heavily using multiprocessing (Especially for Dag FileProcessor) - basically each file is processed in a new forked python process, so gil and singletons have nothing to do with it.

BTW. Horizontal scaling of DAG file processor would really be needed if your Python Parsing gets a lot of CPU cycles. And mostly when you have big number of smal Python files.

[stephen-bracken]

Hi all, is this still a relevant issue? With the introduction of dag bundles / AIP-67 it makes sense to me to have a deployment pattern where we have one dag processor per dag bundle, which would help to isolate teams from noisy neighbour issues caused by parsing dags in other teams. I am interested in the feature regardless because I have had issues with teams creating very heavy (dynamic) dags in the past, so being able to scale horizontally would be a good addition here.