UI - N+1 Calls in the Tasks Tab

[nclaeys]

Apache Airflow version

Other Airflow 3 version (please specify below)

If "Other Airflow 3 version" selected, which one?

3.1.3

What happened?

When browsing the Airflow UI and looking at the tasks tab, it triggers n+1 queries for every task in your dag. If your dag has 200+ tasks, it does 200+ queries which should not be the case. The first 30 might succeed but afterwards the calls fail due to a sqlalchemy timeout.

problematic UI path: <airflowBaseUrl>/airflow/dags/<some-dag>/tasks

UI:

Logs of the api-server where only 1 user is using it:

api-server.log

What you think should happen instead?

Visualizing and working with dags that have many tasks should work. The UI should not do a call for every task in the dag as that can never scale. Redesign of parts of the UI might be needed.

I saw in the issues that work has been done to fix n+1 queries (#57561), but it seems this path has been missed.

How to reproduce

Create a dag with 200 tasks in it and make sure there are a couple of dagruns (5 or so).
Go to the dag (gridview) and then click on tasks tab on the right panel. See previous screenshot.
Check the logs and you will see that 200 calls are triggered to fetch task instances for every task in the dag.

The 200 calls done against the backend look as follows:
<airflowBaseUrl>/airflow/api/v2/dags/<some-dag>/dagRuns/~/taskInstances?task_id=<some-task>&limit=14&order_by=-run_after

Operating System

ubuntu

Versions of Apache Airflow Providers

apache-airflow==3.1.3
apache-airflow-core==3.1.3
apache-airflow-providers-cncf-kubernetes==10.6.0
apache-airflow-providers-common-compat==1.9.0
apache-airflow-providers-common-io==1.6.5
apache-airflow-providers-common-sql==1.29.0
apache-airflow-providers-openlineage==2.9.0
apache-airflow-providers-postgres==6.4.1
apache-airflow-providers-smtp==2.3.2
apache-airflow-providers-standard==1.9.2
apache-airflow-task-sdk==1.1.3

Deployment

Other Docker-based deployment

Deployment details

We deploy airflow on kubernetes similarly to the officical Airflow helm chart but than using our own operator.

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[sania-2912]

Hi, I’d be happy to take a look at this issue.

Since it’s currently unassigned, I can start investigating it. From an initial look, it seems that the Tasks tab in the UI may be triggering one API call per task, which can lead to N+1 calls for larger DAGs and cause scalability issues.

I’ll begin by profiling the UI/API interaction and share my findings here before proposing or opening a PR.
Thanks!

[potiuk]

I think it's worth checking if it is the same for 3.1.5 ? This is the latest version released - and the report is about 3.1.3 which is quite "old news" already and a number of things in UI has been fixed.

[potiuk]

Can you confirm @nclaeys that it's the same in 3.1.5 ?

[tirkarthi]

I think this is an intentional design in #46296 . In my initial implementation in #44604 I used the all task instances for a dag with dag run id as "~" which returned only 100 task instances. Perhaps this page can use a UI specific API since all the fields from public API for a task instance are not used here.

cc: @bbovenzi

[nclaeys]

@potiuk I can confirm the same logic still exists on Airflow 3.1.5. I have just tested it.

[Shally-Katariya]

Hi! I recently contributed a fix related to unnecessary polling in the Airflow UI
(HITL polling on the DAG page).

I’m interested in helping further with UI performance issues, and I’ve been
looking into the N+1 calls behavior on the Tasks tab.

I plan to first understand the current data-fetching flow and the linked PR (#59814),
and I can help with testing, review, or follow-up improvements if needed.