Fix n+1 queries in ui/dags endpoint and use a compact response model. #57425
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t trigger n+1 queries to other tables.
tirkarthi
requested review from
bbovenzi,
bugraoz93,
ephraimbuddy,
jason810496,
jscheffl,
pierrejeambrun,
rawwar,
ryanahamilton and
shubhamraj-git
as code owners
boring-cyborg
bot
added
area:API
Member
pierrejeambrun
left a comment
pierrejeambrun
left a comment
There was a problem hiding this comment.
That's a good idea, we were probably missing some loading options in the queries, and if we don't need them.
Just one question regarding backward compatibility, otherwise LGTM.
pierrejeambrun
approved these changes
Oct 28, 2025
Member
|
What I plan to do, and we could do here is add some |
pierrejeambrun
added
the
backport-to-v3-1-test
Backport successfully created: v3-1-test
|
Lzzz666
pushed a commit
to Lzzz666/airflow
that referenced
this pull request
Oct 30, 2025
…apache#57425) * Use DAGRunLightResponse to stop loading unused fields in response that trigger n+1 queries to other tables. * Update frontend type and tests. * Fix MyPy error.
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The recent dag runs query was triggering access to task_instances, task_instance_histories, deadlines and dag_run_note tables as part of
model_validateusingDagRunResponseand each attribute access caused a query resulting n+1 queries for each dagrun in the recent runs response. The recent runs response included a lot of fields that are unused in the UI. This PR simplifies the response to selectively query the fields and usesDAGRunLightResponseto includedurationfield which fulfills all the fields required in the frontend.Additionally, the
pending_actionsfilter to filter by required actions had condition ashas_pending_actions.value is not Falseand in case of normal dags list page the has_pending_actions.value is None and thusNone is not Falsealso lead to evaluating query where it was not required. This also improved the performance.With the removal of n+1 queries and unused fields the overall response was significantly improved up to 7-8x where page that took 1s second to load for 15 dags and 2100 dagruns now loaded in around 100ms. This is also more noticeable in large environments where the n+1 queries are not executed. The http response size was also reduced with removal of unused fields in the response.
pydantic/pydantic#8192
sqlalchemy/sqlalchemy#10120
https://docs.sqlalchemy.org/en/20/orm/queryguide/relationships.html#sqlalchemy.orm.noload
Closes #57418