Extend airflow_ti_running metrics by scheduled, queued and deferred #58819
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AutomationDev85
requested review from
XD-DENG,
ashb,
hussein-awala,
jedcunningham and
jscheffl
as code owners
boring-cyborg
bot
added
area:ConfigTemplates
area:helm-chart
jscheffl
reviewed
Nov 28, 2025
Contributor
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
In general this all makes sense. But the configuration can not be "just" renamed as well as some mypy static fails. I assume you know how to fix.
jscheffl
reviewed
Nov 28, 2025
AutomationDev85
force-pushed
the
feature/task-instance-metric-add-more-states
branch
2 times, most recently
from
b09efdf to
928210a
Compare
AutomationDev85
force-pushed
the
feature/task-instance-metric-add-more-states
branch
from
5f73cee to
62d61ac
Compare
jscheffl
force-pushed
the
feature/task-instance-metric-add-more-states
branch
from
62d61ac to
877752b
Compare
jscheffl
approved these changes
Dec 7, 2025
Contributor
jscheffl
left a comment
jscheffl
left a comment
There was a problem hiding this comment.
Looks good for me, some other pair of eyes might be good, otherwise LGTM. Will elave it open for a moment for others to review prior merge.
TempestShaw
pushed a commit
to TempestShaw/airflow
that referenced
this pull request
Dec 24, 2025
…pache#58819) * ti metric export of scheduled, queued and deferred * Added deprecated warning for new option * only add task_id to metric --------- Co-authored-by: AutomationDev85 <AutomationDev85>
96 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
While working with deferred tasks, we found that the airflow_ti_running metric does not include task instances in the deferred state. This change extends metric export to also cover scheduled, queued, and deferred states. With these metrics you can build dashboards that show per-DAG load in running and deferred states, and monitor how many tasks per DAG are waiting for resources in scheduled and queued states.
Change Summary
Add ti.scheduled, ti.queued and ti.deferred metrics to the code which exports the ti.running
Update description with new metrics.
Add fix for task_id to replace . because task group tasks are breaking the stasd exporting because of using . separator.
Add ti.scheduled, ti.queued and ti.deferred metrics alongside ti.running.
Update metric descriptions to reflect the new states.
Replace . with __ in task_id for StatsD compatibility (task groups use . as a separator, which breaks StatsD’s dot-delimited metric hierarchy).