Deadline Alert triggered after dag failure

[litesh1998]

Apache Airflow version

Other Airflow 3 version (please specify below)

If "Other Airflow 3 version" selected, which one?

3.1.5

What happened?

I have a dag with deadline alert configured at 180 minutes, look below for code. The dag failed within 10 mins after a single retry but I still got the deadline alert 3 hrs later
deadline=DeadlineAlert( reference=DeadlineReference.DAGRUN_QUEUED_AT, interval=timedelta(minutes=DEADLINE_ALERT_MINUTES), callback=AsyncCallback( send_rich_deadline_email, kwargs={ "dag_id": f"{CLIENT_NAME}_{SERVER_NAME}_{JOB_NAME}", "emails": EMAIL_ALERTS, "deadline_min": DEADLINE_ALERT_MINUTES, },

What you think should happen instead?

I should not have got a deadline alert. The issue is I have 4 dags that failed similarly but only one issued a deadline alert

How to reproduce

Create a dag with the following, fail it before the deadline
deadline=DeadlineAlert( reference=DeadlineReference.DAGRUN_QUEUED_AT, interval=timedelta(minutes=DEADLINE_ALERT_MINUTES), callback=AsyncCallback( send_rich_deadline_email, kwargs={ "dag_id": f"{CLIENT_NAME}_{SERVER_NAME}_{JOB_NAME}", "emails": EMAIL_ALERTS, "deadline_min": DEADLINE_ALERT_MINUTES, }

Operating System

RHEL 9.5

Versions of Apache Airflow Providers

Package Name | Version | Description
apache-airflow-providers-common-compat | 1.11.0 | Common Compatibility Provider - providing compatibility code for previous Airflow versions
apache-airflow-providers-common-io | 1.7.0 | Common IO Provider
apache-airflow-providers-common-sql | 1.30.2 | Common SQL Provider https://en.wikipedia.org/wiki/SQL
apache-airflow-providers-fab | 3.1.1 | Flask App Builder https://flask-appbuilder.readthedocs.io/
apache-airflow-providers-postgres | 6.5.1 | PostgreSQL https://www.postgresql.org/
apache-airflow-providers-smtp | 2.4.1 | Simple Mail Transfer Protocol (SMTP) https://tools.ietf.org/html/rfc5321
apache-airflow-providers-standard | 1.10.2 | Airflow Standard Provider

Deployment

Virtualenv installation

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[arjav1528]

Hi, I'd like to submit a PR on this

[tirkarthi]

cc: @ramitkataria @ferruzzi

[Arunodoy18]

Thanks for reporting this. One clarification that might help triage this:

• Does the deadline alert fire even when the DagRun is already in a FAILED state?
• Is the reference timestamp (DAGRUN_QUEUED_AT) expected to still trigger alerts post-failure, or should alerts be suppressed once a terminal state is reached?

I’m trying to understand whether this is a bug or expected behavior before attempting a fix.

[tirkarthi]

deadlines are pruned only when the dagrun succeeds. In case dagrun fails the deadline remains in the db to be picked up by the scheduler to handle deadline miss. Is this an expected behavior as ?

mysql> select *, now() from deadline limit 1 offset 1 \G;
*************************** 1. row ***************************
               id: 019bfab13d55744d9a2f62505186e5e6
        dagrun_id: 182
    deadline_time: 2026-01-26 19:55:38.195294
           missed: 0
      callback_id: 019bfab13d5472dbaaa0a8118b988a1d
deadline_alert_id: 019bfaad9e5e79eea4c061ac9f48adf6
       created_at: 2026-01-26 19:54:38.229966
  last_updated_at: 2026-01-26 19:54:38.229976
            now(): 2026-01-26 19:55:25
1 row in set (0.00 sec)

mysql> select * from dag_run where id = 182 \G;
*************************** 1. row ***************************
                      id: 182
                  dag_id: gh60927
               queued_at: 2026-01-26 19:54:38.195294
            logical_date: 2026-01-26 19:54:36.000000
              start_date: 2026-01-26 19:54:39.375827
                end_date: 2026-01-26 19:55:01.806481
                   state: failed
                  run_id: manual__2026-01-26T14:24:38.181308+00:00
         creating_job_id: NULL
                run_type: manual
            triggered_by: UI
    triggering_user_name: admin
                    conf: {}
     data_interval_start: 2026-01-26 19:54:36.000000
       data_interval_end: 2026-01-26 19:54:36.000000
               run_after: 2026-01-26 19:54:38.181308
last_scheduling_decision: 2026-01-26 19:55:01.803274
         log_template_id: 1
              updated_at: 2026-01-26 19:55:01.807175
            clear_number: 0
             backfill_id: NULL
          bundle_version: NULL
     scheduled_by_job_id: 7
         context_carrier: {"__var": {}, "__type": "dict"}
             span_status: ended
  created_dag_version_id: 019bfaad9e7975349bc777ebf366a7aa
           partition_key: NULL
1 row in set (0.00 sec)

ERROR: 
No query specified

After the dagrun is marked as failed the deadline is not pruned and is handled by scheduler to be marked as missed.

mysql> select *, now() from deadline limit 1 offset 1 \G;
*************************** 1. row ***************************
               id: 019bfab13d55744d9a2f62505186e5e6
        dagrun_id: 182
    deadline_time: 2026-01-26 19:55:38.195294
           missed: 1
      callback_id: 019bfab13d5472dbaaa0a8118b988a1d
deadline_alert_id: 019bfaad9e5e79eea4c061ac9f48adf6
       created_at: 2026-01-26 19:54:38.229966
  last_updated_at: 2026-01-26 19:55:38.663270
            now(): 2026-01-26 19:55:45
1 row in set (0.00 sec)

airflow/airflow-core/src/airflow/models/dagrun.py

Lines 1266 to 1276 in 4e8274c

	if dag.deadline:
	# The dagrun has succeeded. If there were any Deadlines for it which were not breached, they are no longer needed.
	deadline_alerts = [
	DeadlineAlertModel.get_by_id(alert_id, session) for alert_id in dag.deadline
	]
	if any(
	deadline_alert.reference_class in DeadlineReference.TYPES.DAGRUN
	for deadline_alert in deadline_alerts
	):
	Deadline.prune_deadlines(session=session, conditions={DagRun.id: self.id})

prune_deadlines indicates only successful dagruns are pruned.

airflow/airflow-core/src/airflow/models/deadline.py

Lines 153 to 163 in 4e8274c

	def prune_deadlines(cls, *, session: Session, conditions: dict[Mapped, Any]) -> int:
	"""
	Remove deadlines from the table which match the provided conditions and return the number removed.
	NOTE: This should only be used to remove deadlines which are associated with
	successful events (DagRuns, etc). If the deadline was missed, it will be
	handled by the scheduler.
	:param conditions: Dictionary of conditions to evaluate against.
	:param session: Session to use.
	"""

[ferruzzi]

deadlines are pruned only when the dagrun succeeds. In case dagrun fails the deadline remains in the db to be picked up by the scheduler to handle deadline miss.

That is the expected behavior, yes.

Essentially the workflow is expected to be:

When the dag run starts, check if there is a relevant deadline. If so then calculate and store that timestmp.
Each scheduler pass, check if any of those timestamps are in the past and handle them (run the callback and mark it is done)
When the dag run finishes, if it finished before the deadline then remove the deadline as it is no longer needed.

I remember @ramitkataria was tinkering with the prune logic in #57215 and I was refactoring the database in #58248. If you are willing to look into it, I'd appreciate the help and suggest those as good places to start. Feel free to message me on Slack if you want to ask anything, I'm using the same handle there.

Edit to clarify: The intention is that the deadline's callback should be executed on the scheduler pass after the deadline's timestamp has passed, NOT when the dag run finishes. If the calculated deadline is at 12:00:00 and the scheduler is configured for a 5-second loop, then the callback should be started absolutely no later than 12:00:05 at the worst case.

[ferruzzi]

One POSSIBLE thought is that it might be possible that the email server is introducing lag? I generally use the SlackNotifier when testing deadlines stuff and that is nearly instantaneous, but when I tried using the SmtpNotifier I saw something like a 10-minute delay, presumably because of email server settings, spam filters, etc, but I'm not an SMTP expert by any means so I'm just speculating on the reasons there. Could that maybe be what you are seeing?

[ramitkataria]

Right now, the expected behaviour of deadline alerts is to execute the callback when the Dag run is not in a success state by the deadline time. So it makes sense that you're still getting an alert. There is no special handling for when you manually mark a Dagrun as failed and I don't think we should have it in the backend anyway. I think the solution here would be to add a checkbox in the UI to delete the deadline when you manually set the status to failed

[tirkarthi]

If this is intentional then I feel the docs should be updated to indicate that the state should be finished/completed successfully to be pruned and not just finished/completed. Failed is also a terminal state.

https://airflow.apache.org/docs/apache-airflow/stable/howto/deadline-alerts.html

Below is an example Dag implementation. If the Dag has not finished 15 minutes after it was queued, send a Slack message:

Here’s an example using the Slack Notifier if the Dag run has not finished within 30 minutes of it being queued:

In this case, if a Dag is scheduled to run daily at midnight, the deadline would be triggered if the Dag hasn’t completed by 1:00 AM. This is useful for ensuring that scheduled jobs complete within a certain timeframe after their intended start time

[ferruzzi]

I think I'm still confused here, I'm not sure we are all talking about the same thing.

1. You create a Dag with a 180 minute deadline using the DAGRUN_QUEUED reference with a callback to send an email
2. You run the Dag and it gets queued at 12:00:00

Scenario 0: All goes well and the Dag successfully finishes before the deadline
Expected behavior: The deadline gets pruned as part of the dag run cleanup step when the dag run finishes and nothing happens.

Scenario 1: Some task in the Dag fails at 12:05:00
Expected behavior: At some time between 12:05:00 and 12:05:05 that email will be sent because the Dag did not complete before 13:30:00

Scenario 2: You cancel or manually mark it as failed at 12:05:00
Expected behavior: At some time between 12:05:00 and 12:05:05 that email will be sent because the Dag did not complete before 13:30:00

Scenario 3: Some task in the Dag freezes and it is still running at 13:30:00
Expected behavior: At some time between 13:39:00 and 13:30:05 that email will be sent because the Dag did not complete before 13:30:00

Can you confirm which of those aren't what you'd expect, or if it isn't one of those then maybe clarify what you are seeing?

[tirkarthi]

Thanks @ferruzzi , I think the confusion is around "completed". Does it mean only success state or dagrun reaching terminal state including failed. Looks like there was some discussion around this in AIP. If this is expected behavior to run deadline alerts on failed dagruns too then I think this can be a document issue or closed as expected behavior.

On the name side of things: my issue with alert: alert makes me thing it would also cover cases where the dag fails, but that's not what it is. It's only for SLA/lateness

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=323488182&focusedCommentId=327977705#comment-327977705