Skip to content

Conversation

@Jkhall81
Copy link
Contributor

@Jkhall81 Jkhall81 commented Jan 1, 2026

fix(assets): queue asset updates for stale DAGs

closes: #59337

Summary

This PR ensures that asset updates are recorded in the AssetDagRunQueue even if a DAG is currently in a "stale" state. By allowing these updates to be queued, DAGs will automatically trigger their missed runs once they are successfully parsed and become healthy again, rather than remaining idle.

Changes

  • Backend: Modified AssetManager.register_asset_change in manager.py to remove is_stale filtering while maintaining is_paused checks across direct, alias, and name/URI asset references.
  • Database: Refined the SQLAlchemy query for asset name/URI references to explicitly exclude paused DAGs for logic consistency.
  • Tests: Added a regression test in test_manager.py that mocks a stale DAG state and verifies it is correctly added to the AssetDagRunQueue upon an asset event.

@Jkhall81
Copy link
Contributor Author

Jkhall81 commented Jan 1, 2026

I have verified that the logic changes in airflow/assets/manager.py successfully address the issue in #59337 by allowing asset updates to be queued for DAGs even when they are in a 'stale' state.

But there are failures in the following tests:

  • test_asset_listener_on_asset_changed_gets_calls
  • test_no_create_dag_runs_when_dag_disabled[active] (across Postgres, MySQL, and Sqlite)

These are negative tests designed to assert that stale DAGs are ignored. Since that behavior was changed in the PR, these assertions are now failing.

If code change in the PR is accepted, then I can reverse the assertions of these four tests so they align with the fix.

@kaxil kaxil requested review from Lee-W and uranusjr January 23, 2026 17:49
@Lee-W Lee-W self-assigned this Jan 24, 2026
@Lee-W Lee-W force-pushed the fix-stale-dag-asset-scheduling-59337 branch from 47ae90c to 2c2dc44 Compare January 26, 2026 09:44
Copy link
Member

@Lee-W Lee-W left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase from the latest main and fix a few nitpicks. I'm going to merge it tomorrow if no one objects.

@Lee-W Lee-W added backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch area:data-aware-scheduling assets, datasets, AIP-48 labels Jan 26, 2026
@Lee-W Lee-W force-pushed the fix-stale-dag-asset-scheduling-59337 branch from 2c2dc44 to 196e862 Compare January 26, 2026 10:10
@Lee-W Lee-W merged commit 73451c5 into apache:main Jan 27, 2026
89 checks passed
@github-actions
Copy link

Backport failed to create: v3-1-test. View the failure log Run details

Status Branch Result
v3-1-test Commit Link

You can attempt to backport this manually by running:

cherry_picker 73451c5 v3-1-test

This should apply the commit to the v3-1-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

Lee-W added a commit to astronomer/airflow that referenced this pull request Jan 27, 2026
…e#60022)

(cherry picked from commit 73451c5)

Co-authored-by: Jason K Hall <106277735+Jkhall81@users.noreply.github.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit to astronomer/airflow that referenced this pull request Jan 27, 2026
…e#60022)

(cherry picked from commit 73451c5)

Co-authored-by: Jason K Hall <106277735+Jkhall81@users.noreply.github.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit to astronomer/airflow that referenced this pull request Jan 27, 2026
…e#60022)

(cherry picked from commit 73451c5)

Co-authored-by: Jason K Hall <106277735+Jkhall81@users.noreply.github.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit to astronomer/airflow that referenced this pull request Jan 27, 2026
…e#60022)

(cherry picked from commit 73451c5)

Co-authored-by: Jason K Hall <106277735+Jkhall81@users.noreply.github.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit to astronomer/airflow that referenced this pull request Jan 27, 2026
…e#60022)

(cherry picked from commit 73451c5)

Co-authored-by: Jason K Hall <106277735+Jkhall81@users.noreply.github.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit to astronomer/airflow that referenced this pull request Jan 28, 2026
…e#60022)

(cherry picked from commit 73451c5)

Co-authored-by: Jason K Hall <106277735+Jkhall81@users.noreply.github.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W added a commit that referenced this pull request Jan 28, 2026
…1106)

Co-authored-by: Jason K Hall <106277735+Jkhall81@users.noreply.github.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Fix asset scheduling for stale DAGs (#59337) (#60022)
shreyas-dev pushed a commit to shreyas-dev/airflow that referenced this pull request Jan 29, 2026
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@amoghrajesh amoghrajesh added this to the Airflow 3.1.7 milestone Jan 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:data-aware-scheduling assets, datasets, AIP-48 backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants