-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Fix asset scheduling for stale DAGs (#59337) #60022
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix asset scheduling for stale DAGs (#59337) #60022
Conversation
|
I have verified that the logic changes in But there are failures in the following tests:
These are negative tests designed to assert that stale DAGs are ignored. Since that behavior was changed in the PR, these assertions are now failing. If code change in the PR is accepted, then I can reverse the assertions of these four tests so they align with the fix. |
47ae90c to
2c2dc44
Compare
Lee-W
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rebase from the latest main and fix a few nitpicks. I'm going to merge it tomorrow if no one objects.
2c2dc44 to
196e862
Compare
Backport failed to create: v3-1-test. View the failure log Run details
You can attempt to backport this manually by running: cherry_picker 73451c5 v3-1-testThis should apply the commit to the v3-1-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fix(assets): queue asset updates for stale DAGs
closes: #59337
Summary
This PR ensures that asset updates are recorded in the
AssetDagRunQueueeven if a DAG is currently in a "stale" state. By allowing these updates to be queued, DAGs will automatically trigger their missed runs once they are successfully parsed and become healthy again, rather than remaining idle.Changes
AssetManager.register_asset_changeinmanager.pyto removeis_stalefiltering while maintainingis_pausedchecks across direct, alias, and name/URI asset references.test_manager.pythat mocks a stale DAG state and verifies it is correctly added to theAssetDagRunQueueupon an asset event.