-
Notifications
You must be signed in to change notification settings - Fork 16.4k
Add REST API endpoint to materialize asset #46718
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This is done by triggering the DAG that has the given asset as a task outlet. 409 is returned if more than one DAG uses the asset as outlet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code looks good.
To what extend do we really need that endpoint ? While we can just query the asset details (get_asset), check the producing_tasks dag_ids and manually trigger one of those Run via the POST DagRun API if we only have 1 dag affected.
This operates across DagRun, Assets resources and return a 'DagRun' while we are querying the asset resource. All that seems a little confusing to me.
Does it have to be part of the public API ? Or can this be private if really required ?
|
Yeah, why is this needed? We can already manually trigger a dag run or manually create an asset event through the API. |
|
It’s in the vision of the data awareness proposal. The motivation is to be able to treat assets as the central concept like DAGs. When you want an asset to be refreshed, you can operate on the asset directly, instead of needing to go through the thought process of finding what DAG contains a task that refreshes the asset, and triggers that DAG instead (DAG is the central concept in this thought process). Yeah we definitely don’t really need this, and we probably don’t need to do anything in the UI for 3.0. This is just a new thing that promotes the asset centric idea. |
pierrejeambrun
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the details.
This is done by triggering the DAG that has the given asset as a task outlet. 409 is returned if more than one DAG uses the asset as outlet.
This is done by triggering the DAG that has the given asset as a task outlet. 409 is returned if more than one DAG uses the asset as outlet.
This is done by triggering the DAG that has the given asset as a task outlet. 409 is returned if more than one DAG uses the asset as outlet.
Close #42318.