-
Notifications
You must be signed in to change notification settings - Fork 903
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Skip node at runtime #2410
Comments
How would you like this to work if it existed? Is it based on a condition or is it known pre-run? |
What would work well in the case above is that a node can be skipped (simply a boolean flag), that can be set in the Indeed based on a condition only known at runtime. In the referenced issue this would be a cache hit, however I can imagine use cases with other conditions. (Note that Github Actions, Azure DevOps pipelines and related tools do support this and could be a source of inspiration.) |
I think there's going to be people who disagree (e.g. @idanov) but personally I like this idea and think doing it with hooks feels very natural. Three other ideas that are already possible but I suspect won't offer the full flexible dynamic functionality you'd like. They could also be used in combination:
|
Hi? Options:
|
Our team is working on a kedro runner for this. PyCodeHash was just released and solves the heavy lifting of hashing functions and datasets. |
There currently is no way (that I know of) to skip a node at runtime (e.g. from a hook), without failing the pipeline run.
Is there already an idiomatic way of doing so? e.g. build a custom runner, with a function similar to
run_only_missing
?Alternatives considered:
If not, is this something that is welcome to be contributed? It could be a fairly simple and generic addition. (Happy to add)
(related to #2307)
The text was updated successfully, but these errors were encountered: