Description
Is your feature request related to a problem or challenge?
We have done the first step in #16196 for pipeline breaking cancellation support, this epic trace the remaining sub-task for the remaining improvement. cc @ozankabak @alamb @pepijnve
-
Adding a few tests (maybe SLT?) that show YieldStreamExec being inserted. Also add logs related to built-in YiedStream.
-
Improving the documentation to make it clear that any leaf (source) that already yields just has to implement
fn with_cooperative_yields(self: Arc<Self>) -> Option<Arc<dyn ExecutionPlan>> {
Some(self)
}
to signal the planner that no YieldStream combinator is necessary.
-
Improving InsertYieldExec rule by means of an API that exposes input and output pipelining behaviors of operators effectively
-
Investigating whether any already-existing manual yielding (for example, like the one in RepartitionExec) can now be removed
-
We will think about supporting cases involving non-volcano (e.g. spill) data flow.
-
Fix the corner case provided in this link: https://gist.github.com/pepijnve/0e1a66f98033c6c44c62a51fb9dbae5a
Feel free to add more tasks, thanks!
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response