Skip to content

[Epic] Pipeline breaking cancellation support and improvement #16353

Open
@zhuqi-lucas

Description

@zhuqi-lucas

Is your feature request related to a problem or challenge?

We have done the first step in #16196 for pipeline breaking cancellation support, this epic trace the remaining sub-task for the remaining improvement. cc @ozankabak @alamb @pepijnve

  • Adding a few tests (maybe SLT?) that show YieldStreamExec being inserted. Also add logs related to built-in YiedStream.

  • Improving the documentation to make it clear that any leaf (source) that already yields just has to implement

fn with_cooperative_yields(self: Arc<Self>) -> Option<Arc<dyn ExecutionPlan>> {
    Some(self)
}

to signal the planner that no YieldStream combinator is necessary.

  • Improving InsertYieldExec rule by means of an API that exposes input and output pipelining behaviors of operators effectively

  • Investigating whether any already-existing manual yielding (for example, like the one in RepartitionExec) can now be removed

  • We will think about supporting cases involving non-volcano (e.g. spill) data flow.

  • Fix the corner case provided in this link: https://gist.github.com/pepijnve/0e1a66f98033c6c44c62a51fb9dbae5a

Feel free to add more tasks, thanks!

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions