Skip to content

[multistage] Follow-ups for Multistage Mailbox Stability #10424

@ankitsultana

Description

@ankitsultana

I recently raised a few PRs to fix a bunch of stability issues in the multistage engine.

There are still some issues pending though. Using this issue to track them.

  • Complete the design doc for the changes in https://github.com/apache/pinot/pull/10322/files
  • Address out of order cancellation for both in-memory and gRPC mailbox.
  • Handle gRPC sending mailbox cancellation when stream is created but no message has been sent (I think there's a case where only onCompleted is called and onNext is never called).
  • Handle gRPC sending mailbox cancellation when stream is never created so the receiver never gets any notification. A fix for this was added in [multistage] Handle Stream Cancellations for Unstarted Streams #10425 but we should discuss this in the design doc and see whether that solution needs to be improved.
  • Ensure error message is sent upstream in case of cancellations. This PR added a potential fix for this but there are some more cases: [multistage] Handle Stream Cancellations for Unstarted Streams #10425
  • Add UTs which use MultiplexingMailboxService. Right now QueryRunnerTest uses only gRPC.
  • Add Callbacks for complete events.

Metadata

Metadata

Assignees

No one assigned

    Labels

    multi-stageRelated to the multi-stage query engine

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions