Add streaming query handler #5717
Conversation
| import java.util.List; | ||
|
|
||
|
|
||
| public class StreamingSelectionOnlyOperator extends BaseOperator<IntermediateResultsBlock> { |
There was a problem hiding this comment.
Can you please import Pinot code style in config/codestyle-intellij.xml or config/codestyle-eclipse.xml and reformat the files?
| QueryContext queryContext, | ||
| List<ExpressionContext> expressions, | ||
| TransformOperator transformOperator, | ||
| StreamObserver<Server.ServerResponse> streamObserver) { |
There was a problem hiding this comment.
Let's not pass the streamObserver to the segment level operator because multiple segments will be processed in parallel, but calls to streamObserver.onNext() need to be synchronized.
We should create a StreamingCombineOperator (instance level operator) to keep fetching IntermediateResultsBlock from this operator and handle the calls to streamObserver.onNext().
...-core/src/main/java/org/apache/pinot/core/operator/query/StreamingSelectionOnlyOperator.java
Outdated
Show resolved
Hide resolved
...-core/src/main/java/org/apache/pinot/core/operator/query/StreamingSelectionOnlyOperator.java
Outdated
Show resolved
Hide resolved
pinot-core/src/main/java/org/apache/pinot/core/plan/maker/InstancePlanMakerImplV2.java
Outdated
Show resolved
Hide resolved
pinot-server/src/main/java/org/apache/pinot/server/starter/grpc/PinotQueryService.java
Outdated
Show resolved
Hide resolved
pinot-server/src/main/java/org/apache/pinot/server/starter/grpc/PinotQueryService.java
Outdated
Show resolved
Hide resolved
| // TODO: implement, follow up whether to use ServerQueryRequest | ||
| } | ||
|
|
||
| public DataTable processQuery(ServerQueryRequest queryRequest, |
There was a problem hiding this comment.
This method should not return anything (we can directly implement submit())
There was a problem hiding this comment.
Will be pushing an update for this as well.
| private long _defaultTimeOutMs = CommonConstants.Server.DEFAULT_QUERY_EXECUTOR_TIMEOUT_MS; | ||
| private ServerMetrics _serverMetrics; | ||
|
|
||
| public synchronized void init(PinotConfiguration config, InstanceDataManager instanceDataManager, |
There was a problem hiding this comment.
It will be very hard to reuse the current QueryScheduler because that is not designed for streaming API. So for the first version we can just launch an ExecutorService within this class and use it to execute queries without introducing query scheduling.
There was a problem hiding this comment.
Makes sense, will be pushing an update for this.
Jackie-Jiang
force-pushed
the
streaming_endpoint
branch
from
a04618e to
e6df0c7
Compare
elonazoulay
force-pushed
the
streaming_endpoint
branch
2 times, most recently
from
a04618e to
f886794
Compare
Jackie-Jiang
force-pushed
the
streaming_endpoint
branch
5 times, most recently
from
8bc3d7b to
db95533
Compare
Jackie-Jiang
force-pushed
the
streaming_endpoint
branch
from
db95533 to
477af0b
Compare
pinot-common/src/main/java/org/apache/pinot/common/utils/CommonConstants.java
Show resolved
Hide resolved
pinot-core/src/main/java/org/apache/pinot/core/transport/grpc/GrpcRequestBuilder.java
Outdated
Show resolved
Hide resolved
pinot-core/src/main/java/org/apache/pinot/core/transport/grpc/GrpcQueryClient.java
Outdated
Show resolved
Hide resolved
Jackie-Jiang
force-pushed
the
streaming_endpoint
branch
from
477af0b to
854bfd0
Compare
Description
Add a description of your PR here.
A good description should include pointers to an issue or design document, etc.
Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
backward-incompat, and complete the section below on Release Notes)Does this PR fix a zero-downtime upgrade introduced earlier?
backward-incompat, and complete the section below on Release Notes)Does this PR otherwise need attention when creating release notes? Things to consider:
release-notesand complete the section on Release Notes)Release Notes
If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of the
next release.
If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text
Documentation
If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document