Skip to content

[draft][multistage] Support multiple semi join#11922

Closed
walterddr wants to merge 6 commits intoapache:masterfrom
walterddr:poc_rel_distribution_multi_semi
Closed

[draft][multistage] Support multiple semi join#11922
walterddr wants to merge 6 commits intoapache:masterfrom
walterddr:poc_rel_distribution_multi_semi

Conversation

@walterddr
Copy link
Contributor

@walterddr walterddr commented Nov 1, 2023

This PR enables multiple SEMI join can be planned within the same leaf-stage.

Dep

TODO

  • modified dynamic broadcast to stack semi joins together in dynamic broadcast fashion
  • modify the detection logic to be reused in both (1) broker detection and (2) server to determine leaf PinotQuery boundary

Rong Rong and others added 6 commits October 31, 2023 15:42
connected with QueryRunner

refactor and simplify leaf stage compilation

fix linter and further refactoring

simplified visitor and context class, add javadoc

fix server plan request gen issues
[stash] add other nodes

[stash] adding elimination rules

[stash] fix RelOptUtil usage by utilizing direct method in RelNode

[stash] enable exchange elimination rule as well

[stash] adjust tests

[stash] adding table scan special handling too

[bugfix] change mapping for join to be correct on left only

[stash] fix build

fix test
@codecov-commenter
Copy link

Codecov Report

Merging #11922 (fc5fd92) into master (8d49b11) will decrease coverage by 33.79%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master   #11922       +/-   ##
=============================================
- Coverage     61.40%   27.62%   -33.79%     
+ Complexity     1145      201      -944     
=============================================
  Files          2378     2380        +2     
  Lines        128851   128939       +88     
  Branches      19926    19942       +16     
=============================================
- Hits          79127    35625    -43502     
- Misses        44002    90632    +46630     
+ Partials       5722     2682     -3040     
Flag Coverage Δ
custom-integration1 ?
integration ?
integration1 ?
integration2 ?
java-11 ?
java-21 27.62% <0.00%> (-33.67%) ⬇️
skip-bytebuffers-false ?
skip-bytebuffers-true 27.62% <0.00%> (-33.63%) ⬇️
temurin 27.62% <0.00%> (-33.79%) ⬇️
unittests 27.62% <0.00%> (-33.78%) ⬇️
unittests1 ?
unittests2 27.62% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
...ery/optimizer/filter/NumericalFilterOptimizer.java 0.00% <ø> (-81.96%) ⬇️
...va/org/apache/pinot/query/runtime/QueryRunner.java 0.00% <0.00%> (-80.71%) ⬇️
...ot/common/request/context/RequestContextUtils.java 0.00% <0.00%> (-67.64%) ⬇️
...ot/common/request/context/predicate/Predicate.java 0.00% <0.00%> (-100.00%) ⬇️
...src/main/java/org/apache/pinot/sql/FilterKind.java 0.00% <0.00%> (-100.00%) ⬇️
...he/pinot/query/planner/logical/LogicalPlanner.java 0.00% <0.00%> (-54.55%) ⬇️
...ava/org/apache/pinot/core/plan/FilterPlanNode.java 0.00% <0.00%> (-82.23%) ⬇️
...n/request/context/predicate/ConstantPredicate.java 0.00% <0.00%> (ø)
...org/apache/pinot/query/context/PlannerContext.java 0.00% <0.00%> (-100.00%) ⬇️
...ot/query/runtime/plan/OpChainExecutionContext.java 0.00% <0.00%> (-96.16%) ⬇️
... and 5 more

... and 1159 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@walterddr
Copy link
Contributor Author

closing this one as it is now part of #11937

@walterddr walterddr closed this Nov 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants