[multi-stage] Query compilation support for SetOperations: UNION/INTERSECT/MINUS(EXCEPT) in multi-stage engine#10535
Merged
xiangfu0 merged 1 commit intoapache:masterfrom Apr 20, 2023
Conversation
xiangfu0
changed the title
xiangfu0
changed the title
xiangfu0
force-pushed
the
v2-set-op-support
branch
8 times, most recently
from
b78280d to
c2487f0
Compare
Codecov Report
@@ Coverage Diff @@
## master #10535 +/- ##
============================================
- Coverage 70.35% 68.52% -1.84%
+ Complexity 6499 6486 -13
============================================
Files 2107 2108 +1
Lines 113460 113535 +75
Branches 17101 17117 +16
============================================
- Hits 79822 77795 -2027
- Misses 28038 30185 +2147
+ Partials 5600 5555 -45
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 167 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
xiangfu0
force-pushed
the
v2-set-op-support
branch
from
c2487f0 to
f13a273
Compare
xiangfu0
force-pushed
the
v2-set-op-support
branch
2 times, most recently
from
4f9e11b to
5e95130
Compare
walterddr
reviewed
Apr 10, 2023
pinot-common/src/main/java/org/apache/pinot/sql/parsers/CalciteSqlParser.java
Outdated
Show resolved
Hide resolved
pinot-common/src/test/java/org/apache/pinot/sql/parsers/CalciteSqlCompilerTest.java
Outdated
Show resolved
Hide resolved
...query-planner/src/main/java/org/apache/pinot/query/planner/logical/StageMetadataVisitor.java
Outdated
Show resolved
Hide resolved
xiangfu0
force-pushed
the
v2-set-op-support
branch
3 times, most recently
from
37fe4a4 to
00f2011
Compare
walterddr
reviewed
Apr 19, 2023
...uery-planner/src/main/java/org/apache/pinot/query/planner/logical/ShuffleRewriteVisitor.java
Outdated
Show resolved
Hide resolved
walterddr
reviewed
Apr 19, 2023
...y-planner/src/main/java/org/apache/pinot/query/planner/physical/DispatchablePlanVisitor.java
Outdated
Show resolved
Hide resolved
walterddr
reviewed
Apr 19, 2023
Contributor
There was a problem hiding this comment.
@ankitsultana can you take a look and see if this works with your expectation? even if we do not allow visitSetOp is a valid current limitation IMO
Contributor
Author
There was a problem hiding this comment.
I added a new field for SetOp. Partitioning is a must to ensure the correctness.
Contributor
|
@xiangfu0 : Can you add tests which demonstrate what the plan is going to look like for queries with set-ops? |
xiangfu0
force-pushed
the
v2-set-op-support
branch
3 times, most recently
from
8d49e66 to
0d612d1
Compare
Contributor
Author
Added a SetOpPlans.json file |
xiangfu0
force-pushed
the
v2-set-op-support
branch
8 times, most recently
from
1a2580f to
8d844d5
Compare
Contributor
There was a problem hiding this comment.
@xiangfu0 : Shouldn't intersect require an exchange?
If the row col1=10, col2=11 in tables a and b are on different servers, if we don't shuffle using the intersection key (or broadcast one side) we may end up dropping them from the result-set.
Contributor
Author
There was a problem hiding this comment.
That was planned to be added in the next PR:
...main/java/org/apache/pinot/query/planner/physical/colocated/GreedyShuffleRewriteVisitor.java
Outdated
Show resolved
Hide resolved
…n in query planner
xiangfu0
force-pushed
the
v2-set-op-support
branch
from
8d844d5 to
2f56bf2
Compare
ankitsultana
approved these changes
Apr 20, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Query compilation support for SetOperations: UNION/INTERSECT/MINUS(EXCEPT) in multi-stage engine.
Will have a few follow up PRs for the implementations of:
UNION ALL, UNION, INTERSECT, MINUS(EXCEPT)