-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-12218] Fixes ORC conjunction predicate push down #10377
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-12218] Fixes ORC conjunction predicate push down #10377
Conversation
|
Test build #48001 has finished for PR 10377 at commit
|
|
Test build #48005 has finished for PR 10377 at commit
|
|
Test build #48006 has finished for PR 10377 at commit
|
|
sorry.. My commit introduced a conflict... |
fe1ab88 to
e5913d2
Compare
|
@yhuai Nvm, rebased. |
|
Test build #48042 has finished for PR 10377 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to mention for And Or Not? createFilter is dealing with the top level filters (they will be connected by AND), right? I think it is important to emphasize that createFilter is for top level filters.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
buildSearchArgument is recursive, so nested And/Or/Not within top level filters are also covered.
|
Test build #48270 has finished for PR 10377 at commit
|
f877ffa to
6c59503
Compare
|
Test build #48344 has finished for PR 10377 at commit
|
|
LGTM. Merging to master. |
|
Thanks for the review! |
This PR is a follow-up of PR #10362.
Two major changes:
PR #10362 stops converting an
ANDpredicate if any branch is inconvertible. On the other hand,OrcFilterscombines all filters into a single big conjunction first and then tries to convert it into ORCSearchArgument. This means, if any filter is inconvertible, no filters can be pushed down. This PR fixes this issue by finding out all convertible filters first before doing the actual conversion.The reason behind the current implementation is mostly due to the limitation of ORC
SearchArgumentbuilder, which is documented in this PR in detail.2. Copied the
ANDpredicate fix for ORC from #10362 to avoid merge conflict.Same as #10362, this PR targets master (2.0.0-SNAPSHOT), branch-1.6, and branch-1.5.