-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for pushdown like filter (endsWith and contains) #9683
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yabola This is very cool, could we break up the PR though for easier review? Let me take a deeper look at the code before I propose a way to break it up into separate PRs. On the surface though it seems like we should be able to break apart the expression changes in API/Core and then have separate PRs for the file formats and Spark integration?
Also, I'll need to think more if we can actually support this for delete files. If not, this will need to only be applied for CoW tables. For example, for agg pushdown, we don't support MoR tables (well unless it's compacted) |
I will break up my PR. Thanks for your advice. |
@amogh-jahagirdar I checked the code. When there are delete rows, it won't pushdown agg and I have tested it. iceberg/spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java Lines 244 to 247 in 5f577f1
|
This PR supports pushing down
endsWith
(like %x) andcontains
(like %x%) to Iceberg. The benefits are:endWiths
orcontains
could not be pushed down, so it could not pushdown agg).Before this PR, iceberg only support startWith.