-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Filter push down #1641
Labels
Comments
How to support pushdown.For instance, push down, SolutionsThere are 2 possible solutions, (1) Create UDF and register through FunctionCatalog. (2) Add query optimization rule to rewrite Filter. The PR below is solution (1).
Demo
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Predicate Push Down
Spark provide
predicates: Array[Predicate]
to DataSource, DataSource decide which predicate could be pushed down. I will use example to describe what predicate is provide by Spark.Spark SQL in ANSI mode
Spark SQL has an ANSI mode configuration
spark.sql.ansi.enabled
, by default is false.The ANSI mode could change the filter push down logic. The following example explain it.
The V2ExpressionBuilder class decide which expression could be push down. The V2ExpressionBuilder is not stable yet. More expression push down is adding to it.
The text was updated successfully, but these errors were encountered: