Remove qualifiers on pushed down predicates / Fix parquet pruning#689
Merged
alamb merged 3 commits intoapache:masterfrom Jul 7, 2021
Merged
Remove qualifiers on pushed down predicates / Fix parquet pruning#689alamb merged 3 commits intoapache:masterfrom
alamb merged 3 commits intoapache:masterfrom
Conversation
alamb
force-pushed
the
alamb/fix_parquet_pruning
branch
from
eb980a5 to
432ce23
Compare
alamb
commented
Jul 6, 2021
| // Remove all qualifiers from the scan as the provider | ||
| // doesn't know (nor should care) how the relation was | ||
| // referred to in the query | ||
| let filters = unnormalize_cols(filters.iter().cloned()); |
Contributor
Author
There was a problem hiding this comment.
Here is the actual fix for parquet pruning.
| } | ||
|
|
||
| #[test] | ||
| fn normalize_cols() { |
Contributor
Author
There was a problem hiding this comment.
I added some additional unit test coverage here for normalize when I was writing the unnormalize versions
datafusion/src/logical_plan/expr.rs
Outdated
| /// Recursively call [`Column::normalize`] on all Column expressions | ||
| /// in the `expr` expression tree. | ||
| pub fn normalize_col(e: Expr, schemas: &[&DFSchemaRef]) -> Result<Expr> { | ||
| pub fn normalize_col(expr: Expr, schemas: &[&DFSchemaRef]) -> Result<Expr> { |
Contributor
Author
There was a problem hiding this comment.
Changed to use the name expr to be more consistent with the rest of the codebase
Contributor
Author
|
FYI @houqp |
Dandandan
reviewed
Jul 6, 2021
Dandandan
approved these changes
Jul 6, 2021
alamb
force-pushed
the
alamb/fix_parquet_pruning
branch
from
20f7472 to
de6de17
Compare
alamb
force-pushed
the
alamb/fix_parquet_pruning
branch
from
de6de17 to
c2971f9
Compare
H0TB0X420
pushed a commit
to H0TB0X420/datafusion
that referenced
this pull request
Oct 7, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes ##656
Based on #657 (test for parquet pruning) so review that first
Rationale for this change
Predicate pushdown for parquet is currently broken because the filtering logic gets columns that refer to
foo.barbut the table provider only knows about columns (e.g. only aboutbar)What changes are included in this PR?
unnormalize_exprfunction to remove qualifiers from expressionsAre there any user-facing changes?
Pruning works again