-
Notifications
You must be signed in to change notification settings - Fork 2k
Open
Description
Currently if you have a filter or projection such as select struct_col[field] from t it gets pushed down into ParquetOpener but:
ParquetOpenerexplicitly rejects filters referencing struct fields:datafusion/datafusion/datasource-parquet/src/row_filter.rs
Lines 62 to 65 in c63ca33
//! List-aware predicates (for example, `array_has`, `array_has_all`, and //! `array_has_any`) can be evaluated directly during Parquet decoding. Struct //! columns and other nested projections that are not explicitly supported will //! continue to be evaluated after the batches are materialized. - Projections read the entire column instead of the single leaf column (no special handling)
I tried to fix these in ParquetOpener and found that we'd get errors from arrow-rs because of apache/arrow-rs#9066.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels