Skip to content

Read single struct fields in ParquetOpener #20603

@adriangb

Description

@adriangb

Currently if you have a filter or projection such as select struct_col[field] from t it gets pushed down into ParquetOpener but:

  • ParquetOpener explicitly rejects filters referencing struct fields:
    //! List-aware predicates (for example, `array_has`, `array_has_all`, and
    //! `array_has_any`) can be evaluated directly during Parquet decoding. Struct
    //! columns and other nested projections that are not explicitly supported will
    //! continue to be evaluated after the batches are materialized.
  • Projections read the entire column instead of the single leaf column (no special handling)

I tried to fix these in ParquetOpener and found that we'd get errors from arrow-rs because of apache/arrow-rs#9066.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions