Skip to content

parquet page level skipping (page index pruning) doesn't work with evolved schemas #5104

@alamb

Description

@alamb

Describe the bug
parquet page level skipping (page index pruning) panic's with evolved schemas

To Reproduce

Enable setting like

        pub enable_page_index: bool, default = true

And then run test:

cargo test -p datafusion --lib -- evolved_schema_disjoint_schema_filter

Which fails

test physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter ... FAILED

failures:

---- physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter stdout ----
thread 'physical_plan::file_format::parquet::tests::evolved_schema_disjoint_schema_filter' panicked at 'index out of bounds: the len is 1 but the index is 1', /Users/alamb/.cargo/registry/src/github.com-1ecc6299db9ec823/parquet-31.0.0/src/file/metadata.rs:251:10

Expected behavior
Test should pass

Additional context
I found this while working to enable this feature by default
#5099

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions