Skip to content

Panic/dropped data when reading parquet files with incompatible shemas #1651

@alamb

Description

@alamb

@thinkharderdev added the ability to read from multiple parquet files that have different schemas in #1622

However, if some of the files have "incompatible" schemas (e.g. a different type), the data can be ignored from that file.

Expected behavior would be to have the plan actually return an error rather than silently ignored.

the issue is that the panic is on the reader thread. In effect it basically just causes the read on that partition to get dropped. We could test for a panic by calling read_partition directly from the test if you'd rather do it that way.

Originally posted by @thinkharderdev in #1622 (comment)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions