Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Raise proper error for mismatching parquet schema instead of panicking #17321

Merged
merged 3 commits into from
Jul 2, 2024

Conversation

nameexhaustion
Copy link
Collaborator

@nameexhaustion nameexhaustion commented Jul 1, 2024

This has been reported quite a bit lately, so got around to it

Fixes #13436, fixes #17067, fixes #17255, fixes #13438

Maybe also #17311, but I don't have a repro for that one

@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars labels Jul 1, 2024
@etiennebacher
Copy link

Hi, just a quick note that in the first post you need to have one "fixes" per issue number to close them, e.g. "fixes #A, #B, #C" will only close #A :)

Copy link

codecov bot commented Jul 1, 2024

Codecov Report

Attention: Patch coverage is 81.36646% with 30 lines in your changes missing coverage. Please review.

Project coverage is 80.73%. Comparing base (41610e3) to head (ae742b6).
Report is 4 commits behind head on main.

Files Patch % Lines
crates/polars-core/src/schema.rs 65.62% 22 Missing ⚠️
crates/polars-arrow/src/datatypes/schema.rs 71.42% 4 Missing ⚠️
crates/polars-io/src/parquet/read/reader.rs 93.75% 2 Missing ⚠️
crates/polars-plan/src/plans/conversion/scans.rs 0.00% 1 Missing ⚠️
crates/polars-plan/src/plans/functions/count.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main   #17321   +/-   ##
=======================================
  Coverage   80.72%   80.73%           
=======================================
  Files        1475     1475           
  Lines      193333   193383   +50     
  Branches     2760     2760           
=======================================
+ Hits       156077   156119   +42     
- Misses      36746    36754    +8     
  Partials      510      510           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@nameexhaustion
Copy link
Collaborator Author

Hi, just a quick note that in the first post you need to have one "fixes" per issue number to close them, e.g. "fixes #A, #B, #C" will only close #A :)

Very cool tip, thanks 🙂

@ritchie46 ritchie46 merged commit 8eef76e into pola-rs:main Jul 2, 2024
30 checks passed
@ion-elgreco
Copy link
Contributor

@nameexhaustion @ritchie46, but this doesn't add support read parquets with mixed schema's though. This is something pyarrow supports

@stinodego
Copy link
Member

stinodego commented Jul 2, 2024

Yeah we should be careful not to close issues when a panic has been turned into an error, if it should not throw an error in the first place. In this case, mismatching schemas indicate schema evolution and we may have to actually support this.

@nameexhaustion nameexhaustion deleted the parquet-schema branch July 8, 2024 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix Bug fix python Related to Python Polars rust Related to Rust Polars
Projects
None yet
5 participants