WIP: Upgrade arrow/parquet to 53.1.0
#12724
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
functions
sqllogictest
alamb
commented
Oct 2, 2024
| &suffix[metadata_start - footer_start..suffix_len - 8], | ||
| )?) | ||
| } | ||
| ParquetMetaDataReader::new() |
There was a problem hiding this comment.
We can finally have nice things -- this code is way simpler now thanks to all @etseidl's work upstream on apache/arrow-rs#6447
alamb
commented
Oct 2, 2024
| @@ -61,7 +61,7 @@ logical_plan | |||
| physical_plan | |||
| 01)CoalesceBatchesExec: target_batch_size=8192 | |||
| 02)--FilterExec: column1@0 != 42 | |||
| 03)----ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..87], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:87..174], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:174..261], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:261..347]]}, projection=[column1], predicate=column1@0 != 42, pruning_predicate=CASE WHEN column1_null_count@2 = column1_row_count@3 THEN false ELSE column1_min@0 != 42 OR 42 != column1_max@1 END, required_guarantees=[column1 not in (42)] | |||
| 03)----ParquetExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..88], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:88..176], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:176..264], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:264..351]]}, projection=[column1], predicate=column1@0 != 42, pruning_predicate=CASE WHEN column1_null_count@2 = column1_row_count@3 THEN false ELSE column1_min@0 != 42 OR 42 != column1_max@1 END, required_guarantees=[column1 not in (42)] | |||
There was a problem hiding this comment.
these files are slightly different sizes (larger) due to apache/arrow-rs#6490
alamb
commented
Oct 2, 2024
| // specific language governing permissions and limitations | ||
| // under the License. | ||
|
|
||
| //! Common utilities for implementing regex functions |
There was a problem hiding this comment.
@tlm365 ported this upstream to arrow-rs in apache/arrow-rs#6376 ❤️
alamb
commented
Oct 2, 2024
| @@ -771,7 +771,7 @@ mod tests { | |||
| "c7: Int64", | |||
| "c8: Int64", | |||
| "c9: Int64", | |||
| "c10: Int64", | |||
| "c10: Utf8", | |||
There was a problem hiding this comment.
This is due to apache/arrow-rs#6481 -- the fix for #3174 (see #3174 (comment))
alamb
force-pushed
the
alamb/prep_53.1.0
branch
from
4c03f29
to
f66be51
Compare
alamb
commented
Oct 2, 2024
| @@ -103,65 +99,6 @@ impl PhysicalExpr for IsNullExpr { | |||
| } | |||
| } | |||
|
|
|||
| /// workaround <https://github.com/apache/arrow-rs/issues/6017>, | |||
There was a problem hiding this comment.
Use the implementation added in apache/arrow-rs#6303 from @gstvg
3 tasks
alamb
force-pushed
the
alamb/prep_53.1.0
branch
from
f66be51
to
dbee260
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Related to apache/arrow-rs#6340
Closes #3174
Rationale for this change
Keep up with dependencies
I am creating this PR in advance as a way to test the arrow release candidate as well
What changes are included in this PR?
53.1.0Are these changes tested?
Are there any user-facing changes?