Skip to content

chore(deps): update arrow-rs to v57 (major) #5482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

chore(deps): update arrow-rs to v57 (major) #5482

wants to merge 1 commit into from

Conversation

@renovate
Copy link
Contributor

@renovate renovate bot commented Nov 24, 2025 •
edited
Loading

This PR contains the following updates:

Package Type Update Change
arrow-arith workspace.dependencies major 56 -> 57
arrow-array workspace.dependencies major 56 -> 57
arrow-buffer workspace.dependencies major 56 -> 57
arrow-cast workspace.dependencies major 56 -> 57
arrow-data workspace.dependencies major 56 -> 57
arrow-ipc workspace.dependencies major 56 -> 57
arrow-ord workspace.dependencies major 56 -> 57
arrow-schema workspace.dependencies major 56 -> 57
arrow-select workspace.dependencies major 56 -> 57
arrow-string workspace.dependencies major 56 -> 57
parquet workspace.dependencies major 56 -> 57

Release Notes

apache/arrow-rs (arrow-arith)

v57.1.0

Compare Source

Full Changelog

Implemented enhancements:

  • Eliminate bound checks in filter kernels #​8865 [arrow]
  • Respect page index policy option for ParquetObjectReader when it's not skip #​8856 [parquet]
  • Speed up collect_bool and remove unsafe #​8848 [arrow]
  • Error reading parquet FileMetaData with empty lists encoded as element-type=0 #​8826 [parquet]
  • ValueStatistics methods can't be used from generic context in external crate #​8823 [parquet]
  • Custom Pretty-Printing Implementation for Column when Formatting Record Batches #​8821 [arrow]
  • Parquet-concat: supports bloom filter and page index #​8804 [parquet]
  • [Parquet] virtual row number support #​7299
  • [Variant] Enforce shredded-type validation in shred_variant #​8795 [arrow]
  • Simplify decision logic to call FilterBuilder::optimize or not #​8781 [arrow]
  • [Variant] Add variant to arrow for DataType::{Binary, LargeBinary, BinaryView} #​8767 [arrow]
  • Provide algorithm that allows zipping arrays whose values are not prealigned #​8752 [arrow]
  • [Parquet] ParquetMetadataReader decodes too much metadata under point-get scenerio #​8751 [parquet]
  • arrow-json supports encoding binary arrays, but not decoding #​8736 [arrow]
  • Allow FilterPredicate instances to be reused for RecordBatches #​8692 [arrow]
  • ArrowJsonBatch::from_batch is incomplete #​8684 [arrow]
  • parquet-layout: More info about layout including footer size, page index, bloom filter? #​8682 [parquet]
  • Rewrite ParquetRecordBatchStream (async API) in terms of the PushDecoder #​8677 [parquet]
  • [JSON] Add encoding for binary view #​8674 [arrow]
  • Refactor arrow-cast decimal casting to unify the rescale logic used in Parquet variant casts #​8670 [arrow]
  • [Variant] Support Uuid/FixedSizeBinary(16) shredding #​8665
  • [Parquet]There should be an encoding counter to know how many encodings the repo supports in total #​8662 [parquet]
  • Improve parse_data_type for List, ListView, LargeList, LargeListView, FixedSizeList, Union, Map, RunEndCoded. #​8648 [arrow]
  • [Variant] Support variant to arrow primitive support null/time/decimal_* #​8637
  • Return error from RleDecoder::reset rather than panic #​8632 [parquet]
  • Add bitwise ops on BooleanBufferBuilder and MutableBuffer that mutate directly the buffer #​8618 [arrow]
  • [Variant] Add variant_to_arrow Utf-8, LargeUtf8, Utf8View types support #​8567 [arrow]

Fixed bugs:

  • Regression: Parsing List(Int64) results in nullable list in 57.0.0 and a non-nullable list in 57.1.0 #​8883
  • Regression: FixedSlizeList data type parsing fails on 57.1.0 #​8880
  • (dyn ArrayFormatterFactory + 'static) can't be safely shared between threads #​8875
  • RowNumber reader has wrong row group ordering #​8864 [parquet]
  • ThriftMetadataWriter::write_column_indexes cannot handle a ColumnIndexMetaData::NONE #​8815 [parquet]
  • "Archery test With other arrows" Integration test failing on main: #​8813 [arrow]
  • [Parquet] Writing in 57.0.0 seems 10% slower than 56.0.0 #​8783 [parquet]
  • Parquet reader cannot handle files with unknown logical types #​8776 [parquet]
  • zip now treats nulls as false in provided mask regardless of the underlying bit value #​8721 [arrow]
  • [avro] Incorrect version in crate.io landing page #​8691 [arrow]
  • Array: ViewType gc() has bug when array sum length exceed i32::MAX #​8681 [arrow]
  • Parquet 56: encounter error: item_reader def levels are None when reading nested field with row filter #​8657 [parquet]
  • Degnerate and non-nullable FixedSizeListArrays are not handled #​8623 [arrow]
  • [Parquet]Performance Degradation with RowFilter on Unsorted Columns due to Fragmented ReadPlan #​8565 [parquet]

Documentation updates:

Performance improvements:

Closed issues:

  • Variant to NullType conversion ignores strict casting #​8810
  • Unify display representation for Field #​8784
  • Misleading configuration name: skip_arrow_metadata #​8780
  • Inconsistent display for types with Metadata #​8761 [arrow]
  • Internal arrow-integration-test crate is linked from arrow docs #​8739 [arrow]
  • Add benchmark for RunEndEncoded casting #​8709 [arrow]
  • [Varaint] Support VariantArray::value to return a Result<Variant> #​8672

Merged pull requests:

* This Changelog was automatically generated by github_changelog_generator

v57.0.0: arrow 57.0.0

Compare Source

Changelog
57.0.0 (2025-10-19)

Full Changelog

Breaking changes:

Implemented enhancements:

  • Added Avro support (new arrow-avro crate) #​4886
  • parquet-rewrite: supports compression level and write batch size #​8639
  • Error not panic when int96 stastistics aren't size 12 #​8614 [parquet]
  • [Variant] Make VariantArray iterable #​8612
  • [Variant] impl PartialEq for VariantArray #​8610
  • [Variant] Remove potential panics when probing VariantArray #​8609
  • [Variant] Remove ceremony of going from list of Variant to VariantArray #​8606
  • Eliminate redundant validation in RecordBatch::project #​8591 [arrow]
  • [PARQUET][BENCH] Arrow writer bench with compression and/or page v2 #​8559 [parquet]
  • [Variant] casting functions are confusingly named #​8531 [parquet]
  • Support writing GeospatialStatistics in Parquet writer #​8523 [parquet]
  • [thrift-remodel] Optimize convert_row_groups #​8517 [parquet]
  • [Variant] Add variant to arrow primitive support for boolean/timestamp/time #​8515
  • Test thrift-remodel branch with DataFusion #​8513 [parquet]
  • Make UnionArray::is_dense Method Public #​8503 [arrow]
  • Add append_n method to FixedSizeBinaryDictionaryBuilder #​8497 [arrow]
  • [Parquet] Reduce size of ParquetMetadata when encryption feature is enabled #​8469 [parquet]
  • [Parquet] Remove useless mut requirements in geting bloom filter function #​8461 [parquet]
  • Change serde dependency to serde_core where applicable #​8451 [arrow]
  • [Parquet] Split ParquetMetadataReader into IO/decoder state machine and thrift parsing #​8439 [parquet]
  • Remove compiler warning for redundant config enablement #​8412 [arrow]
  • Add geospatial statistics creation support for GEOMETRY/GEOGRAPHY Parquet logical types #​8411 [arrow]
  • arrow_json lacks with_timestamp_format functions like arrow_csv had offered #​8398 [arrow]
  • Unify API for writing column chunks / row groups in parallel #​8389 [parquet]
  • Reuse zstd context in arrow IPC writer #​8386 [arrow] [arrow-flight]
  • [Variant] Support reading/writing Parquet Variant LogicalType #​8370 [parquet]
  • [Variant] Implement a shred_variant function #​8361
  • [Parquet] Expose ReadPlan and ReadPlanBuilder #​8347 [parquet]
  • [Variant] [Shredding] Support typed_access for List #​8337 [parquet]
  • [Variant] [Shredding] Support typed_access for Struct #​8336 [parquet]
  • [Variant] [Shredding] Support typed_access for Time64(Microsecond) #​8334 [parquet]
  • [Variant] [Shredding] Support typed_access for Decimal128 #​8332 [parquet]
  • [Variant] [Shredding] Support typed_access for Timestamp(Microsecond, _) and Timestamp(Nanosecond, _) #​8331 [parquet]
  • [Variant] [Shredding] Support typed_access for Date32 #​8330 [parquet]
  • [Variant] Support strict casting for all data types #​8303
  • [Variant] Support typed access for string types in variant_get #​8285
  • [Variant]: Implement DataType::FixedSizeList support for cast_to_variant kernel #​8281

Fixed bugs:

  • Fix arrow-avro Writer Documentation related to AvroBinaryFormat #​8631 [arrow]
  • Decimal -> Decimal cast wrongly fails for large scale reduction #​8579 [arrow]
  • [Parquet] Avoid fetching multiple pages when max_predicate_cache_sizeis 0 #​8542 [parquet]
  • DataType parsing no longer works correctly for old formatted timestamps #​8539 [parquet] [arrow]
  • [Parquet] ArrowWriter flush does not work #​8534 [parquet]
  • arrow::compute::interleave fails with struct arrays with no fields #​8533 [arrow]
  • [Parquet] Over memory consumation for writer page v1 compressed #​8526 [parquet]
  • Incorrect Behavior of Collecting a filtered iterator to a BooleanArray #​8505 [arrow]
  • [Parquet] ProjectionMask::columns name handling is bug prone #​8443 [parquet]
  • [Variant] Shredded typed_value columns must have valid variant types #​8435 [parquet]
  • cargo test -p parquet fails with default ulimit #​8406 [parquet]
  • Column with List(Struct) causes failed to decode level data for struct array #​8404 [parquet]
  • Binaryview Utf8 Cast Issue #​8403 [arrow]
  • Decimal precision validation displays value without accounting for scale #​8382 [arrow]
  • [Variant] VariantArray::data_type returns StructType, causing Array::as_struct to panic #​8319 [parquet]
  • [Variant] writing a VariantArray to parquet panics #​8296 [parquet]

Documentation updates:

Performance improvements:

Closed issues:

  • Utf-8, LargeUtf8, Utf8View #​8601
  • [Variant] Improve the get type logic for DataType in variant to arrow row builder #​8538
  • Add a README.md for arrow-avro #​8504 [arrow]
  • Fix UnionArray references to "positive" values #​8418 [arrow]
  • [Variant] metadata field should be marked is non-nullable #​8410 [parquet]
  • [Avro] Example read_with_utf8view.rs fails to run with error "Error: ParseError("Unexpected EOF while reading Avro header")" #​8380 [arrow]
  • [Geospatial]: Add CI checks for parquet-geospatial crate #​8377
  • [Geospatial] Create new parquet-geometry crate #​8374

Merged pull requests:

Configuration

📅 Schedule: Branch creation - Between 12:00 AM and 03:59 AM, only on Monday ( * 0-3 * * 1 ) (UTC), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about these updates again.

  • If you want to rebase/retry this PR, check this box

This PR was generated by Mend Renovate. View the repository job log.

@renovate
Copy link
Contributor Author

renovate bot commented Nov 24, 2025 •
edited
Loading

⚠️ Artifact update problem

Renovate failed to update an artifact related to this branch. You probably do not want to merge this PR as-is.

♻ Renovate will retry this branch, including artifacts, only when one of the following happens:

  • any of the package files in this branch needs updating, or
  • the branch becomes conflicted, or
  • you click the rebase/retry checkbox if found above, or
  • you rename this PR's title to start with "rebase!" to trigger it manually

The artifact failure details are included below:

File name: Cargo.lock
Command failed: cargo update --config net.git-fetch-with-cli=true --manifest-path Cargo.toml --package arrow-arith@56.2.0 --precise 57.1.0
    Updating crates.io index
error: failed to select a version for the requirement `arrow-arith = "^56.1"`
candidate versions found which didn't match: 57.1.0
location searched: crates.io index
required by package `lance v0.39.0`
    ... which satisfies dependency `lance = "^0.39.0"` of package `bench-vortex v0.1.0 (/tmp/renovate/repos/github/vortex-data/vortex/bench-vortex)`

@renovate renovate bot added the chore Release label indicating a trivial change label Nov 24, 2025
@codspeed-hq
Copy link

codspeed-hq bot commented Nov 24, 2025 •
edited
Loading

CodSpeed Performance Report

Merging #5482 will degrade performances by 51.14%

Comparing renovate/major-57-arrow-rs (ab3e18f) with develop (5458ab2)

Summary

⚡ 4 improvements
❌ 111 regressions
✅ 1390 untouched
🆕 7 new
⏩ 157 skipped1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
❌ new_alp_prim_test_between[f32, 32768] 376.2 µs 503.1 µs -25.21%
❌ old_alp_prim_test_between[f32, 16384] 192 µs 269.5 µs -28.78%
❌ old_alp_prim_test_between[f32, 32768] 366.6 µs 498 µs -26.39%
❌ old_alp_prim_test_between[f64, 16384] 232 µs 326.2 µs -28.88%
❌ old_alp_prim_test_between[f64, 32768] 356.7 µs 546.5 µs -34.73%
❌ old_bp_prim_test_between[i16, 16384] 149.1 µs 240.6 µs -38.02%
❌ old_bp_prim_test_between[i16, 2048] 96.6 µs 108.7 µs -11.06%
❌ old_bp_prim_test_between[i16, 32768] 207.6 µs 391.1 µs -46.91%
❌ old_bp_prim_test_between[i32, 16384] 168.1 µs 245.3 µs -31.46%
❌ old_bp_prim_test_between[i32, 2048] 96.9 µs 109.1 µs -11.17%
❌ old_bp_prim_test_between[i32, 32768] 258.3 µs 398.7 µs -35.21%
❌ old_bp_prim_test_between[i64, 16384] 206.7 µs 302.9 µs -31.74%
❌ old_bp_prim_test_between[i64, 2048] 105.8 µs 117.7 µs -10.08%
❌ old_bp_prim_test_between[i64, 32768] 320.2 µs 511.7 µs -37.42%
❌ old_raw_prim_test_between[f32, 16384] 149.1 µs 216.6 µs -31.17%
❌ old_raw_prim_test_between[f32, 32768] 219 µs 352.3 µs -37.83%
❌ old_raw_prim_test_between[f64, 16384] 178.4 µs 252.6 µs -29.35%
❌ old_raw_prim_test_between[f64, 2048] 88.7 µs 98.9 µs -10.3%
❌ old_raw_prim_test_between[f64, 32768] 277.3 µs 424 µs -34.61%
❌ old_raw_prim_test_between[i32, 16384] 139.5 µs 217.5 µs -35.88%
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Footnotes

  1. 157 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@renovate renovate bot changed the title chore(deps): update arrow-rs to v57 (major) Update arrow-rs to v57 (major) Nov 24, 2025
@renovate renovate bot force-pushed the renovate/major-57-arrow-rs branch from a0f5d8f to f3fe6c4 Compare November 24, 2025 22:41
@renovate renovate bot force-pushed the renovate/major-57-arrow-rs branch from f3fe6c4 to ab3e18f Compare November 24, 2025 23:09
@renovate renovate bot changed the title Update arrow-rs to v57 (major) chore(deps): update arrow-rs to v57 (major) Nov 26, 2025
@renovate renovate bot changed the title chore(deps): update arrow-rs to v57 (major) Update arrow-rs to v57 (major) Nov 27, 2025
@renovate renovate bot changed the title Update arrow-rs to v57 (major) chore(deps): update arrow-rs to v57 (major) Nov 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

chore Release label indicating a trivial change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant