WIP: Update to arrow-rs / parquet `54.1.0` #14328

alamb · 2025-01-27T18:54:40Z

Which issue does this PR close?

Related to Release arrow-rs / parquet minor version 54.1.0 (Jan 2025) arrow-rs-object-store#27

Rationale for this change

Keep up with dependencies

What changes are included in this PR?

Update version of arrow-rs used in DataFusion (currently pinned until release)
Update for API changes

Are these changes tested?

By existing CI

Are there any user-facing changes?

alamb · 2025-01-27T18:56:14Z

datafusion/optimizer/src/unwrap_cast_in_comparison.rs

            // value is `-999`
-            MIN_DECIMAL_FOR_EACH_PRECISION[*precision as usize - 1],
-            MAX_DECIMAL_FOR_EACH_PRECISION[*precision as usize - 1],
+            MIN_DECIMAL128_FOR_EACH_PRECISION[*precision as usize],


Due to in Expose min/max values for Decimal128/256 and improve docs arrow-rs#6992

datafusion/core/src/datasource/physical_plan/arrow_file.rs

                        decoder.read_dictionary(
                            dict_block,
-                            &Buffer::from_bytes(dict_result.into()),
+                            &Buffer::from(dict_result),


alamb · 2025-01-27T18:58:14Z

datafusion/core/src/datasource/physical_plan/arrow_file.rs

-                            dict_block,
-                            &Buffer::from_bytes(dict_result.into()),
-                        )?;
+                        decoder


Due to Improve Buffer documentation, deprecate Buffer::from_bytes add From<Bytes> and From<bytes::Bytes> impls arrow-rs#6939

alamb · 2025-01-27T20:19:10Z

https://github.com/apache/datafusion/actions/runs/12996259121/job/36244947022?pr=14328 appears to be a real difference

     Running bin/sqllogictests.rs (target/ci/deps/sqllogictests-71f9d985f37bce43)
External error: query result mismatch:
[SQL] SELECT
  column1,
  lag(column2) OVER (order by column1),
  lead(column2) OVER (order by column1)
FROM array_data;
[Diff] (-expected|+actual)
    a NULL [4, 5, 6]
    b [1, 2, 3] [7, 8, 9]
-   c [4, 5, 6] NULL
+   c [1, 2, 3] NULL
at test_files/window.slt:4853

External error: query result mismatch:
[SQL] select array_append(column1, column2) from nested_arrays;
[Diff] (-expected|+actual)
-   [[1, 2, 3], [2, 9, 1], [7, 8, 9], [1, 2, 3], [1, 7, 4], [4, 5, 6], [7, 8, 9]]
-   [[4, 5, 6], [10, 11, 12], [4, 9, 8], [7, 8, 9], [10, 11, 12], [1, 8, 7], [10, 11, 12]]
+   [[1, 2, 3], [2, 9, 1], [7, 8, 9], [1, 2, 3], [1, 7, 4], [4, 5, 6], [4, 5, 6]]
+   [[10, 11, 12], [4, 9, 8], [7, 8, 9], [10, 11, 12], [1, 8, 7], [7, 8, 9], [10, 11, 12]]
at test_files/array.slt:2424

I am git bisecting to figure out what is going on

alamb · 2025-01-27T20:41:57Z

According to git bisect the error was introduced in this PR

feat(arrow-select): concat kernel will merge dictionary values for list of dictionaries arrow-rs#6893

andrewlamb@Andrews-MacBook-Pro-2:~/Software/arrow-rs$ git bisect bad
27992680d5677b629d7b4ca11d3f94c2fe905f1a is the first bad commit
commit 27992680d5677b629d7b4ca11d3f94c2fe905f1a (HEAD)
Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com>
Date:   Sat Jan 4 12:24:48 2025 +0200

    feat(arrow-select): `concat` kernel will merge dictionary values for list of dictionaries (#6893)

    * feat(arrow-select): make list of dictionary merge dictionary keys

    TODO:
    - [ ] Add support to nested lists
    - [ ] Add more tests
    - [ ] Fix failing test

    * fix concat lists of dictionaries

    * format

    * remove unused import

    * improve test helper

    * feat: add merge offset buffers into one

    * format

    * add reproduction tst

    * recommit

    * fix clippy

    * fix clippy

    * fix clippy

    * improve offsets code according to code review

    * use concat dictionaries

    * add specialize code to concat lists to be able to use the concat dictionary logic

    * remove the use of ArrayData
    *

You can verify this by applying a diff such as this locally:

diff --git a/Cargo.toml b/Cargo.toml
index 1c8f9f0a1..8025d32b8 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -190,15 +190,15 @@ unexpected_cfgs = { level = "warn", check-cfg = ["cfg(tarpaulin)"] }
 unused_qualifications = "deny"

 [patch.crates-io]
-arrow = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-array = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-buffer = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-cast = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-data = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-ipc = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-schema = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-select = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-string = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-ord = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-arrow-flight = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
-parquet = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }
+arrow = { path="/Users/andrewlamb/Software/arrow-rs/arrow" }
+arrow-array = { path="/Users/andrewlamb/Software/arrow-rs/arrow-array" }
+arrow-buffer = { path="/Users/andrewlamb/Software/arrow-rs/arrow-buffer" }
+arrow-cast = { path="/Users/andrewlamb/Software/arrow-rs/arrow-cast" }
+arrow-data = { path="/Users/andrewlamb/Software/arrow-rs/arrow-data" }
+arrow-ipc = { path="/Users/andrewlamb/Software/arrow-rs/arrow-ipc" }
+arrow-schema = { path="/Users/andrewlamb/Software/arrow-rs/arrow-schema" }
+arrow-select = { path="/Users/andrewlamb/Software/arrow-rs/arrow-select" }
+arrow-string = { path="/Users/andrewlamb/Software/arrow-rs/arrow-string" }
+arrow-ord = { path="/Users/andrewlamb/Software/arrow-rs/arrow-ord" }
+arrow-flight = { path="/Users/andrewlamb/Software/arrow-rs/arrow-flight" }
+parquet = { path="/Users/andrewlamb/Software/arrow-rs/parquet" }

And then running

$ cargo test --test sqllogictests -- window

From the list of commits on https://github.com/apache/arrow-rs/commits/main/?after=6aaff7e38a573f797b31f89f869c3706cbe26e37+34 you can see:

apache/arrow-rs@2799268 FAILS
apache/arrow-rs@debc5bf PASSES (one PR before)

comphead · 2025-01-30T18:35:38Z

is it still WIP?

comphead · 2025-01-30T18:37:35Z

Cargo.toml

 unused_qualifications = "deny"
+
+[patch.crates-io]
+arrow = { git = "https://github.com/apache/arrow-rs.git", rev = "b1bfa0f5852536ef2edb43ac8ddb66c39e8a7233" }


Oh arrow-rs doesn't plan to publish the official patch?

Yeah -- it isn't officially released yet (I expect it to be done in a day or two)

https://github.com/apache/arrow-rs/issues/6929

Sorry -- I should have maked this PR as draft

github-actions bot added optimizer Optimizer rules core Core DataFusion crate labels Jan 27, 2025

alamb commented Jan 27, 2025

View reviewed changes

datafusion/core/src/datasource/physical_plan/arrow_file.rs Outdated

decoder.read_dictionary(

dict_block,

&Buffer::from_bytes(dict_result.into()),

&Buffer::from(dict_result),

This comment was marked as outdated.

Sign in to view

alamb commented Jan 27, 2025

View reviewed changes

alamb force-pushed the alamb/update_arrow_44.1 branch from 6ce98ad to a0097a2 Compare January 29, 2025 12:55

alamb added 4 commits January 29, 2025 07:56

TEMP: Pin to pre-release arrow-rs

7dbcfc7

Update to latest arrow patch

0c0d15e

Update cargo.lock

1b1b93d

chore: update for deprecated APIs

7758936

alamb force-pushed the alamb/update_arrow_44.1 branch from a0097a2 to 7758936 Compare January 29, 2025 12:56

comphead reviewed Jan 30, 2025

View reviewed changes

alamb marked this pull request as draft January 30, 2025 20:01

alamb mentioned this pull request Feb 3, 2025

chore: Upgrade to arrow/parquet 54.1.0 and fix clippy/ci #14415

Merged

alamb closed this in #14415 Feb 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP: Update to arrow-rs / parquet `54.1.0` #14328

WIP: Update to arrow-rs / parquet `54.1.0` #14328

Uh oh!

alamb commented Jan 27, 2025

Uh oh!

alamb Jan 27, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

alamb Jan 27, 2025

Uh oh!

alamb commented Jan 27, 2025

Uh oh!

alamb commented Jan 27, 2025 •

edited

Loading

Uh oh!

comphead commented Jan 30, 2025

Uh oh!

comphead Jan 30, 2025

Uh oh!

alamb Jan 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

WIP: Update to arrow-rs / parquet 54.1.0 #14328

WIP: Update to arrow-rs / parquet 54.1.0 #14328

Uh oh!

Conversation

alamb commented Jan 27, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb Jan 27, 2025

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

alamb Jan 27, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 27, 2025

Uh oh!

alamb commented Jan 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

comphead commented Jan 30, 2025

Uh oh!

comphead Jan 30, 2025

Choose a reason for hiding this comment

Uh oh!

alamb Jan 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

WIP: Update to arrow-rs / parquet `54.1.0` #14328

WIP: Update to arrow-rs / parquet `54.1.0` #14328

alamb commented Jan 27, 2025 •

edited

Loading