WIP: Update to arrow-rs / parquet 54.1.0

[alamb]

Which issue does this PR close?

• Related to Release arrow-rs / parquet minor version 54.1.0 (Jan 2025) arrow-rs-object-store#27

Rationale for this change

Keep up with dependencies

What changes are included in this PR?

1. Update version of arrow-rs used in DataFusion (currently pinned until release)
2. Update for API changes

Are these changes tested?

By existing CI

Are there any user-facing changes?

[alamb]

• Due to in Expose min/max values for Decimal128/256 and improve docs arrow-rs#6992

[alamb]

• Due to Improve Buffer documentation, deprecate Buffer::from_bytes add From<Bytes> and From<bytes::Bytes> impls arrow-rs#6939

[alamb]

https://github.com/apache/datafusion/actions/runs/12996259121/job/36244947022?pr=14328 appears to be a real difference

 Running bin/sqllogictests.rs (target/ci/deps/sqllogictests-71f9d985f37bce43)
External error: query result mismatch:
[SQL] SELECT
column1,
lag(column2) OVER (order by column1),
lead(column2) OVER (order by column1)
FROM array_data;
[Diff] (-expected|+actual)
a NULL [4, 5, 6]
b [1, 2, 3] [7, 8, 9]
- c [4, 5, 6] NULL
+ c [1, 2, 3] NULL
at test_files/window.slt:4853
External error: query result mismatch:
[SQL] select array_append(column1, column2) from nested_arrays;
[Diff] (-expected|+actual)
- [[1, 2, 3], [2, 9, 1], [7, 8, 9], [1, 2, 3], [1, 7, 4], [4, 5, 6], [7, 8, 9]]
- [[4, 5, 6], [10, 11, 12], [4, 9, 8], [7, 8, 9], [10, 11, 12], [1, 8, 7], [10, 11, 12]]
+ [[1, 2, 3], [2, 9, 1], [7, 8, 9], [1, 2, 3], [1, 7, 4], [4, 5, 6], [4, 5, 6]]
+ [[10, 11, 12], [4, 9, 8], [7, 8, 9], [10, 11, 12], [1, 8, 7], [7, 8, 9], [10, 11, 12]]
at test_files/array.slt:2424

I am git bisecting to figure out what is going on

[alamb]

According to git bisect the error was introduced in this PR

• feat(arrow-select): concat kernel will merge dictionary values for list of dictionaries arrow-rs#6893

andrewlamb@Andrews-MacBook-Pro-2:~/Software/arrow-rs$ git bisect bad
27992680d5677b629d7b4ca11d3f94c2fe905f1a is the first bad commit
commit 27992680d5677b629d7b4ca11d3f94c2fe905f1a (HEAD)
Author: Raz Luvaton <16746759+rluvaton@users.noreply.github.com>
Date: Sat Jan 4 12:24:48 2025 +0200
feat(arrow-select): `concat` kernel will merge dictionary values for list of dictionaries (#6893)
* feat(arrow-select): make list of dictionary merge dictionary keys
TODO:
- [ ] Add support to nested lists
- [ ] Add more tests
- [ ] Fix failing test
* fix concat lists of dictionaries
* format
* remove unused import
* improve test helper
* feat: add merge offset buffers into one
* format
* add reproduction tst
* recommit
* fix clippy
* fix clippy
* fix clippy
* improve offsets code according to code review
* use concat dictionaries
* add specialize code to concat lists to be able to use the concat dictionary logic
* remove the use of ArrayData
* 

You can verify this by applying a diff such as this locally:

diff --git a/Cargo.toml b/Cargo.toml
index 1c8f9f0a1..8025d32b8 100644
--- a/Cargo.toml+++ b/Cargo.toml@@ -190,15 +190,15 @@ unexpected_cfgs = { level = "warn", check-cfg = ["cfg(tarpaulin)"] }
unused_qualifications = "deny"
[patch.crates-io]
-arrow = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-array = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-buffer = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-cast = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-data = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-ipc = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-schema = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-select = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-string = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-ord = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-arrow-flight = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }-parquet = { git = "https://github.com/apache/arrow-rs.git", rev = "6aaff7e38a573f797b31f89f869c3706cbe26e37" }+arrow = { path="/Users/andrewlamb/Software/arrow-rs/arrow" }+arrow-array = { path="/Users/andrewlamb/Software/arrow-rs/arrow-array" }+arrow-buffer = { path="/Users/andrewlamb/Software/arrow-rs/arrow-buffer" }+arrow-cast = { path="/Users/andrewlamb/Software/arrow-rs/arrow-cast" }+arrow-data = { path="/Users/andrewlamb/Software/arrow-rs/arrow-data" }+arrow-ipc = { path="/Users/andrewlamb/Software/arrow-rs/arrow-ipc" }+arrow-schema = { path="/Users/andrewlamb/Software/arrow-rs/arrow-schema" }+arrow-select = { path="/Users/andrewlamb/Software/arrow-rs/arrow-select" }+arrow-string = { path="/Users/andrewlamb/Software/arrow-rs/arrow-string" }+arrow-ord = { path="/Users/andrewlamb/Software/arrow-rs/arrow-ord" }+arrow-flight = { path="/Users/andrewlamb/Software/arrow-rs/arrow-flight" }+parquet = { path="/Users/andrewlamb/Software/arrow-rs/parquet" }

And then running

$ cargo test --test sqllogictests -- window

From the list of commits on https://github.com/apache/arrow-rs/commits/main/?after=6aaff7e38a573f797b31f89f869c3706cbe26e37+34 you can see:

• apache/arrow-rs@2799268 FAILS
• apache/arrow-rs@debc5bf PASSES (one PR before)

[comphead]

is it still WIP?

[comphead]

Oh arrow-rs doesn't plan to publish the official patch?

[alamb]

Yeah -- it isn't officially released yet (I expect it to be done in a day or two)

• https://github.com/apache/arrow-rs/issues/6929

Sorry -- I should have maked this PR as draft