Upgrade DataFusion to arrow-rs/parquet 57.2.0 by alamb · Pull Request #19355 · apache/datafusion

alamb · 2025-12-16T15:01:40Z

Which issue does this PR close?

Related to Release arrow-rs / parquet Minor version 57.2.0 (December 2025) arrow-rs#8465
ClosesCoercion from Dictionary(UInt8, LargeUtf8) to Utf8View #19290

Rationale for this change

Upgrade to latest arrow version

I made this PR early to test the arrow release with DataFusion

What changes are included in this PR?

Update arrow
Updates for API

Are these changes tested?

Yes by CI

Are there any user-facing changes?

No

alamb · 2025-12-18T21:06:35Z

-# Upstream arrow-rs issue: https://github.com/apache/arrow-rs/issues/8841
-# This should succeed after we receive the fix
-query error Arrow error: Compute error: Internal Error: Cannot cast BinaryView to BinaryArray of expected type
+query I


@Jefffrey fixed them upstream ❤️

I think this fix will also close #19290; quickly tested checking this PR branch out and trying the test in the issue and it seems to execute successfully

I updated the PR description to close #19290 as well

alamb · 2025-12-18T21:07:01Z

 # coerce structs with different field orders,
-# (note the *value*s are from column2 but the field name is 'xxx', as the coerced
-# type takes the field name from the last argument (column3)
+# should keep the same field values


These tests now work thanks to @brancz in
Struct casting field order arrow-rs#8871

alamb · 2025-12-18T21:07:07Z

 drop table t;

-query error DataFusion error: Optimizer rule 'simplify_expressions' failed[\s\S]*Arrow error: Cast error: Cannot cast string 'a' to value of Float64 type
+statement ok


These tests now work thanks to @brancz in
Struct casting field order arrow-rs#8871

Dandandan · 2025-12-21T12:50:15Z

run benchmarks

Jefffrey · 2025-12-21T14:19:33Z

FYI @tobixdev can keep an eye on this PR as it should include the fix from arrow-rs needed to do #19085 -> can start it once this PR merges

alamb · 2025-12-23T14:37:06Z

run benchmarks

alamb-ghbot · 2025-12-23T15:18:24Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_57.2.0
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_update_arrow_57.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2767.72 ms │ 2671.31 ms │ no change │
│ QQuery 1 │ 1251.11 ms │ 1188.18 ms │ +1.05x faster │
│ QQuery 2 │ 2449.97 ms │ 2294.54 ms │ +1.07x faster │
│ QQuery 3 │ 1166.19 ms │ 1150.24 ms │ no change │
│ QQuery 4 │ 2274.51 ms │ 2284.93 ms │ no change │
│ QQuery 5 │ 28696.11 ms │ 28601.98 ms │ no change │
│ QQuery 6 │ 3914.27 ms │ 3971.77 ms │ no change │
│ QQuery 7 │ 3651.74 ms │ 3616.21 ms │ no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 46171.62ms │
│ Total Time (alamb_update_arrow_57.2) │ 45779.17ms │
│ Average Time (HEAD) │ 5771.45ms │
│ Average Time (alamb_update_arrow_57.2) │ 5722.40ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 6 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_update_arrow_57.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.07 ms │ 2.32 ms │ 1.12x slower │
│ QQuery 1 │ 48.99 ms │ 49.41 ms │ no change │
│ QQuery 2 │ 137.78 ms │ 132.68 ms │ no change │
│ QQuery 3 │ 149.93 ms │ 155.75 ms │ no change │
│ QQuery 4 │ 1092.93 ms │ 1100.46 ms │ no change │
│ QQuery 5 │ 1509.24 ms │ 1485.47 ms │ no change │
│ QQuery 6 │ 2.02 ms │ 2.05 ms │ no change │
│ QQuery 7 │ 54.26 ms │ 55.34 ms │ no change │
│ QQuery 8 │ 1441.32 ms │ 1459.08 ms │ no change │
│ QQuery 9 │ 1789.92 ms │ 1858.92 ms │ no change │
│ QQuery 10 │ 352.62 ms │ 367.90 ms │ no change │
│ QQuery 11 │ 404.84 ms │ 420.42 ms │ no change │
│ QQuery 12 │ 1362.12 ms │ 1389.97 ms │ no change │
│ QQuery 13 │ 2027.32 ms │ 2022.58 ms │ no change │
│ QQuery 14 │ 1227.95 ms │ 1258.69 ms │ no change │
│ QQuery 15 │ 1251.14 ms │ 1253.71 ms │ no change │
│ QQuery 16 │ 2668.86 ms │ 2682.87 ms │ no change │
│ QQuery 17 │ 2667.08 ms │ 2677.97 ms │ no change │
│ QQuery 18 │ 5631.29 ms │ 4942.96 ms │ +1.14x faster │
│ QQuery 19 │ 119.93 ms │ 123.27 ms │ no change │
│ QQuery 20 │ 1949.89 ms │ 1838.56 ms │ +1.06x faster │
│ QQuery 21 │ 2242.78 ms │ 2173.30 ms │ no change │
│ QQuery 22 │ 3898.87 ms │ 3720.39 ms │ no change │
│ QQuery 23 │ 22288.53 ms │ 12109.53 ms │ +1.84x faster │
│ QQuery 24 │ 212.76 ms │ 207.87 ms │ no change │
│ QQuery 25 │ 480.56 ms │ 479.09 ms │ no change │
│ QQuery 26 │ 229.81 ms │ 221.38 ms │ no change │
│ QQuery 27 │ 2808.68 ms │ 2702.34 ms │ no change │
│ QQuery 28 │ 24994.86 ms │ 21730.94 ms │ +1.15x faster │
│ QQuery 29 │ 988.56 ms │ 987.75 ms │ no change │
│ QQuery 30 │ 1366.47 ms │ 1332.78 ms │ no change │
│ QQuery 31 │ 1375.51 ms │ 1304.00 ms │ +1.05x faster │
│ QQuery 32 │ 5458.33 ms │ 4682.87 ms │ +1.17x faster │
│ QQuery 33 │ 6464.69 ms │ 5781.71 ms │ +1.12x faster │
│ QQuery 34 │ 6078.81 ms │ 5871.69 ms │ no change │
│ QQuery 35 │ 1963.68 ms │ 1936.37 ms │ no change │
│ QQuery 36 │ 68.30 ms │ 67.42 ms │ no change │
│ QQuery 37 │ 45.32 ms │ 46.41 ms │ no change │
│ QQuery 38 │ 68.03 ms │ 68.16 ms │ no change │
│ QQuery 39 │ 104.91 ms │ 104.49 ms │ no change │
│ QQuery 40 │ 28.10 ms │ 27.21 ms │ no change │
│ QQuery 41 │ 22.06 ms │ 21.91 ms │ no change │
│ QQuery 42 │ 20.37 ms │ 19.70 ms │ no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 107101.47ms │
│ Total Time (alamb_update_arrow_57.2) │ 90877.71ms │
│ Average Time (HEAD) │ 2490.73ms │
│ Average Time (alamb_update_arrow_57.2) │ 2113.44ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 35 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_update_arrow_57.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 137.38 ms │ 132.73 ms │ no change │
│ QQuery 2 │ 29.66 ms │ 30.00 ms │ no change │
│ QQuery 3 │ 35.07 ms │ 38.82 ms │ 1.11x slower │
│ QQuery 4 │ 29.02 ms │ 28.79 ms │ no change │
│ QQuery 5 │ 88.44 ms │ 88.90 ms │ no change │
│ QQuery 6 │ 19.93 ms │ 19.63 ms │ no change │
│ QQuery 7 │ 235.82 ms │ 227.84 ms │ no change │
│ QQuery 8 │ 35.73 ms │ 37.59 ms │ 1.05x slower │
│ QQuery 9 │ 101.05 ms │ 107.10 ms │ 1.06x slower │
│ QQuery 10 │ 63.69 ms │ 64.63 ms │ no change │
│ QQuery 11 │ 19.00 ms │ 18.98 ms │ no change │
│ QQuery 12 │ 51.72 ms │ 52.51 ms │ no change │
│ QQuery 13 │ 47.56 ms │ 47.23 ms │ no change │
│ QQuery 14 │ 13.90 ms │ 13.72 ms │ no change │
│ QQuery 15 │ 24.23 ms │ 24.41 ms │ no change │
│ QQuery 16 │ 25.06 ms │ 24.36 ms │ no change │
│ QQuery 17 │ 150.67 ms │ 150.91 ms │ no change │
│ QQuery 18 │ 282.61 ms │ 271.37 ms │ no change │
│ QQuery 19 │ 48.16 ms │ 36.40 ms │ +1.32x faster │
│ QQuery 20 │ 61.49 ms │ 49.56 ms │ +1.24x faster │
│ QQuery 21 │ 319.20 ms │ 336.62 ms │ 1.05x slower │
│ QQuery 22 │ 17.60 ms │ 17.32 ms │ no change │
└──────────────┴───────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1836.99ms │
│ Total Time (alamb_update_arrow_57.2) │ 1819.42ms │
│ Average Time (HEAD) │ 83.50ms │
│ Average Time (alamb_update_arrow_57.2) │ 82.70ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 4 │
│ Queries with No Change │ 16 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴───────────┘

alamb · 2025-12-24T11:56:12Z

run benchmarks

alamb-ghbot · 2025-12-24T12:36:17Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_57.2.0
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_update_arrow_57.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2700.49 ms │ 2692.25 ms │ no change │
│ QQuery 1 │ 1263.34 ms │ 1195.98 ms │ +1.06x faster │
│ QQuery 2 │ 2480.36 ms │ 2333.51 ms │ +1.06x faster │
│ QQuery 3 │ 1162.40 ms │ 1144.25 ms │ no change │
│ QQuery 4 │ 2306.63 ms │ 2258.38 ms │ no change │
│ QQuery 5 │ 28497.50 ms │ 28794.82 ms │ no change │
│ QQuery 6 │ 3967.24 ms │ 3968.46 ms │ no change │
│ QQuery 7 │ 3740.76 ms │ 3744.01 ms │ no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 46118.73ms │
│ Total Time (alamb_update_arrow_57.2) │ 46131.65ms │
│ Average Time (HEAD) │ 5764.84ms │
│ Average Time (alamb_update_arrow_57.2) │ 5766.46ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 6 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_update_arrow_57.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.38 ms │ 2.13 ms │ +1.11x faster │
│ QQuery 1 │ 49.21 ms │ 49.38 ms │ no change │
│ QQuery 2 │ 133.40 ms │ 135.12 ms │ no change │
│ QQuery 3 │ 154.79 ms │ 157.63 ms │ no change │
│ QQuery 4 │ 1105.55 ms │ 1125.54 ms │ no change │
│ QQuery 5 │ 1516.54 ms │ 1494.42 ms │ no change │
│ QQuery 6 │ 2.06 ms │ 2.10 ms │ no change │
│ QQuery 7 │ 56.58 ms │ 56.37 ms │ no change │
│ QQuery 8 │ 1485.93 ms │ 1437.34 ms │ no change │
│ QQuery 9 │ 1838.47 ms │ 1822.70 ms │ no change │
│ QQuery 10 │ 358.08 ms │ 366.87 ms │ no change │
│ QQuery 11 │ 409.45 ms │ 413.04 ms │ no change │
│ QQuery 12 │ 1387.36 ms │ 1361.49 ms │ no change │
│ QQuery 13 │ 2037.73 ms │ 2041.79 ms │ no change │
│ QQuery 14 │ 1293.00 ms │ 1289.01 ms │ no change │
│ QQuery 15 │ 1304.98 ms │ 1285.26 ms │ no change │
│ QQuery 16 │ 2745.69 ms │ 2707.38 ms │ no change │
│ QQuery 17 │ 2710.65 ms │ 2664.66 ms │ no change │
│ QQuery 18 │ 5585.94 ms │ 5010.55 ms │ +1.11x faster │
│ QQuery 19 │ 123.53 ms │ 123.08 ms │ no change │
│ QQuery 20 │ 1914.61 ms │ 1877.57 ms │ no change │
│ QQuery 21 │ 2197.97 ms │ 2162.31 ms │ no change │
│ QQuery 22 │ 3795.84 ms │ 3722.56 ms │ no change │
│ QQuery 23 │ 16226.93 ms │ 12140.04 ms │ +1.34x faster │
│ QQuery 24 │ 216.07 ms │ 213.70 ms │ no change │
│ QQuery 25 │ 467.36 ms │ 475.61 ms │ no change │
│ QQuery 26 │ 229.42 ms │ 233.52 ms │ no change │
│ QQuery 27 │ 2766.30 ms │ 2672.27 ms │ no change │
│ QQuery 28 │ 24764.37 ms │ 21932.32 ms │ +1.13x faster │
│ QQuery 29 │ 988.07 ms │ 991.64 ms │ no change │
│ QQuery 30 │ 1338.37 ms │ 1318.09 ms │ no change │
│ QQuery 31 │ 1380.64 ms │ 1339.78 ms │ no change │
│ QQuery 32 │ 5129.67 ms │ 4956.29 ms │ no change │
│ QQuery 33 │ 6197.64 ms │ 5667.21 ms │ +1.09x faster │
│ QQuery 34 │ 6276.32 ms │ 5994.54 ms │ no change │
│ QQuery 35 │ 1970.40 ms │ 1892.81 ms │ no change │
│ QQuery 36 │ 67.22 ms │ 69.20 ms │ no change │
│ QQuery 37 │ 45.04 ms │ 45.99 ms │ no change │
│ QQuery 38 │ 66.49 ms │ 67.71 ms │ no change │
│ QQuery 39 │ 106.13 ms │ 100.96 ms │ no change │
│ QQuery 40 │ 27.49 ms │ 26.55 ms │ no change │
│ QQuery 41 │ 23.85 ms │ 23.47 ms │ no change │
│ QQuery 42 │ 19.86 ms │ 20.00 ms │ no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 100517.36ms │
│ Total Time (alamb_update_arrow_57.2) │ 91490.02ms │
│ Average Time (HEAD) │ 2337.61ms │
│ Average Time (alamb_update_arrow_57.2) │ 2127.67ms │
│ Queries Faster │ 5 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 38 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_update_arrow_57.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 136.35 ms │ 136.96 ms │ no change │
│ QQuery 2 │ 28.61 ms │ 29.09 ms │ no change │
│ QQuery 3 │ 38.54 ms │ 38.63 ms │ no change │
│ QQuery 4 │ 28.57 ms │ 28.40 ms │ no change │
│ QQuery 5 │ 87.49 ms │ 87.27 ms │ no change │
│ QQuery 6 │ 19.85 ms │ 19.76 ms │ no change │
│ QQuery 7 │ 237.65 ms │ 239.63 ms │ no change │
│ QQuery 8 │ 34.95 ms │ 35.45 ms │ no change │
│ QQuery 9 │ 107.19 ms │ 107.03 ms │ no change │
│ QQuery 10 │ 63.60 ms │ 62.76 ms │ no change │
│ QQuery 11 │ 17.62 ms │ 17.92 ms │ no change │
│ QQuery 12 │ 52.18 ms │ 51.66 ms │ no change │
│ QQuery 13 │ 47.47 ms │ 47.88 ms │ no change │
│ QQuery 14 │ 14.18 ms │ 13.66 ms │ no change │
│ QQuery 15 │ 24.01 ms │ 24.27 ms │ no change │
│ QQuery 16 │ 24.90 ms │ 25.16 ms │ no change │
│ QQuery 17 │ 152.13 ms │ 154.29 ms │ no change │
│ QQuery 18 │ 284.74 ms │ 279.15 ms │ no change │
│ QQuery 19 │ 37.80 ms │ 37.16 ms │ no change │
│ QQuery 20 │ 49.15 ms │ 52.53 ms │ 1.07x slower │
│ QQuery 21 │ 307.03 ms │ 315.54 ms │ no change │
│ QQuery 22 │ 17.40 ms │ 17.60 ms │ no change │
└──────────────┴───────────┴─────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1811.41ms │
│ Total Time (alamb_update_arrow_57.2) │ 1821.81ms │
│ Average Time (HEAD) │ 82.34ms │
│ Average Time (alamb_update_arrow_57.2) │ 82.81ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴───────────┘

alamb · 2026-01-06T22:52:41Z

run benchmarks

alamb-ghbot · 2026-01-06T23:57:37Z

🤖 ./gh_compare_branch.shgh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_57.2.0 (70e2841) to 166ef81diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-07T00:40:11Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_update_arrow_57.2.0
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_update_arrow_57.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2419.66 ms │ 2435.88 ms │ no change │
│ QQuery 1 │ 928.43 ms │ 931.95 ms │ no change │
│ QQuery 2 │ 1919.23 ms │ 1852.60 ms │ no change │
│ QQuery 3 │ 1176.41 ms │ 1134.39 ms │ no change │
│ QQuery 4 │ 2319.77 ms │ 2343.65 ms │ no change │
│ QQuery 5 │ 28296.40 ms │ 28333.13 ms │ no change │
│ QQuery 6 │ 3812.68 ms │ 4011.85 ms │ 1.05x slower │
│ QQuery 7 │ 3909.11 ms │ 3801.45 ms │ no change │
└──────────────┴─────────────┴─────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 44781.68ms │
│ Total Time (alamb_update_arrow_57.2) │ 44844.90ms │
│ Average Time (HEAD) │ 5597.71ms │
│ Average Time (alamb_update_arrow_57.2) │ 5605.61ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 7 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_update_arrow_57.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.50 ms │ 1.65 ms │ 1.10x slower │
│ QQuery 1 │ 49.50 ms │ 53.56 ms │ 1.08x slower │
│ QQuery 2 │ 134.56 ms │ 137.00 ms │ no change │
│ QQuery 3 │ 152.97 ms │ 161.67 ms │ 1.06x slower │
│ QQuery 4 │ 1090.28 ms │ 1202.82 ms │ 1.10x slower │
│ QQuery 5 │ 1388.56 ms │ 1473.87 ms │ 1.06x slower │
│ QQuery 6 │ 1.46 ms │ 1.79 ms │ 1.22x slower │
│ QQuery 7 │ 53.23 ms │ 59.79 ms │ 1.12x slower │
│ QQuery 8 │ 1441.71 ms │ 1559.23 ms │ 1.08x slower │
│ QQuery 9 │ 1787.15 ms │ 2104.78 ms │ 1.18x slower │
│ QQuery 10 │ 334.39 ms │ 365.40 ms │ 1.09x slower │
│ QQuery 11 │ 386.11 ms │ 424.23 ms │ 1.10x slower │
│ QQuery 12 │ 1263.44 ms │ 1401.23 ms │ 1.11x slower │
│ QQuery 13 │ 1950.70 ms │ 2079.84 ms │ 1.07x slower │
│ QQuery 14 │ 1228.14 ms │ 1352.82 ms │ 1.10x slower │
│ QQuery 15 │ 1245.15 ms │ 1339.71 ms │ 1.08x slower │
│ QQuery 16 │ 2540.26 ms │ 2690.26 ms │ 1.06x slower │
│ QQuery 17 │ 2519.94 ms │ 2661.98 ms │ 1.06x slower │
│ QQuery 18 │ 5735.86 ms │ 5209.22 ms │ +1.10x faster │
│ QQuery 19 │ 119.75 ms │ 124.82 ms │ no change │
│ QQuery 20 │ 1886.02 ms │ 1904.50 ms │ no change │
│ QQuery 21 │ 2207.11 ms │ 2240.74 ms │ no change │
│ QQuery 22 │ 3710.17 ms │ 3890.36 ms │ no change │
│ QQuery 23 │ 17570.69 ms │ 12478.67 ms │ +1.41x faster │
│ QQuery 24 │ 213.61 ms │ 223.01 ms │ no change │
│ QQuery 25 │ 455.25 ms │ 471.34 ms │ no change │
│ QQuery 26 │ 208.03 ms │ 218.49 ms │ 1.05x slower │
│ QQuery 27 │ 2818.97 ms │ 2791.64 ms │ no change │
│ QQuery 28 │ 24790.93 ms │ 22004.60 ms │ +1.13x faster │
│ QQuery 29 │ 979.69 ms │ 951.72 ms │ no change │
│ QQuery 30 │ 1352.14 ms │ 1321.24 ms │ no change │
│ QQuery 31 │ 1345.29 ms │ 1343.04 ms │ no change │
│ QQuery 32 │ 5214.55 ms │ 4666.46 ms │ +1.12x faster │
│ QQuery 33 │ 5663.96 ms │ 5429.21 ms │ no change │
│ QQuery 34 │ 6339.20 ms │ 5848.00 ms │ +1.08x faster │
│ QQuery 35 │ 2102.60 ms │ 1905.58 ms │ +1.10x faster │
│ QQuery 36 │ 68.62 ms │ 66.24 ms │ no change │
│ QQuery 37 │ 45.75 ms │ 47.01 ms │ no change │
│ QQuery 38 │ 66.90 ms │ 64.31 ms │ no change │
│ QQuery 39 │ 108.48 ms │ 103.79 ms │ no change │
│ QQuery 40 │ 28.04 ms │ 26.79 ms │ no change │
│ QQuery 41 │ 23.76 ms │ 22.50 ms │ +1.06x faster │
│ QQuery 42 │ 20.97 ms │ 19.84 ms │ +1.06x faster │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 100645.39ms │
│ Total Time (alamb_update_arrow_57.2) │ 92444.76ms │
│ Average Time (HEAD) │ 2340.59ms │
│ Average Time (alamb_update_arrow_57.2) │ 2149.88ms │
│ Queries Faster │ 8 │
│ Queries Slower │ 18 │
│ Queries with No Change │ 17 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_update_arrow_57.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 150.55 ms │ 119.27 ms │ +1.26x faster │
│ QQuery 2 │ 31.78 ms │ 26.97 ms │ +1.18x faster │
│ QQuery 3 │ 39.93 ms │ 37.16 ms │ +1.07x faster │
│ QQuery 4 │ 29.91 ms │ 28.85 ms │ no change │
│ QQuery 5 │ 89.36 ms │ 86.82 ms │ no change │
│ QQuery 6 │ 20.29 ms │ 19.77 ms │ no change │
│ QQuery 7 │ 248.61 ms │ 225.85 ms │ +1.10x faster │
│ QQuery 8 │ 34.57 ms │ 33.26 ms │ no change │
│ QQuery 9 │ 112.16 ms │ 104.82 ms │ +1.07x faster │
│ QQuery 10 │ 65.13 ms │ 63.24 ms │ no change │
│ QQuery 11 │ 18.27 ms │ 17.53 ms │ no change │
│ QQuery 12 │ 51.40 ms │ 51.26 ms │ no change │
│ QQuery 13 │ 48.50 ms │ 47.23 ms │ no change │
│ QQuery 14 │ 14.33 ms │ 13.11 ms │ +1.09x faster │
│ QQuery 15 │ 25.41 ms │ 24.14 ms │ no change │
│ QQuery 16 │ 25.40 ms │ 24.06 ms │ +1.06x faster │
│ QQuery 17 │ 166.39 ms │ 157.40 ms │ +1.06x faster │
│ QQuery 18 │ 296.26 ms │ 274.99 ms │ +1.08x faster │
│ QQuery 19 │ 39.33 ms │ 37.84 ms │ no change │
│ QQuery 20 │ 50.56 ms │ 50.52 ms │ no change │
│ QQuery 21 │ 354.49 ms │ 322.95 ms │ +1.10x faster │
│ QQuery 22 │ 17.62 ms │ 17.69 ms │ no change │
└──────────────┴───────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1930.25ms │
│ Total Time (alamb_update_arrow_57.2) │ 1784.72ms │
│ Average Time (HEAD) │ 87.74ms │
│ Average Time (alamb_update_arrow_57.2) │ 81.12ms │
│ Queries Faster │ 10 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 12 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴───────────┘

alamb · 2026-01-07T11:47:26Z

run benchmark clickbench_partitioned

alamb-ghbot · 2026-01-07T11:47:33Z

🤖 ./gh_compare_branch.shgh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/update_arrow_57.2.0 (70e2841) to 166ef81diff using: clickbench_partitioned
Results will be posted here when complete

alamb · 2026-01-10T12:32:08Z

 ScalarValue::try_new_null(&DataType::Map(map_field_ref, false)).unwrap(),
 ScalarValue::try_new_null(&DataType::Union(
- UnionFields::new(vec![42], vec![field_ref]),
+ UnionFields::try_new(vec![42], vec![field_ref]).unwrap(),


Changes due to this one from @friendlymatthew
Add validated constructors for UnionFields arrow-rs#8891

alamb · 2026-01-11T12:46:31Z

-# Upstream arrow-rs issue: https://github.com/apache/arrow-rs/issues/8841
-# This should succeed after we receive the fix
-query error Arrow error: Compute error: Internal Error: Cannot cast BinaryView to BinaryArray of expected type
+query I


I updated the PR description to close #19290 as well

Jefffrey · 2026-01-11T12:49:28Z

 let type_ids = 0_i8..fields.len() as i8;
- DataType::Union(UnionFields::new(type_ids, fields), UnionMode::Dense)
+ DataType::Union(
+ UnionFields::try_new(type_ids, fields).unwrap(),


Should we propagate this unwrap?

Since the type_ids are autoincrementing, we can probably use UnionFields::from_fields(fields) and avoid the Result
UnionFields::from_fields is a convenience constructor that will infallibly construct the UnionFields (since the only check we really need is to ensure type id uniqueness). https://docs.rs/arrow/latest/arrow/datatypes/struct.UnionFields.html#method.from_fields

Good call -- I did this in 2cb7928

friendlymatthew · 2026-01-11T12:51:48Z

 let type_ids = 0_i8..fields.len() as i8;
- DataType::Union(UnionFields::new(type_ids, fields), UnionMode::Dense)
+ DataType::Union(
+ UnionFields::try_new(type_ids, fields).unwrap(),


Since the type_ids are autoincrementing, we can probably use UnionFields::from_fields(fields) and avoid the Result
UnionFields::from_fields is a convenience constructor that will infallibly construct the UnionFields (since the only check we really need is to ensure type id uniqueness). https://docs.rs/arrow/latest/arrow/datatypes/struct.UnionFields.html#method.from_fields

….2.0

alamb · 2026-01-13T16:14:41Z

Let's do this!

…s methods (#19797) These PRs are available to us now as part of upgrade to arrow-s 57.2.0 (#19355): - apache/arrow-rs#8993 - apache/arrow-rs#9040 Make use of them in some refactorings here.

…#19728) ## Which issue does this PR close? - Follow on to #19355 - related to apache/arrow-rs#8466 - Closes#17455 ## Rationale for this change Keep datafusion up to date (and test Arrow using DataFusion tests) ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Daniël Heres <danielheres@gmail.com>

Includes fix for FixedSizeBinary LEFT JOIN bug - apache/arrow-rs#8981 Cherry-picked test and API updates from - apache#19355

## Which issue does this PR close? - Related to apache/arrow-rs#8465 - Closesapache#19290 ## Rationale for this change Upgrade to latest arrow version I made this PR early to test the arrow release with DataFusion ## What changes are included in this PR? 1. Update arrow 2. Updates for API ## Are these changes tested? Yes by CI ## Are there any user-facing changes? No

…s methods (apache#19797) These PRs are available to us now as part of upgrade to arrow-s 57.2.0 (apache#19355): - apache/arrow-rs#8993 - apache/arrow-rs#9040 Make use of them in some refactorings here.

…apache#19728) ## Which issue does this PR close? - Follow on to apache#19355 - related to apache/arrow-rs#8466 - Closesapache#17455 ## Rationale for this change Keep datafusion up to date (and test Arrow using DataFusion tests) ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Daniël Heres <danielheres@gmail.com>

github-actionsBot added sqllogictest SQL Logic Tests (.slt) common Related to common crate proto Related to proto crate functions Changes to functions implementation datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Dec 16, 2025

This was referenced Dec 16, 2025
Regression in struct casting in 57.2.0 (not yet released) apache/arrow-rs#9005
Closed
arrow-cast: Bring back in-order field casting for StructArrayapache/arrow-rs#9007
Merged

alamb force-pushed the alamb/update_arrow_57.2.0 branch from 7ec33db to 24d3a15Compare December 18, 2025 21:04

alamb commented Dec 18, 2025
View reviewed changes

This was referenced Dec 18, 2025
Test upgrade to object store 0.13.0 #19392
Closed
Migrate Avro reader to arrow-avro and remove internal conversion code #17861
Merged

This comment was marked as outdated.
Sign in to view

alamb mentioned this pull request Jan 6, 2026
Update version to 57.2.0, add CHANGELOG apache/arrow-rs#9103
Merged

alamb force-pushed the alamb/update_arrow_57.2.0 branch from 24d3a15 to ff98f2fCompare January 6, 2026 22:52

alamb force-pushed the alamb/update_arrow_57.2.0 branch 2 times, most recently from 4155056 to 70e2841Compare January 6, 2026 22:55

alamb mentioned this pull request Jan 6, 2026
Release arrow-rs / parquet Minor version 57.2.0 (December 2025) apache/arrow-rs#8465
Closed
9 tasks

alamb force-pushed the alamb/update_arrow_57.2.0 branch from ca11966 to 4cf6e96Compare January 11, 2026 12:42

alamb marked this pull request as ready for review January 11, 2026 12:45

alamb changed the title ~~WIP: Upgrade DataFusion to arrow-rs/parquet 57.2.0~~Upgrade DataFusion to arrow-rs/parquet 57.2.0Jan 11, 2026

alamb commented Jan 11, 2026
View reviewed changes

Jefffrey approved these changes Jan 11, 2026
View reviewed changes

friendlymatthew approved these changes Jan 11, 2026
View reviewed changes

alamb added 2 commits January 12, 2026 13:46

Merge remote-tracking branch 'apache/main' into alamb/update_arrow_57…
34cc90c
….2.0

Use UnionFields::from_fields
2cb7928

Jefffrey mentioned this pull request Jan 13, 2026
chore(deps): bump the arrow-parquet group with 7 updates #19779
Closed

brancz mentioned this pull request Jan 13, 2026
Support aggregating by ListView#19782
Closed

Merge remote-tracking branch 'apache/main' into alamb/update_arrow_57…
7a73f0b
….2.0

alamb added this pull request to the merge queue Jan 13, 2026

Merged via the queue into apache:main with commit 617700dJan 13, 2026
35 checks passed

This was referenced Jan 14, 2026
LEFT JOIN with FixedSizeBinary join keys returns incorrect values instead of NULL #19067
Closed
chore: refactor scalarvalue/encoding using available upstream arrow-rs methods #19797
Merged

pepijnve mentioned this pull request Jan 17, 2026
Array SLT fails when run using dev profile #19867
Closed

erratic-pattern added a commit to influxdata/arrow-datafusion that referenced this pull request Mar 4, 2026
chore: bump arrow/parquet to 57.3.0
6a0206e
Includes fix for FixedSizeBinary LEFT JOIN bug - apache/arrow-rs#8981 Cherry-picked test and API updates from - apache#19355

erratic-pattern added a commit to influxdata/arrow-datafusion that referenced this pull request Mar 17, 2026
chore: bump arrow/parquet to 57.3.0
a03e445
Includes fix for FixedSizeBinary LEFT JOIN bug - apache/arrow-rs#8981 Cherry-picked test and API updates from - apache#19355

erratic-pattern added a commit to influxdata/arrow-datafusion that referenced this pull request Mar 17, 2026
chore: bump arrow/parquet to 57.3.0
7f67875
Includes fix for FixedSizeBinary LEFT JOIN bug - apache/arrow-rs#8981 Cherry-picked test and API updates from - apache#19355

Uh oh!

Conversation

alamb commented Dec 16, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Dandandan commented Dec 21, 2025

Uh oh!

Jefffrey commented Dec 21, 2025

Uh oh!

alamb commented Dec 23, 2025

Uh oh!

This comment was marked as outdated.

alamb-ghbot commented Dec 23, 2025

Uh oh!

alamb commented Dec 24, 2025

Uh oh!

This comment was marked as outdated.

alamb-ghbot commented Dec 24, 2025

Uh oh!

alamb commented Jan 6, 2026

Uh oh!

alamb-ghbot commented Jan 6, 2026

Uh oh!

alamb-ghbot commented Jan 7, 2026

Uh oh!

alamb commented Jan 7, 2026

Uh oh!

alamb-ghbot commented Jan 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

friendlymatthewJan 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

friendlymatthewJan 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alamb commented Dec 16, 2025•
edited
Loading

friendlymatthewJan 11, 2026•
edited
Loading

friendlymatthewJan 11, 2026•
edited
Loading