Update to arrow/parquet 55.2.0 by alamb · Pull Request #16575 · apache/datafusion

alamb · 2025-06-26T16:57:21Z

Which issue does this PR close?

ClosesPin to pre-release arrow / parquet version 55.2.2 #16492

Rationale for this change

I would like to use new features in this minor version of arrow

What changes are included in this PR?

Update versions
Update some plans

Are these changes tested?

By existing CI tests

Are there any user-facing changes?

No

Update to released version

alamb · 2025-06-26T16:57:44Z

 01)CoalesceBatchesExec: target_batch_size=8192
 02)--FilterExec: column1@0 != 42
-03)----DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..137], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:137..274], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:274..411], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:411..547]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]
+03)----DataSourceExec: file_groups={4 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:0..141], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:141..282], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:282..423], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/repartition_scan/parquet_table/2.parquet:423..563]]}, projection=[column1], file_type=parquet, predicate=column1@0 != 42, pruning_predicate=column1_null_count@2 != row_count@3 AND (column1_min@0 != 42 OR 42 != column1_max@1), required_guarantees=[column1 not in (42)]


Some files can get (slightly) larger due to
Ensure page encoding statistics are written to Parquet file arrow-rs#7643

alamb · 2025-06-26T16:58:22Z

 07)│ DataSourceExec │
 08)│ -------------------- │
-09)│ bytes: 1072 │
+09)│ bytes: 1040 │


I believe these files are now slightly smaller due to
Truncate Parquet page data page statistics arrow-rs#7555

alamb · 2025-06-26T18:58:35Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/arrow_55.2.0_upgrade_real (96f17a7) to 3649dc8diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-06-26T19:40:28Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_arrow_55.2.0_upgrade_real
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2039.62 ms │ 1872.95 ms │ +1.09x faster │
│ QQuery 1 │ 721.85 ms │ 777.35 ms │ 1.08x slower │
│ QQuery 2 │ 1381.49 ms │ 1446.61 ms │ no change │
│ QQuery 3 │ 654.10 ms │ 652.88 ms │ no change │
│ QQuery 4 │ 1371.50 ms │ 1357.93 ms │ no change │
│ QQuery 5 │ 15268.50 ms │ 15149.55 ms │ no change │
│ QQuery 6 │ 2003.63 ms │ 2109.24 ms │ 1.05x slower │
│ QQuery 7 │ 1990.88 ms │ 1939.62 ms │ no change │
└──────────────┴─────────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 25431.57ms │
│ Total Time (alamb_arrow_55.2) │ 25306.14ms │
│ Average Time (HEAD) │ 3178.95ms │
│ Average Time (alamb_arrow_55.2) │ 3163.27ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 5 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.89 ms │ 2.36 ms │ +1.22x faster │
│ QQuery 1 │ 33.80 ms │ 33.78 ms │ no change │
│ QQuery 2 │ 83.92 ms │ 81.07 ms │ no change │
│ QQuery 3 │ 100.08 ms │ 99.54 ms │ no change │
│ QQuery 4 │ 593.50 ms │ 600.12 ms │ no change │
│ QQuery 5 │ 868.64 ms │ 873.19 ms │ no change │
│ QQuery 6 │ 2.35 ms │ 2.31 ms │ no change │
│ QQuery 7 │ 39.00 ms │ 38.57 ms │ no change │
│ QQuery 8 │ 910.66 ms │ 853.04 ms │ +1.07x faster │
│ QQuery 9 │ 1206.41 ms │ 1175.13 ms │ no change │
│ QQuery 10 │ 262.29 ms │ 264.12 ms │ no change │
│ QQuery 11 │ 286.10 ms │ 293.71 ms │ no change │
│ QQuery 12 │ 885.55 ms │ 900.68 ms │ no change │
│ QQuery 13 │ 1296.31 ms │ 1252.96 ms │ no change │
│ QQuery 14 │ 824.80 ms │ 822.52 ms │ no change │
│ QQuery 15 │ 814.70 ms │ 792.63 ms │ no change │
│ QQuery 16 │ 1637.84 ms │ 1640.59 ms │ no change │
│ QQuery 17 │ 1622.59 ms │ 1609.69 ms │ no change │
│ QQuery 18 │ 2943.17 ms │ 2920.87 ms │ no change │
│ QQuery 19 │ 89.06 ms │ 84.25 ms │ +1.06x faster │
│ QQuery 20 │ 1123.94 ms │ 1201.05 ms │ 1.07x slower │
│ QQuery 21 │ 1247.75 ms │ 1370.46 ms │ 1.10x slower │
│ QQuery 22 │ 2093.21 ms │ 2250.95 ms │ 1.08x slower │
│ QQuery 23 │ 7358.96 ms │ 7703.96 ms │ no change │
│ QQuery 24 │ 441.26 ms │ 476.62 ms │ 1.08x slower │
│ QQuery 25 │ 378.28 ms │ 403.63 ms │ 1.07x slower │
│ QQuery 26 │ 516.91 ms │ 522.28 ms │ no change │
│ QQuery 27 │ 1521.77 ms │ 1545.69 ms │ no change │
│ QQuery 28 │ 11907.33 ms │ 12078.02 ms │ no change │
│ QQuery 29 │ 531.94 ms │ 517.15 ms │ no change │
│ QQuery 30 │ 777.41 ms │ 796.89 ms │ no change │
│ QQuery 31 │ 826.40 ms │ 826.07 ms │ no change │
│ QQuery 32 │ 2523.61 ms │ 2586.72 ms │ no change │
│ QQuery 33 │ 3205.36 ms │ 3278.81 ms │ no change │
│ QQuery 34 │ 3233.91 ms │ 3260.97 ms │ no change │
│ QQuery 35 │ 1260.12 ms │ 1273.89 ms │ no change │
│ QQuery 36 │ 120.05 ms │ 121.39 ms │ no change │
│ QQuery 37 │ 51.68 ms │ 52.34 ms │ no change │
│ QQuery 38 │ 115.67 ms │ 119.73 ms │ no change │
│ QQuery 39 │ 191.50 ms │ 199.61 ms │ no change │
│ QQuery 40 │ 42.32 ms │ 41.76 ms │ no change │
│ QQuery 41 │ 38.11 ms │ 39.42 ms │ no change │
│ QQuery 42 │ 32.89 ms │ 32.93 ms │ no change │
└──────────────┴─────────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 54044.06ms │
│ Total Time (alamb_arrow_55.2) │ 55041.47ms │
│ Average Time (HEAD) │ 1256.84ms │
│ Average Time (alamb_arrow_55.2) │ 1280.03ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 5 │
│ Queries with No Change │ 35 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 102.37 ms │ 99.18 ms │ no change │
│ QQuery 2 │ 20.18 ms │ 20.97 ms │ no change │
│ QQuery 3 │ 33.81 ms │ 32.28 ms │ no change │
│ QQuery 4 │ 19.03 ms │ 18.95 ms │ no change │
│ QQuery 5 │ 50.69 ms │ 50.58 ms │ no change │
│ QQuery 6 │ 12.31 ms │ 12.04 ms │ no change │
│ QQuery 7 │ 91.24 ms │ 89.10 ms │ no change │
│ QQuery 8 │ 25.64 ms │ 25.26 ms │ no change │
│ QQuery 9 │ 55.69 ms │ 56.86 ms │ no change │
│ QQuery 10 │ 44.96 ms │ 43.69 ms │ no change │
│ QQuery 11 │ 11.63 ms │ 11.51 ms │ no change │
│ QQuery 12 │ 35.82 ms │ 34.84 ms │ no change │
│ QQuery 13 │ 26.43 ms │ 26.10 ms │ no change │
│ QQuery 14 │ 10.36 ms │ 10.00 ms │ no change │
│ QQuery 15 │ 20.58 ms │ 19.92 ms │ no change │
│ QQuery 16 │ 19.62 ms │ 18.33 ms │ +1.07x faster │
│ QQuery 17 │ 100.60 ms │ 97.29 ms │ no change │
│ QQuery 18 │ 206.16 ms │ 198.20 ms │ no change │
│ QQuery 19 │ 26.31 ms │ 25.28 ms │ no change │
│ QQuery 20 │ 32.10 ms │ 32.71 ms │ no change │
│ QQuery 21 │ 150.21 ms │ 149.40 ms │ no change │
│ QQuery 22 │ 15.01 ms │ 14.63 ms │ no change │
└──────────────┴───────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1110.72ms │
│ Total Time (alamb_arrow_55.2) │ 1087.10ms │
│ Average Time (HEAD) │ 50.49ms │
│ Average Time (alamb_arrow_55.2) │ 49.41ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴───────────┘

alamb · 2025-06-26T19:40:30Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/arrow_55.2.0_upgrade_real (96f17a7) to 3649dc8diff
Benchmarks: clickbench_1
Results will be posted here when complete

alamb · 2025-06-26T19:50:21Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_arrow_55.2.0_upgrade_real
--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 0.57 ms │ 0.63 ms │ 1.10x slower │
│ QQuery 1 │ 72.79 ms │ 73.72 ms │ no change │
│ QQuery 2 │ 107.20 ms │ 110.38 ms │ no change │
│ QQuery 3 │ 127.56 ms │ 121.40 ms │ no change │
│ QQuery 4 │ 743.49 ms │ 724.97 ms │ no change │
│ QQuery 5 │ 889.36 ms │ 887.14 ms │ no change │
│ QQuery 6 │ 0.73 ms │ 0.66 ms │ +1.10x faster │
│ QQuery 7 │ 82.26 ms │ 84.35 ms │ no change │
│ QQuery 8 │ 931.38 ms │ 923.17 ms │ no change │
│ QQuery 9 │ 1289.32 ms │ 1245.75 ms │ no change │
│ QQuery 10 │ 294.88 ms │ 299.20 ms │ no change │
│ QQuery 11 │ 322.21 ms │ 324.07 ms │ no change │
│ QQuery 12 │ 887.91 ms │ 897.96 ms │ no change │
│ QQuery 13 │ 1237.34 ms │ 1292.74 ms │ no change │
│ QQuery 14 │ 829.66 ms │ 828.48 ms │ no change │
│ QQuery 15 │ 833.36 ms │ 823.00 ms │ no change │
│ QQuery 16 │ 1653.50 ms │ 1656.66 ms │ no change │
│ QQuery 17 │ 1626.75 ms │ 1630.99 ms │ no change │
│ QQuery 18 │ 2978.61 ms │ 2992.06 ms │ no change │
│ QQuery 19 │ 122.62 ms │ 124.94 ms │ no change │
│ QQuery 20 │ 1168.75 ms │ 1208.68 ms │ no change │
│ QQuery 21 │ 1363.30 ms │ 1421.08 ms │ no change │
│ QQuery 22 │ 2327.65 ms │ 2453.46 ms │ 1.05x slower │
│ QQuery 23 │ 7931.55 ms │ 8328.22 ms │ 1.05x slower │
│ QQuery 24 │ 486.73 ms │ 502.95 ms │ no change │
│ QQuery 25 │ 419.88 ms │ 436.25 ms │ no change │
│ QQuery 26 │ 554.22 ms │ 576.22 ms │ no change │
│ QQuery 27 │ 1682.67 ms │ 1699.77 ms │ no change │
│ QQuery 28 │ 12494.41 ms │ 12490.94 ms │ no change │
│ QQuery 29 │ 550.53 ms │ 556.93 ms │ no change │
│ QQuery 30 │ 815.78 ms │ 834.62 ms │ no change │
│ QQuery 31 │ 873.14 ms │ 884.87 ms │ no change │
│ QQuery 32 │ 2586.87 ms │ 2577.80 ms │ no change │
│ QQuery 33 │ 3270.00 ms │ 3380.92 ms │ no change │
│ QQuery 34 │ 3345.68 ms │ 3386.54 ms │ no change │
│ QQuery 35 │ 1334.93 ms │ 1334.78 ms │ no change │
│ QQuery 36 │ 170.46 ms │ 175.19 ms │ no change │
│ QQuery 37 │ 102.06 ms │ 107.55 ms │ 1.05x slower │
│ QQuery 38 │ 175.96 ms │ 175.25 ms │ no change │
│ QQuery 39 │ 257.08 ms │ 272.73 ms │ 1.06x slower │
│ QQuery 40 │ 87.27 ms │ 87.36 ms │ no change │
│ QQuery 41 │ 81.80 ms │ 85.82 ms │ no change │
│ QQuery 42 │ 78.65 ms │ 74.49 ms │ +1.06x faster │
└──────────────┴─────────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 57190.90ms │
│ Total Time (alamb_arrow_55.2) │ 58094.74ms │
│ Average Time (HEAD) │ 1330.02ms │
│ Average Time (alamb_arrow_55.2) │ 1351.04ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 5 │
│ Queries with No Change │ 36 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘

alamb · 2025-06-26T19:50:24Z

🤖 ./gh_compare_branch_bench.shBenchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/arrow_55.2.0_upgrade_real (96f17a7) to 3649dc8diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_arrow_55.2.0_upgrade_real
Results will be posted here when complete

alamb · 2025-06-26T20:47:00Z

🤖: Benchmark completed

Details

group alamb_arrow_55.2.0_upgrade_real main
----- ------------------------------- ----
logical_aggregate_with_join 1.02 641.0±3.38µs ? ?/sec 1.00 630.9±2.69µs ? ?/sec
logical_select_all_from_1000 1.01 11.3±0.07ms ? ?/sec 1.00 11.1±0.03ms ? ?/sec
logical_select_one_from_700 1.01 420.3±6.54µs ? ?/sec 1.00 414.6±1.90µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.01 379.7±1.77µs ? ?/sec 1.00 374.5±4.12µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.02 366.6±1.55µs ? ?/sec 1.00 359.0±1.69µs ? ?/sec
physical_intersection 1.02 848.3±9.03µs ? ?/sec 1.00 835.5±9.19µs ? ?/sec
physical_join_consider_sort 1.02 1417.6±7.19µs ? ?/sec 1.00 1388.4±5.81µs ? ?/sec
physical_join_distinct 1.02 358.4±3.57µs ? ?/sec 1.00 351.3±3.02µs ? ?/sec
physical_many_self_joins 1.02 10.6±0.05ms ? ?/sec 1.00 10.4±0.11ms ? ?/sec
physical_plan_clickbench_all 1.10 205.5±1.90ms ? ?/sec 1.00 187.5±2.10ms ? ?/sec
physical_plan_clickbench_q1 1.04 2.7±0.03ms ? ?/sec 1.00 2.6±0.03ms ? ?/sec
physical_plan_clickbench_q10 1.09 3.6±0.04ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q11 1.08 3.9±0.07ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q12 1.06 4.0±0.07ms ? ?/sec 1.00 3.8±0.09ms ? ?/sec
physical_plan_clickbench_q13 1.06 3.6±0.05ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q14 1.04 3.8±0.06ms ? ?/sec 1.00 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q15 1.02 3.7±0.06ms ? ?/sec 1.00 3.6±0.12ms ? ?/sec
physical_plan_clickbench_q16 1.05 3.5±0.04ms ? ?/sec 1.00 3.3±0.08ms ? ?/sec
physical_plan_clickbench_q17 1.04 3.6±0.06ms ? ?/sec 1.00 3.4±0.06ms ? ?/sec
physical_plan_clickbench_q18 1.04 3.0±0.04ms ? ?/sec 1.00 2.9±0.04ms ? ?/sec
physical_plan_clickbench_q19 1.04 4.0±0.06ms ? ?/sec 1.00 3.9±0.06ms ? ?/sec
physical_plan_clickbench_q2 1.10 3.3±0.09ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q20 1.07 2.8±0.07ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q21 1.04 3.1±0.04ms ? ?/sec 1.00 3.0±0.03ms ? ?/sec
physical_plan_clickbench_q22 1.12 4.0±0.19ms ? ?/sec 1.00 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q23 1.05 4.1±0.07ms ? ?/sec 1.00 3.9±0.06ms ? ?/sec
physical_plan_clickbench_q24 1.03 4.5±0.05ms ? ?/sec 1.00 4.4±0.06ms ? ?/sec
physical_plan_clickbench_q25 1.03 3.2±0.04ms ? ?/sec 1.00 3.1±0.05ms ? ?/sec
physical_plan_clickbench_q26 1.03 3.0±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q27 1.11 3.4±0.13ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q28 1.06 4.1±0.07ms ? ?/sec 1.00 3.9±0.07ms ? ?/sec
physical_plan_clickbench_q29 1.09 4.9±0.10ms ? ?/sec 1.00 4.5±0.07ms ? ?/sec
physical_plan_clickbench_q3 1.13 3.3±0.05ms ? ?/sec 1.00 2.9±0.04ms ? ?/sec
physical_plan_clickbench_q30 1.02 13.2±0.17ms ? ?/sec 1.00 12.9±0.15ms ? ?/sec
physical_plan_clickbench_q31 1.03 4.0±0.07ms ? ?/sec 1.00 3.9±0.06ms ? ?/sec
physical_plan_clickbench_q32 1.10 4.3±0.19ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q33 1.03 3.5±0.07ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q34 1.06 3.3±0.20ms ? ?/sec 1.00 3.1±0.03ms ? ?/sec
physical_plan_clickbench_q35 1.05 3.3±0.05ms ? ?/sec 1.00 3.1±0.04ms ? ?/sec
physical_plan_clickbench_q36 1.00 3.9±0.06ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q37 1.01 4.0±0.06ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q38 1.02 4.0±0.07ms ? ?/sec 1.00 3.9±0.05ms ? ?/sec
physical_plan_clickbench_q39 1.17 4.3±0.05ms ? ?/sec 1.00 3.7±0.04ms ? ?/sec
physical_plan_clickbench_q4 1.08 2.8±0.10ms ? ?/sec 1.00 2.6±0.04ms ? ?/sec
physical_plan_clickbench_q40 1.09 4.7±0.07ms ? ?/sec 1.00 4.3±0.06ms ? ?/sec
physical_plan_clickbench_q41 1.07 4.2±0.06ms ? ?/sec 1.00 3.9±0.07ms ? ?/sec
physical_plan_clickbench_q42 1.05 4.0±0.05ms ? ?/sec 1.00 3.8±0.06ms ? ?/sec
physical_plan_clickbench_q43 1.08 4.5±0.22ms ? ?/sec 1.00 4.2±0.07ms ? ?/sec
physical_plan_clickbench_q44 1.02 2.8±0.02ms ? ?/sec 1.00 2.7±0.04ms ? ?/sec
physical_plan_clickbench_q45 1.08 2.9±0.16ms ? ?/sec 1.00 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q46 1.14 3.6±0.04ms ? ?/sec 1.00 3.1±0.04ms ? ?/sec
physical_plan_clickbench_q47 1.15 4.3±0.05ms ? ?/sec 1.00 3.8±0.06ms ? ?/sec
physical_plan_clickbench_q48 1.14 5.1±0.12ms ? ?/sec 1.00 4.4±0.06ms ? ?/sec
physical_plan_clickbench_q49 1.09 5.1±0.08ms ? ?/sec 1.00 4.7±0.07ms ? ?/sec
physical_plan_clickbench_q5 1.05 3.0±0.04ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q50 1.10 4.6±0.06ms ? ?/sec 1.00 4.2±0.06ms ? ?/sec
physical_plan_clickbench_q51 1.11 3.5±0.06ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q6 1.06 3.0±0.03ms ? ?/sec 1.00 2.8±0.02ms ? ?/sec
physical_plan_clickbench_q7 1.06 2.7±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.08 3.7±0.05ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q9 1.05 3.4±0.05ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_tpcds_all 1.01 1070.3±8.58ms ? ?/sec 1.00 1055.0±4.92ms ? ?/sec
physical_plan_tpch_all 1.04 65.6±1.55ms ? ?/sec 1.00 63.0±0.30ms ? ?/sec
physical_plan_tpch_q1 1.01 2.1±0.01ms ? ?/sec 1.00 2.1±0.02ms ? ?/sec
physical_plan_tpch_q10 1.05 4.0±0.05ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_tpch_q11 1.03 3.4±0.04ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_tpch_q12 1.02 1855.5±15.69µs ? ?/sec 1.00 1819.9±9.99µs ? ?/sec
physical_plan_tpch_q13 1.03 1502.9±54.92µs ? ?/sec 1.00 1460.4±6.23µs ? ?/sec
physical_plan_tpch_q14 1.01 1965.1±10.71µs ? ?/sec 1.00 1947.8±9.44µs ? ?/sec
physical_plan_tpch_q16 1.01 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_tpch_q17 1.04 2.5±0.06ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q18 1.01 2.7±0.01ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_tpch_q19 1.01 3.4±0.02ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec
physical_plan_tpch_q2 1.01 5.6±0.04ms ? ?/sec 1.00 5.5±0.03ms ? ?/sec
physical_plan_tpch_q20 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_tpch_q21 1.03 4.3±0.11ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
physical_plan_tpch_q22 1.08 2.9±0.03ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_tpch_q3 1.01 2.6±0.02ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec
physical_plan_tpch_q4 1.01 1545.3±24.41µs ? ?/sec 1.00 1536.4±11.87µs ? ?/sec
physical_plan_tpch_q5 1.00 3.2±0.02ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q6 1.01 873.1±4.90µs ? ?/sec 1.00 864.6±6.96µs ? ?/sec
physical_plan_tpch_q7 1.06 4.5±0.04ms ? ?/sec 1.00 4.3±0.02ms ? ?/sec
physical_plan_tpch_q8 1.06 5.6±0.03ms ? ?/sec 1.00 5.3±0.03ms ? ?/sec
physical_plan_tpch_q9 1.05 4.4±0.03ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
physical_select_aggregates_from_200 1.00 17.9±0.08ms ? ?/sec 1.00 18.0±0.08ms ? ?/sec
physical_select_all_from_1000 1.02 25.2±0.18ms ? ?/sec 1.00 24.7±0.07ms ? ?/sec
physical_select_one_from_700 1.02 1095.1±7.27µs ? ?/sec 1.00 1073.0±5.64µs ? ?/sec
physical_sorted_union_orderby 1.01 43.9±0.23ms ? ?/sec 1.00 43.5±0.21ms ? ?/sec
physical_theta_join_consider_sort 1.01 1791.5±6.25µs ? ?/sec 1.00 1768.4±20.72µs ? ?/sec
physical_unnest_to_join 1.01 1327.6±5.40µs ? ?/sec 1.00 1312.1±7.89µs ? ?/sec
with_param_values_many_columns 1.01 127.0±1.07µs ? ?/sec 1.00 125.7±0.78µs ? ?/sec

zhuqi-lucas

LGTM, thanks @alamb , there are some performance regression for clickbench, but i may be noise?

alamb · 2025-06-27T14:55:09Z

LGTM, thanks @alamb , there are some performance regression for clickbench, but i may be noise?

Hmm looks non trivial -- I will rerun

alamb · 2025-06-27T16:50:18Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/arrow_55.2.0_upgrade_real (96f17a7) to 3649dc8diff
Benchmarks: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-06-27T17:04:07Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_arrow_55.2.0_upgrade_real
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1913.32 ms │ 1865.92 ms │ no change │
│ QQuery 1 │ 713.59 ms │ 772.48 ms │ 1.08x slower │
│ QQuery 2 │ 1363.12 ms │ 1462.35 ms │ 1.07x slower │
│ QQuery 3 │ 653.42 ms │ 657.41 ms │ no change │
│ QQuery 4 │ 1338.25 ms │ 1350.20 ms │ no change │
│ QQuery 5 │ 15228.64 ms │ 15375.87 ms │ no change │
│ QQuery 6 │ 1990.99 ms │ 2083.72 ms │ no change │
│ QQuery 7 │ 1967.37 ms │ 1994.69 ms │ no change │
└──────────────┴─────────────┴──────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 25168.69ms │
│ Total Time (alamb_arrow_55.2) │ 25562.63ms │
│ Average Time (HEAD) │ 3146.09ms │
│ Average Time (alamb_arrow_55.2) │ 3195.33ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 6 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.71 ms │ 2.25 ms │ +1.20x faster │
│ QQuery 1 │ 32.97 ms │ 35.15 ms │ 1.07x slower │
│ QQuery 2 │ 83.42 ms │ 82.42 ms │ no change │
│ QQuery 3 │ 98.81 ms │ 101.81 ms │ no change │
│ QQuery 4 │ 590.81 ms │ 669.83 ms │ 1.13x slower │
│ QQuery 5 │ 855.03 ms │ 914.58 ms │ 1.07x slower │
│ QQuery 6 │ 2.37 ms │ 2.29 ms │ no change │
│ QQuery 7 │ 39.76 ms │ 38.33 ms │ no change │
│ QQuery 8 │ 853.93 ms │ 873.36 ms │ no change │
│ QQuery 9 │ 1170.27 ms │ 1184.95 ms │ no change │
│ QQuery 10 │ 253.20 ms │ 263.64 ms │ no change │
│ QQuery 11 │ 287.56 ms │ 293.46 ms │ no change │
│ QQuery 12 │ 853.62 ms │ 890.84 ms │ no change │
│ QQuery 13 │ 1272.42 ms │ 1227.21 ms │ no change │
│ QQuery 14 │ 820.10 ms │ 812.10 ms │ no change │
│ QQuery 15 │ 776.69 ms │ 785.43 ms │ no change │
│ QQuery 16 │ 1640.34 ms │ 1621.95 ms │ no change │
│ QQuery 17 │ 1607.04 ms │ 1602.04 ms │ no change │
│ QQuery 18 │ 2935.41 ms │ 2918.67 ms │ no change │
│ QQuery 19 │ 88.01 ms │ 85.69 ms │ no change │
│ QQuery 20 │ 1114.60 ms │ 1173.67 ms │ 1.05x slower │
│ QQuery 21 │ 1244.49 ms │ 1336.03 ms │ 1.07x slower │
│ QQuery 22 │ 2073.69 ms │ 2177.59 ms │ 1.05x slower │
│ QQuery 23 │ 7279.85 ms │ 7658.51 ms │ 1.05x slower │
│ QQuery 24 │ 447.58 ms │ 467.54 ms │ no change │
│ QQuery 25 │ 373.22 ms │ 403.37 ms │ 1.08x slower │
│ QQuery 26 │ 514.99 ms │ 529.93 ms │ no change │
│ QQuery 27 │ 1517.55 ms │ 1543.37 ms │ no change │
│ QQuery 28 │ 11877.48 ms │ 11684.47 ms │ no change │
│ QQuery 29 │ 536.12 ms │ 537.93 ms │ no change │
│ QQuery 30 │ 769.65 ms │ 784.99 ms │ no change │
│ QQuery 31 │ 817.10 ms │ 834.81 ms │ no change │
│ QQuery 32 │ 2503.28 ms │ 2487.83 ms │ no change │
│ QQuery 33 │ 3137.64 ms │ 3221.20 ms │ no change │
│ QQuery 34 │ 3179.58 ms │ 3207.61 ms │ no change │
│ QQuery 35 │ 1251.77 ms │ 1225.26 ms │ no change │
│ QQuery 36 │ 118.46 ms │ 120.14 ms │ no change │
│ QQuery 37 │ 51.75 ms │ 52.24 ms │ no change │
│ QQuery 38 │ 115.68 ms │ 121.69 ms │ 1.05x slower │
│ QQuery 39 │ 192.83 ms │ 195.73 ms │ no change │
│ QQuery 40 │ 41.36 ms │ 40.21 ms │ no change │
│ QQuery 41 │ 38.07 ms │ 41.36 ms │ 1.09x slower │
│ QQuery 42 │ 31.81 ms │ 32.19 ms │ no change │
└──────────────┴─────────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 53493.01ms │
│ Total Time (alamb_arrow_55.2) │ 54283.65ms │
│ Average Time (HEAD) │ 1244.02ms │
│ Average Time (alamb_arrow_55.2) │ 1262.41ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 10 │
│ Queries with No Change │ 32 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 100.75 ms │ 99.09 ms │ no change │
│ QQuery 2 │ 21.11 ms │ 21.62 ms │ no change │
│ QQuery 3 │ 32.06 ms │ 33.03 ms │ no change │
│ QQuery 4 │ 18.37 ms │ 18.63 ms │ no change │
│ QQuery 5 │ 49.30 ms │ 49.10 ms │ no change │
│ QQuery 6 │ 12.10 ms │ 11.97 ms │ no change │
│ QQuery 7 │ 90.16 ms │ 88.98 ms │ no change │
│ QQuery 8 │ 24.19 ms │ 25.04 ms │ no change │
│ QQuery 9 │ 54.65 ms │ 54.52 ms │ no change │
│ QQuery 10 │ 42.66 ms │ 42.82 ms │ no change │
│ QQuery 11 │ 11.23 ms │ 11.39 ms │ no change │
│ QQuery 12 │ 34.51 ms │ 34.14 ms │ no change │
│ QQuery 13 │ 26.20 ms │ 26.05 ms │ no change │
│ QQuery 14 │ 9.77 ms │ 9.80 ms │ no change │
│ QQuery 15 │ 18.74 ms │ 19.52 ms │ no change │
│ QQuery 16 │ 18.77 ms │ 18.24 ms │ no change │
│ QQuery 17 │ 94.49 ms │ 95.19 ms │ no change │
│ QQuery 18 │ 198.24 ms │ 184.03 ms │ +1.08x faster │
│ QQuery 19 │ 24.83 ms │ 24.83 ms │ no change │
│ QQuery 20 │ 31.79 ms │ 30.94 ms │ no change │
│ QQuery 21 │ 148.45 ms │ 145.14 ms │ no change │
│ QQuery 22 │ 15.62 ms │ 14.48 ms │ +1.08x faster │
└──────────────┴───────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1077.99ms │
│ Total Time (alamb_arrow_55.2) │ 1058.55ms │
│ Average Time (HEAD) │ 49.00ms │
│ Average Time (alamb_arrow_55.2) │ 48.12ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 20 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴───────────┘

alamb · 2025-06-27T17:04:09Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/arrow_55.2.0_upgrade_real (96f17a7) to 3649dc8diff
Benchmarks: clickbench_1
Results will be posted here when complete

alamb · 2025-06-27T17:13:53Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_arrow_55.2.0_upgrade_real
--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 0.56 ms │ 0.62 ms │ 1.10x slower │
│ QQuery 1 │ 71.30 ms │ 72.41 ms │ no change │
│ QQuery 2 │ 107.34 ms │ 111.30 ms │ no change │
│ QQuery 3 │ 121.93 ms │ 124.30 ms │ no change │
│ QQuery 4 │ 641.22 ms │ 631.99 ms │ no change │
│ QQuery 5 │ 848.89 ms │ 872.64 ms │ no change │
│ QQuery 6 │ 0.68 ms │ 0.65 ms │ no change │
│ QQuery 7 │ 77.22 ms │ 80.99 ms │ no change │
│ QQuery 8 │ 885.04 ms │ 879.72 ms │ no change │
│ QQuery 9 │ 1239.18 ms │ 1243.99 ms │ no change │
│ QQuery 10 │ 291.03 ms │ 300.47 ms │ no change │
│ QQuery 11 │ 318.13 ms │ 339.56 ms │ 1.07x slower │
│ QQuery 12 │ 875.32 ms │ 864.88 ms │ no change │
│ QQuery 13 │ 1248.43 ms │ 1251.37 ms │ no change │
│ QQuery 14 │ 804.13 ms │ 823.16 ms │ no change │
│ QQuery 15 │ 811.54 ms │ 826.16 ms │ no change │
│ QQuery 16 │ 1629.96 ms │ 1686.09 ms │ no change │
│ QQuery 17 │ 1627.45 ms │ 1619.67 ms │ no change │
│ QQuery 18 │ 2929.92 ms │ 2972.99 ms │ no change │
│ QQuery 19 │ 132.73 ms │ 126.66 ms │ no change │
│ QQuery 20 │ 1152.66 ms │ 1195.01 ms │ no change │
│ QQuery 21 │ 1345.79 ms │ 1399.30 ms │ no change │
│ QQuery 22 │ 2313.33 ms │ 2437.10 ms │ 1.05x slower │
│ QQuery 23 │ 7882.99 ms │ 8172.66 ms │ no change │
│ QQuery 24 │ 483.90 ms │ 498.62 ms │ no change │
│ QQuery 25 │ 416.63 ms │ 428.36 ms │ no change │
│ QQuery 26 │ 553.91 ms │ 563.05 ms │ no change │
│ QQuery 27 │ 1651.48 ms │ 1695.69 ms │ no change │
│ QQuery 28 │ 12660.69 ms │ 12382.36 ms │ no change │
│ QQuery 29 │ 561.98 ms │ 563.12 ms │ no change │
│ QQuery 30 │ 799.64 ms │ 822.62 ms │ no change │
│ QQuery 31 │ 845.51 ms │ 863.06 ms │ no change │
│ QQuery 32 │ 2545.88 ms │ 2548.07 ms │ no change │
│ QQuery 33 │ 3265.95 ms │ 3288.58 ms │ no change │
│ QQuery 34 │ 3342.47 ms │ 3315.20 ms │ no change │
│ QQuery 35 │ 1305.97 ms │ 1326.26 ms │ no change │
│ QQuery 36 │ 179.11 ms │ 172.32 ms │ no change │
│ QQuery 37 │ 101.55 ms │ 104.25 ms │ no change │
│ QQuery 38 │ 166.68 ms │ 179.22 ms │ 1.08x slower │
│ QQuery 39 │ 239.39 ms │ 261.90 ms │ 1.09x slower │
│ QQuery 40 │ 86.70 ms │ 75.58 ms │ +1.15x faster │
│ QQuery 41 │ 82.66 ms │ 80.56 ms │ no change │
│ QQuery 42 │ 75.30 ms │ 76.93 ms │ no change │
└──────────────┴─────────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 56722.19ms │
│ Total Time (alamb_arrow_55.2) │ 57279.44ms │
│ Average Time (HEAD) │ 1319.12ms │
│ Average Time (alamb_arrow_55.2) │ 1332.08ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 5 │
│ Queries with No Change │ 37 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘

alamb · 2025-06-27T17:13:56Z

🤖 ./gh_compare_branch_bench.shBenchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/arrow_55.2.0_upgrade_real (96f17a7) to 3649dc8diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_arrow_55.2.0_upgrade_real
Results will be posted here when complete

alamb · 2025-06-27T17:34:32Z

Hmm, next up will be to try and reproduce the slowdown locally 🤔

alamb · 2025-06-27T17:48:55Z

🤖: Benchmark completed

Details

group alamb_arrow_55.2.0_upgrade_real main
----- ------------------------------- ----
logical_aggregate_with_join 1.01 636.6±3.27µs ? ?/sec 1.00 627.8±3.60µs ? ?/sec
logical_select_all_from_1000 1.01 11.2±0.07ms ? ?/sec 1.00 11.1±0.03ms ? ?/sec
logical_select_one_from_700 1.01 418.5±2.54µs ? ?/sec 1.00 414.7±1.54µs ? ?/sec
logical_trivial_join_high_numbered_columns 1.01 375.3±4.09µs ? ?/sec 1.00 373.4±1.41µs ? ?/sec
logical_trivial_join_low_numbered_columns 1.00 361.2±1.39µs ? ?/sec 1.00 360.2±5.60µs ? ?/sec
physical_intersection 1.02 850.9±3.83µs ? ?/sec 1.00 838.1±4.39µs ? ?/sec
physical_join_consider_sort 1.01 1403.6±8.18µs ? ?/sec 1.00 1383.3±6.74µs ? ?/sec
physical_join_distinct 1.01 355.0±2.69µs ? ?/sec 1.00 352.0±1.54µs ? ?/sec
physical_many_self_joins 1.04 10.6±0.06ms ? ?/sec 1.00 10.2±0.03ms ? ?/sec
physical_plan_clickbench_all 1.03 193.1±3.69ms ? ?/sec 1.00 187.4±1.29ms ? ?/sec
physical_plan_clickbench_q1 1.01 2.5±0.03ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q10 1.00 3.4±0.02ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q11 1.00 3.6±0.03ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q12 1.00 3.7±0.03ms ? ?/sec 1.01 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q13 1.00 3.4±0.04ms ? ?/sec 1.00 3.3±0.03ms ? ?/sec
physical_plan_clickbench_q14 1.00 3.6±0.03ms ? ?/sec 1.00 3.6±0.03ms ? ?/sec
physical_plan_clickbench_q15 1.01 3.4±0.03ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q16 1.00 3.3±0.03ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_clickbench_q17 1.00 3.4±0.02ms ? ?/sec 1.00 3.4±0.02ms ? ?/sec
physical_plan_clickbench_q18 1.00 2.9±0.02ms ? ?/sec 1.02 2.9±0.05ms ? ?/sec
physical_plan_clickbench_q19 1.00 3.8±0.02ms ? ?/sec 1.01 3.8±0.05ms ? ?/sec
physical_plan_clickbench_q2 1.01 2.9±0.03ms ? ?/sec 1.00 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q20 1.00 2.6±0.01ms ? ?/sec 1.02 2.7±0.03ms ? ?/sec
physical_plan_clickbench_q21 1.00 2.9±0.02ms ? ?/sec 1.04 3.1±0.11ms ? ?/sec
physical_plan_clickbench_q22 1.00 3.5±0.03ms ? ?/sec 1.03 3.6±0.05ms ? ?/sec
physical_plan_clickbench_q23 1.00 3.8±0.02ms ? ?/sec 1.02 3.9±0.08ms ? ?/sec
physical_plan_clickbench_q24 1.00 4.3±0.06ms ? ?/sec 1.03 4.4±0.06ms ? ?/sec
physical_plan_clickbench_q25 1.00 3.0±0.02ms ? ?/sec 1.02 3.1±0.05ms ? ?/sec
physical_plan_clickbench_q26 1.00 2.9±0.02ms ? ?/sec 1.01 2.9±0.03ms ? ?/sec
physical_plan_clickbench_q27 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q28 1.02 3.9±0.13ms ? ?/sec 1.00 3.9±0.04ms ? ?/sec
physical_plan_clickbench_q29 1.01 4.6±0.18ms ? ?/sec 1.00 4.5±0.06ms ? ?/sec
physical_plan_clickbench_q3 1.01 2.9±0.05ms ? ?/sec 1.00 2.9±0.02ms ? ?/sec
physical_plan_clickbench_q30 1.01 12.9±0.20ms ? ?/sec 1.00 12.8±0.14ms ? ?/sec
physical_plan_clickbench_q31 1.07 4.1±0.11ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q32 1.02 3.9±0.10ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q33 1.02 3.4±0.07ms ? ?/sec 1.00 3.4±0.03ms ? ?/sec
physical_plan_clickbench_q34 1.04 3.2±0.07ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q35 1.06 3.3±0.08ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q36 1.04 4.0±0.07ms ? ?/sec 1.00 3.8±0.03ms ? ?/sec
physical_plan_clickbench_q37 1.02 3.9±0.06ms ? ?/sec 1.00 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q38 1.02 4.0±0.07ms ? ?/sec 1.00 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q39 1.03 3.8±0.07ms ? ?/sec 1.00 3.7±0.05ms ? ?/sec
physical_plan_clickbench_q4 1.02 2.6±0.07ms ? ?/sec 1.00 2.6±0.02ms ? ?/sec
physical_plan_clickbench_q40 1.01 4.4±0.06ms ? ?/sec 1.00 4.3±0.04ms ? ?/sec
physical_plan_clickbench_q41 1.01 3.9±0.07ms ? ?/sec 1.00 3.9±0.03ms ? ?/sec
physical_plan_clickbench_q42 1.01 3.9±0.05ms ? ?/sec 1.00 3.8±0.04ms ? ?/sec
physical_plan_clickbench_q43 1.03 4.3±0.08ms ? ?/sec 1.00 4.2±0.05ms ? ?/sec
physical_plan_clickbench_q44 1.08 2.9±0.07ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q45 1.09 3.0±0.07ms ? ?/sec 1.00 2.7±0.02ms ? ?/sec
physical_plan_clickbench_q46 1.02 3.2±0.04ms ? ?/sec 1.00 3.1±0.02ms ? ?/sec
physical_plan_clickbench_q47 1.05 3.9±0.09ms ? ?/sec 1.00 3.7±0.03ms ? ?/sec
physical_plan_clickbench_q48 1.04 4.6±0.10ms ? ?/sec 1.00 4.4±0.04ms ? ?/sec
physical_plan_clickbench_q49 1.06 5.0±0.11ms ? ?/sec 1.00 4.7±0.04ms ? ?/sec
physical_plan_clickbench_q5 1.00 2.8±0.02ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q50 1.07 4.5±0.10ms ? ?/sec 1.00 4.2±0.04ms ? ?/sec
physical_plan_clickbench_q51 1.06 3.4±0.09ms ? ?/sec 1.00 3.2±0.03ms ? ?/sec
physical_plan_clickbench_q6 1.00 2.8±0.03ms ? ?/sec 1.00 2.8±0.03ms ? ?/sec
physical_plan_clickbench_q7 1.00 2.5±0.02ms ? ?/sec 1.00 2.5±0.02ms ? ?/sec
physical_plan_clickbench_q8 1.00 3.4±0.03ms ? ?/sec 1.00 3.4±0.04ms ? ?/sec
physical_plan_clickbench_q9 1.01 3.2±0.02ms ? ?/sec 1.00 3.2±0.02ms ? ?/sec
physical_plan_tpcds_all 1.00 1035.3±3.60ms ? ?/sec 1.00 1034.5±6.63ms ? ?/sec
physical_plan_tpch_all 1.00 62.3±0.26ms ? ?/sec 1.00 62.2±0.28ms ? ?/sec
physical_plan_tpch_q1 1.01 2.1±0.01ms ? ?/sec 1.00 2.0±0.01ms ? ?/sec
physical_plan_tpch_q10 1.00 3.8±0.01ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec
physical_plan_tpch_q11 1.00 3.3±0.03ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec
physical_plan_tpch_q12 1.00 1803.9±7.91µs ? ?/sec 1.00 1804.5±7.61µs ? ?/sec
physical_plan_tpch_q13 1.00 1449.2±9.67µs ? ?/sec 1.00 1445.5±7.47µs ? ?/sec
physical_plan_tpch_q14 1.00 1932.3±11.37µs ? ?/sec 1.00 1925.9±10.81µs ? ?/sec
physical_plan_tpch_q16 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_tpch_q17 1.01 2.4±0.01ms ? ?/sec 1.00 2.4±0.01ms ? ?/sec
physical_plan_tpch_q18 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q19 1.00 3.3±0.01ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec
physical_plan_tpch_q2 1.02 5.6±0.03ms ? ?/sec 1.00 5.5±0.05ms ? ?/sec
physical_plan_tpch_q20 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q21 1.00 4.1±0.02ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
physical_plan_tpch_q22 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec
physical_plan_tpch_q3 1.01 2.5±0.02ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec
physical_plan_tpch_q4 1.02 1559.4±15.80µs ? ?/sec 1.00 1525.1±10.55µs ? ?/sec
physical_plan_tpch_q5 1.00 3.1±0.03ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec
physical_plan_tpch_q6 1.01 866.3±7.20µs ? ?/sec 1.00 862.0±4.13µs ? ?/sec
physical_plan_tpch_q7 1.00 4.2±0.02ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec
physical_plan_tpch_q8 1.00 5.2±0.03ms ? ?/sec 1.00 5.2±0.02ms ? ?/sec
physical_plan_tpch_q9 1.00 4.1±0.02ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec
physical_select_aggregates_from_200 1.01 17.9±0.07ms ? ?/sec 1.00 17.7±0.08ms ? ?/sec
physical_select_all_from_1000 1.02 25.0±0.14ms ? ?/sec 1.00 24.6±0.06ms ? ?/sec
physical_select_one_from_700 1.02 1093.1±7.37µs ? ?/sec 1.00 1067.4±5.99µs ? ?/sec
physical_sorted_union_orderby 1.02 43.5±0.27ms ? ?/sec 1.00 42.7±0.24ms ? ?/sec
physical_theta_join_consider_sort 1.02 1789.5±13.56µs ? ?/sec 1.00 1755.0±14.64µs ? ?/sec
physical_unnest_to_join 1.01 1328.8±7.15µs ? ?/sec 1.00 1309.8±9.43µs ? ?/sec
with_param_values_many_columns 1.00 126.0±1.01µs ? ?/sec 1.01 126.8±0.53µs ? ?/sec

alamb · 2025-06-27T18:24:31Z

I am preparing to compare locally with 3649dc8 (the merge-base of this branch and main)

(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Software/datafusion$ git checkout `git merge-base alamb/alamb/arrow_55.2.0_upgrade_real apache/main`
Note: switching to '3649dc80cef2cdcd2df750e90e6ffd94ae6ae808'.
....
cargo build --profile=profiling -p datafusion-cli && cp target/profiling/datafusion-cli ~/Downloads/datafusion-cli-arrow-55.1.0

comphead

thanks @alamb lgtm
and thanks for test diff explanation

alamb · 2025-06-27T19:25:15Z

I couldn't really reproduce anything locally but the variation is pretty noisy with q21

./datafusion-cli-arrow-55.1.0 -f q21-times-10.sql |grep Elapsed
Elapsed 0.831 seconds.
Elapsed 0.783 seconds.
Elapsed 0.792 seconds.
Elapsed 0.809 seconds.
Elapsed 0.818 seconds.
Elapsed 0.790 seconds.
Elapsed 0.788 seconds.
Elapsed 0.804 seconds.
Elapsed 0.803 seconds.
Elapsed 0.793 seconds.

$ ./datafusion-cli-alamb_arrow_55.2.0_upgrade_real -f q21-times-10.sql |grep Elapsed
Elapsed 0.870 seconds.
Elapsed 0.788 seconds.
Elapsed 0.787 seconds.
Elapsed 0.798 seconds.
Elapsed 0.799 seconds.
Elapsed 0.776 seconds.
Elapsed 0.777 seconds.
Elapsed 0.802 seconds.
Elapsed 0.804 seconds.
Elapsed 0.835 seconds.

When looking at that query however, I found some other performance improvement:

Improve performance of ClickBench Q21 by removing the cast #16591

alamb · 2025-06-27T20:15:01Z

I also don't really see any difference with q4 (another that was reported 10% slower)

(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ ./datafusion-cli-alamb_arrow_55.2.0_upgrade_real -f q4-times-10.sql | grep Elapsed
Elapsed 0.312 seconds.
Elapsed 0.271 seconds.
Elapsed 0.266 seconds.
Elapsed 0.265 seconds.
Elapsed 0.273 seconds.
Elapsed 0.277 seconds.
Elapsed 0.267 seconds.
Elapsed 0.273 seconds.
Elapsed 0.269 seconds.
Elapsed 0.276 seconds.
(venv) andrewlamb@Andrews-MacBook-Pro-2:~/Downloads$ ./datafusion-cli-arrow-55.1.0 -f q4-times-10.sql |grep Elapsed
Elapsed 0.324 seconds.
Elapsed 0.289 seconds.
Elapsed 0.266 seconds.
Elapsed 0.269 seconds.
Elapsed 0.263 seconds.
Elapsed 0.272 seconds.
Elapsed 0.267 seconds.
Elapsed 0.275 seconds.
Elapsed 0.268 seconds.
Elapsed 0.262 seconds.

I will next see if I can reproduce the same thing on the actual benchmark machine

alamb · 2025-06-27T20:56:25Z

Here are my benchmark results on the same benchmark machine:

q4:

alamb@aal-dev:~/55.2.0_upgrade_testing$ ./datafusion-cli-arrow-55.1.0 -f q4-times10.sql | grep Elapsed
Elapsed 0.813 seconds.
Elapsed 0.800 seconds.
Elapsed 0.777 seconds.
Elapsed 0.739 seconds.
Elapsed 0.756 seconds.
Elapsed 0.776 seconds.
Elapsed 0.748 seconds.
Elapsed 0.742 seconds.
Elapsed 0.736 seconds.
Elapsed 0.741 seconds.
alamb@aal-dev:~/55.2.0_upgrade_testing$ ./datafusion-cli-arrow-55.2.0 -f q4-times10.sql | grep Elapsed
Elapsed 0.816 seconds.
Elapsed 0.810 seconds.
Elapsed 0.818 seconds.
Elapsed 0.771 seconds.
Elapsed 0.735 seconds.
Elapsed 0.757 seconds.
Elapsed 0.737 seconds.
Elapsed 0.739 seconds.
Elapsed 0.737 seconds.
Elapsed 0.744 seconds.

q21

alamb@aal-dev:~/55.2.0_upgrade_testing$ ./datafusion-cli-arrow-55.1.0 -f q4-times10.sql | grep Elapsed
Elapsed 0.813 seconds.
Elapsed 0.800 seconds.
Elapsed 0.777 seconds.
Elapsed 0.739 seconds.
Elapsed 0.756 seconds.
Elapsed 0.776 seconds.
Elapsed 0.748 seconds.
Elapsed 0.742 seconds.
Elapsed 0.736 seconds.
Elapsed 0.741 seconds.
alamb@aal-dev:~/55.2.0_upgrade_testing$ ./datafusion-cli-arrow-55.2.0 -f q4-times10.sql | grep Elapsed
Elapsed 0.816 seconds.
Elapsed 0.810 seconds.
Elapsed 0.818 seconds.
Elapsed 0.771 seconds.
Elapsed 0.735 seconds.
Elapsed 0.757 seconds.
Elapsed 0.737 seconds.
Elapsed 0.739 seconds.
Elapsed 0.737 seconds.
Elapsed 0.744 seconds.
alamb@aal-dev:~/55.2.0_upgrade_testing$ ./datafusion-cli-arrow-55.1.0 -f q21-times10.sql | grep Elapsed
Elapsed 9.460 seconds.
Elapsed 1.825 seconds.
Elapsed 1.816 seconds.
Elapsed 1.859 seconds.
Elapsed 1.766 seconds.
Elapsed 1.803 seconds.
Elapsed 1.780 seconds.
Elapsed 1.796 seconds.
Elapsed 1.806 seconds.
Elapsed 1.830 seconds.
alamb@aal-dev:~/55.2.0_upgrade_testing$ ./datafusion-cli-arrow-55.2.0 -f q21-times10.sql | grep Elapsed
Elapsed 1.906 seconds.
Elapsed 1.841 seconds.
Elapsed 1.791 seconds.
Elapsed 1.842 seconds.
Elapsed 1.834 seconds.
Elapsed 1.794 seconds.
Elapsed 1.833 seconds.
Elapsed 1.823 seconds.
Elapsed 1.793 seconds.
Elapsed 1.802 seconds.

(basically no change ...)

So I am stumped

alamb · 2025-06-29T12:13:22Z

An update here is I am working on some scripts to start measuring performance over time of datafusion so we can get a better handle on how performance is changing over time

zhuqi-lucas · 2025-06-30T11:27:09Z

An update here is I am working on some scripts to start measuring performance over time of datafusion so we can get a better handle on how performance is changing over time

This is great for us to monitor performance!

alamb · 2025-06-30T11:57:52Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/arrow_55.2.0_upgrade_real (96f17a7) to 3649dc8diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-06-30T12:38:27Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_arrow_55.2.0_upgrade_real
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1978.71 ms │ 1983.73 ms │ no change │
│ QQuery 1 │ 721.61 ms │ 772.62 ms │ 1.07x slower │
│ QQuery 2 │ 1401.77 ms │ 1464.18 ms │ no change │
│ QQuery 3 │ 647.93 ms │ 641.63 ms │ no change │
│ QQuery 4 │ 1333.18 ms │ 1329.49 ms │ no change │
│ QQuery 5 │ 15133.77 ms │ 15045.60 ms │ no change │
│ QQuery 6 │ 1988.09 ms │ 2076.03 ms │ no change │
│ QQuery 7 │ 2010.53 ms │ 1963.98 ms │ no change │
└──────────────┴─────────────┴──────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 25215.60ms │
│ Total Time (alamb_arrow_55.2) │ 25277.27ms │
│ Average Time (HEAD) │ 3151.95ms │
│ Average Time (alamb_arrow_55.2) │ 3159.66ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 7 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.32 ms │ 2.55 ms │ 1.10x slower │
│ QQuery 1 │ 34.53 ms │ 34.71 ms │ no change │
│ QQuery 2 │ 81.96 ms │ 83.19 ms │ no change │
│ QQuery 3 │ 98.65 ms │ 96.91 ms │ no change │
│ QQuery 4 │ 600.21 ms │ 586.70 ms │ no change │
│ QQuery 5 │ 839.62 ms │ 855.19 ms │ no change │
│ QQuery 6 │ 2.36 ms │ 2.22 ms │ +1.06x faster │
│ QQuery 7 │ 39.60 ms │ 39.42 ms │ no change │
│ QQuery 8 │ 852.85 ms │ 834.24 ms │ no change │
│ QQuery 9 │ 1164.97 ms │ 1151.93 ms │ no change │
│ QQuery 10 │ 257.19 ms │ 256.65 ms │ no change │
│ QQuery 11 │ 284.33 ms │ 288.22 ms │ no change │
│ QQuery 12 │ 876.34 ms │ 877.73 ms │ no change │
│ QQuery 13 │ 1245.55 ms │ 1222.01 ms │ no change │
│ QQuery 14 │ 800.20 ms │ 812.34 ms │ no change │
│ QQuery 15 │ 767.98 ms │ 771.52 ms │ no change │
│ QQuery 16 │ 1605.70 ms │ 1615.51 ms │ no change │
│ QQuery 17 │ 1578.02 ms │ 1600.82 ms │ no change │
│ QQuery 18 │ 2913.53 ms │ 2891.11 ms │ no change │
│ QQuery 19 │ 82.85 ms │ 83.03 ms │ no change │
│ QQuery 20 │ 1119.44 ms │ 1185.32 ms │ 1.06x slower │
│ QQuery 21 │ 1256.13 ms │ 1329.78 ms │ 1.06x slower │
│ QQuery 22 │ 2062.91 ms │ 2198.62 ms │ 1.07x slower │
│ QQuery 23 │ 7328.97 ms │ 7534.78 ms │ no change │
│ QQuery 24 │ 448.30 ms │ 465.27 ms │ no change │
│ QQuery 25 │ 378.59 ms │ 395.31 ms │ no change │
│ QQuery 26 │ 514.57 ms │ 527.80 ms │ no change │
│ QQuery 27 │ 1519.52 ms │ 1590.31 ms │ no change │
│ QQuery 28 │ 11861.77 ms │ 11878.02 ms │ no change │
│ QQuery 29 │ 527.52 ms │ 515.61 ms │ no change │
│ QQuery 30 │ 785.46 ms │ 788.29 ms │ no change │
│ QQuery 31 │ 808.81 ms │ 822.58 ms │ no change │
│ QQuery 32 │ 2557.21 ms │ 2516.31 ms │ no change │
│ QQuery 33 │ 3173.45 ms │ 3251.74 ms │ no change │
│ QQuery 34 │ 3257.37 ms │ 3248.34 ms │ no change │
│ QQuery 35 │ 1256.93 ms │ 1251.09 ms │ no change │
│ QQuery 36 │ 115.06 ms │ 120.49 ms │ no change │
│ QQuery 37 │ 50.09 ms │ 52.92 ms │ 1.06x slower │
│ QQuery 38 │ 115.83 ms │ 117.30 ms │ no change │
│ QQuery 39 │ 184.51 ms │ 202.77 ms │ 1.10x slower │
│ QQuery 40 │ 40.20 ms │ 40.64 ms │ no change │
│ QQuery 41 │ 37.96 ms │ 37.76 ms │ no change │
│ QQuery 42 │ 33.11 ms │ 32.48 ms │ no change │
└──────────────┴─────────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 53562.52ms │
│ Total Time (alamb_arrow_55.2) │ 54209.55ms │
│ Average Time (HEAD) │ 1245.64ms │
│ Average Time (alamb_arrow_55.2) │ 1260.69ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 6 │
│ Queries with No Change │ 36 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 100.64 ms │ 97.82 ms │ no change │
│ QQuery 2 │ 20.84 ms │ 20.80 ms │ no change │
│ QQuery 3 │ 32.26 ms │ 32.28 ms │ no change │
│ QQuery 4 │ 18.81 ms │ 18.34 ms │ no change │
│ QQuery 5 │ 50.07 ms │ 50.06 ms │ no change │
│ QQuery 6 │ 12.01 ms │ 12.05 ms │ no change │
│ QQuery 7 │ 92.07 ms │ 91.63 ms │ no change │
│ QQuery 8 │ 24.32 ms │ 25.09 ms │ no change │
│ QQuery 9 │ 54.14 ms │ 53.69 ms │ no change │
│ QQuery 10 │ 43.40 ms │ 42.70 ms │ no change │
│ QQuery 11 │ 11.20 ms │ 11.28 ms │ no change │
│ QQuery 12 │ 34.38 ms │ 34.05 ms │ no change │
│ QQuery 13 │ 26.67 ms │ 25.53 ms │ no change │
│ QQuery 14 │ 9.83 ms │ 9.90 ms │ no change │
│ QQuery 15 │ 19.85 ms │ 19.07 ms │ no change │
│ QQuery 16 │ 19.12 ms │ 18.02 ms │ +1.06x faster │
│ QQuery 17 │ 95.81 ms │ 94.41 ms │ no change │
│ QQuery 18 │ 188.64 ms │ 194.78 ms │ no change │
│ QQuery 19 │ 25.97 ms │ 24.98 ms │ no change │
│ QQuery 20 │ 32.53 ms │ 31.53 ms │ no change │
│ QQuery 21 │ 141.93 ms │ 146.01 ms │ no change │
│ QQuery 22 │ 14.94 ms │ 14.66 ms │ no change │
└──────────────┴───────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1069.44ms │
│ Total Time (alamb_arrow_55.2) │ 1068.67ms │
│ Average Time (HEAD) │ 48.61ms │
│ Average Time (alamb_arrow_55.2) │ 48.58ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴───────────┘

alamb · 2025-06-30T12:38:30Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/arrow_55.2.0_upgrade_real (96f17a7) to 3649dc8diff using: clickbench_1
Results will be posted here when complete

alamb · 2025-06-30T12:48:24Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_arrow_55.2.0_upgrade_real
--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ alamb_arrow_55.2 ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 0.60 ms │ 0.56 ms │ +1.07x faster │
│ QQuery 1 │ 74.85 ms │ 73.00 ms │ no change │
│ QQuery 2 │ 108.60 ms │ 108.95 ms │ no change │
│ QQuery 3 │ 129.34 ms │ 127.64 ms │ no change │
│ QQuery 4 │ 637.62 ms │ 631.53 ms │ no change │
│ QQuery 5 │ 866.10 ms │ 855.62 ms │ no change │
│ QQuery 6 │ 0.66 ms │ 0.65 ms │ no change │
│ QQuery 7 │ 82.24 ms │ 81.25 ms │ no change │
│ QQuery 8 │ 900.73 ms │ 870.95 ms │ no change │
│ QQuery 9 │ 1229.84 ms │ 1194.27 ms │ no change │
│ QQuery 10 │ 284.54 ms │ 295.59 ms │ no change │
│ QQuery 11 │ 325.07 ms │ 326.03 ms │ no change │
│ QQuery 12 │ 875.44 ms │ 860.67 ms │ no change │
│ QQuery 13 │ 1242.09 ms │ 1205.29 ms │ no change │
│ QQuery 14 │ 809.58 ms │ 814.53 ms │ no change │
│ QQuery 15 │ 818.09 ms │ 809.99 ms │ no change │
│ QQuery 16 │ 1652.10 ms │ 1619.74 ms │ no change │
│ QQuery 17 │ 1610.13 ms │ 1611.01 ms │ no change │
│ QQuery 18 │ 2924.30 ms │ 2924.66 ms │ no change │
│ QQuery 19 │ 127.40 ms │ 120.40 ms │ +1.06x faster │
│ QQuery 20 │ 1143.44 ms │ 1208.32 ms │ 1.06x slower │
│ QQuery 21 │ 1335.84 ms │ 1401.33 ms │ no change │
│ QQuery 22 │ 2322.35 ms │ 2442.41 ms │ 1.05x slower │
│ QQuery 23 │ 7921.41 ms │ 8182.17 ms │ no change │
│ QQuery 24 │ 485.77 ms │ 493.02 ms │ no change │
│ QQuery 25 │ 416.03 ms │ 426.08 ms │ no change │
│ QQuery 26 │ 556.08 ms │ 566.43 ms │ no change │
│ QQuery 27 │ 1639.87 ms │ 1675.49 ms │ no change │
│ QQuery 28 │ 12231.89 ms │ 12226.72 ms │ no change │
│ QQuery 29 │ 570.58 ms │ 564.21 ms │ no change │
│ QQuery 30 │ 798.08 ms │ 819.68 ms │ no change │
│ QQuery 31 │ 839.61 ms │ 854.93 ms │ no change │
│ QQuery 32 │ 2541.36 ms │ 2520.78 ms │ no change │
│ QQuery 33 │ 3291.32 ms │ 3281.86 ms │ no change │
│ QQuery 34 │ 3566.07 ms │ 3300.54 ms │ +1.08x faster │
│ QQuery 35 │ 1293.52 ms │ 1285.57 ms │ no change │
│ QQuery 36 │ 170.78 ms │ 178.72 ms │ no change │
│ QQuery 37 │ 100.71 ms │ 103.44 ms │ no change │
│ QQuery 38 │ 176.01 ms │ 154.63 ms │ +1.14x faster │
│ QQuery 39 │ 253.78 ms │ 258.68 ms │ no change │
│ QQuery 40 │ 82.48 ms │ 87.26 ms │ 1.06x slower │
│ QQuery 41 │ 81.05 ms │ 81.52 ms │ no change │
│ QQuery 42 │ 77.47 ms │ 77.55 ms │ no change │
└──────────────┴─────────────┴──────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 56594.83ms │
│ Total Time (alamb_arrow_55.2) │ 56723.66ms │
│ Average Time (HEAD) │ 1316.16ms │
│ Average Time (alamb_arrow_55.2) │ 1319.15ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 36 │
│ Queries with Failure │ 0 │
└─────────────────────────────────┴────────────┘

alamb · 2025-06-30T18:31:19Z

I am going to merge this PR in to unblock other things (like the PR for external parquet indexes)

I am not really sure about the performance impact as I see some non trivial fluctuation.

However, I have been working on a system to record and report DataFusion performance overtime. See #5504 for more detail or get a sneak peek here:

https://github.com/alamb/datafusion-benchmarking

If we have lost any performance let's plan to get it back over time

alamb · 2025-06-30T19:05:59Z

🚀

alamb added 2 commits June 26, 2025 12:55

Update to arrow/parquet 55.2.0
5ff7585
Update to released version

Update plans
96f17a7

github-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Jun 26, 2025

alamb commented Jun 26, 2025
View reviewed changes

alamb marked this pull request as ready for review June 26, 2025 16:59

This was referenced Jun 26, 2025
Add an example of embedding indexes inside a parquet file #16395
Merged
Pin to pre-release arrow / parquet version 55.2.2#16492
Closed

zhuqi-lucas approved these changes Jun 27, 2025
View reviewed changes

This comment was marked as outdated.
Sign in to view

alamb mentioned this pull request Jun 27, 2025
chore(deps): bump the arrow-parquet group with 7 updates #16581
Closed

comphead approved these changes Jun 27, 2025
View reviewed changes

alamb mentioned this pull request Jun 27, 2025
Improve performance of ClickBench Q21 by removing the cast #16591
Closed

Merge branch 'main' into alamb/arrow_55.2.0_upgrade_real
44da7a8

alamb merged commit f65da24 into apache:mainJun 30, 2025
29 checks passed

alamb deleted the alamb/arrow_55.2.0_upgrade_real branch June 30, 2025 19:06

alamb mentioned this pull request Jun 30, 2025
chore(deps): bump the arrow-parquet group with 7 updates #16621
Closed

Uh oh!

Conversation

alamb commented Jun 26, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alambJun 26, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alambJun 26, 2025

Choose a reason for hiding this comment

Uh oh!

alamb commented Jun 26, 2025

Uh oh!

alamb commented Jun 26, 2025

Uh oh!

alamb commented Jun 26, 2025

Uh oh!

alamb commented Jun 26, 2025

Uh oh!

alamb commented Jun 26, 2025

Uh oh!

alamb commented Jun 26, 2025

Uh oh!

zhuqi-lucas left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

This comment was marked as outdated.

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 27, 2025

Uh oh!

alamb commented Jun 29, 2025

Uh oh!

zhuqi-lucas commented Jun 30, 2025

Uh oh!

alamb commented Jun 30, 2025

Uh oh!

alamb commented Jun 30, 2025

Uh oh!

alamb commented Jun 30, 2025

Uh oh!

alamb commented Jun 30, 2025

Uh oh!

alamb commented Jun 30, 2025

Uh oh!

Uh oh!

alamb commented Jun 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

alambJun 26, 2025•
edited
Loading