Use the upstream arrow-rs coalesce kernel by zhuqi-lucas · Pull Request #17193 · apache/datafusion

zhuqi-lucas · 2025-08-14T15:11:53Z

Which issue does this PR close?

Use the upstream arrow-rs coalesce kernel, and support LimitedBatchCoalesce for datafusion

Rationale for this change

Use the upstream arrow-rs coalesce kernel, also it will support LimitedBatchCoalesce for datafusion. There are some future work based this, for example Push limit into joins which will optimize join.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

…oalesce

…eam_arrow_coalesce

…rmance

zhuqi-lucas · 2025-10-28T05:28:26Z

Hi @alamb I am reviving this PR since the upstream arrow-rs has be upgraded, thanks!

alamb · 2025-10-28T20:19:55Z

Hi @alamb I am reviving this PR since the upstream arrow-rs has be upgraded, thanks!

Thank you @zhuqi-lucas -- I was just thinking the other day that we (I really) have dropped the ball on the coalesce kernel. I feel like once we get it into DataFusion we can then drive improvements upstream in arrow...

alamb · 2025-10-28T22:04:05Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing test_optimize_performance (d303a4d) to 2d004afdiff using: tpch_mem
Results will be posted here when complete

alamb · 2025-10-28T22:38:36Z

🤖: Benchmark completed

Details

Comparing HEAD and test_optimize_performance
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ test_optimize_performance ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 165.36 ms │ 139.21 ms │ +1.19x faster │
│ QQuery 2 │ 26.07 ms │ 27.02 ms │ no change │
│ QQuery 3 │ 41.53 ms │ 37.80 ms │ +1.10x faster │
│ QQuery 4 │ 27.70 ms │ 28.87 ms │ no change │
│ QQuery 5 │ 77.59 ms │ 86.36 ms │ 1.11x slower │
│ QQuery 6 │ 19.51 ms │ 19.09 ms │ no change │
│ QQuery 7 │ 259.48 ms │ 223.90 ms │ +1.16x faster │
│ QQuery 8 │ 35.96 ms │ 32.23 ms │ +1.12x faster │
│ QQuery 9 │ 110.90 ms │ 97.19 ms │ +1.14x faster │
│ QQuery 10 │ 59.72 ms │ 63.75 ms │ 1.07x slower │
│ QQuery 11 │ 17.08 ms │ 17.37 ms │ no change │
│ QQuery 12 │ 51.09 ms │ 52.85 ms │ no change │
│ QQuery 13 │ 47.12 ms │ 46.62 ms │ no change │
│ QQuery 14 │ 13.75 ms │ 13.98 ms │ no change │
│ QQuery 15 │ 24.58 ms │ 24.44 ms │ no change │
│ QQuery 16 │ 24.85 ms │ 25.24 ms │ no change │
│ QQuery 17 │ 148.38 ms │ 150.48 ms │ no change │
│ QQuery 18 │ 325.68 ms │ 268.59 ms │ +1.21x faster │
│ QQuery 19 │ 35.89 ms │ 37.68 ms │ no change │
│ QQuery 20 │ 48.33 ms │ 48.37 ms │ no change │
│ QQuery 21 │ 343.21 ms │ 321.81 ms │ +1.07x faster │
│ QQuery 22 │ 20.77 ms │ 20.99 ms │ no change │
└──────────────┴───────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1924.55ms │
│ Total Time (test_optimize_performance) │ 1783.83ms │
│ Average Time (HEAD) │ 87.48ms │
│ Average Time (test_optimize_performance) │ 81.08ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 13 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────┴───────────┘

alamb · 2025-10-28T22:38:39Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing test_optimize_performance (d303a4d) to 2d004afdiff using: clickbench_partitioned
Results will be posted here when complete

alamb · 2025-10-28T22:55:29Z

🤖: Benchmark completed

Details

Comparing HEAD and test_optimize_performance
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ test_optimize_performance ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.08 ms │ 2.22 ms │ 1.07x slower │
│ QQuery 1 │ 51.76 ms │ 49.19 ms │ no change │
│ QQuery 2 │ 138.58 ms │ 136.48 ms │ no change │
│ QQuery 3 │ 163.87 ms │ 159.43 ms │ no change │
│ QQuery 4 │ 1062.09 ms │ 1083.48 ms │ no change │
│ QQuery 5 │ 1504.52 ms │ 1519.72 ms │ no change │
│ QQuery 6 │ 2.13 ms │ 2.17 ms │ no change │
│ QQuery 7 │ 54.99 ms │ 56.21 ms │ no change │
│ QQuery 8 │ 1453.12 ms │ 1434.03 ms │ no change │
│ QQuery 9 │ 1771.36 ms │ 1795.54 ms │ no change │
│ QQuery 10 │ 380.31 ms │ 367.38 ms │ no change │
│ QQuery 11 │ 437.98 ms │ 420.04 ms │ no change │
│ QQuery 12 │ 1389.21 ms │ 1356.83 ms │ no change │
│ QQuery 13 │ 2154.30 ms │ 2102.75 ms │ no change │
│ QQuery 14 │ 1278.18 ms │ 1265.16 ms │ no change │
│ QQuery 15 │ 1199.29 ms │ 1255.16 ms │ no change │
│ QQuery 16 │ 2745.53 ms │ 2673.37 ms │ no change │
│ QQuery 17 │ 2686.26 ms │ 2671.54 ms │ no change │
│ QQuery 18 │ 5441.17 ms │ 5012.66 ms │ +1.09x faster │
│ QQuery 19 │ 126.65 ms │ 126.00 ms │ no change │
│ QQuery 20 │ 2076.29 ms │ 2017.20 ms │ no change │
│ QQuery 21 │ 2409.47 ms │ 2349.88 ms │ no change │
│ QQuery 22 │ 4146.20 ms │ 3961.03 ms │ no change │
│ QQuery 23 │ 13031.20 ms │ 12723.77 ms │ no change │
│ QQuery 24 │ 216.08 ms │ 208.33 ms │ no change │
│ QQuery 25 │ 495.59 ms │ 462.90 ms │ +1.07x faster │
│ QQuery 26 │ 212.81 ms │ 213.01 ms │ no change │
│ QQuery 27 │ 2953.33 ms │ 2862.80 ms │ no change │
│ QQuery 28 │ 22788.11 ms │ 22901.94 ms │ no change │
│ QQuery 29 │ 1012.52 ms │ 1010.36 ms │ no change │
│ QQuery 30 │ 1325.59 ms │ 1303.61 ms │ no change │
│ QQuery 31 │ 1339.31 ms │ 1388.16 ms │ no change │
│ QQuery 32 │ 4638.08 ms │ 4939.60 ms │ 1.07x slower │
│ QQuery 33 │ 5956.32 ms │ 5823.50 ms │ no change │
│ QQuery 34 │ 6161.05 ms │ 6107.98 ms │ no change │
│ QQuery 35 │ 2023.78 ms │ 1995.03 ms │ no change │
│ QQuery 36 │ 121.87 ms │ 123.41 ms │ no change │
│ QQuery 37 │ 53.46 ms │ 50.81 ms │ no change │
│ QQuery 38 │ 122.16 ms │ 124.81 ms │ no change │
│ QQuery 39 │ 202.66 ms │ 198.31 ms │ no change │
│ QQuery 40 │ 39.94 ms │ 43.28 ms │ 1.08x slower │
│ QQuery 41 │ 39.29 ms │ 36.95 ms │ +1.06x faster │
│ QQuery 42 │ 31.41 ms │ 32.66 ms │ no change │
└──────────────┴─────────────┴───────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 95439.88ms │
│ Total Time (test_optimize_performance) │ 94368.69ms │
│ Average Time (HEAD) │ 2219.53ms │
│ Average Time (test_optimize_performance) │ 2194.62ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 37 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────┴────────────┘

alamb · 2025-10-28T22:55:33Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing test_optimize_performance (d303a4d) to 2d004afdiff using: clickbench_extended
Results will be posted here when complete

alamb · 2025-10-28T23:03:42Z

🤖: Benchmark completed

Details

Comparing HEAD and test_optimize_performance
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ test_optimize_performance ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 2723.99 ms │ 2724.43 ms │ no change │
│ QQuery 1 │ 1252.31 ms │ 1279.79 ms │ no change │
│ QQuery 2 │ 2472.33 ms │ 2436.89 ms │ no change │
│ QQuery 3 │ 1192.10 ms │ 1194.83 ms │ no change │
│ QQuery 4 │ 2230.66 ms │ 2304.44 ms │ no change │
│ QQuery 5 │ 27861.61 ms │ 28255.38 ms │ no change │
│ QQuery 6 │ 4213.00 ms │ 4240.26 ms │ no change │
│ QQuery 7 │ 3679.97 ms │ 3683.50 ms │ no change │
└──────────────┴─────────────┴───────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 45625.96ms │
│ Total Time (test_optimize_performance) │ 46119.50ms │
│ Average Time (HEAD) │ 5703.25ms │
│ Average Time (test_optimize_performance) │ 5764.94ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└──────────────────────────────────────────┴────────────┘

zhuqi-lucas · 2025-10-29T03:35:47Z

Thank you @alamb for benchmark, the result is good!

alamb · 2025-10-29T11:04:19Z

I agree the benchmarks look good to me (no significant difference in performance)

@zhuqi-lucas would it be ok to update the title of this PR and its description? I think the core change here is to use the upstream 'coalesce' kernel. Is that your understanding too?

zhuqi-lucas · 2025-10-29T11:37:38Z

I agree the benchmarks look good to me (no significant difference in performance)
@zhuqi-lucas would it be ok to update the title of this PR and its description? I think the core change here is to use the upstream 'coalesce' kernel. Is that your understanding too?

Updated the title now, thank you @alamb .

alamb

Thank you @zhuqi-lucas

2010YOUY01

This is great!

zhuqi-lucas · 2025-10-31T09:48:20Z

Thank you @alamb @2010YOUY01 for review! Merged now.

alamb · 2025-10-31T10:14:28Z

Awesome -- one step closer

## Which issue does this PR close? Use the upstream arrow-rs coalesce kernel, and support LimitedBatchCoalesce for datafusion ## Rationale for this change Use the upstream arrow-rs coalesce kernel, also it will support LimitedBatchCoalesce for datafusion. There are some future work based this, for example [Push limit into joins ](apache#18295) which will optimize join. ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?   --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Daniël Heres <danielheres@gmail.com>

alamband others added 30 commits June 4, 2025 09:01

Pin toapache/arrow-rs#7597
9a161d2

Update pin
083931d

Use upstream BatchCoalescer
e79454f

Update the pin
4e8e1ce

Update tests
9e20973

Update rev
8918b3c

Update rev
49cb62e

New rev
140ee9c

New rev
5d5683c

New rev
f79dd09

cargo fmt
1c44c5c

update pin
ea8b700

Merge branch 'main' into alamb/test_upstream_coalesce
f2fc00b

Merge branch 'main' into alamb/test_upstream_coalesce
a36065e

Merge remote-tracking branch 'apache/main' into alamb/test_upstream_c…
423137a
…oalesce

Temp pin toapache/arrow-rs#7650
1c61513

Update plans for smaller parquet files
ed31ce1

Merge remote-tracking branch 'apache/main' into alamb/test_upstream_c…
5349c73
…oalesce

update pin
c5bb25e

Merge remote-tracking branch 'upstream/main' into revive_to_use_upstr…
2f94a22
…eam_arrow_coalesce

fix test
fe7e6a3

fix
0832ff4

Merge branch 'main' into revive_to_use_upstream_arrow_coalesce
396ef3c

fix test
7e6ced0

fix
3ccea48

fix
bac0197

Merge branch 'main' into revive_to_use_upstream_arrow_coalesce
64fc038

Address comments
8bbadaf

Merge remote-tracking branch 'upstream/main' into revive_to_use_upstr…
c88085b
…eam_arrow_coalesce

address new comments
940d49d

Merge remote-tracking branch 'upstream/main' into test_optimize_perfo…
d303a4d
…rmance

zhuqi-lucas force-pushed the test_optimize_performance branch from 8dec409 to d303a4dCompare October 28, 2025 05:23

zhuqi-lucas changed the title ~~Testing: Try test optimize performance for coalesce~~Use the upstream arrow-rs coalesce kernelOct 29, 2025

Merge branch 'main' into test_optimize_performance
bf916f6

alamb approved these changes Oct 29, 2025
View reviewed changes

Merge branch 'main' into test_optimize_performance
7c01073

2010YOUY01 approved these changes Oct 30, 2025
View reviewed changes

2010YOUY01 mentioned this pull request Oct 30, 2025
Push limit into joins #18295
Open

Merge branch 'main' into test_optimize_performance
fad15e1

zhuqi-lucas added this pull request to the merge queue Oct 31, 2025

Merged via the queue into apache:main with commit b6e5732Oct 31, 2025
37 checks passed

alamb mentioned this pull request Nov 17, 2025
[EPIC] Remove CoalesceBatchesExec operator #18779
Closed
12 tasks

mbutrovich mentioned this pull request Nov 21, 2025
Sort Merge Join: Reduce batch concatenation, use BatchCoalescer, new benchmarks (TPC-H Q21 SMJ up to ~4000x faster) #18875
Merged

haohuaijin mentioned this pull request Dec 9, 2025
perf: gc stringview before send recordbatch to leader openobserve/openobserve#9525
Merged

Uh oh!

Conversation

zhuqi-lucas commented Aug 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

zhuqi-lucas commented Oct 28, 2025

Uh oh!

alamb commented Oct 28, 2025

Uh oh!

alamb commented Oct 28, 2025

Uh oh!

alamb commented Oct 28, 2025

Uh oh!

alamb commented Oct 28, 2025

Uh oh!

alamb commented Oct 28, 2025

Uh oh!

alamb commented Oct 28, 2025

Uh oh!

alamb commented Oct 28, 2025

Uh oh!

zhuqi-lucas commented Oct 29, 2025

Uh oh!

alamb commented Oct 29, 2025

Uh oh!

zhuqi-lucas commented Oct 29, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

2010YOUY01 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhuqi-lucas commented Oct 31, 2025

Uh oh!

alamb commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhuqi-lucas commented Aug 14, 2025•
edited
Loading