remove repartition exec from coalesce batches optimizer by jizezhang · Pull Request #19239 · apache/datafusion

jizezhang · 2025-12-09T15:49:04Z

Which issue does this PR close?

ClosesIntegrate BatchCoalescer into RepartitionExec #18782.

Rationale for this change

What changes are included in this PR?

Removes RepartitionExec from CoalesceBatches optimizer.

Are these changes tested?

Yes

Are there any user-facing changes?

No

alamb · 2025-12-11T22:52:03Z

run benchmarks

alamb · 2025-12-11T22:52:11Z

run benchmark tpch_mem10

alamb-ghbot · 2025-12-11T23:24:30Z

🤖 ./gh_compare_branch.shgh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (446f6cf) to 20870c1diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2025-12-12T00:04:14Z

🤖: Benchmark completed

Details

Comparing HEAD and jizezhang_main
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ jizezhang_main ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 2733.19 ms │ 2720.03 ms │ no change │
│ QQuery 1 │ 1281.78 ms │ 1252.52 ms │ no change │
│ QQuery 2 │ 2377.84 ms │ 2424.84 ms │ no change │
│ QQuery 3 │ 1160.92 ms │ 1168.87 ms │ no change │
│ QQuery 4 │ 2324.43 ms │ 2290.43 ms │ no change │
│ QQuery 5 │ 28592.61 ms │ 28760.63 ms │ no change │
│ QQuery 6 │ 4037.19 ms │ 3970.23 ms │ no change │
│ QQuery 7 │ 3695.67 ms │ 3737.12 ms │ no change │
└──────────────┴─────────────┴────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 46203.63ms │
│ Total Time (jizezhang_main) │ 46324.67ms │
│ Average Time (HEAD) │ 5775.45ms │
│ Average Time (jizezhang_main) │ 5790.58ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└───────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ jizezhang_main ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.57 ms │ 2.25 ms │ +1.14x faster │
│ QQuery 1 │ 52.05 ms │ 52.40 ms │ no change │
│ QQuery 2 │ 134.07 ms │ 134.10 ms │ no change │
│ QQuery 3 │ 157.59 ms │ 154.23 ms │ no change │
│ QQuery 4 │ 1177.17 ms │ 1092.07 ms │ +1.08x faster │
│ QQuery 5 │ 1555.61 ms │ 1502.54 ms │ no change │
│ QQuery 6 │ 2.14 ms │ 2.19 ms │ no change │
│ QQuery 7 │ 56.24 ms │ 57.00 ms │ no change │
│ QQuery 8 │ 1508.89 ms │ 1430.78 ms │ +1.05x faster │
│ QQuery 9 │ 1970.06 ms │ 1881.53 ms │ no change │
│ QQuery 10 │ 371.09 ms │ 374.63 ms │ no change │
│ QQuery 11 │ 416.93 ms │ 423.24 ms │ no change │
│ QQuery 12 │ 1440.40 ms │ 1365.87 ms │ +1.05x faster │
│ QQuery 13 │ 2049.91 ms │ 2021.63 ms │ no change │
│ QQuery 14 │ 1300.92 ms │ 1248.05 ms │ no change │
│ QQuery 15 │ 1307.90 ms │ 1243.04 ms │ no change │
│ QQuery 16 │ 2665.57 ms │ 2700.25 ms │ no change │
│ QQuery 17 │ 2677.16 ms │ 2689.07 ms │ no change │
│ QQuery 18 │ 5058.27 ms │ 5181.21 ms │ no change │
│ QQuery 19 │ 126.21 ms │ 124.54 ms │ no change │
│ QQuery 20 │ 1896.68 ms │ 1911.49 ms │ no change │
│ QQuery 21 │ 2174.96 ms │ 2189.22 ms │ no change │
│ QQuery 22 │ 3676.22 ms │ 3730.96 ms │ no change │
│ QQuery 23 │ 12348.34 ms │ 12302.89 ms │ no change │
│ QQuery 24 │ 208.97 ms │ 211.21 ms │ no change │
│ QQuery 25 │ 464.76 ms │ 471.47 ms │ no change │
│ QQuery 26 │ 220.82 ms │ 208.49 ms │ +1.06x faster │
│ QQuery 27 │ 2708.31 ms │ 2710.37 ms │ no change │
│ QQuery 28 │ 23987.85 ms │ 24389.78 ms │ no change │
│ QQuery 29 │ 974.58 ms │ 940.27 ms │ no change │
│ QQuery 30 │ 1352.37 ms │ 1309.12 ms │ no change │
│ QQuery 31 │ 1365.80 ms │ 1327.44 ms │ no change │
│ QQuery 32 │ 5194.56 ms │ 4840.81 ms │ +1.07x faster │
│ QQuery 33 │ 5948.34 ms │ 5925.71 ms │ no change │
│ QQuery 34 │ 6060.74 ms │ 6119.94 ms │ no change │
│ QQuery 35 │ 2005.86 ms │ 1924.80 ms │ no change │
│ QQuery 36 │ 118.66 ms │ 117.42 ms │ no change │
│ QQuery 37 │ 54.36 ms │ 55.94 ms │ no change │
│ QQuery 38 │ 117.99 ms │ 116.94 ms │ no change │
│ QQuery 39 │ 188.85 ms │ 196.87 ms │ no change │
│ QQuery 40 │ 46.58 ms │ 43.80 ms │ +1.06x faster │
│ QQuery 41 │ 40.04 ms │ 41.09 ms │ no change │
│ QQuery 42 │ 34.13 ms │ 33.88 ms │ no change │
└──────────────┴─────────────┴────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 95220.52ms │
│ Total Time (jizezhang_main) │ 94800.51ms │
│ Average Time (HEAD) │ 2214.43ms │
│ Average Time (jizezhang_main) │ 2204.66ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 36 │
│ Queries with Failure │ 0 │
└───────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ jizezhang_main ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1 │ 129.27 ms │ 132.98 ms │ no change │
│ QQuery 2 │ 27.66 ms │ 27.10 ms │ no change │
│ QQuery 3 │ 37.19 ms │ 37.92 ms │ no change │
│ QQuery 4 │ 28.76 ms │ 28.28 ms │ no change │
│ QQuery 5 │ 87.45 ms │ 86.95 ms │ no change │
│ QQuery 6 │ 19.85 ms │ 19.87 ms │ no change │
│ QQuery 7 │ 227.31 ms │ 227.85 ms │ no change │
│ QQuery 8 │ 36.11 ms │ 35.44 ms │ no change │
│ QQuery 9 │ 106.23 ms │ 101.00 ms │ no change │
│ QQuery 10 │ 63.19 ms │ 63.64 ms │ no change │
│ QQuery 11 │ 17.70 ms │ 18.54 ms │ no change │
│ QQuery 12 │ 52.76 ms │ 52.15 ms │ no change │
│ QQuery 13 │ 47.73 ms │ 48.17 ms │ no change │
│ QQuery 14 │ 13.77 ms │ 14.03 ms │ no change │
│ QQuery 15 │ 24.71 ms │ 24.06 ms │ no change │
│ QQuery 16 │ 25.16 ms │ 24.86 ms │ no change │
│ QQuery 17 │ 150.79 ms │ 150.25 ms │ no change │
│ QQuery 18 │ 279.74 ms │ 278.43 ms │ no change │
│ QQuery 19 │ 37.56 ms │ 37.52 ms │ no change │
│ QQuery 20 │ 49.26 ms │ 48.52 ms │ no change │
│ QQuery 21 │ 313.83 ms │ 322.16 ms │ no change │
│ QQuery 22 │ 17.49 ms │ 17.33 ms │ no change │
└──────────────┴───────────┴────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1793.52ms │
│ Total Time (jizezhang_main) │ 1797.05ms │
│ Average Time (HEAD) │ 81.52ms │
│ Average Time (jizezhang_main) │ 81.68ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 22 │
│ Queries with Failure │ 0 │
└───────────────────────────────┴───────────┘

alamb-ghbot · 2025-12-12T00:04:17Z

🤖 ./gh_compare_branch.shgh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (446f6cf) to 20870c1diff using: tpch_mem10
Results will be posted here when complete

alamb-ghbot · 2025-12-12T00:04:20Z

🤖: Benchmark completed

Details

Comparing HEAD and jizezhang_main
--------------------
Benchmark tpch_mem_sf10.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ jizezhang_main ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ FAIL │ FAIL │ incomparable │
│ QQuery 2 │ FAIL │ FAIL │ incomparable │
│ QQuery 3 │ FAIL │ FAIL │ incomparable │
│ QQuery 4 │ FAIL │ FAIL │ incomparable │
│ QQuery 5 │ FAIL │ FAIL │ incomparable │
│ QQuery 6 │ FAIL │ FAIL │ incomparable │
│ QQuery 7 │ FAIL │ FAIL │ incomparable │
│ QQuery 8 │ FAIL │ FAIL │ incomparable │
│ QQuery 9 │ FAIL │ FAIL │ incomparable │
│ QQuery 10 │ FAIL │ FAIL │ incomparable │
│ QQuery 11 │ FAIL │ FAIL │ incomparable │
│ QQuery 12 │ FAIL │ FAIL │ incomparable │
│ QQuery 13 │ FAIL │ FAIL │ incomparable │
│ QQuery 14 │ FAIL │ FAIL │ incomparable │
│ QQuery 15 │ FAIL │ FAIL │ incomparable │
│ QQuery 16 │ FAIL │ FAIL │ incomparable │
│ QQuery 17 │ FAIL │ FAIL │ incomparable │
│ QQuery 18 │ FAIL │ FAIL │ incomparable │
│ QQuery 19 │ FAIL │ FAIL │ incomparable │
│ QQuery 20 │ FAIL │ FAIL │ incomparable │
│ QQuery 21 │ FAIL │ FAIL │ incomparable │
│ QQuery 22 │ FAIL │ FAIL │ incomparable │
└──────────────┴──────┴────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ Total Time (HEAD) │ 0.00ms │
│ Total Time (jizezhang_main) │ 0.00ms │
│ Average Time (HEAD) │ 0.00ms │
│ Average Time (jizezhang_main) │ 0.00ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 0 │
│ Queries with Failure │ 22 │
└───────────────────────────────┴────────┘

Copilot

Pull request overview

This PR removes RepartitionExec from the CoalesceBatches optimizer by updating test expectations to reflect the optimizer change. The CoalesceBatchesExec node is no longer inserted after RepartitionExec operations in the physical execution plans.

Removes CoalesceBatchesExec optimizer behavior after repartitioning operations
Updates all affected SQL logic test files to reflect the new physical plan structure
Includes minor whitespace corrections in test files

Reviewed changes

Copilot reviewed 53 out of 55 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
window_limits.slt	Removed CoalesceBatchesExec from window aggregate plans
window.slt	Removed CoalesceBatchesExec from multiple window and aggregate operations
unnest.slt	Removed CoalesceBatchesExec from unnest aggregate plans
union.slt	Removed CoalesceBatchesExec from union and aggregate plans
tpch/plans/*.slt.part	Removed CoalesceBatchesExec from all TPC-H query plans (Q1-Q22)
timestamps.slt	Removed extra spaces in error message assertions
spark/math/hex.slt	Fixed line number formatting
subquery*.slt	Removed CoalesceBatchesExec from subquery aggregate plans
select.slt	Removed CoalesceBatchesExec from aggregate with limit plans
repartition.slt	Removed CoalesceBatchesExec from repartition aggregate plans
qualify.slt	Removed CoalesceBatchesExec from qualify window plans
push_down_filter.slt	Removed CoalesceBatchesExec from grouping sets plans
preserve_file_partitioning.slt	Removed CoalesceBatchesExec from partitioned file plans
predicates.slt	Removed CoalesceBatchesExec from join filter pushdown plans
order.slt	Removed CoalesceBatchesExec from ordered aggregate plans
limit.slt	Removed CoalesceBatchesExec from limited aggregate plans
joins.slt & join.slt.part	Removed CoalesceBatchesExec from various join plans
insert*.slt	Removed CoalesceBatchesExec from insert with window plans
explain_tree.slt	Removed CoalesceBatchesExec from tree visualization
distinct_on.slt	Removed CoalesceBatchesExec from distinct aggregate plans
cte.slt	Removed CoalesceBatchesExec from recursive CTE plans
count_star_rule.slt	Removed CoalesceBatchesExec from count aggregate plans

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

alamb

This is great -- thank you so much @jizezhang 🙏 I am sorry it took so long to review.

I'll plan to merge this tomorrow. Whoever merges this should make sure to rerun the tests as I think there is a non trivial chance of conflicts

alamb · 2025-12-11T22:53:23Z

- target_batch_size,
- ))))
- } else if let Some(async_exec) = plan_any.downcast_ref::<AsyncFuncExec>() {
+ if let Some(async_exec) = plan_any.downcast_ref::<AsyncFuncExec>() {


looks like the only thing left is AsyncFuncExec and we could entirely remove the CoalesceBatches pass (and maybe even structure)

alamb · 2025-12-14T11:27:51Z

-54)----------------------FilterExec: r_name@1 = AMERICA, projection=[r_regionkey@0]
-55)------------------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
-56)--------------------------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/tpch/data/region.tbl]]}, projection=[r_regionkey, r_name], file_type=csv, has_header=false
+05)--------RepartitionExec: partitioning=Hash([o_year@0], 4), input_partitions=4


these complex query plans are so much easier to read now

Dandandan · 2025-12-15T12:47:50Z

run benchmark clickbench_partitioned

alamb-ghbot · 2025-12-15T12:47:56Z

🤖 ./gh_compare_branch.shgh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (d685866) to 033116ediff using: clickbench_partitioned
Results will be posted here when complete

alamb-ghbot · 2025-12-15T13:19:26Z

🤖: Benchmark completed

Details

Comparing HEAD and jizezhang_main
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ jizezhang_main ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.12 ms │ 2.20 ms │ no change │
│ QQuery 1 │ 49.90 ms │ 50.17 ms │ no change │
│ QQuery 2 │ 135.88 ms │ 135.00 ms │ no change │
│ QQuery 3 │ 153.88 ms │ 156.47 ms │ no change │
│ QQuery 4 │ 1090.93 ms │ 1086.08 ms │ no change │
│ QQuery 5 │ 1433.46 ms │ 1467.81 ms │ no change │
│ QQuery 6 │ 2.20 ms │ 2.22 ms │ no change │
│ QQuery 7 │ 55.37 ms │ 54.06 ms │ no change │
│ QQuery 8 │ 1426.54 ms │ 1421.77 ms │ no change │
│ QQuery 9 │ 1824.69 ms │ 1827.51 ms │ no change │
│ QQuery 10 │ 354.93 ms │ 360.16 ms │ no change │
│ QQuery 11 │ 416.23 ms │ 414.75 ms │ no change │
│ QQuery 12 │ 1315.38 ms │ 1327.46 ms │ no change │
│ QQuery 13 │ 2030.94 ms │ 2032.11 ms │ no change │
│ QQuery 14 │ 1236.37 ms │ 1244.11 ms │ no change │
│ QQuery 15 │ 1248.80 ms │ 1227.81 ms │ no change │
│ QQuery 16 │ 2639.01 ms │ 2661.50 ms │ no change │
│ QQuery 17 │ 2623.62 ms │ 2648.13 ms │ no change │
│ QQuery 18 │ 5802.31 ms │ 5078.57 ms │ +1.14x faster │
│ QQuery 19 │ 122.89 ms │ 125.37 ms │ no change │
│ QQuery 20 │ 1925.23 ms │ 1816.50 ms │ +1.06x faster │
│ QQuery 21 │ 2183.29 ms │ 2184.08 ms │ no change │
│ QQuery 22 │ 3762.73 ms │ 3730.48 ms │ no change │
│ QQuery 23 │ 18873.98 ms │ 12326.09 ms │ +1.53x faster │
│ QQuery 24 │ 219.05 ms │ 223.43 ms │ no change │
│ QQuery 25 │ 461.98 ms │ 455.19 ms │ no change │
│ QQuery 26 │ 223.39 ms │ 218.96 ms │ no change │
│ QQuery 27 │ 2765.04 ms │ 2686.97 ms │ no change │
│ QQuery 28 │ 23681.66 ms │ 23350.57 ms │ no change │
│ QQuery 29 │ 987.66 ms │ 962.29 ms │ no change │
│ QQuery 30 │ 1313.97 ms │ 1298.99 ms │ no change │
│ QQuery 31 │ 1326.60 ms │ 1344.03 ms │ no change │
│ QQuery 32 │ 4965.74 ms │ 5040.46 ms │ no change │
│ QQuery 33 │ 5944.09 ms │ 5822.75 ms │ no change │
│ QQuery 34 │ 5960.38 ms │ 5866.65 ms │ no change │
│ QQuery 35 │ 1931.03 ms │ 1890.30 ms │ no change │
│ QQuery 36 │ 116.81 ms │ 118.06 ms │ no change │
│ QQuery 37 │ 54.45 ms │ 55.22 ms │ no change │
│ QQuery 38 │ 115.15 ms │ 117.00 ms │ no change │
│ QQuery 39 │ 186.79 ms │ 188.61 ms │ no change │
│ QQuery 40 │ 46.25 ms │ 45.71 ms │ no change │
│ QQuery 41 │ 41.04 ms │ 41.15 ms │ no change │
│ QQuery 42 │ 35.18 ms │ 34.14 ms │ no change │
└──────────────┴─────────────┴────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 101086.95ms │
│ Total Time (jizezhang_main) │ 93140.89ms │
│ Average Time (HEAD) │ 2350.86ms │
│ Average Time (jizezhang_main) │ 2166.07ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 40 │
│ Queries with Failure │ 0 │
└───────────────────────────────┴─────────────┘

alamb · 2025-12-15T19:53:31Z

I merged up to resolve some conflicts

alamb · 2025-12-16T14:10:46Z

run benchmarks

alamb · 2025-12-16T14:11:11Z

(I don't think this PR makes any meaningful difference on performance, as expected)

alamb · 2025-12-16T14:11:25Z

Thank you @jizezhang @Dandandan

alamb-ghbot · 2025-12-16T19:09:21Z

🤖 ./gh_compare_branch.shgh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (3d70d93) to 199cf06diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2025-12-16T19:50:30Z

🤖: Benchmark completed

Details

Comparing HEAD and jizezhang_main
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ jizezhang_main ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0 │ 2752.93 ms │ 2833.55 ms │ no change │
│ QQuery 1 │ 1334.02 ms │ 1313.99 ms │ no change │
│ QQuery 2 │ 2503.21 ms │ 2515.26 ms │ no change │
│ QQuery 3 │ 1137.68 ms │ 1166.77 ms │ no change │
│ QQuery 4 │ 2333.81 ms │ 2321.91 ms │ no change │
│ QQuery 5 │ 29034.68 ms │ 29119.55 ms │ no change │
│ QQuery 6 │ 4165.77 ms │ 4185.77 ms │ no change │
│ QQuery 7 │ 3770.29 ms │ 3678.40 ms │ no change │
└──────────────┴─────────────┴────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 47032.39ms │
│ Total Time (jizezhang_main) │ 47135.20ms │
│ Average Time (HEAD) │ 5879.05ms │
│ Average Time (jizezhang_main) │ 5891.90ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 8 │
│ Queries with Failure │ 0 │
└───────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ jizezhang_main ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.46 ms │ 2.28 ms │ +1.08x faster │
│ QQuery 1 │ 50.46 ms │ 49.18 ms │ no change │
│ QQuery 2 │ 136.12 ms │ 141.33 ms │ no change │
│ QQuery 3 │ 153.27 ms │ 156.12 ms │ no change │
│ QQuery 4 │ 1151.18 ms │ 1091.50 ms │ +1.05x faster │
│ QQuery 5 │ 1590.18 ms │ 1522.21 ms │ no change │
│ QQuery 6 │ 2.31 ms │ 2.12 ms │ +1.09x faster │
│ QQuery 7 │ 57.88 ms │ 57.94 ms │ no change │
│ QQuery 8 │ 1556.44 ms │ 1482.48 ms │ no change │
│ QQuery 9 │ 1947.87 ms │ 1932.25 ms │ no change │
│ QQuery 10 │ 381.38 ms │ 394.58 ms │ no change │
│ QQuery 11 │ 433.58 ms │ 438.70 ms │ no change │
│ QQuery 12 │ 1418.75 ms │ 1390.06 ms │ no change │
│ QQuery 13 │ 2069.43 ms │ 2003.36 ms │ no change │
│ QQuery 14 │ 1332.84 ms │ 1292.79 ms │ no change │
│ QQuery 15 │ 1342.43 ms │ 1264.79 ms │ +1.06x faster │
│ QQuery 16 │ 2809.11 ms │ 2746.40 ms │ no change │
│ QQuery 17 │ 2761.72 ms │ 2781.23 ms │ no change │
│ QQuery 18 │ 5907.30 ms │ 5189.89 ms │ +1.14x faster │
│ QQuery 19 │ 125.46 ms │ 126.28 ms │ no change │
│ QQuery 20 │ 1972.57 ms │ 1886.73 ms │ no change │
│ QQuery 21 │ 2277.15 ms │ 2240.41 ms │ no change │
│ QQuery 22 │ 3916.07 ms │ 3869.49 ms │ no change │
│ QQuery 23 │ 17593.38 ms │ 12708.17 ms │ +1.38x faster │
│ QQuery 24 │ 232.17 ms │ 208.77 ms │ +1.11x faster │
│ QQuery 25 │ 483.36 ms │ 467.84 ms │ no change │
│ QQuery 26 │ 221.06 ms │ 225.65 ms │ no change │
│ QQuery 27 │ 2820.67 ms │ 2725.06 ms │ no change │
│ QQuery 28 │ 25232.60 ms │ 24332.50 ms │ no change │
│ QQuery 29 │ 983.71 ms │ 983.07 ms │ no change │
│ QQuery 30 │ 1391.12 ms │ 1345.42 ms │ no change │
│ QQuery 31 │ 1418.65 ms │ 1344.50 ms │ +1.06x faster │
│ QQuery 32 │ 5239.80 ms │ 4909.99 ms │ +1.07x faster │
│ QQuery 33 │ 6444.06 ms │ 5842.59 ms │ +1.10x faster │
│ QQuery 34 │ 6505.13 ms │ 6010.55 ms │ +1.08x faster │
│ QQuery 35 │ 2059.10 ms │ 1928.39 ms │ +1.07x faster │
│ QQuery 36 │ 69.64 ms │ 67.15 ms │ no change │
│ QQuery 37 │ 46.41 ms │ 45.93 ms │ no change │
│ QQuery 38 │ 68.21 ms │ 67.84 ms │ no change │
│ QQuery 39 │ 101.59 ms │ 104.09 ms │ no change │
│ QQuery 40 │ 26.71 ms │ 29.41 ms │ 1.10x slower │
│ QQuery 41 │ 24.16 ms │ 23.76 ms │ no change │
│ QQuery 42 │ 20.85 ms │ 23.18 ms │ 1.11x slower │
└──────────────┴─────────────┴────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 104378.34ms │
│ Total Time (jizezhang_main) │ 95456.00ms │
│ Average Time (HEAD) │ 2427.40ms │
│ Average Time (jizezhang_main) │ 2219.91ms │
│ Queries Faster │ 12 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 29 │
│ Queries with Failure │ 0 │
└───────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ jizezhang_main ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 133.92 ms │ 135.04 ms │ no change │
│ QQuery 2 │ 30.58 ms │ 30.82 ms │ no change │
│ QQuery 3 │ 41.51 ms │ 33.86 ms │ +1.23x faster │
│ QQuery 4 │ 30.71 ms │ 30.13 ms │ no change │
│ QQuery 5 │ 91.80 ms │ 90.41 ms │ no change │
│ QQuery 6 │ 19.98 ms │ 19.90 ms │ no change │
│ QQuery 7 │ 253.11 ms │ 250.10 ms │ no change │
│ QQuery 8 │ 39.48 ms │ 37.46 ms │ +1.05x faster │
│ QQuery 9 │ 114.49 ms │ 113.47 ms │ no change │
│ QQuery 10 │ 67.41 ms │ 63.91 ms │ +1.05x faster │
│ QQuery 11 │ 18.61 ms │ 18.11 ms │ no change │
│ QQuery 12 │ 54.01 ms │ 53.18 ms │ no change │
│ QQuery 13 │ 51.73 ms │ 52.95 ms │ no change │
│ QQuery 14 │ 14.59 ms │ 14.64 ms │ no change │
│ QQuery 15 │ 25.75 ms │ 24.91 ms │ no change │
│ QQuery 16 │ 25.84 ms │ 25.45 ms │ no change │
│ QQuery 17 │ 161.77 ms │ 158.14 ms │ no change │
│ QQuery 18 │ 302.07 ms │ 303.76 ms │ no change │
│ QQuery 19 │ 38.79 ms │ 39.23 ms │ no change │
│ QQuery 20 │ 52.62 ms │ 50.85 ms │ no change │
│ QQuery 21 │ 343.00 ms │ 313.00 ms │ +1.10x faster │
│ QQuery 22 │ 17.93 ms │ 17.45 ms │ no change │
└──────────────┴───────────┴────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1929.70ms │
│ Total Time (jizezhang_main) │ 1876.78ms │
│ Average Time (HEAD) │ 87.71ms │
│ Average Time (jizezhang_main) │ 85.31ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 18 │
│ Queries with Failure │ 0 │
└───────────────────────────────┴───────────┘

remove repartition exec from coalesce batches optimizer
3869a78

github-actionsBot added optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Dec 9, 2025

update tpch plans
446f6cf

jizezhang marked this pull request as ready for review December 10, 2025 06:24

jizezhang mentioned this pull request Dec 10, 2025
Integrate BatchCoalescer into RepartitionExec#18782
Closed

alamb mentioned this pull request Dec 11, 2025
Andrew Lamb Weekly-ish Open Source plan - 2025-12-08 #19210
Closed
40 tasks

alamb added 2 commits December 14, 2025 06:18

Merge remote-tracking branch 'apache/main' into jizezhang/main
6d92a19

Update slt
d685866

alamb requested a review from Copilot December 14, 2025 11:25

Copilotstarted reviewing on behalf of alamb December 14, 2025 11:26 View session

CopilotAI reviewed Dec 14, 2025
View reviewed changes

alamb approved these changes Dec 14, 2025
View reviewed changes

alamb mentioned this pull request Dec 15, 2025
Integrate BatchCoalescer into AsyncFuncExec and remove from CoalesceBatches optimization rule #19331
Closed

Dandandan approved these changes Dec 15, 2025
View reviewed changes

Merge remote-tracking branch 'apache/main' into jizezhang/main
f3f300e

Merge branch 'main' into main
3d70d93

alamb added this pull request to the merge queue Dec 16, 2025

Merged via the queue into apache:main with commit 89af694Dec 16, 2025
31 checks passed

alamb mentioned this pull request Dec 17, 2025
Hash RepartitionExec should buffer/emit batches based on target batch size #405
Closed

Uh oh!

Conversation

jizezhang commented Dec 9, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Dec 11, 2025

Uh oh!

alamb commented Dec 11, 2025

Uh oh!

alamb-ghbot commented Dec 11, 2025

Uh oh!

alamb-ghbot commented Dec 12, 2025

Uh oh!

alamb-ghbot commented Dec 12, 2025

Uh oh!

alamb-ghbot commented Dec 12, 2025

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alambDec 11, 2025

Choose a reason for hiding this comment

Uh oh!

alambDec 14, 2025

Choose a reason for hiding this comment

Uh oh!

Dandandan commented Dec 15, 2025

Uh oh!

alamb-ghbot commented Dec 15, 2025

Uh oh!

alamb-ghbot commented Dec 15, 2025

Uh oh!

alamb commented Dec 15, 2025

Uh oh!

alamb commented Dec 16, 2025

Uh oh!

alamb commented Dec 16, 2025

Uh oh!

alamb commented Dec 16, 2025

Uh oh!

Uh oh!

alamb-ghbot commented Dec 16, 2025

Uh oh!

alamb-ghbot commented Dec 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jizezhang commented Dec 9, 2025•
edited
Loading