Row group limit pruning for row groups that entirely match predicates by xudong963 · Pull Request #18868 · apache/datafusion

xudong963 · 2025-11-21T09:43:44Z

Which issue does this PR close?

Part of [EPIC] Support limit pruning #18860

Rationale for this change

See #18860 (comment)

What changes are included in this PR?

How to decide if we can do limit pruning without messing up the sql semantics.
Add logic to decide if a row group is fully matched, all rows in the row group are matched the predicated.
Use the fully matched row groups to return limit rows.

Are these changes tested?

Yes

Are there any user-facing changes?

No, no new configs, or API change

xudong963 · 2025-11-25T05:37:47Z

I also like to construct some good-fit cases to show some benchmark results.

alamb · 2025-11-25T22:37:10Z

I plan to review this PR carefully tomorrow

alamb

Thank you very much for this PR @xudong963

The idea is very cool, and a nicely done PR; I have several suggestions, but I think it is very close.

Suggestions:

Rename the flag to preserve_order
Make this new flag on TableScan visible in EXPLAIN plans so we have a better chance of understanding when it is being applied/not applied
Implement an end to end test for the behavior you want to see (specifically, a test that shows a query on a parquet file and demonstrates that only a single row group is fetched with this flag, and more than one is fetched without the flag)

In my mind, 3 is the most important

alamb · 2025-11-27T11:56:22Z

🤖 ./gh_compare_branch.shBenchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing row_group_limit_pruning (24ae747) to f1ecaccdiff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-11-27T12:54:32Z

🤖: Benchmark completed

Details

Comparing HEAD and row_group_limit_pruning
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row_group_limit_pruning ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2564.84 ms │ 2711.60 ms │ 1.06x slower │
│ QQuery 1 │ 1177.86 ms │ 1306.25 ms │ 1.11x slower │
│ QQuery 2 │ 2412.87 ms │ 2423.96 ms │ no change │
│ QQuery 3 │ 1156.85 ms │ 1145.44 ms │ no change │
│ QQuery 4 │ 2318.20 ms │ 2308.57 ms │ no change │
│ QQuery 5 │ 28372.67 ms │ 27912.28 ms │ no change │
│ QQuery 6 │ 4041.22 ms │ 4170.72 ms │ no change │
│ QQuery 7 │ 3812.31 ms │ 3670.54 ms │ no change │
└──────────────┴─────────────┴─────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 45856.83ms │
│ Total Time (row_group_limit_pruning) │ 45649.37ms │
│ Average Time (HEAD) │ 5732.10ms │
│ Average Time (row_group_limit_pruning) │ 5706.17ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 6 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row_group_limit_pruning ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2.11 ms │ 2.20 ms │ no change │
│ QQuery 1 │ 49.14 ms │ 49.42 ms │ no change │
│ QQuery 2 │ 134.43 ms │ 137.18 ms │ no change │
│ QQuery 3 │ 165.90 ms │ 166.30 ms │ no change │
│ QQuery 4 │ 1082.62 ms │ 1093.44 ms │ no change │
│ QQuery 5 │ 1509.20 ms │ 1523.29 ms │ no change │
│ QQuery 6 │ 2.23 ms │ 2.12 ms │ no change │
│ QQuery 7 │ 53.46 ms │ 53.16 ms │ no change │
│ QQuery 8 │ 1464.07 ms │ 1443.88 ms │ no change │
│ QQuery 9 │ 1920.32 ms │ 1861.46 ms │ no change │
│ QQuery 10 │ 368.92 ms │ 369.59 ms │ no change │
│ QQuery 11 │ 425.35 ms │ 415.20 ms │ no change │
│ QQuery 12 │ 1360.27 ms │ 1366.53 ms │ no change │
│ QQuery 13 │ 2088.62 ms │ 2099.38 ms │ no change │
│ QQuery 14 │ 1291.80 ms │ 1276.05 ms │ no change │
│ QQuery 15 │ 1287.95 ms │ 1249.38 ms │ no change │
│ QQuery 16 │ 2687.28 ms │ 2705.34 ms │ no change │
│ QQuery 17 │ 2666.35 ms │ 2685.82 ms │ no change │
│ QQuery 18 │ 5334.10 ms │ 5051.38 ms │ +1.06x faster │
│ QQuery 19 │ 128.58 ms │ 129.60 ms │ no change │
│ QQuery 20 │ 2025.19 ms │ 1974.90 ms │ no change │
│ QQuery 21 │ 2326.74 ms │ 2256.83 ms │ no change │
│ QQuery 22 │ 3910.56 ms │ 3883.88 ms │ no change │
│ QQuery 23 │ 16847.81 ms │ 12664.80 ms │ +1.33x faster │
│ QQuery 24 │ 218.91 ms │ 218.97 ms │ no change │
│ QQuery 25 │ 470.06 ms │ 470.62 ms │ no change │
│ QQuery 26 │ 211.18 ms │ 219.75 ms │ no change │
│ QQuery 27 │ 2828.80 ms │ 2821.30 ms │ no change │
│ QQuery 28 │ 23333.65 ms │ 24002.05 ms │ no change │
│ QQuery 29 │ 1010.28 ms │ 990.00 ms │ no change │
│ QQuery 30 │ 1361.32 ms │ 1340.64 ms │ no change │
│ QQuery 31 │ 1401.38 ms │ 1386.63 ms │ no change │
│ QQuery 32 │ 4833.23 ms │ 4590.26 ms │ +1.05x faster │
│ QQuery 33 │ 5922.93 ms │ 5861.58 ms │ no change │
│ QQuery 34 │ 6218.31 ms │ 6093.69 ms │ no change │
│ QQuery 35 │ 1945.21 ms │ 1892.18 ms │ no change │
│ QQuery 36 │ 118.71 ms │ 122.83 ms │ no change │
│ QQuery 37 │ 52.92 ms │ 53.12 ms │ no change │
│ QQuery 38 │ 119.74 ms │ 120.43 ms │ no change │
│ QQuery 39 │ 195.76 ms │ 198.67 ms │ no change │
│ QQuery 40 │ 40.90 ms │ 45.00 ms │ 1.10x slower │
│ QQuery 41 │ 39.19 ms │ 40.69 ms │ no change │
│ QQuery 42 │ 31.96 ms │ 33.45 ms │ no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 99487.42ms │
│ Total Time (row_group_limit_pruning) │ 94963.00ms │
│ Average Time (HEAD) │ 2313.66ms │
│ Average Time (row_group_limit_pruning) │ 2208.44ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 39 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row_group_limit_pruning ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 130.90 ms │ 136.37 ms │ no change │
│ QQuery 2 │ 28.85 ms │ 29.40 ms │ no change │
│ QQuery 3 │ 35.92 ms │ 38.73 ms │ 1.08x slower │
│ QQuery 4 │ 29.91 ms │ 29.59 ms │ no change │
│ QQuery 5 │ 87.85 ms │ 88.68 ms │ no change │
│ QQuery 6 │ 19.75 ms │ 19.33 ms │ no change │
│ QQuery 7 │ 224.01 ms │ 217.44 ms │ no change │
│ QQuery 8 │ 33.21 ms │ 32.74 ms │ no change │
│ QQuery 9 │ 103.44 ms │ 100.86 ms │ no change │
│ QQuery 10 │ 64.34 ms │ 64.25 ms │ no change │
│ QQuery 11 │ 17.90 ms │ 19.12 ms │ 1.07x slower │
│ QQuery 12 │ 52.62 ms │ 53.57 ms │ no change │
│ QQuery 13 │ 47.65 ms │ 47.74 ms │ no change │
│ QQuery 14 │ 13.71 ms │ 13.79 ms │ no change │
│ QQuery 15 │ 24.87 ms │ 24.60 ms │ no change │
│ QQuery 16 │ 25.17 ms │ 25.16 ms │ no change │
│ QQuery 17 │ 149.35 ms │ 151.78 ms │ no change │
│ QQuery 18 │ 278.08 ms │ 276.50 ms │ no change │
│ QQuery 19 │ 37.01 ms │ 38.28 ms │ no change │
│ QQuery 20 │ 49.78 ms │ 48.67 ms │ no change │
│ QQuery 21 │ 317.59 ms │ 310.12 ms │ no change │
│ QQuery 22 │ 18.42 ms │ 18.49 ms │ no change │
└──────────────┴───────────┴─────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1790.33ms │
│ Total Time (row_group_limit_pruning) │ 1785.22ms │
│ Average Time (HEAD) │ 81.38ms │
│ Average Time (row_group_limit_pruning) │ 81.15ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 20 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴───────────┘

xudong963 · 2025-11-28T02:06:36Z

Thank you very much for this PR @xudong963
The idea is very cool, and a nicely done PR; I have several suggestions, but I think it is very close.
Suggestions:
Rename the flag to preserve_order
Make this new flag on TableScan visible in EXPLAIN plans so we have a better chance of understanding when it is being applied/not applied
Implement an end to end test for the behavior you want to see (specifically, a test that shows a query on a parquet file and demonstrates that only a single row group is fetched with this flag, and more than one is fetched without the flag)
In my mind, 3 is the most important

Thanks for the review @alamb . I addressed some suggestions:

Renamed the flag to preseerve_orderecbfff7
Avoided the second recursive call in pushdown limit and maintained a state to decide if set preserve_order as true. e919879
Extracted the logic that finds fully matched rg as a function: 647618a
Added the end-to-end sqllogictests for limit pruning 63cd878
Extracted the not simplifier as a new PR: Support simplify not for physical expr #18970

Things later the PR:

Make a TableScanBuilder or something (as a follow on PR) and deprecate TableScan::try_new
Add the preserve_order to explain. The reason I plan to do it in another PR is that I guess it'll make lots of changes for tests, which will expand the PR.

alamb · 2025-12-01T17:45:52Z

I was mentioning this PR to @crepererum and he also noted there is another interesting potential optimization when we know the predicate is true for the entire row group: we can skip evaluating the predicate for the row group entirely (as the filter can often be quite expensive itself)

@xudong963 is this something else you are contemplating? This PR likely lays the foundation for such an optimization. If you think it is worthwhile I'll file a ticket

alamb · 2025-12-01T17:46:29Z

Extracted the not simplifier as a new PR: #18970

I just reviewed this -- it looks nice 👍

xudong963 · 2026-01-16T01:28:43Z

I plan to merge the PR today if there are no more comments.

Thanks for the review @alamb @adriangb!

adriangb · 2026-01-16T14:12:37Z

Should we also have an issue to track using this idea / machinery to avoid running filters that we know match all rows for a file during filter pushdown?

xudong963 · 2026-01-16T14:35:19Z

Should we also have an issue to track using this idea / machinery to avoid running filters that we know match all rows for a file during filter pushdown?

Yes, this one #19028

alamb · 2026-01-16T21:17:04Z

This is super exciting work -- thank you so much @xudong963 and @adriangb

Interestingly I was just speaking about this feature with @NGA-TRAN and @gene-bordegaray (who I ran into at https://nedbday.github.io/2026/). Very cool

alamb · 2026-01-16T21:17:34Z

run benchmarks

alamb-ghbot · 2026-01-16T21:17:51Z

🤖 ./gh_compare_branch.shgh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing row_group_limit_pruning (fd6e7c8) to 7b0ed2ddiff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-16T21:57:47Z

🤖: Benchmark completed

Details

Comparing HEAD and row_group_limit_pruning
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row_group_limit_pruning ┃ Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2475.29 ms │ 2360.49 ms │ no change │
│ QQuery 1 │ 948.89 ms │ 932.32 ms │ no change │
│ QQuery 2 │ 1821.81 ms │ 1924.93 ms │ 1.06x slower │
│ QQuery 3 │ 1143.63 ms │ 1085.76 ms │ +1.05x faster │
│ QQuery 4 │ 2267.23 ms │ 2207.05 ms │ no change │
│ QQuery 5 │ 28296.37 ms │ 28352.43 ms │ no change │
│ QQuery 6 │ 4057.65 ms │ 3914.06 ms │ no change │
│ QQuery 7 │ 3080.12 ms │ 2998.42 ms │ no change │
└──────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 44091.00ms │
│ Total Time (row_group_limit_pruning) │ 43775.46ms │
│ Average Time (HEAD) │ 5511.38ms │
│ Average Time (row_group_limit_pruning) │ 5471.93ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 6 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row_group_limit_pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.92 ms │ 2.08 ms │ 1.08x slower │
│ QQuery 1 │ 48.35 ms │ 51.21 ms │ 1.06x slower │
│ QQuery 2 │ 133.42 ms │ 132.67 ms │ no change │
│ QQuery 3 │ 155.75 ms │ 155.52 ms │ no change │
│ QQuery 4 │ 1032.39 ms │ 1151.10 ms │ 1.11x slower │
│ QQuery 5 │ 1354.01 ms │ 1423.64 ms │ 1.05x slower │
│ QQuery 6 │ 1.87 ms │ 1.87 ms │ no change │
│ QQuery 7 │ 55.71 ms │ 56.27 ms │ no change │
│ QQuery 8 │ 1425.78 ms │ 1499.70 ms │ 1.05x slower │
│ QQuery 9 │ 1785.56 ms │ 1865.44 ms │ no change │
│ QQuery 10 │ 344.35 ms │ 353.01 ms │ no change │
│ QQuery 11 │ 399.04 ms │ 402.44 ms │ no change │
│ QQuery 12 │ 1264.11 ms │ 1345.76 ms │ 1.06x slower │
│ QQuery 13 │ 1962.17 ms │ 1976.80 ms │ no change │
│ QQuery 14 │ 1239.72 ms │ 1298.12 ms │ no change │
│ QQuery 15 │ 1238.12 ms │ 1336.95 ms │ 1.08x slower │
│ QQuery 16 │ 2625.57 ms │ 2619.81 ms │ no change │
│ QQuery 17 │ 2541.94 ms │ 2601.39 ms │ no change │
│ QQuery 18 │ 5560.19 ms │ 4867.23 ms │ +1.14x faster │
│ QQuery 19 │ 123.69 ms │ 122.27 ms │ no change │
│ QQuery 20 │ 1950.22 ms │ 1907.30 ms │ no change │
│ QQuery 21 │ 2214.38 ms │ 2213.92 ms │ no change │
│ QQuery 22 │ 3805.63 ms │ 3830.18 ms │ no change │
│ QQuery 23 │ 16609.81 ms │ 12241.74 ms │ +1.36x faster │
│ QQuery 24 │ 214.49 ms │ 213.07 ms │ no change │
│ QQuery 25 │ 479.21 ms │ 472.03 ms │ no change │
│ QQuery 26 │ 220.73 ms │ 233.49 ms │ 1.06x slower │
│ QQuery 27 │ 2745.83 ms │ 2714.54 ms │ no change │
│ QQuery 28 │ 23408.34 ms │ 23501.94 ms │ no change │
│ QQuery 29 │ 967.65 ms │ 947.36 ms │ no change │
│ QQuery 30 │ 1282.57 ms │ 1312.39 ms │ no change │
│ QQuery 31 │ 1319.43 ms │ 1322.87 ms │ no change │
│ QQuery 32 │ 4461.63 ms │ 4063.49 ms │ +1.10x faster │
│ QQuery 33 │ 5612.08 ms │ 5449.24 ms │ no change │
│ QQuery 34 │ 5888.10 ms │ 5852.38 ms │ no change │
│ QQuery 35 │ 1918.26 ms │ 2005.16 ms │ no change │
│ QQuery 36 │ 71.51 ms │ 69.48 ms │ no change │
│ QQuery 37 │ 45.91 ms │ 47.99 ms │ no change │
│ QQuery 38 │ 66.16 ms │ 66.89 ms │ no change │
│ QQuery 39 │ 101.54 ms │ 105.81 ms │ no change │
│ QQuery 40 │ 28.33 ms │ 29.19 ms │ no change │
│ QQuery 41 │ 23.45 ms │ 25.40 ms │ 1.08x slower │
│ QQuery 42 │ 20.36 ms │ 20.53 ms │ no change │
└───────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 96749.32ms │
│ Total Time (row_group_limit_pruning) │ 91909.66ms │
│ Average Time (HEAD) │ 2249.98ms │
│ Average Time (row_group_limit_pruning) │ 2137.43ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 9 │
│ Queries with No Change │ 31 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row_group_limit_pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1 │ 103.92 ms │ 101.44 ms │ no change │
│ QQuery 2 │ 34.24 ms │ 34.60 ms │ no change │
│ QQuery 3 │ 35.48 ms │ 40.77 ms │ 1.15x slower │
│ QQuery 4 │ 30.57 ms │ 31.75 ms │ no change │
│ QQuery 5 │ 91.96 ms │ 94.16 ms │ no change │
│ QQuery 6 │ 20.82 ms │ 21.12 ms │ no change │
│ QQuery 7 │ 164.94 ms │ 158.31 ms │ no change │
│ QQuery 8 │ 39.30 ms │ 42.88 ms │ 1.09x slower │
│ QQuery 9 │ 104.78 ms │ 109.03 ms │ no change │
│ QQuery 10 │ 68.56 ms │ 72.47 ms │ 1.06x slower │
│ QQuery 11 │ 19.45 ms │ 19.03 ms │ no change │
│ QQuery 12 │ 52.08 ms │ 52.78 ms │ no change │
│ QQuery 13 │ 49.28 ms │ 50.75 ms │ no change │
│ QQuery 14 │ 15.90 ms │ 15.57 ms │ no change │
│ QQuery 15 │ 31.05 ms │ 30.51 ms │ no change │
│ QQuery 16 │ 29.64 ms │ 28.78 ms │ no change │
│ QQuery 17 │ 145.50 ms │ 149.75 ms │ no change │
│ QQuery 18 │ 287.48 ms │ 286.50 ms │ no change │
│ QQuery 19 │ 41.67 ms │ 40.20 ms │ no change │
│ QQuery 20 │ 57.11 ms │ 55.88 ms │ no change │
│ QQuery 21 │ 187.56 ms │ 192.69 ms │ no change │
│ QQuery 22 │ 22.75 ms │ 22.99 ms │ no change │
└───────────┴───────────┴─────────────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1634.02ms │
│ Total Time (row_group_limit_pruning) │ 1651.95ms │
│ Average Time (HEAD) │ 74.27ms │
│ Average Time (row_group_limit_pruning) │ 75.09ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 3 │
│ Queries with No Change │ 19 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴───────────┘

alamb · 2026-01-16T22:29:07Z

run benchmarks

alamb-ghbot · 2026-01-16T22:29:22Z

🤖 ./gh_compare_branch.shgh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing row_group_limit_pruning (fd6e7c8) to 7b0ed2ddiff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb-ghbot · 2026-01-16T22:54:01Z

🤖: Benchmark completed

Details

Comparing HEAD and row_group_limit_pruning
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row_group_limit_pruning ┃ Change ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 2397.88 ms │ 2286.49 ms │ no change │
│ QQuery 1 │ 918.99 ms │ 947.76 ms │ no change │
│ QQuery 2 │ 1849.34 ms │ 1950.88 ms │ 1.05x slower │
│ QQuery 3 │ 1139.98 ms │ 1081.48 ms │ +1.05x faster │
│ QQuery 4 │ 2216.64 ms │ 2208.60 ms │ no change │
│ QQuery 5 │ 28423.59 ms │ 28344.83 ms │ no change │
│ QQuery 6 │ 4043.86 ms │ 3881.62 ms │ no change │
│ QQuery 7 │ 2669.09 ms │ 2701.79 ms │ no change │
└──────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 43659.35ms │
│ Total Time (row_group_limit_pruning) │ 43403.45ms │
│ Average Time (HEAD) │ 5457.42ms │
│ Average Time (row_group_limit_pruning) │ 5425.43ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 6 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row_group_limit_pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.95 ms │ 2.18 ms │ 1.12x slower │
│ QQuery 1 │ 51.06 ms │ 51.70 ms │ no change │
│ QQuery 2 │ 133.87 ms │ 135.89 ms │ no change │
│ QQuery 3 │ 157.12 ms │ 153.69 ms │ no change │
│ QQuery 4 │ 1105.21 ms │ 1065.12 ms │ no change │
│ QQuery 5 │ 1365.77 ms │ 1385.32 ms │ no change │
│ QQuery 6 │ 1.86 ms │ 1.94 ms │ no change │
│ QQuery 7 │ 54.07 ms │ 58.69 ms │ 1.09x slower │
│ QQuery 8 │ 1432.00 ms │ 1405.40 ms │ no change │
│ QQuery 9 │ 1809.05 ms │ 1788.70 ms │ no change │
│ QQuery 10 │ 351.19 ms │ 361.45 ms │ no change │
│ QQuery 11 │ 400.64 ms │ 434.68 ms │ 1.08x slower │
│ QQuery 12 │ 1263.94 ms │ 1260.64 ms │ no change │
│ QQuery 13 │ 1953.43 ms │ 1968.05 ms │ no change │
│ QQuery 14 │ 1255.54 ms │ 1237.41 ms │ no change │
│ QQuery 15 │ 1233.38 ms │ 1201.28 ms │ no change │
│ QQuery 16 │ 2613.36 ms │ 2473.27 ms │ +1.06x faster │
│ QQuery 17 │ 2515.40 ms │ 2489.38 ms │ no change │
│ QQuery 18 │ 5426.26 ms │ 4761.28 ms │ +1.14x faster │
│ QQuery 19 │ 121.28 ms │ 122.62 ms │ no change │
│ QQuery 20 │ 1919.05 ms │ 1930.49 ms │ no change │
│ QQuery 21 │ 2182.90 ms │ 2209.94 ms │ no change │
│ QQuery 22 │ 3805.87 ms │ 3795.62 ms │ no change │
│ QQuery 23 │ 15725.52 ms │ 12202.85 ms │ +1.29x faster │
│ QQuery 24 │ 219.75 ms │ 206.09 ms │ +1.07x faster │
│ QQuery 25 │ 472.27 ms │ 469.00 ms │ no change │
│ QQuery 26 │ 215.34 ms │ 232.55 ms │ 1.08x slower │
│ QQuery 27 │ 2675.63 ms │ 2684.74 ms │ no change │
│ QQuery 28 │ 23283.38 ms │ 23563.82 ms │ no change │
│ QQuery 29 │ 967.07 ms │ 963.93 ms │ no change │
│ QQuery 30 │ 1258.63 ms │ 1282.73 ms │ no change │
│ QQuery 31 │ 1328.71 ms │ 1308.06 ms │ no change │
│ QQuery 32 │ 4153.72 ms │ 4296.85 ms │ no change │
│ QQuery 33 │ 5674.39 ms │ 5372.83 ms │ +1.06x faster │
│ QQuery 34 │ 5758.60 ms │ 5695.52 ms │ no change │
│ QQuery 35 │ 1906.86 ms │ 1864.12 ms │ no change │
│ QQuery 36 │ 66.31 ms │ 69.28 ms │ no change │
│ QQuery 37 │ 46.80 ms │ 47.16 ms │ no change │
│ QQuery 38 │ 66.98 ms │ 67.05 ms │ no change │
│ QQuery 39 │ 101.17 ms │ 104.37 ms │ no change │
│ QQuery 40 │ 27.51 ms │ 26.25 ms │ no change │
│ QQuery 41 │ 23.91 ms │ 23.92 ms │ no change │
│ QQuery 42 │ 19.86 ms │ 21.37 ms │ 1.08x slower │
└───────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 95146.60ms │
│ Total Time (row_group_limit_pruning) │ 90797.24ms │
│ Average Time (HEAD) │ 2212.71ms │
│ Average Time (row_group_limit_pruning) │ 2111.56ms │
│ Queries Faster │ 5 │
│ Queries Slower │ 5 │
│ Queries with No Change │ 33 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ row_group_limit_pruning ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 108.17 ms │ 101.37 ms │ +1.07x faster │
│ QQuery 2 │ 33.11 ms │ 33.72 ms │ no change │
│ QQuery 3 │ 41.20 ms │ 36.68 ms │ +1.12x faster │
│ QQuery 4 │ 30.96 ms │ 31.48 ms │ no change │
│ QQuery 5 │ 95.06 ms │ 94.54 ms │ no change │
│ QQuery 6 │ 21.19 ms │ 21.34 ms │ no change │
│ QQuery 7 │ 161.13 ms │ 165.88 ms │ no change │
│ QQuery 8 │ 41.79 ms │ 43.90 ms │ 1.05x slower │
│ QQuery 9 │ 107.83 ms │ 109.50 ms │ no change │
│ QQuery 10 │ 71.53 ms │ 73.04 ms │ no change │
│ QQuery 11 │ 18.88 ms │ 19.33 ms │ no change │
│ QQuery 12 │ 53.78 ms │ 53.00 ms │ no change │
│ QQuery 13 │ 52.15 ms │ 49.89 ms │ no change │
│ QQuery 14 │ 15.88 ms │ 15.96 ms │ no change │
│ QQuery 15 │ 31.11 ms │ 30.87 ms │ no change │
│ QQuery 16 │ 29.44 ms │ 30.40 ms │ no change │
│ QQuery 17 │ 142.50 ms │ 150.77 ms │ 1.06x slower │
│ QQuery 18 │ 291.64 ms │ 288.67 ms │ no change │
│ QQuery 19 │ 41.61 ms │ 39.91 ms │ no change │
│ QQuery 20 │ 56.25 ms │ 57.32 ms │ no change │
│ QQuery 21 │ 194.42 ms │ 199.17 ms │ no change │
│ QQuery 22 │ 22.64 ms │ 22.61 ms │ no change │
└───────────┴───────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1662.27ms │
│ Total Time (row_group_limit_pruning) │ 1669.36ms │
│ Average Time (HEAD) │ 75.56ms │
│ Average Time (row_group_limit_pruning) │ 75.88ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 18 │
│ Queries with Failure │ 0 │
└────────────────────────────────────────┴───────────┘

NGA-TRAN · 2026-01-18T12:59:43Z

Thanks @alamb for letting us know this great feature.
@xudong963 Your comment — #18860 (comment) — is especially helpful . It’s always exciting to see optimizations like this; they’re part of what makes working in query optimization so much fun.

…apache#18868)  - Part of apache#18860  See apache#18860 (comment)  1. How to decide if we can do limit pruning without messing up the sql semantics. 2. Add logic to decide if a row group is fully matched, all rows in the row group are matched the predicated. 3. Use the fully matched row groups to return limit rows.  Yes   No, no new configs, or API change

…apache#18868) ## Which issue does this PR close?  - Part of apache#18860 ## Rationale for this change  See apache#18860 (comment) ## What changes are included in this PR?  1. How to decide if we can do limit pruning without messing up the sql semantics. 2. Add logic to decide if a row group is fully matched, all rows in the row group are matched the predicated. 3. Use the fully matched row groups to return limit rows. ## Are these changes tested?  Yes ## Are there any user-facing changes?   No, no new configs, or API change

xudong963 marked this pull request as draft November 21, 2025 09:43

github-actionsBot added physical-expr Changes to the physical-expr crates core Core DataFusion crate execution Related to the execution crate datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Nov 21, 2025

xudong963 commented Nov 21, 2025
View reviewed changes

Comment threaddatafusion/datasource-parquet/src/opener.rs Outdated

xudong963 commented Nov 21, 2025
View reviewed changes

Comment threaddatafusion/datasource-parquet/src/row_group_filter.rs Outdated

github-actionsBot added logical-expr Logical plan and expressions optimizer Optimizer rules catalog Related to the catalog crate proto Related to proto crate labels Nov 25, 2025

xudong963 force-pushed the row_group_limit_pruning branch from 52f012f to d075c86Compare November 25, 2025 05:26

xudong963 marked this pull request as ready for review November 25, 2025 05:31

xudong963 commented Nov 25, 2025
View reviewed changes

Comment threaddatafusion/core/tests/parquet/row_group_pruning.rs
Comment threaddatafusion/core/tests/parquet/mod.rs
Comment threaddatafusion/datasource-parquet/src/opener.rs Outdated
Comment threaddatafusion/datasource-parquet/src/row_group_filter.rs

xudong963 force-pushed the row_group_limit_pruning branch from aaf76c0 to cb5711dCompare November 25, 2025 06:32

alamb mentioned this pull request Nov 25, 2025
Andrew Lamb Weekly-ish Open Source plan - 2025-11-24 #18888
Closed
37 tasks

alamb reviewed Nov 26, 2025
View reviewed changes

xudong963 force-pushed the row_group_limit_pruning branch from fdef903 to 7e1fc4fCompare November 27, 2025 13:31

xudong963 mentioned this pull request Nov 27, 2025
Support simplify not for physical expr #18970
Merged

github-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Nov 28, 2025

xudong963 force-pushed the row_group_limit_pruning branch from 90295f9 to 63cd878Compare November 28, 2025 01:58

alamb mentioned this pull request Dec 1, 2025
Andrew Lamb Weekly-ish Open Source plan - 2025-12-01 #19016
Closed
42 tasks

xudong963 force-pushed the row_group_limit_pruning branch from 2e82b97 to ca7de4fCompare January 13, 2026 08:01

fix doc test
d2b84d4

adriangb approved these changes Jan 15, 2026
View reviewed changes

update doc
6c515b2

xudong963 force-pushed the row_group_limit_pruning branch from 70375e5 to 6c515b2Compare January 15, 2026 06:33

Merge branch 'main' into row_group_limit_pruning
d80a354

Merge branch 'main' into row_group_limit_pruning
fd6e7c8

xudong963 mentioned this pull request Jan 16, 2026
Find a way to reduce the cost of making new pruning predicate during limit pruning #19847
Open

xudong963 added this pull request to the merge queue Jan 16, 2026

Merged via the queue into apache:main with commit ca904b3Jan 16, 2026
33 checks passed

apache deleted a comment from alamb-ghbot Jan 16, 2026

This was referenced Jan 27, 2026
Cherry pick limit pruning from upstream massive-com/arrow-datafusion#29
Merged
refactor: restrict ParquetOpener field visibility to pub(crate)#20060
Closed

alamb mentioned this pull request Mar 20, 2026
Release DataFusion 54.0.0 (Apr 2026 / May 2026) #21080
Closed
55 tasks

hhhizzz mentioned this pull request May 4, 2026
Project: make Parquet predicate pushdown safe to enable by default in query engines hhhizzz/arrow-rs#11
Closed
6 tasks

Uh oh!

Conversation

xudong963 commented Nov 21, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xudong963 commented Nov 25, 2025

Uh oh!

alamb commented Nov 25, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alamb commented Nov 27, 2025

Uh oh!

alamb commented Nov 27, 2025

Uh oh!

xudong963 commented Nov 28, 2025

Uh oh!

alamb commented Dec 1, 2025

Uh oh!

alamb commented Dec 1, 2025

Uh oh!

xudong963 commented Jan 16, 2026

Uh oh!

Uh oh!

adriangb commented Jan 16, 2026

Uh oh!

xudong963 commented Jan 16, 2026

Uh oh!

alamb commented Jan 16, 2026

Uh oh!

alamb commented Jan 16, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

alamb commented Jan 16, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

alamb-ghbot commented Jan 16, 2026

Uh oh!

NGA-TRAN commented Jan 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

xudong963 commented Nov 21, 2025•
edited
Loading

alamb commented Jan 16, 2026•
edited
Loading

alamb commented Jan 16, 2026•
edited
Loading