Uh oh!
There was an error while loading. Please reload this page.
perf(parquet): skip RowFilter on statically fully-matched row groups - #23696
Open
zhuqi-lucas wants to merge 13 commits into
Open
perf(parquet): skip RowFilter on statically fully-matched row groups#23696zhuqi-lucas wants to merge 13 commits into
zhuqi-lucas wants to merge 13 commits into
Conversation
zhuqi-lucas
marked this pull request as draft
July 19, 2026 03:28
zhuqi-lucas
commented
Jul 19, 2026
ContributorAuthor
run benchmarks |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpcds File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpch File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Parquet scan performance by avoiding per-row RowFilter evaluation for row groups proven (via statistics/static pruning) to be fully matched, and by enabling cheap per-row-group rebuilds of the row filter using prebuilt candidates. This builds on arrow-rs’s peek_next_row_group() to keep DataFusion’s row-group plan aligned with the decoder’s actual frontier.
Changes:
- Track per-row-group
fully_matchedstate in the Parquet access plan and preserve it through reorder/reverse operations. - Toggle the decoder’s
RowFilterat row-group boundaries (empty filter for fully-matched RGs; real filter otherwise) and add a new metric for observability. - Refactor row-filter construction to prebuild filter candidates once per file and rebuild cheaply per row group; update integration + SQL logic tests accordingly.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| datafusion/datasource-parquet/src/access_plan.rs | Carries per-RG fully_matched flags into the prepared plan and strips empty-selection RGs to keep plan/decoder alignment. |
| datafusion/datasource-parquet/src/metrics.rs | Adds row_filter_skipped_fully_matched metric to Parquet file metrics. |
| datafusion/datasource-parquet/src/push_decoder.rs | Adds per-RG plan entries and implements boundary-time RowFilter toggling using peek_next_row_group(). |
| datafusion/datasource-parquet/src/row_filter.rs | Splits row-filter building into “prebuild once per file” + “instantiate per RG” and expands nested/struct field planning logic. |
| datafusion/datasource-parquet/src/opener/mod.rs | Wires the prebuilt row-filter context into decoder construction and propagates new metric/state into the stream. |
| datafusion/datasource-parquet/src/sort.rs | Updates prepared-plan reverse test to reflect stripping of all-empty selections. |
| datafusion/core/tests/parquet/dynamic_row_group_pruning.rs | Adds an end-to-end integration test asserting fully-matched RGs trigger row-filter skipping behavior. |
| datafusion/sqllogictest/test_files/push_down_filter_parquet.slt | Updates EXPLAIN ANALYZE expectations to include the new metric. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+64
to
+68
| /// Number of row groups for which the per-row | ||
| /// [`RowFilter`](parquet::arrow::arrow_reader::RowFilter) was skipped | ||
| /// because the static stats proved every row of the RG satisfies the | ||
| /// predicate. The decoder is rebuilt at the boundary with an empty | ||
| /// row filter so the upcoming RG decodes without per-row evaluation. |
ContributorAuthor
There was a problem hiding this comment.
Fixed in fea0264 — the doc now states explicitly that this counts suppression events, not row groups: a run of consecutive fully-matched RGs shares a single toggle.
| && (!DataType::is_nested(return_type) | ||
| || self.is_nested_type_supported(return_type)) | ||
| { | ||
| // try to resolve all field name arguments to strinrg literals |
Comment on lines
+192
to
+196
| /// The result of resolving which Parquet leaf columns and Arrow schema fields | ||
| /// are needed to evaluate an expression against a Parquet file | ||
| /// | ||
| /// This is the shared output of the column resolution pipeline used by both | ||
| /// the row filter to build `ArrowPredicate`s and the opener to build `ProjectionMask`s |
ContributorAuthor
There was a problem hiding this comment.
Fixed in fea0264 — row_filter.rs now reuses the shared ParquetReadPlan / PushdownChecker machinery from projection_read_plan.rs instead of carrying its own copy (~700 lines removed).
| .map(|p| replace_columns_with_literals(p, &literal_columns)) | ||
| .transpose()?; | ||
| } | ||
ContributorAuthor
There was a problem hiding this comment.
Good catch — this was real: the rebase had silently dropped the input_file_name() rewrite. Restored in fea0264 along with a regression test (test_input_file_name_projection).
zhuqi-lucas
commented
Jul 19, 2026
ContributorAuthor
run benchmarks env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true |
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpch File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: tpcds File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
zhuqi-lucas
commented
Jul 19, 2026
ContributorAuthor
run benchmark sort_tpch env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (a3047ed) to 67947b6 (merge-base) diff using: sort_tpch File an issue against this benchmark runner |
adriangbot
commented
Jul 19, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagesort_tpch — base (merge-base)
sort_tpch — branch
File an issue against this benchmark runner |
zhuqi-lucasforce-pushed
the
perf/parquet-skip-filter-for-full-match
branch
from
July 31, 2026 14:13
a3047ed to
fea0264CompareBuilds on the `peek_next_row_group` API landed in apache/arrow-rs#10158. When Parquet stats prove that every row of a row group already satisfies the pushdown predicate (`fully_matched`), running the per-row `RowFilter` inside that row group is pure overhead — every row passes anyway. This change installs an *empty* `RowFilter` on fully-matched runs and only pays the row-level machinery on RGs that still need filtering. * `access_plan.rs` — `ParquetAccessPlan` now tracks per-RG `fully_matched` state, produced during static pruning. * `metrics.rs` — new `row_filter_skipped_fully_matched` counter to observe how often the toggle fires. * `push_decoder.rs` — `RgPlanEntry` carries the per-RG toggle; `RowFilterContext` swaps between the real filter (from the `prebuild_row_filter_candidates` cache) and an empty filter as the decoder crosses row-group boundaries. * `row_filter.rs` — split `build_row_filter` into `prebuild_*` and `row_filter_from_prebuilt`: the expensive tree walk + candidate construction runs once per file, and the cheap per-RG bind runs at each toggle. Preserves the existing public `build_row_filter` API for non-toggling callers. * `opener/mod.rs` — wires the prebuilt cache into the decoder builder; skips page-index loading for fully-matched RGs (their per-row filter is a no-op, so page pruning saves nothing). * `sort.rs` — sort-order-aware RG reorder preserves the per-RG toggle through reordering. * `fully_matched_rgs_skip_row_filter` (`dynamic_row_group_pruning.rs`) — 4 RGs of 3 rows each, predicate `v >= 3` makes RGs 1..=3 fully matched. Asserts: (a) results are correct, (b) `row_filter_skipped_fully_matched` fires at least once at the RG 0 → RG 1 boundary. * All 6 pre-existing `dynamic_row_group_pruning` tests still pass. * Full `datasource-parquet` lib suite (158 tests) passes.
…t_file_name handling lost in rebase Three fixes on top of the rebase onto latest main: 1. row_filter.rs carried a stale pre-apache#23638 copy of the column-resolution machinery (ParquetReadPlan, PushdownChecker, StructFieldAccess, build_filter_schema, etc.) that main has since moved into projection_read_plan.rs. Rebuilt the file on main's shared-module version, keeping only this PR's real additions (PrebuiltRowFilterCandidate, prebuild_row_filter_candidates, row_filter_from_prebuilt, and the RowFilterGenerator removal): 2194 → 1483 lines, -798 net. The stray "strinrg" typo lived in the deleted copy. 2. The rebase silently dropped main's input_file_name() handling in ParquetMorselizer (import, the per-file literal rewrite in open(), and the test_input_file_name_projection test) because this PR's restructure of open() overlapped the region apache#23638 later touched. Restored all three; the test passes again. 3. metrics: the row_filter_skipped_fully_matched doc claimed to count row groups, but the counter increments once per suppression event (a run of consecutive fully-matched RGs shares a single toggle). Doc now states the actual semantics. Verified: 158 datasource-parquet tests pass (with PARQUET_TEST_DATA), cargo doc clean, clippy clean on touched files.
…redAccessPlan::new, regenerate slt for new metric, fix redundant doc link
adriangb
commented
Aug 12, 2026
Contributor
Looks good to me! I'd like to do some more benchmark runs, I do see regressions in the previous runs. I'd also like to know if there are existing benchmarks we expect to improve mechanistically and check that they do indeed improve; otherwise we should add benchmarks first. |
ContributorAuthor
Thanks @adriangb — filed the benchmark: #24328. It's a |
…down re-read Adds the reporter's fixture as an slt case: the filter column (search_phrase) differs from the sort column (event_time), and one row group has an empty post-predicate selection invisible to statistics. Before the rg_plan/ decoder-frontier sync, a runtime prune re-read an already-delivered row group, duplicating rows and dropping the top-k tail. The query now returns the correct p0, p4096..p4104.
ContributorAuthor
adriangb
commented
Aug 14, 2026
Contributor
run benchmark clickbench_pushdown clickbench_sorted |
adriangb
commented
Aug 14, 2026
Contributor
run benchmark clickbench_pushdown clickbench_sorted env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true |
adriangbot
commented
Aug 14, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (5392336) to 0a429a3 (merge-base) diff Run configurationrun benchmark clickbench_pushdownResults will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 14, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (5392336) to 0a429a3 (merge-base) diff Run configurationrun benchmark clickbench_pushdownenv:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 14, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (5392336) to 0a429a3 (merge-base) diff Run configurationrun benchmark clickbench_sortedResults will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 14, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (5392336) to 0a429a3 (merge-base) diff Run configurationrun benchmark clickbench_sortedenv:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 14, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (5392336) to 0a429a3 (merge-base) diff Run configurationrun benchmark clickbench_pushdownenv:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"CPU Details (lscpu)DetailsResource Usageclickbench_pushdown — base (merge-base)
clickbench_pushdown — branch
File an issue against this benchmark runner |
adriangbot
commented
Aug 14, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (5392336) to 0a429a3 (merge-base) diff Run configurationrun benchmark clickbench_pushdownCPU Details (lscpu)DetailsResource Usageclickbench_pushdown — base (merge-base)
clickbench_pushdown — branch
File an issue against this benchmark runner |
adriangbot
commented
Aug 14, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (5392336) to 0a429a3 (merge-base) diff Run configurationrun benchmark clickbench_sortedenv:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"CPU Details (lscpu)DetailsResource Usageclickbench_sorted — base (merge-base)
clickbench_sorted — branch
File an issue against this benchmark runner |
adriangbot
commented
Aug 14, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (5392336) to 0a429a3 (merge-base) diff Run configurationrun benchmark clickbench_sortedCPU Details (lscpu)DetailsResource Usageclickbench_sorted — base (merge-base)
clickbench_sorted — branch
File an issue against this benchmark runner |
Smallfu666 pushed a commit
to Smallfu666/datafusion
that referenced
this pull request
Aug 14, 2026
…ter skip (apache#24328) ## Which issue does this PR close? - Part of apache#23696 — a benchmark that mechanistically exercises the per-RG fully-matched `RowFilter` skip. ## Rationale for this change apache#23696 adds a per-row-group fully-matched `RowFilter` skip, but **none of the existing benchmarks exercise it**: - `sort_tpch` / `tpch` don't enable `pushdown_filters` by default, so there is no `RowFilter` to skip; - ClickBench's `URL LIKE …` / equality predicates rarely make a row group's min/max fall entirely inside the satisfying range, so fully-matched RGs are rare. As raised in review (we should verify the optimization improves something mechanistically, otherwise add a benchmark first), this adds a suite that **necessarily** triggers the skip. ## What changes are included in this PR? A new `sql_benchmarks/parquet_row_filter_skip/` suite: - The **load SQL** enables `pushdown_filters` and `COPY`s a clustered Parquet file — a fixed-width, zero-padded, monotonically increasing string key (`skey`) so each row group holds a disjoint, sorted range — plus 14 payload columns. - The **query** applies a low-selectivity range filter (`skey >= '0000100000'`, `skey` not projected). The first row group straddles the threshold; every later RG is fully matched by statistics, so the per-row `RowFilter` is skipped on the fully-matched run (and `skey` isn't decoded there). - `bench.sh` integration: `./bench.sh run parquet_row_filter_skip`, data generated inline by the load SQL. Knobs: `PRED_ROWS` (row count), `RG_SIZE` (parquet row-group size). ## Are these changes tested? Smoke-tested locally via `cargo bench --bench sql -- --test`. Local A/B (main vs apache#23696, 10M rows / 10 RGs / `skey >= '0000100000'`): - **9 of 10 row groups fully matched** → `row_filter_skipped_fully_matched=9`; - **~18% faster** with the optimization (main ~0.143s → branch ~0.117s); an int64-key variant is ~12%. Once this lands, `run benchmark parquet_row_filter_skip` will compare any PR (e.g. apache#23696) against `main` in CI. ## Are there any user-facing changes? No — benchmark only. --------- Co-authored-by: Claude <noreply@anthropic.com>
tjnangosha pushed a commit
to tjnangosha/datafusion
that referenced
this pull request
Aug 14, 2026
…ts from re-reading already-delivered row groups (apache#24352) (apache#24354) ## Which issue does this PR close? - Closesapache#24352. ## Rationale for this change With `datafusion.execution.parquet.pushdown_filters = true` and TopK dynamic filter pushdown (both on by default), a query of the shape `SELECT b FROM t WHERE <predicate on a> ORDER BY b LIMIT k` can silently return **wrong results** — one source row emitted several times and the true tail of the top-k missing — with no error or warning. Root cause (thanks to @hhhizzz's very detailed report + fixture in apache#24352): a row group whose post-predicate selection is empty is silently finished by arrow-rs **without handing back a reader**. `PushDecoderStreamState` pops its `rg_plan` **only** when a reader is returned, so after a silently-finished RG the plan trails the decoder by one. When the runtime row-group pruner then rebuilds the decoder (`into_builder().with_row_groups(...)`) from the stale `rg_plan`, it re-includes an already-delivered row group, whose rows are emitted a second time and displace the genuine top-k in the heap. ## What changes are included in this PR? - `push_decoder.rs`: before each boundary prune/rebuild, `rg_plan` is synced to the row group the decoder will actually emit next via `peek_next_row_group()` (`sync_rg_plan_to_decoder_frontier` / `advance_rg_plan_to`), dropping entries for silently-finished row groups so a rebuild can never re-include a delivered group. A rebuild frontier naming an RG not in the plan is now an internal error instead of a silent plan drain. ## Are these changes tested? - Adds @hhhizzz's fixture as an slt regression test in `dynamic_row_group_pruning.slt` (filter column `search_phrase` differs from the sort column `event_time`, one row group has an empty post-predicate selection invisible to statistics). It now returns the correct `p0 p4096 p4097 … p4104` (was the buggy `p0 p4096 p4096 …`). - clippy clean; `datasource-parquet` unit tests and the sqllogictest suite pass locally. ## Are there any user-facing changes? Fixes silently-wrong query results; no API change. ## Note This is the standalone bug fix extracted from apache#23696 (per review discussion in apache#24352): the same `rg_plan` ↔ decoder-frontier sync, on its own so it merges fast and is easy to backport. apache#23696 will rebase on top so it carries only the fully-matched `RowFilter` skip performance optimization. cc @alamb@adriangb@hhhizzz
…ilter-for-full-match # Conflicts: # datafusion/core/tests/parquet/dynamic_row_group_pruning.rs # datafusion/datasource-parquet/src/opener/mod.rs # datafusion/datasource-parquet/src/push_decoder.rs # datafusion/sqllogictest/test_files/dynamic_row_group_pruning.slt
zhuqi-lucasforce-pushed
the
perf/parquet-skip-filter-for-full-match
branch
from
August 15, 2026 07:48
5392336 to
4f7306cComparezhuqi-lucas
commented
Aug 15, 2026
ContributorAuthor
run benchmark clickbench_pushdown clickbench_sorted env:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: true |
adriangbot
commented
Aug 15, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (4f7306c) to b9399dc (merge-base) diff Run configurationrun benchmark clickbench_pushdownenv:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 15, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing perf/parquet-skip-filter-for-full-match (4f7306c) to b9399dc (merge-base) diff Run configurationrun benchmark clickbench_sortedenv:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"Results will be posted here when complete File an issue against this benchmark runner |
adriangbot
commented
Aug 15, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (4f7306c) to b9399dc (merge-base) diff Run configurationrun benchmark clickbench_pushdownenv:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"CPU Details (lscpu)DetailsResource Usageclickbench_pushdown — base (merge-base)
clickbench_pushdown — branch
File an issue against this benchmark runner |
adriangbot
commented
Aug 15, 2026
🤖 Benchmark completed (GKE) | trigger Instance: Comparing perf/parquet-skip-filter-for-full-match (4f7306c) to b9399dc (merge-base) diff Run configurationrun benchmark clickbench_sortedenv:
DATAFUSION_EXECUTION_PARQUET_PUSHDOWN_FILTERS: "true"CPU Details (lscpu)DetailsResource Usageclickbench_sorted — base (merge-base)
clickbench_sorted — branch
File an issue against this benchmark runner |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
fully_matchedRowFilter skip · needs arrow-rs#10158 (peek_next_row_group) #23067.Rationale for this change
Builds on the
peek_next_row_groupAPI landed in apache/arrow-rs#10158 (in releasedarrow-rs 59.1.0). When Parquet stats prove that every row of a row group already satisfies the pushdown predicate (fully_matched), running the per-rowRowFilterinside that row group is pure overhead — every row passes anyway. This change installs an emptyRowFilteron fully-matched runs and only pays the row-level machinery on RGs that still need filtering.Two-phase reads for those RGs also skip fetching the filter columns when they aren't in the projection — this pattern is common on time-partitioned tables (
WHERE ts >= X AND ts < Ywhere the RG's min/max is contained in[X, Y)) and on the middle stretch of aTopKscan where the dynamic threshold is still loose.What changes are included in this PR?
access_plan.rs—ParquetAccessPlannow tracks per-RGfully_matchedstate, produced during static pruning.metrics.rs— newrow_filter_skipped_fully_matchedcounter to observe how often the toggle fires.push_decoder.rs—RgPlanEntrycarries the per-RG toggle;RowFilterContextswaps between the real filter (from theprebuild_row_filter_candidatescache) and an empty filter as the decoder crosses row-group boundaries viapeek_next_row_group().row_filter.rs— splitbuild_row_filterintoprebuild_row_filter_candidates(once per file: tree walk + candidate construction) androw_filter_from_prebuilt(per RG boundary: cheap bind against the fresh array readers). The existing publicbuild_row_filterAPI is preserved for callers that don't need per-RG toggling.opener/mod.rs— wires the prebuilt cache into the decoder builder; skips page-index loading for fully-matched RGs (their per-row filter is a no-op, so page pruning saves nothing).sort.rs— sort-order-aware RG reorder preserves the per-RG toggle through reordering.Are these changes tested?
Yes.
fully_matched_rgs_skip_row_filter(datafusion/core/tests/parquet/dynamic_row_group_pruning.rs) — 4 RGs of 3 rows each, predicatev >= 3makes RG 0 a straddler and RGs 1..=3 fully matched. Asserts (a) results are correct and (b)row_filter_skipped_fully_matchedfires at least once at the RG 0 → RG 1 boundary.dynamic_row_group_pruning.slt— anEXPLAIN ANALYZEcase (WHERE v >= 4withpushdown_filters=true) that surfacesrow_filter_skipped_fully_matched=1in the scan metrics, so the optimization is observable from SQL and any regression shows up as the count dropping back to 0.dynamic_row_group_pruningintegration tests still pass.datafusion-datasource-parquetlib suite passes.Are there any user-facing changes?
row_filter_skipped_fully_matchedmetric appears inEXPLAIN ANALYZEon parquet scans (0 when the toggle didn't fire).build_row_filterstill exists and behaves identically for non-toggling callers.