Uh oh!
There was an error while loading. Please reload this page.
[SPARK-57205][SQL] Add SupportsScanMerging to merge equivalent V2 file scans - #56264
[SPARK-57205][SQL] Add SupportsScanMerging to merge equivalent V2 file scans#56264LuciferYang wants to merge 2 commits into
Conversation
fbaa5f0 to
7c1e41aCompare…e scans ### What changes were proposed in this pull request? This adds a `SupportsScanMerging` mix-in for DSv2 `Scan` and teaches `MergeSubplans` (through `PlanMerger`) to use it. When two scans of the same table differ only in their projected columns and/or pushed filters, the optimizer asks the source to fuse them into one scan that covers both, and then deduplicates the subplans built on top. The interface is implemented once on the `FileScan` base, so every built-in file format (Parquet, ORC, CSV, JSON, Text, Avro) participates. Two scans merge when they read the same data (same file index, schema, options and partition filters); the merged scan reads the union of the two read schemas. When the pushed data filters differ, the merged scan widens them to `OR(f1, f2)` and reads a superset -- the exact per-side predicate is still enforced by the post-scan `Filter`, which `MergeSubplans` turns into per-aggregate `FILTER (WHERE ...)` clauses. That widening is off by default, gated by `spark.sql.files.scanMerge.ignorePushedDataFilters`. Merging is declined when an aggregate is pushed into the scan, since there is then no post-scan `Filter` to separate the two sides. ### Why are the changes needed? V1 file sources already get this. Their filter and column pushdown happens during physical planning, so when `MergeSubplans` runs the logical plan still has plain `Filter`/`Project` nodes over identical relation leaves, which the existing rules merge. DSv2 bakes pushdown into `DataSourceV2ScanRelation` during logical optimization, so two subqueries that differ only in `WHERE` or `SELECT` become structurally different leaves and cannot be merged. For example TPC-DS q9 (fifteen scalar subqueries over `store_sales`) collapses to a single scan under V1 but reads the table fifteen times under V2. This closes that gap. ### Does this PR introduce _any_ user-facing change? No. The connector interface and the config are internal, and the config defaults to off, so the default plan is unchanged. ### How was this patch tested? New tests in `PlanMergeSuite` run each query through both the V1 and V2 file-source paths, with AQE on and off, and assert identical results and identical merge structure. The merge-semantics cases cover differing columns, differing data filters, differing partition filters, filter propagation through a join, and a multi-subquery composition, plus negative cases; the format-coverage cases run a representative query over Parquet, ORC and JSON. `MergeSubplansSuite`, `DataSourceV2Suite` and the Parquet/ORC V2 aggregate-pushdown suites still pass. I also checked that V1 and V2 produce identical results and merge structure on the affected TPC-DS queries (q2, q9, q28, q59, q77a, q88, q90) at scale factor 1. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 4.8
7c1e41a to
511e5d1CompareLuciferYang
commented
Jun 4, 2026
cc @peter-toth Do you have any better ideas or suggestions regarding this optimization capability? Thanks ~ |
peter-toth
commented
Jun 4, 2026
Thanks for picking this up @LuciferYang. I haven't done a thorough review of the PR yet, but I read it through carefully alongside my old #37711 and wanted to share a thought on the approach before getting into the details. The shape here is essentially the same as #37711 (which never landed), generalized in two important ways: the merge logic now lives once on One thought on the design direction, and an alternative worth at least considering before we commit to this shape: The current shape (call it Option A) The merge body lives on the connector side, in
An alternative (Option B) SPARK-56385 (April 2026) added
The strict/relaxed split and the Trade-off
I don't have a position yet on whether to swap this PR for B or land A first and follow up — wanted to put the alternative on the table since it changes who pays the cost (Spark once vs. every connector). If you'd like to experiment with Option B yourself, please go ahead; if you'd prefer, I can sketch it as a draft so we can compare side by side. Let me know which you'd like. cc @cloud-fan |
@peter-toth Sorry for missing PR #37711 earlier, and many thanks for your review and suggestions. You're right that with the body on One thing worth pinning down in the Option B sketch: the holder's A merge driven off the relation's The strict column-union case (same pushed filters, differ only in projected columns) is universally safe and is the only on-by-default path. That's where B helps most, since it generalizes to every V2 source with no per-connector code. The relaxed OR-widen only makes sense for sources that keep a post-scan residual There's also a layering issue that affects where the generic body can live. I'd propose we land A first (complete, tested, ships the on-by-default strict column-union case), then generalize in a follow-up that (1) adds a generic Spark-side strict merge in sql/core behind the marker interface, (2) adds the Either way, yes please -- I'd like to take you up on the offer to sketch B. Seeing the two side by side would make the strict-vs-relaxed boundary concrete and show whether our follow-ups converge. If you can wire B so the post-scan residual is visible to the leaf merge, that dissolves my main concern -- I may well be missing a simpler path. cc @cloud-fan |
peter-toth
commented
Jun 5, 2026
Thanks for the thorough writeup, @LuciferYang -- you're right on all the specifics, and the Agreed that the relation's Where I'm more optimistic is the conclusion. The predicate isn't lost, it's just not on the relation: for best-effort sources it's the post-scan You're right about layering too: the legacy I don't want to block your PR on a hunch, though. Let me spend a bit of time next week sketching the generic version so we can put the two side by side, as you suggested -- then we can see whether the strict-vs-relaxed boundary really dissolves, and decide together whether the DSv2 file sources are worth their own code on top of the generic core, or whether landing this first and generalizing later is cleaner after all. I'll report back here. |
…-56385 pushedFilters dependency, adapted to 4.0.x - backport SPARK-56385 pushedFilters field on DataSourceV2ScanRelation (surgical: field + doCanonicalize + V2ScanRelationPushDown population; drop join-pushdown entanglement) - update ~12 positional DataSourceV2ScanRelation patterns to 6-arg - cherry-pick apache#56264: FileScan union (SupportsRuntimeCatalystFilters + SupportsScanMerging), PlanMerger coexists with SPARK-40193 filter-prop - strip bindingPolicy from new conf; drop ParquetScan variant comparison (no variant pushdown in 4.0.x)
Any updates? @peter-toth Thanks ~ |
peter-toth
commented
Jun 25, 2026
Sorry @LuciferYang, I had to deal with some urgent security issues in the previous weeks, and I'm off this week, but I already started working on this. |
LuciferYang
commented
Jun 25, 2026
Thanks for your feedback, looking forward to your PR. |
LuciferYang
commented
Jul 9, 2026
Here I come again... |
peter-toth
commented
Jul 9, 2026
I'm very sorry @LuciferYang, I could not wrap up my PR yet due to internal activities last and this week. I will be off for a few days next week, so let me try to conclude my version by the end of next week and come back to you. |
peter-toth
commented
Jul 19, 2026
@LuciferYang I opened #57360 with the generic, Spark-side alternative we discussed. It's still a draft, but feel free to take a look whenever. |
…MergeSubplans ### What changes were proposed in this pull request? This adds a generic, Spark-side merge of equivalent DataSource V2 scans in the `MergeSubplans` rule. When two subplans read the same V2 table, their scans can be fused into a single scan. V1 file sources already get this (via `FileSourceStrategy`), but every V2 scan was built independently, so the same table could be read many times. A source opts in with a new **`TableCapability.SCAN_MERGING`** capability. Making it a table capability (rather than a marker on `Scan`) lets Spark decide before `build()`, and lets a V1-fallback source — whose `Scan` Spark wraps in an internal `V1ScanWrapper` — still opt in. `PlanMerger` fuses two `DataSourceV2ScanRelation`s only when **all** of the following hold: - they read the same relation — same table, catalog, identifier and **options** (compared by the relation's canonical form); - both relations' tables declare the `SCAN_MERGING` capability; - both scans are **mergeable**: each came out of the plain column-pruning + filter-pushdown path, carrying only pushdowns a rebuilt scan can reproduce. A non-reproducible pushdown — a pushed **aggregate, join, variant extraction, limit, offset, sort/top-N, or table sample** — makes a scan non-mergeable. This is tracked by a **default-safe `mergeableScan` flag** on `DataSourceV2ScanRelation`: it defaults to `false` and is granted only on the plain-scan path, so any other current or future build site is non-mergeable unless it explicitly opts in; - neither reports **key-grouped partitioning** or an **ordering** — the rebuilt scan doesn't reconstruct these, so the merge is declined rather than silently dropping them (preserving them is a follow-up); - each scan reads a **subset of its relation's columns at the same type** — a column read at a narrower type via nested-schema pruning carries extractor ordinals against that narrow layout, so merging it (the rebuilt scan reads the full type) would read the wrong field; such a pair is declined (widening to the union of nested fields is a follow-up); - their **fully-pushed (strict) filters are equal** (differing-strict merge is a follow-up). The two scans may still differ in **projected columns** (the merged scan reads the union) and in **best-effort / post-scan filters** (OR-widened above the merged scan by symmetric filter propagation). When these hold, `PlanMerger` rebuilds the merged scan by driving the real `V2ScanRelationPushDown` (through a new `rebuildScan` entry point) over the union of columns and the (equal) strict filters, and verifies each strict filter comes back fully enforced **and that the rebuilt scan is itself mergeable**. Reusing the production pushdown end to end means the merged scan goes through the same filter translation, column pruning and iterative `PartitionPredicate` second pass. When the merged scan sits under a `Filter`, the build is **deferred to that Filter** so it's built once with the strict filters plus the best-effort OR pruning together. Mechanically, a small `MergeContext` threads down through the recursion, and `TryMergeResult` gains a `dsv2DeferredScan` field carrying the pending scan's union columns and strict filters back up to that `Filter`. A separate `dsv2Merged` fact propagates the "a DSv2 merge happened below" signal all the way up (past the innermost `Filter` that builds the scan), so a stacked **outer** `Filter` — e.g. a DPP or runtime-bloom `Filter` that `PartitionPruning` / `InjectRuntimeFilter` insert above the scan-adjacent one after pushdown — still recognizes the merge and can OR-widen. Because a DSv2 merge's strict filters are equal on both sides, the best-effort OR-widen (which only broadens the enclosing `Filter` condition offered to the source for data skipping — the `Filter` above still guarantees correctness) is allowed under a **new DSv2-specific config** `spark.sql.optimizer.mergeSubplans.filterPropagation.dsv2SymmetricFilterPropagation.enabled` even when the general symmetric propagation is off. It ships **default `false`** for now; it can be flipped on in a later release once more sources have exercised the path. To make the merge sound, this PR also completes `DataSourceV2ScanRelation.pushedFilters`, which previously dropped fully-pushed filters referencing a column pruned out of the scan output (e.g. an unselected partition column the source enforces internally); the merge compares and re-enforces this set, so it must be complete. This widens the field's effect slightly beyond the merge: `pushedFilters` participates in `canonicalized`, which backs `sameResult`, so two scans differing only in a filter over a pruned-out column are no longer `sameResult`. Physical reuse is unaffected (`BatchScanExec.doCanonicalize` doesn't consult the field), but `CacheManager.lookupCachedData` uses `sameResult`, so such a query may correctly stop hitting the cache — the previous behavior treated the two as equivalent, which was a wrong hit. `DataSourceV2Suite` pins this canonicalization inequality. Non-deterministic filters cannot reach the merge: they aren't pushed to V2 sources (SPARK-58207). This is the generic alternative to #56264, which added scan merging per file format; here any V2 source gets it with a single capability. Note on structure: the first commit is a pure move of `MergeSubplans`/`PlanMerger` (and suites) from `sql/catalyst` to a new `sql/core` `execution.planmerging` package — required because the merge now calls sql/core-only pushdown — and can be split into its own PR. The move does **not** change the rule's batch position (it was and is registered only in `SparkOptimizer`); because the rule's fully-qualified class name changed, a 4.2→4.3 migration-guide note is added (mirroring the `DynamicJoinSelection` rename), since `spark.sql.optimizer.excludedRules` matches by that name. The remaining commits are the feature, the build-deferral, the `TableCapability` opt-in, the outer-`Filter` `dsv2Merged` fact, and several self-review passes. ### Why are the changes needed? Under DataSource V2, subplans that read the same table each build their own scan, so the table is scanned once per subplan. The motivating case is TPC-DS q9, whose 15 scalar subqueries over `store_sales` collapse to a single scan under V1 but read the table 15 times under V2. This closes that gap generically for any V2 source. ### Does this PR introduce _any_ user-facing change? No behavior change for existing sources: the merge is opt-in through the new `TableCapability.SCAN_MERGING` capability, which no built-in source declares, so plans and results are unchanged. This PR adds one new SQL config (`spark.sql.optimizer.mergeSubplans.filterPropagation.dsv2SymmetricFilterPropagation.enabled`, default `false`), which only affects sources that opt into `SCAN_MERGING`. ### How was this patch tested? New tests: - `MergeSubplansSuite` (plan tests, `comparePlans` where the merge changes the plan): column-union merge; differing-filter OR-widen (2- and 3-way, including a reuse-branch case); a stacked outer-`Filter` pair fusing via the propagated `dsv2Merged` fact; identical-filter re-push; equal-strict re-push (2- and 3-way); mixed equal-strict + differing-post-scan; the DSv2 symmetric config gate (merges when it is on and the general one is off, not when both are off); the decline gates (no `SCAN_MERGING` capability, a merge-blocking pushdown, a column read at a nested-pruned type, reported key-grouped partitioning/ordering, a strict filter not re-enforced on rebuild); a `mergeableScan` classification check driving a pushed limit/offset/top-N/sample through the real pushdown; a non-deterministic pruning-predicate drop; a negative canonicalization check; and an invariant assertion that no placeholder `DataSourceV2Relation` survives a merge. - `DSv2PlanMergingSuite` (end-to-end with a real session): two scalar subqueries whose partition filter is strict only via the iterative second pass fuse into one scan (`checkAnswer` verifies correctness); two subqueries with different partition filters are not fused; and two subqueries with differing best-effort filters OR-widen into one scan. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes#57360 from peter-toth/SPARK-40259-dsv2-planmerger-support. Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Peter Toth <peter.toth@gmail.com>
…MergeSubplans ### What changes were proposed in this pull request? This adds a generic, Spark-side merge of equivalent DataSource V2 scans in the `MergeSubplans` rule. When two subplans read the same V2 table, their scans can be fused into a single scan. V1 file sources already get this (via `FileSourceStrategy`), but every V2 scan was built independently, so the same table could be read many times. A source opts in with a new **`TableCapability.SCAN_MERGING`** capability. Making it a table capability (rather than a marker on `Scan`) lets Spark decide before `build()`, and lets a V1-fallback source — whose `Scan` Spark wraps in an internal `V1ScanWrapper` — still opt in. `PlanMerger` fuses two `DataSourceV2ScanRelation`s only when **all** of the following hold: - they read the same relation — same table, catalog, identifier and **options** (compared by the relation's canonical form); - both relations' tables declare the `SCAN_MERGING` capability; - both scans are **mergeable**: each came out of the plain column-pruning + filter-pushdown path, carrying only pushdowns a rebuilt scan can reproduce. A non-reproducible pushdown — a pushed **aggregate, join, variant extraction, limit, offset, sort/top-N, or table sample** — makes a scan non-mergeable. This is tracked by a **default-safe `mergeableScan` flag** on `DataSourceV2ScanRelation`: it defaults to `false` and is granted only on the plain-scan path, so any other current or future build site is non-mergeable unless it explicitly opts in; - neither reports **key-grouped partitioning** or an **ordering** — the rebuilt scan doesn't reconstruct these, so the merge is declined rather than silently dropping them (preserving them is a follow-up); - each scan reads a **subset of its relation's columns at the same type** — a column read at a narrower type via nested-schema pruning carries extractor ordinals against that narrow layout, so merging it (the rebuilt scan reads the full type) would read the wrong field; such a pair is declined (widening to the union of nested fields is a follow-up); - their **fully-pushed (strict) filters are equal** (differing-strict merge is a follow-up). The two scans may still differ in **projected columns** (the merged scan reads the union) and in **best-effort / post-scan filters** (OR-widened above the merged scan by symmetric filter propagation). When these hold, `PlanMerger` rebuilds the merged scan by driving the real `V2ScanRelationPushDown` (through a new `rebuildScan` entry point) over the union of columns and the (equal) strict filters, and verifies each strict filter comes back fully enforced **and that the rebuilt scan is itself mergeable**. Reusing the production pushdown end to end means the merged scan goes through the same filter translation, column pruning and iterative `PartitionPredicate` second pass. When the merged scan sits under a `Filter`, the build is **deferred to that Filter** so it's built once with the strict filters plus the best-effort OR pruning together. Mechanically, a small `MergeContext` threads down through the recursion, and `TryMergeResult` gains a `dsv2DeferredScan` field carrying the pending scan's union columns and strict filters back up to that `Filter`. A separate `dsv2Merged` fact propagates the "a DSv2 merge happened below" signal all the way up (past the innermost `Filter` that builds the scan), so a stacked **outer** `Filter` — e.g. a DPP or runtime-bloom `Filter` that `PartitionPruning` / `InjectRuntimeFilter` insert above the scan-adjacent one after pushdown — still recognizes the merge and can OR-widen. Because a DSv2 merge's strict filters are equal on both sides, the best-effort OR-widen (which only broadens the enclosing `Filter` condition offered to the source for data skipping — the `Filter` above still guarantees correctness) is allowed under a **new DSv2-specific config** `spark.sql.optimizer.mergeSubplans.filterPropagation.dsv2SymmetricFilterPropagation.enabled` even when the general symmetric propagation is off. It ships **default `false`** for now; it can be flipped on in a later release once more sources have exercised the path. To make the merge sound, this PR also completes `DataSourceV2ScanRelation.pushedFilters`, which previously dropped fully-pushed filters referencing a column pruned out of the scan output (e.g. an unselected partition column the source enforces internally); the merge compares and re-enforces this set, so it must be complete. This widens the field's effect slightly beyond the merge: `pushedFilters` participates in `canonicalized`, which backs `sameResult`, so two scans differing only in a filter over a pruned-out column are no longer `sameResult`. Physical reuse is unaffected (`BatchScanExec.doCanonicalize` doesn't consult the field), but `CacheManager.lookupCachedData` uses `sameResult`, so such a query may correctly stop hitting the cache — the previous behavior treated the two as equivalent, which was a wrong hit. `DataSourceV2Suite` pins this canonicalization inequality. Non-deterministic filters cannot reach the merge: they aren't pushed to V2 sources (SPARK-58207). This is the generic alternative to #56264, which added scan merging per file format; here any V2 source gets it with a single capability. Note on structure: the first commit is a pure move of `MergeSubplans`/`PlanMerger` (and suites) from `sql/catalyst` to a new `sql/core` `execution.planmerging` package — required because the merge now calls sql/core-only pushdown — and can be split into its own PR. The move does **not** change the rule's batch position (it was and is registered only in `SparkOptimizer`); because the rule's fully-qualified class name changed, a 4.2→4.3 migration-guide note is added (mirroring the `DynamicJoinSelection` rename), since `spark.sql.optimizer.excludedRules` matches by that name. The remaining commits are the feature, the build-deferral, the `TableCapability` opt-in, the outer-`Filter` `dsv2Merged` fact, and several self-review passes. ### Why are the changes needed? Under DataSource V2, subplans that read the same table each build their own scan, so the table is scanned once per subplan. The motivating case is TPC-DS q9, whose 15 scalar subqueries over `store_sales` collapse to a single scan under V1 but read the table 15 times under V2. This closes that gap generically for any V2 source. ### Does this PR introduce _any_ user-facing change? No behavior change for existing sources: the merge is opt-in through the new `TableCapability.SCAN_MERGING` capability, which no built-in source declares, so plans and results are unchanged. This PR adds one new SQL config (`spark.sql.optimizer.mergeSubplans.filterPropagation.dsv2SymmetricFilterPropagation.enabled`, default `false`), which only affects sources that opt into `SCAN_MERGING`. ### How was this patch tested? New tests: - `MergeSubplansSuite` (plan tests, `comparePlans` where the merge changes the plan): column-union merge; differing-filter OR-widen (2- and 3-way, including a reuse-branch case); a stacked outer-`Filter` pair fusing via the propagated `dsv2Merged` fact; identical-filter re-push; equal-strict re-push (2- and 3-way); mixed equal-strict + differing-post-scan; the DSv2 symmetric config gate (merges when it is on and the general one is off, not when both are off); the decline gates (no `SCAN_MERGING` capability, a merge-blocking pushdown, a column read at a nested-pruned type, reported key-grouped partitioning/ordering, a strict filter not re-enforced on rebuild); a `mergeableScan` classification check driving a pushed limit/offset/top-N/sample through the real pushdown; a non-deterministic pruning-predicate drop; a negative canonicalization check; and an invariant assertion that no placeholder `DataSourceV2Relation` survives a merge. - `DSv2PlanMergingSuite` (end-to-end with a real session): two scalar subqueries whose partition filter is strict only via the iterative second pass fuse into one scan (`checkAnswer` verifies correctness); two subqueries with different partition filters are not fused; and two subqueries with differing best-effort filters OR-widen into one scan. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) Closes#57360 from peter-toth/SPARK-40259-dsv2-planmerger-support. Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Peter Toth <peter.toth@gmail.com> (cherry picked from commit 4dcdd4d) Signed-off-by: Peter Toth <peter.toth@gmail.com>
Close this PR, as the solution provided by @peter-toth is more complete. |
What changes were proposed in this pull request?
This adds a
SupportsScanMergingmix-in for DSv2Scanand wires it intoMergeSubplansthroughPlanMerger. When two scans of the same table differ only in their projected columns and/or pushed filters, the optimizer asks the source to fuse them into a single scan that covers both, then deduplicates the subplans built on top. The interface is implemented once on theFileScanbase, so every built-in file format (Parquet, ORC, CSV, JSON, Text, Avro) participates. Two scans merge when they read the same data (same file index, schema, options and partition filters), and the merged scan reads the union of their read schemas. When the pushed data filters differ, the merged scan widens them toOR(f1, f2)and reads a superset; the exact per-side predicate is still enforced by the post-scanFilter, whichMergeSubplansrewrites into per-aggregateFILTER (WHERE ...)clauses. That widening is off by default and gated byspark.sql.files.scanMerge.ignorePushedDataFilters. Merging is declined when an aggregate is pushed into the scan, because there is then no post-scanFilterleft to separate the two sides.Why are the changes needed?
V1 file sources already merge in this case. Their filter and column pushdown happens during physical planning, so when
MergeSubplansruns the logical plan still has plainFilter/Projectnodes over identical relation leaves, which the existing rules combine. DSv2 instead bakes pushdown intoDataSourceV2ScanRelationduring logical optimization, so two subqueries that differ only inWHEREorSELECTbecome structurally different leaves and cannot be merged. TPC-DS q9 is a good example: its fifteen scalar subqueries overstore_salescollapse to a single scan under V1 but read the table fifteen times under V2. This change brings V2 to parity.Does this PR introduce any user-facing change?
No. The connector interface and the config are both internal, and the config defaults to off, so the default plan is unchanged.
How was this patch tested?
New tests in
PlanMergeSuiterun each query through both the V1 and V2 file-source paths, with AQE on and off, and assert that the results and the merge structure are identical. They cover differing columns, differing data filters, differing partition filters, filter propagation through a join, and a multi-subquery composition, plus the negative cases, and they run a representative query across Parquet, ORC and JSON for format coverage.MergeSubplansSuite,DataSourceV2Suite, and the Parquet/ORC V2 aggregate-pushdown suites still pass. I also verified that V1 and V2 produce identical results and merge structure on the affected TPC-DS queries (q2, q9, q28, q59, q77a, q88, q90) at scale factor 1.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8