fix: preserve MERGE target qualifier bindings

[wirybeaver]

Which issue does this PR close?

• Part of Support MERGE INTO #20746.
• Follow-up to feat: Add SQL planner, physical planner, and TableProvider hook for MERGE INTO #22988.

Rationale for this change

#22988 deliberately rejected two valid MERGE forms to avoid silently changing expression meaning.

Limitation 1: target-correlated subquery with an aliased target

MERGE INTO target AS t
USING source AS s
ON EXISTS (SELECT1FROM source AS x WHEREx.id=t.id)
WHEN MATCHED THEN DELETE;

t.id inside the subquery becomes outer_ref(t.id). The old top-level canonicalizer only rewrote Expr::Column(t.id) to target.id, leaving the outer reference inconsistent with the schema later rebuilt from DmlStatement.table_name.

Limitation 2: source qualifier equals the target table name

MERGE INTO target AS t
USING source AS target
ONt.id=target.id
WHEN MATCHED THEN DELETE;

Canonicalizing target t.id to target.id collapses both operands onto the source qualifier and can turn the condition into target.id = target.id.

A recursive string rewrite is not safe: qualifiers are scope-local, so an inner relation can legally shadow t. It also cannot solve the second limitation because both relations would still have the same qualifier after rewriting.

What changes are included in this PR?

Solution

Keep provider identity and the SQL-visible target qualifier as separate plan state:

SQL: MERGE INTO target AS t USING source AS target
SQL binding
+-------------------+
| |
target table source relation
provider: target qualifier: target
qualifier: t
| |
+---------+---------+
|
MERGE expression schema
+-----------+-----------+
| t.* | target.* |
| index 0.. | index N.. |
+-----------+-----------+
|
TableProvider::merge_into
Logical / protobuf representation
DmlStatement.table_name / DmlNode.table_name
= target (provider identity)
MergeIntoOp.target_qualifier /
MergeIntoOpNode.target_qualifier
= t (SQL-visible binding)

Planning now proceeds as follows:

1. Resolve target to the target provider, while retaining alias t as the visible qualifier.
2. Store t on MergeIntoOp; do not canonicalize target columns or recursively rewrite subquery plans.
3. Build one target-first expression schema (t.*, then source fields) and use it consistently in SQL planning, analyzer/optimizer rules, physical planning, and programmatic plans.
4. Reject only a source qualifier that collides with the visible target qualifier in the outer MERGE scope. A source qualifier matching the real table name remains valid when the target is aliased.
5. Pass the preserved schema and expressions to TableProvider::merge_into, where target and source columns resolve to distinct physical indices.

Protobuf change and 55.0/56.0 compatibility

MergeIntoOpNode gains optional target_qualifier field 3. This field is necessary because DmlNode.table_name contains provider identity and cannot also represent alias t; without it, a proto round trip loses the binding needed to resolve MERGE expressions.

Compatibility is directional:

• 55.0 payload -> 56.0 reader: supported. The field is absent, so the new reader falls back to DmlNode.table_name, matching 55.0's canonicalized representation.
• 56.0 payload -> 55.0 reader: unsupported. A 55.0 reader ignores the unknown field but alias-preserving expressions still require it, so expressions can fail resolution or be misbound.

Rust API compatibility

DataFusion 55.0 released MergeIntoOp with public struct-literal construction: MergeIntoOp { on, clauses }. This PR makes the struct non-exhaustive, adds private target-qualifier state, and requires MergeIntoOp::new(target_qualifier, on, clauses). Therefore existing 55.0 downstream struct literals will not compile unchanged against 56.0. This breaking change is documented in the 56.0 upgrade guide and is appropriate for the next major release.

The PR also:

• removes alias canonicalization and the recursive target-correlation guard;
• documents that providers may receive residual subqueries; and
• adds coverage for direct correlations, nested shadowing, lateral and LIMIT scopes, qualifier collisions, quoted/qualified identifiers, proto fallback/round trips, and physical column indices.

Are these changes tested?

• cargo fmt --all
• cargo clippy --all-targets --all-features -- -D warnings
• ./ci/scripts/doc_prettier_check.sh --write --allow-dirty
• RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption

Are there any user-facing changes?

Yes. Both valid MERGE forms above now plan successfully and reach TableProvider::merge_into. The Rust and protobuf compatibility constraints for upgrading from 55.0 to 56.0 are documented above.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

 Cloning apache/main
Building datafusion v55.0.0 (current)
Built [ 46.249s] (current)
Parsing datafusion v55.0.0 (current)
Parsed [ 0.029s] (current)
Building datafusion v55.0.0 (baseline)
Built [ 46.277s] (baseline)
Parsing datafusion v55.0.0 (baseline)
Parsed [ 0.031s] (baseline)
Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.769s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 94.959s] datafusion
Building datafusion-expr v55.0.0 (current)
Built [ 22.799s] (current)
Parsing datafusion-expr v55.0.0 (current)
Parsed [ 0.066s] (current)
Building datafusion-expr v55.0.0 (baseline)
Built [ 23.014s] (baseline)
Parsing datafusion-expr v55.0.0 (baseline)
Parsed [ 0.066s] (baseline)
Checking datafusion-expr v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 1.460s] 223 checks: 221 pass, 1 fail, 1 warn, 31 skip
--- failure struct_marked_non_exhaustive: struct marked #[non_exhaustive] ---
Description:
A public struct has been marked #[non_exhaustive], which will prevent it from being constructed using a struct literal outside of its crate. It previously had no private fields, so a struct literal could be used to construct it outside its crate.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#attr-adding-non-exhaustive
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/struct_marked_non_exhaustive.ron
Failed in:
struct MergeIntoOp in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:320
struct MergeIntoOp in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:320
struct MergeIntoOp in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:320
struct MergeIntoOp in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:320
--- warning partial_ord_struct_fields_reordered: struct fields reordered in #[derive(PartialOrd)] struct ---
Description:
A public struct that derives PartialOrd had its fields reordered. #[derive(PartialOrd)] uses the field order to set the struct's ordering behavior, so this change may break downstream code that relies on the previous order.
ref: https://doc.rust-lang.org/std/cmp/trait.PartialOrd.html#derivable
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/partial_ord_struct_fields_reordered.ron
Failed in:
MergeIntoOp.on moved from position 1 to 2, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:324
MergeIntoOp.clauses moved from position 2 to 3, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:326
MergeIntoOp.on moved from position 1 to 2, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:324
MergeIntoOp.clauses moved from position 2 to 3, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:326
MergeIntoOp.on moved from position 1 to 2, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:324
MergeIntoOp.clauses moved from position 2 to 3, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:326
MergeIntoOp.on moved from position 1 to 2, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:324
MergeIntoOp.clauses moved from position 2 to 3, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:326
Summary semver requires new major version: 1 major and 0 minor checks failed
Warning produced 1 major and 0 minor level warnings
Finished [ 48.384s] datafusion-expr
Building datafusion-optimizer v55.0.0 (current)
Built [ 21.325s] (current)
Parsing datafusion-optimizer v55.0.0 (current)
Parsed [ 0.026s] (current)
Building datafusion-optimizer v55.0.0 (baseline)
Built [ 21.432s] (baseline)
Parsing datafusion-optimizer v55.0.0 (baseline)
Parsed [ 0.027s] (baseline)
Checking datafusion-optimizer v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.182s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 43.753s] datafusion-optimizer
Building datafusion-proto v55.0.0 (current)
Built [ 42.703s] (current)
Parsing datafusion-proto v55.0.0 (current)
Parsed [ 0.015s] (current)
Building datafusion-proto v55.0.0 (baseline)
Built [ 43.366s] (baseline)
Parsing datafusion-proto v55.0.0 (baseline)
Parsed [ 0.015s] (baseline)
Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.124s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 87.362s] datafusion-proto
Building datafusion-proto-models v55.0.0 (current)
Built [ 19.823s] (current)
Parsing datafusion-proto-models v55.0.0 (current)
Parsed [ 0.108s] (current)
Building datafusion-proto-models v55.0.0 (baseline)
Built [ 19.810s] (baseline)
Parsing datafusion-proto-models v55.0.0 (baseline)
Parsed [ 0.109s] (baseline)
Checking datafusion-proto-models v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 1.950s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field MergeIntoOpNode.target_qualifier in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:540
field MergeIntoOpNode.target_qualifier in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:540
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 43.051s] datafusion-proto-models
Building datafusion-session v55.0.0 (current)
Built [ 30.320s] (current)
Parsing datafusion-session v55.0.0 (current)
Parsed [ 0.009s] (current)
Building datafusion-session v55.0.0 (baseline)
Built [ 31.322s] (baseline)
Parsing datafusion-session v55.0.0 (baseline)
Parsed [ 0.010s] (baseline)
Checking datafusion-session v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.207s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 62.720s] datafusion-session
Building datafusion-sql v55.0.0 (current)
Built [ 33.475s] (current)
Parsing datafusion-sql v55.0.0 (current)
Parsed [ 0.027s] (current)
Building datafusion-sql v55.0.0 (baseline)
Built [ 33.595s] (baseline)
Parsing datafusion-sql v55.0.0 (baseline)
Parsed [ 0.027s] (baseline)
Checking datafusion-sql v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.256s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 68.286s] datafusion-sql
Building datafusion-sqllogictest v55.0.0 (current)
Built [ 79.848s] (current)
Parsing datafusion-sqllogictest v55.0.0 (current)
Parsed [ 0.020s] (current)
Building datafusion-sqllogictest v55.0.0 (baseline)
Built [ 80.062s] (baseline)
Parsing datafusion-sqllogictest v55.0.0 (baseline)
Parsed [ 0.019s] (baseline)
Checking datafusion-sqllogictest v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.101s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 162.265s] datafusion-sqllogictest

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.85075% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.71%. Comparing base (6ab4ce6) to head (a3ef001).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	0.00%	13 Missing ⚠️
datafusion/expr/src/logical_plan/dml.rs	93.15%	1 Missing and 4 partials ⚠️
datafusion/core/src/physical_planner.rs	50.00%	0 Missing and 3 partials ⚠️
...afusion/optimizer/src/analyzer/function_rewrite.rs	0.00%	2 Missing ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs	75.00%	0 Missing and 1 partial ⚠️
datafusion/optimizer/src/rewrite_set_comparison.rs	50.00%	0 Missing and 1 partial ⚠️
...timizer/src/simplify_expressions/simplify_exprs.rs	50.00%	0 Missing and 1 partial ⚠️
datafusion/proto/src/logical_plan/from_proto.rs	93.33%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #24429 +/- ##
==========================================
- Coverage 81.72% 81.71% -0.01% 
==========================================
Files 1127 1127 Lines 416237 416234 -3 Branches 416237 416234 -3 ==========================================
- Hits 340156 340146 -10 - Misses 56092 56098 +6 - Partials 19989 19990 +1 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[wirybeaver]

@alamb@kosiew@timsaucer Could you take a look of this PR before the Datafusion 55.0.0 release. I think adding the target table's qualifer into the protobuf is a more elegant solution to make code succinct and recursively rewriting subquery.

[kosiew]

@wirybeaver,

Thanks for working on this. This looks good to me.

I like the approach of keeping MERGE's SQL-visible target qualifier separate from the target provider identity. Using that qualifier consistently when rebuilding the MERGE expression schema across the analyzer, optimizer, and physical planner avoids the alias canonicalization issues while preserving the correct SQL scoping semantics.

The protobuf handling also looks good. Persisting the qualifier while falling back to DmlNode.table_name for older payloads keeps the change backward compatible.

The added coverage for aliased targets, source-name collisions, correlated subqueries, qualifier shadowing, and the basic protobuf round trip gives me good confidence in the change.

One note on protobuf coverage: I think keeping the MERGE round-trip test's ON expression flat is appropriate here. A target-correlated subquery cannot currently be serialized because datafusion/proto/src/logical_plan/to_proto.rs does not support OuterReferenceColumn, Exists, or InSubquery. A MERGE round-trip test for that case would fit better with future work adding broader protobuf support for correlated subquery expressions.

Thanks again!

[alamb]

Thanks for this @wirybeaver and @kosiew

The code looks good -- I would just like to request we migrate some of these tests to use slt rather than more rust code

[wirybeaver]

@alamb The testing code has moved to SLT. Thanks for the guide

[kosiew]

@wirybeaver
Can you fix the CI errors?

[kosiew]

Could we move this SQL-only boolean-condition case into merge_into.slt as well? The non-boolean ON and WHEN cases are already covered there, and this test does not inspect bindings, protobuf state, or physical column indices.

That would also let us remove assert_merge_physical_error. I see this as test maintenance rather than a correctness blocker.

[wirybeaver]

Done in a2377f0. I moved the remaining NULL boolean-condition case to merge_into.slt and removed the Rust test and assert_merge_physical_error helper.

[timsaucer]

It looks like this has an API change. Per our documentation https://datafusion.apache.org/contributor-guide/api-health.html and https://datafusion.apache.org/contributor-guide/release_management.html#eligible-for-backport

Specifically the changes in datafusion/proto-models/proto/datafusion.proto which created the need for the upgrade guide entry. I think this is going to have to go into 56.0.0 instead of 55.1.0.

Do others concur?

[kosiew]

@timsaucer

Specifically the changes in datafusion/proto-models/proto/datafusion.proto which created the need for the upgrade guide entry. I think this is going to have to go into 56.0.0 instead of 55.1.0.

Agreed.

[kumarUjjawal]

I think this is going to have to go into 56.0.0 instead of 55.1.0.

Agreed!

[wirybeaver]

I also think 56.0.0 would be a better version to release API change. Does that mean the PR should get merged after 55.1.0 release?

[alamb]

I also think 56.0.0 would be a better version to release API change. Does that mean the PR should get merged after 55.1.0 release?

No, we will make the 55.1.0 release from the branch-55 branch -- main will become 56

[kosiew]

@wirybeaver
Can you resolve the merge conflicts?

[wirybeaver]

Add @kosiew as the co-author, appreciating for his substantial review in the PR #22988 with a bunch of challenging corner cases that inspires me to create this PR.

[wirybeaver]

@timsaucer The doc is moved to 56.0's markdown file