[SPARK-46625][SQL] Push WithCTE into CTEInChildren when materialised by ResolveIdentifierClause

[stevomitric]

What changes were proposed in this pull request?

After ResolveIdentifierClause materialises a PlanWithUnresolvedIdentifier into a CTEInChildren (e.g. InsertIntoStatement), collapse any surrounding WithCTE(c: CTEInChildren, defs) into c.withCTEDefs(defs) - restoring the placement invariant CTESubstitution.withCTEDefs already enforces at substitution time.

Why are the changes needed?

WITH t AS (...)
INSERT [INTO|OVERWRITE] TABLE IDENTIFIER('t')
SELECT * FROM t

produces an analysed plan with WithCTE wrapping the InsertIntoStatement - a structurally invalid shape. Plug-in datasources that re-analyse the InsertIntoStatement's query subtree and throw NoSuchElementException: key not found.
Bug is zero-day issue since SPARK-46625

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New tests.

Was this patch authored or co-authored using generative AI tooling?

Yes.

[cloud-fan]

Problem.WITH t AS (...) INSERT INTO/OVERWRITE TABLE IDENTIFIER(:p) SELECT * FROM t produces a structurally invalid analyzed plan: WithCTE(InsertIntoStatement(...), cteDefs). InsertIntoStatement is a CTEInChildren, and CTESubstitution.withCTEDefs (CTESubstitution.scala:466-471) explicitly maintains the invariant that no WithCTE may wrap a CTEInChildren — but the invariant is decided at substitution time, before ResolveIdentifierClause materializes the leaf PlanWithUnresolvedIdentifier into a CTEInChildren. Downstream consumers that re-analyze the query subtree see orphan CTERelationRefs and trip InlineCTE.buildCTEMap with NoSuchElementException.

Root cause.withIdentClause (AstBuilder.scala:97-107) lifts PlanWithUnresolvedIdentifier above the entire write/CTAS command, so CTESubstitution sees a leaf at that position and can't tell the leaf will resolve into a CTEInChildren. The PR addresses this by re-applying CTESubstitution.withCTEDefs's dispatch after materialization (a post-hoc collapse), but the structural cause is the parser placement — see the inline comment on ResolveIdentifierClause.scala for a redirect to fix this at the source.

[cloud-fan]

This patches the symptom rather than the root cause: withIdentClause lifts PlanWithUnresolvedIdentifier above the whole write/CTAS command, and we then have to undo the placement after materialization. Please change the parser to push the placeholder into the identifier slot of the produced plan (e.g. InsertIntoStatement.table, CreateTableAsSelect.name) instead of wrapping the entire command, and add a parallel handler in this rule — e.g.:

case i @InsertIntoStatement(p: PlanWithUnresolvedIdentifier, _, _, _, _, _, _, _, _)
if p.identifierExpr.resolved =>
i.copy(table = executor.execute(p.planBuilder.apply(
IdentifierResolution.evalIdentifierExpr(p.identifierExpr), p.children)))

Then CTESubstitution sees the actual CTEInChildren from the start and places WithCTE correctly — no post-hoc collapse needed, and the invariant is preserved by construction. The same shape applies to all withIdentClause call sites whose builder produces a CTEInChildren (INSERT, CTAS, RTAS, CACHE TABLE AS — AstBuilder.scala:911-1002, 5640, 5724, 6502). Downstream matchers like case InsertIntoStatement(LogicalRelationWithTable(_), ...) aren't affected because they run after this rule, by which point table is back to a normal resolved relation.

[stevomitric]

I tried this and reverted - the parser-level placement breaks legacy-mode parameter binding (spark.sql.legacy.parameterSubstitution.constantsOnly=true) for INSERT … IDENTIFIER(:p).

The problem: InsertIntoStatement.child = query (statements.scala:210), so .table is not in children. BindParameters.bind (parameters.scala:177-186) walks via resolveOperatorsDown (children-based) and transformExpressionsWithPruning (expressions only) — neither reaches a LogicalPlan-typed non-child field. Tree-pattern propagation in TreeNode.getDefaultTreePatternBits (TreeNode.scala:104-108) also flows only through children, so even containsPattern(PARAMETER) returns false on the wrapping

InsertIntoStatement and the rule prunes itself out. Result: the NamedParameter inside PlanWithUnresolvedIdentifier.identifierExpr is never bound, the placeholder never resolves, and analysis fails with [UNSUPPORTED_INSERT.RDD_BASED]. The non-legacy path works because withIdentClause short-circuits a Literal directly to UnresolvedRelation (AstBuilder.scala:85) and never creates PlanWithUnresolvedIdentifier in the first place.

Making the parser-level placement work would require teaching BindParameters (and tree-pattern propagation) about the non-child placeholder slot on every CTEInChildren write command, CTAS/RTAS name, plus future additions. That's a broader refactor than this bug warrants. Happy to do the deeper refactor as a separate change if you'd prefer.

[cloud-fan]

Once the fix moves into the parser per the comment on ResolveIdentifierClause.scala, please broaden the structural test to cover the other affected commands too — at minimum a WITH t AS (...) CREATE TABLE IDENTIFIER(:p) AS SELECT * FROM t variant, since CTAS goes through the same shape. Also worth knowing: the INSERT INTO/OVERWRITE smoke tests above pass even without the fix (eager command execution doesn't hit the re-analysis path that produces the NoSuchElementException); the structural assertion here is what actually anchors the regression.

[stevomitric]

Added a CTAS variant - CREATE TABLE IDENTIFIER(:p) AS WITH … SELECT * FROM t