[SPARK-58111][SQL] Scan and write schema narrowing for column-level UPDATE in DSv2 by anuragmantri · Pull Request #55518 · apache/spark

anuragmantri · 2026-04-23T18:30:03Z

What changes were proposed in this pull request?

This PR adds an opt-in DSv2 mix-in for connectors to receive narrow rows on column-level UPDATE, so connectors can avoid reading and writing full-table rows when only a subset of columns is being updated.

Public API additions (since 4.3.0):

SupportsColumnUpdates (@Experimental) — a new mix-in on RowLevelOperation. Connectors implement requiredDataAttributes(): NamedReference[] to declare the columns they need in the write payload, and optionally scanOnlyDataAttributes(): NamedReference[] to declare additional columns needed only for scan/write-side planning (e.g. resolving partitioning expressions or write clustering keys) but excluded from the write payload; must not overlap with requiredDataAttributes(), defaults to an empty array.
RowLevelOperationInfo.updatedColumns(): NamedReference[] (@experimental) — non-identity assignment column names (root-column granularity for nested field updates), populated by Spark before the connector's operation builder runs, so the connector can size requiredDataAttributes() accordingly.
LogicalWriteInfo.updateSchema(): Optional<StructType> (@evolving) — narrow row schema for writeUpdate()-bound rows; schema() is empty when there are no INSERT-shaped rows (the common UPDATE-only case), and otherwise still carries the full table shape for INSERT-tagged rows.
DataWriter.writeUpdate(record) and writeUpdate(metadata, record) (@evolving) — narrow write channel; default implementations delegate to write(...) so existing connectors are unaffected.

When a connector mixes in SupportsColumnUpdates, Spark's UPDATE (RewriteUpdateTable) narrows both sides of the plan:

Scan side: the physical scan is narrowed to requiredDataAttributes() plus scanOnlyDataAttributes(), plus any columns referenced by the UPDATE condition or non-identity assignment expressions.
Write side: the row handed to writeUpdate()/DeltaWriter is narrowed to requiredDataAttributes() only; scanOnlyDataAttributes() columns stay in the scan for planning but are excluded from the write payload.

Why are the changes needed?

Schema narrowing helps connectors request for only updated columns enabling efficient column-level updates of wide tables.

Does this PR introduce any user-facing change?

Yes, new public DSv2 connector APIs:

RowLevelOperation gains a new mix-in SupportsColumnUpdates (requiredDataAttributes(), scanOnlyDataAttributes())
RowLevelOperationInfo.updatedColumns()
LogicalWriteInfo.updateSchema()
DataWriter.writeUpdate(...)

How was this patch tested?

New tests in:

DeltaBasedColumnUpdateTableSuite
GroupBasedColumnUpdateTableSuite

Was this patch authored or co-authored using generative AI tooling?

I used Claude Code (Opus 5) to generate code and tests and manually reviewed the generated code.

peter-toth · 2026-04-27T16:26:08Z

+ val required =
+ AttributeSet(dataAttrs) ++ AttributeSet(Seq(cond)) ++ AttributeSet(rowIdAttrs)
+ val narrowOutput = relation.output.filter(required.contains)
+ relation.copy(table = table, output = dedupAttrs(narrowOutput ++ rowIdAttrs ++ metadataAttrs))


Can an attribute in required be missing from relation.output?
rowIdAttrs seems to be added 2 times.
If we already have a dedupAttrs() then probably doesn't make sense build AttributeSets.

Can an attribute in required be missing from relation.output?
No. dataAttrs come from the connector's requiredDataAttributes() which are resolved against relation (via V2ExpressionUtils.resolveRefs), so they're guaranteed to be present. The condition's referenced columns are also table columns from the user's WHERE clause. rowIdAttrs and metadataAttrs can be absent from relation.output (they're resolved separately), but they're not part of the filter. They're appended unconditionally afterward via dedupAttrs(narrowOutput ++ rowIdAttrs ++ metadataAttrs)
rowIdAttrs seems to be added 2 times. If we already have dedupAttrs() then probably doesn't make sense to build AttributeSets.
Agreed. I fixed it.

dongjoon-hyun

Could you resolve the conflicts, @anuragmantri ?

anuragmantri · 2026-05-05T23:38:47Z

Could you resolve the conflicts, @anuragmantri ?

Thanks. I rebased and fixed the conflicts.

dongjoon-hyun · 2026-05-06T00:34:42Z

 return new NamedReference[0];
 }
+
+


nit. Remove redundant empty line.

dongjoon-hyun · 2026-05-06T00:35:34Z

+ * including the columns being updated. If {@link #requiredDataAttributes()} returns an empty
+ * array, Spark sends only the non-identity assigned columns (heuristic path).
+ *
+ * @since 4.2.0


4.2.0 -> 4.3.0

dongjoon-hyun · 2026-05-06T00:35:41Z

+ * <p>
+ * When empty (the default), Spark falls back to sending only the non-identity assigned columns.
+ *
+ * @since 4.2.0


ditto. 4.3.0

dongjoon-hyun · 2026-05-06T00:40:28Z

- val table = buildOperationTable(tbl, UPDATE, CaseInsensitiveStringMap.empty())
+ val updatedCols = assignments.collect {
+ case Assignment(key: AttributeReference, value)
+ if !isIdentityAssignment(key, value) =>


One liner doesn't violate the line-length rule, does it?
-caseAssignment(key: AttributeReference, value) -if!isIdentityAssignment(key, value) =>+caseAssignment(key: AttributeReference, value) if!isIdentityAssignment(key, value) =>

dongjoon-hyun · 2026-05-06T00:44:55Z

+ //
+ // When dataAttrs is non-empty, the relation output is narrowed to include only columns
+ // required for a column-update write. When dataAttrs is empty, the full relation.output is
+ // preserved.


For function description, please follow the community style like the other code path.
/** * ... */

dongjoon-hyun · 2026-05-06T00:45:06Z

+ // When the connector supports column updates and declares required data attributes,
+ // the read relation is narrowed at analysis time so that
+ // GroupBasedRowLevelOperationScanPlanning uses only the needed columns for the scan.
+ // Otherwise the full relation output is used.


For function description, please follow the community style like the other code path.
/** * ... */

dongjoon-hyun · 2026-05-06T00:45:15Z

 WriteDelta(writeRelation, cond, rowDeltaPlan, relation, projections, groupFilterCond)
 }

+ // Builds the row delta projection for the column update path.


For function description, please follow the community style like the other code path.
/** * ... */

dongjoon-hyun · 2026-05-06T00:45:28Z

+ dataAttrsResolved(inRowAttrs)
+ }
+
+ // Validates the narrow-write-schema row projection output.


For function description, please follow the community style like the other code path.
/** * ... */

dongjoon-hyun · 2026-05-06T00:46:36Z

- table.skipSchemaResolution || areCompatible(inRowAttrs, outRowAttrs)
+ table.skipSchemaResolution ||
+ areCompatible(inRowAttrs, outRowAttrs) ||
+ dataAttrsResolved(inRowAttrs)


nit. Please minimize the change of existing code as much as possible like the following.
table.skipSchemaResolution || areCompatible(inRowAttrs, outRowAttrs) || dataAttrsResolved(inRowAttrs)

dongjoon-hyun · 2026-05-06T00:51:39Z

+ * is ignored and the full table row is sent (the default behavior).
+ * <p>
+ * When non-empty, the returned columns become the write schema in declared order.
+ * The connector must declare all columns it wants to receive, including the columns being


This is very strong assumption, but it seems that this PR didn't have a protection. May I ask if we have some kind of assertion or a test coverage for this?

Each column the connector returns passes through V2ExpressionUtils.resolveRefs which throws AnalysisException if the column is non existent.
I added a test test("column-update: requiredDataAttributes throws AnalysisException for invalid column")

dongjoon-hyun · 2026-05-06T00:56:47Z

+ //
+ // ColumnPruning observes exactly these references and narrows the physical scan accordingly.
+ // Connectors that need additional columns in the scan (e.g., partition columns for
+ // distribution) should declare them in requiredDataAttributes().


IIUC, for the correctness, we need to throw AnalysisException if requiredDataAttributes is invalid.

Each column the connector returns passes through V2ExpressionUtils.resolveRefs which throws AnalysisException if the column is non existent.
I added a test test("column-update: requiredDataAttributes throws AnalysisException for invalid column")

dongjoon-hyun · 2026-05-06T00:58:39Z

+ // Connectors that need additional columns in the scan (e.g., partition columns for
+ // distribution) should declare them in requiredDataAttributes().
+ //
+ // Note: AlignUpdateAssignments guarantees all assignment keys are top-level


Do we have a test coverage for this, AlignUpdateAssignments contract?

I added a new test test("column-update: nested struct field update narrows to the root struct column") that updates an inner field in a struct, the AlignUpdateAssignment returns only the root key.

dongjoon-hyun · 2026-05-06T00:59:09Z

+ * whether pk is already in the updated columns list and, if not, add it to
+ * requiredDataAttributes().
+ *
+ * @since 4.2.0


dongjoon-hyun · 2026-05-06T01:01:27Z

- // build a plan to replace read groups in the table
 val writeRelation = relation.copy(table = operationTable)
- val projections = buildReplaceDataProjections(query, relation.output, metadataAttrs)
+ val query = updatedAndRemainingRowsPlan


This looks like duplications: Let's use one variable instead of mixing two variables, updatedAndRemainingRowsPlan and query.

Done, used a single variable

dongjoon-hyun · 2026-05-06T01:03:37Z

+ // GroupBasedRowLevelOperationScanPlanning needs explicit column declarations to narrow.
+ val rowAttrs: Seq[Attribute] = if (isNarrow) connectorDataAttrs else relation.output
+
+ (readRelation, rowAttrs)


Please return metadataAttrs too to avoid the following recomputation in the caller-side.
val metadataAttrs = resolveRequiredMetadataAttrs(relation, operationTable.operation)

I changed this to return metadataAttrs too.

dongjoon-hyun · 2026-05-06T01:04:29Z

+ //
+ // Works for both the full-scan and narrow-scan CoW paths. In the narrow case,
+ // readRelation.output is already restricted by buildCoWReadSetup, so projecting
+ // all plan.output gives the correct narrow write schema.


Use function description style.

dongjoon-hyun · 2026-05-06T01:07:04Z

+ *
+ * @since 4.2.0
+ */
+ default boolean supportsColumnUpdates() {


Given the scope of this PR, shall we mention that DELETE and MERGE ignores this method?

dongjoon-hyun · 2026-05-06T01:07:18Z

+ *
+ * @since 4.2.0
+ */
+ default NamedReference[] requiredDataAttributes() {


Given the scope of this PR, shall we mention that DELETE and MERGE ignores this method?

Even though the scope of this PR is UPDATE only, we'd like this API to work for MERGE as well (DELETE doesn't benefit since it doesn't write data columns). I'm still assessing what it takes and will add a section in the SPIP on how it could be implemented.
Happy to add a "currently only consulted for UPDATE" note in the Javadoc for now and remove it when MERGE support lands.

dongjoon-hyun · 2026-05-06T01:08:26Z

I finished the first round review, @anuragmantri .

anuragmantri

Thanks for the review @dongjoon-hyun. I addressed your comments and cleaned up some AI generated comments which were redundant.

anuragmantri · 2026-05-06T17:53:45Z

 return new NamedReference[0];
 }
+
+


anuragmantri · 2026-05-06T17:53:50Z

+ * including the columns being updated. If {@link #requiredDataAttributes()} returns an empty
+ * array, Spark sends only the non-identity assigned columns (heuristic path).
+ *
+ * @since 4.2.0


anuragmantri · 2026-05-06T22:50:19Z

+ * is ignored and the full table row is sent (the default behavior).
+ * <p>
+ * When non-empty, the returned columns become the write schema in declared order.
+ * The connector must declare all columns it wants to receive, including the columns being


Each column the connector returns passes through V2ExpressionUtils.resolveRefs which throws AnalysisException if the column is non existent.
I added a test test("column-update: requiredDataAttributes throws AnalysisException for invalid column")

anuragmantri · 2026-05-06T22:51:55Z

+ * whether pk is already in the updated columns list and, if not, add it to
+ * requiredDataAttributes().
+ *
+ * @since 4.2.0


anuragmantri · 2026-05-06T22:52:05Z

+ //
+ // When dataAttrs is non-empty, the relation output is narrowed to include only columns
+ // required for a column-update write. When dataAttrs is empty, the full relation.output is
+ // preserved.


anuragmantri · 2026-05-06T23:48:12Z

+ // Connectors that need additional columns in the scan (e.g., partition columns for
+ // distribution) should declare them in requiredDataAttributes().
+ //
+ // Note: AlignUpdateAssignments guarantees all assignment keys are top-level


I added a new test test("column-update: nested struct field update narrows to the root struct column") that updates an inner field in a struct, the AlignUpdateAssignment returns only the root key.

anuragmantri · 2026-05-06T23:50:05Z

+ //
+ // ColumnPruning observes exactly these references and narrows the physical scan accordingly.
+ // Connectors that need additional columns in the scan (e.g., partition columns for
+ // distribution) should declare them in requiredDataAttributes().


Each column the connector returns passes through V2ExpressionUtils.resolveRefs which throws AnalysisException if the column is non existent.
I added a test test("column-update: requiredDataAttributes throws AnalysisException for invalid column")

anuragmantri · 2026-05-07T00:29:51Z

+ dataAttrsResolved(inRowAttrs)
+ }
+
+ // Validates the narrow-write-schema row projection output.


anuragmantri · 2026-05-07T00:30:56Z

- table.skipSchemaResolution || areCompatible(inRowAttrs, outRowAttrs)
+ table.skipSchemaResolution ||
+ areCompatible(inRowAttrs, outRowAttrs) ||
+ dataAttrsResolved(inRowAttrs)


anuragmantri · 2026-05-07T01:00:12Z

+ *
+ * @since 4.2.0
+ */
+ default NamedReference[] requiredDataAttributes() {


Even though the scope of this PR is UPDATE only, we'd like this API to work for MERGE as well (DELETE doesn't benefit since it doesn't write data columns). I'm still assessing what it takes and will add a section in the SPIP on how it could be implemented.
Happy to add a "currently only consulted for UPDATE" note in the Javadoc for now and remove it when MERGE support lands.

anuragmantri · 2026-05-07T01:15:18Z

- .getOrElse {
- throw new AnalysisException(
- errorClass = "_LEGACY_ERROR_TEMP_3075",
- messageParameters = Map(
- "tableAttr" -> tableAttr.toString,
- "scanAttrs" -> scanAttrs.mkString(",")))
- }
 }


I believe this is safe because condition-referenced columns are guaranteed to be in the scan. Please correct me if I'm wrong.

No. Unfortunately, this PR should not remove this because the existing sanity check is used for other code path in the existing test cases. Please recover it.
I guess you may achieve your goal via the following. Please review and revise the following example for your purpose.
privatedefbuildTableToScanAttrMap( tableAttrs: Seq[Attribute], scanAttrs: Seq[Attribute], requiredAttrs: AttributeSet):AttributeMap[Attribute] = { // Table attrs may be legitimately absent from a column-update narrowed scan, so map only// those that have a matching scan attribute. Attrs referenced by the condition must always// be present (computeNarrowReadAttrs keeps them in the scan); failing to map one would// leave a dangling reference in the group filter, so keep the strict check for them.valattrMapping= tableAttrs.flatMap { tableAttr =>valmatched= scanAttrs.find(scanAttr => conf.resolver(scanAttr.name, tableAttr.name)) if (matched.isEmpty && requiredAttrs.contains(tableAttr)) { thrownewAnalysisException( errorClass ="_LEGACY_ERROR_TEMP_3075", messageParameters =Map( "tableAttr"-> tableAttr.toString, "scanAttrs"-> scanAttrs.mkString(","))) } matched.map(scanAttr => tableAttr -> scanAttr) } AttributeMap(attrMapping) }

Makes sense. Similar to other changes for column updates paths, I created a conditional method buildNarrowTableToScanAttrMap() which is called only during column updates and throws when any of the condition references are missing. My rationale is that the runtime filtering applies to only the filters so it is sufficient remap the filters only. Let me know if this understanding is incorrect.

dongjoon-hyun

Thank you for updating, @anuragmantri .

BTW, I cannot find the vote for the mentioned SPIP. Does pass the vote officially, @anuragmantri ? For SPIP, we need an official vote result to move forward including merging something, don't we? (cc @huaxingao as the Shepherd of SPARK-56599 JIRA issue)

What changes were proposed in this pull request?
For SPIP: SPARK-56599

cc @aokolnychyi too because RowLevelOperation.java has been never changed since being added 4 years ago via the following.

#36304

anuragmantri · 2026-05-08T13:29:15Z

Thanks for the review @dongjoon-hyun. For the SPIP, we are waiting for a few more maintainers to also review the design as well as the PR before going for a vote.

Addressed.

dongjoon-hyun

I reviewed the latest head (774412c) focusing on points not covered in the earlier rounds. On top of the above Peter's comment, I added a few inline comments below.

One item with no code anchor: the PR description still says the writeUpdate defaults "delegate to write(...) so existing connectors are unaffected", but after the change from the earlier review the defaults now throw DATA_SOURCE_WRITE_UPDATE_NOT_IMPLEMENTED. Since the PR description becomes the merge commit message, please update it to match the code.

dongjoon-hyun · 2026-08-09T01:27:55Z

+ *
+ * @since 4.3.0
+ */
+ default void writeUpdate(T record) throws IOException {


DATA_SOURCE_WRITE_UPDATE_NOT_IMPLEMENTED is defined and thrown here (and at :103), but no test exercises this path. Could you add a test where a connector mixes in SupportsColumnUpdates without overriding writeUpdate, and assert this error condition is raised? That keeps the new error condition covered and guards the dispatch contract.

dongjoon-hyun · 2026-08-09T01:27:55Z

 ],
 "sqlState" : "KD009"
 },
+ "EMPTY_REQUIRED_DATA_ATTRIBUTES" : {


nit. The five new error conditions for the same SupportsColumnUpdates contract use three different prefixes (EMPTY_REQUIRED_DATA_ATTRIBUTES, REQUIRED_DATA_ATTRIBUTES_*, SPLIT_UPDATE_*). Could you consider a common prefix or a single parent condition with sub-conditions so they group together and are easier to discover?

dongjoon-hyun · 2026-08-09T01:27:55Z

+ validatePartitionAttrsDeclared(operation, relation, connectorDataAttrs, scanOnlyDataAttrs)
+ }
+
+


nit. Redundant empty line.
Suggested change

dongjoon-hyun · 2026-08-09T01:27:55Z

+ /**
+ * Variant of `buildReplaceDataUpdateProjection` for the `SupportsColumnUpdates` narrow-scan
+ * path.
+ * For narrow attributes, looks up assignments by `ExprId` via `AttributeMap`and passes through


nit. Missing space.
Suggested change
*For narrow attributes, looks up assignments by `ExprId` via `AttributeMap`and passes through
*For narrow attributes, looks up assignments by `ExprId` via `AttributeMap`and passes through

dongjoon-hyun · 2026-08-09T01:27:55Z

+ val connectorPassThroughValues = connectorDataAttrs.filterNot(a =>
+ assignedKeyIds.contains(a.exprId) || rowIdAttrSet.contains(a))
+
+ // scanOnlyDataAttrs are never assigned carry them through the write query so


nit. Missing punctuation here and at line 341 (write payload) they are emitted once -> write payload); they are emitted once).
Suggested change
// scanOnlyDataAttrs are never assigned carry them through the write query so
// scanOnlyDataAttrs are never assigned; carry them through the write query so

dongjoon-hyun · 2026-08-09T01:27:55Z

+ assert(actualUpdateSchema == expectedUpdateSchema, "update schema must match")
+ }
+
+ protected def getUpdateSummary(): org.apache.spark.sql.connector.write.UpdateSummary = {


nit. Please import org.apache.spark.sql.connector.write.UpdateSummary instead of using the fully-qualified name inline (here and at line 286).

dongjoon-hyun · 2026-08-09T01:27:56Z


 case class DataAndMetadataWritingSparkTask(
 dataProj: ProjectingInternalRow,
+ updateDataProj: ProjectingInternalRow,


nit (optional). A nullable updateDataProj: ProjectingInternalRow constructor parameter (here and in DataWithProjectionWritingSparkTask) is consistent with the existing orNull usage in the delta tasks, but Option[ProjectingInternalRow] would be more idiomatic for a case class constructor. Feel free to keep as is if you prefer the symmetry.

Stale review.

aokolnychyi · 2026-08-13T23:57:56Z

+ *
+ * @since 4.3.0
+ */
+ default void writeUpdate(T metadata, T record) throws IOException {


I am still thinking about the best possible name for this method. What worries me a bit is that it will also pass copied or re-inserted rows. This is different from DeltaWriter where we have these different methods. I am not sure I can offer a better name, though.

I wonder whether even something like writeColumnUpdate would be more descriptive.

Need to think more, no need to update.

aokolnychyi · 2026-08-14T15:27:00Z

+ * Returns the columns being updated by this operation. Currently populated only for UPDATE;
+ * DELETE and MERGE report an empty array.
+ * <p>
+ * Nested struct field updates are reported at root-column granularity


Is this a temp limitation to simplify implementation?

aokolnychyi · 2026-08-14T15:29:31Z

+ *
+ * @since 4.3.0
+ */
+ default Optional<StructType> updateSchema() {


This could technically also become columnUpdateSchema(), if we decide to do the rename. It does indicate the schema of copied rows in CoW and reinserted rows. One thing I don't like is that it is longer.

aokolnychyi · 2026-08-14T15:33:29Z

+ * <p>
+ * Must not overlap with {@link #requiredDataAttributes()}; defaults to an empty array.
+ */
+ default NamedReference[] scanOnlyDataAttributes() {


I am not sure adding this part is justified. If we need these columns for clustering / ordering as the doc says, this means they have to stay with the plan till the very end. If so, I would prefer to simply include all such data columns in requiredDataColumns and do a projection inside the connector, if needed. Iceberg has all utilities for that, for example.
At this point, I am reluctant to add the complexity to Spark for something we are not sure would be needed. It does complicate the API and also the Spark logic that is already very complicated.
We can always reconsider and add this API later.

aokolnychyi · 2026-08-14T17:33:20Z

+ case other => other
+ }
+ unwrapped match {
+ case attr: Attribute => AttributeSet(Seq(key)).contains(attr)


It is a bit odd to see AttributeSet being used here? Can we do this instead?
protected def isIdentityAssignment(key: Attribute, value: Expression): Boolean = { val valueWithoutAlias = value match { case Alias(child, _) => child case other => other } key.semanticEquals(valueWithoutAlias) }

aokolnychyi · 2026-08-14T17:51:29Z

 case r @ ExtractV2Table(tbl: SupportsRowLevelOperations) =>
 checkNoGeneratedColumns(r, UPDATE)
- val table = buildOperationTable(tbl, UPDATE, r.options)
+ val updatedCols = assignments.collect {


Can we add a helper like below?
private def collectUpdatedAttrs(assignments: Seq[Assignment]): Seq[AttributeReference] = { assignments.collect { case Assignment(key: AttributeReference, value) if !isIdentityAssignment(key, value) => key } }
And then pass AttributeReference to buildOperationTable?
protected def buildOperationTable( table: SupportsRowLevelOperations, command: Command, options: CaseInsensitiveStringMap, updatedAttrs: Seq[AttributeReference] = Nil): RowLevelOperationTable = { val updatedColumns = updatedAttrs.map(attr => FieldReference(Seq(attr.name))) ... }
So that RewriteUpdateTable becomes a bit easier. I also see you may use collectUpdatedAttrs in a few other places.

aokolnychyi · 2026-08-14T18:02:12Z

 val operation = operationTable.operation.asInstanceOf[SupportsDelta]

- // resolve all needed attrs (e.g. row ID and any required metadata attrs)
+ // resolve all needed attrs (e.g. row ID, any required metadata attrs and optionally connector


It is unclear what optionally connector declared attrs means in this case. I'd opt to simplify.
// resolve all needed attrs (e.g. row ID and required data / metadata attrs) // resolve all needed attrs (e.g. row ID or any required metadata attrs)
Or similar

aokolnychyi · 2026-08-14T18:07:51Z

+ * implicitly, so a connector that needs them for partitioning resolution or write-side
+ * clustering must declare them in one of the two methods.
+ */
+ private def validatePartitionAttrsDeclared(


Why do we need this? Like the comment says, if the connector needs, it adds them. Why should Spark be concerned about it?

We can't be very restrictive on the Spark side and make assumptions about what the connectors may want to do.

aokolnychyi · 2026-08-14T19:19:57Z

+ } else Nil
+
+ if (supportsColumnUpdate) {
+ validateUpdatedColumnsSubset(operation, assignments, connectorDataAttrs)


Why not validate validateUpdatedColumnsSubset when we resolve the attributes?

aokolnychyi · 2026-08-14T20:50:57Z

- // construct a read relation and include all required metadata columns
- val readRelation = buildRelationWithAttrs(relation, operationTable, metadataAttrs, rowIdAttrs)
+ if (supportsColumnUpdate && operation.representUpdateAsDeleteAndInsert) {
+ validateNoRowIdReassignment(operation, assignments, rowIdAttrs)


I am not sure I understand this validation. Specifically, validateRowIdDeclared seems wrong. What if my row id is a metadata column?

aokolnychyi · 2026-08-14T23:32:07Z

 val matchedRowsPlan = Filter(cond, readRelation)
- val rowDeltaPlan = if (operation.representUpdateAsDeleteAndInsert) {
- buildDeletesAndInserts(matchedRowsPlan, assignments, rowIdAttrs)
+ val rowDeltaPlan = if (supportsColumnUpdate) {


I am worried about these constant if statements. We will need to find a better structure.

holdenk · 2026-09-01T18:23:42Z

Hey @anuragmantri do you have the cycles to update this?

anuragmantri · 2026-09-01T18:42:51Z

Hey @anuragmantri do you have the cycles to update this?

Yes, I will update this. I will be actively working on this PR. @holdenk

…DSv2

- Rename the five SupportsColumnUpdates error conditions under a common COLUMN_UPDATE_ prefix so they sort and are discoverable together, instead of three unrelated prefixes (EMPTY_REQUIRED_DATA_ATTRIBUTES, REQUIRED_DATA_ATTRIBUTES_*, SPLIT_UPDATE_*). - Add test coverage for DataWriter#writeUpdate's default implementation, which throws DATA_SOURCE_WRITE_UPDATE_NOT_IMPLEMENTED when a connector mixes in SupportsColumnUpdates without overriding it. - Fix a handful of doc/comment nits in RewriteUpdateTable.scala (missing space, redundant blank line, missing punctuation) and use an import for UpdateSummary instead of the fully-qualified name.

Finding 21: connector attribute declarations resolve case-insensitively but resolveRefs keeps the declared spelling, and dedupAttrs keys on exprId, so a mis-cased declaration (e.g. PK instead of pk) silently replaced the table's real attribute name in the narrow scan/write schema, breaking column pruning with INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND. Map each resolved attribute back to the relation's own (by exprId) at all three resolution sites: resolveRequiredDataAttrs, resolveScanOnlyDataAttrs, and RowLevelWrite.projectedDataAttrs. Finding 22: SPLIT_UPDATE_ROW_ID_REASSIGNMENT's message advertised two remedies -- avoid reassigning row ID columns, or declare every table column in requiredDataAttributes() -- but validateNoRowIdReassignment only implemented the first. Skip the check when the declaration covers every column in the relation, since the REINSERT payload is then the full row with the new row-ID value and the DELETE half still carries the original row-ID via newLazyRowIdProjection, making reassignment safe. Written test-first: both fixtures/tests were confirmed to fail against the unmodified code before implementing each fix.

anuragmantri force-pushed the dsv2-required-data-attrs branch from fb14c34 to ae635f4Compare April 23, 2026 20:51

peter-toth reviewed Apr 27, 2026
View reviewed changes

dongjoon-hyun reviewed May 5, 2026
View reviewed changes

anuragmantri force-pushed the dsv2-required-data-attrs branch from ae635f4 to a99bb2dCompare May 5, 2026 23:32

dongjoon-hyun reviewed May 6, 2026
View reviewed changes

dongjoon-hyun previously requested changes May 6, 2026
View reviewed changes

dongjoon-hyun reviewed May 6, 2026
View reviewed changes

anuragmantri commented May 7, 2026
View reviewed changes

dongjoon-hyun reviewed May 8, 2026
View reviewed changes

anuragmantri force-pushed the dsv2-required-data-attrs branch from e806004 to 4060cbfCompare May 29, 2026 06:44

dongjoon-hyun reviewed Aug 9, 2026
View reviewed changes

aokolnychyi reviewed Aug 13, 2026
View reviewed changes

aokolnychyi reviewed Aug 14, 2026
View reviewed changes

anuragmantriand others added 14 commits September 1, 2026 12:37

[SPARK-58111][SQL] Write schema narrowing for column-level UPDATE in …
879f3c8
…DSv2

Add new replace data narrow projection
5a7aba5

Add tests for nested column updates
b56f0bd

Update comments
db5702a

Add mima excludes
52efba7

Scala fmt
ffc3940

Address review comments
fc30ea1

Rebase with master and minor changes
43adda2

Address review comments except finding 9
e71eed4

Add scanOnlyDataAttributes() API
ce65ffd

Address findings 7, 15, 16, 17, 18
6bf3f50

Address Anton's review comments
2466fca

anuragmantri force-pushed the dsv2-required-data-attrs branch from 774412c to 2466fcaCompare September 1, 2026 20:01

		validatePartitionAttrsDeclared(operation, relation, connectorDataAttrs, scanOnlyDataAttrs)
		}

	*For narrow attributes, looks up assignments by `ExprId` via `AttributeMap`and passes through
	*For narrow attributes, looks up assignments by `ExprId` via `AttributeMap`and passes through

	// scanOnlyDataAttrs are never assigned carry them through the write query so
	// scanOnlyDataAttrs are never assigned; carry them through the write query so

Uh oh!

Conversation

anuragmantri commented Apr 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

anuragmantri commented May 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunMay 6, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anuragmantri commented Apr 23, 2026•
edited
Loading

dongjoon-hyunMay 6, 2026•
edited
Loading

anuragmantri left a comment •
edited
Loading