Spark: Make the view stored-schema cast configurable by bmorck · Pull Request #17499 · apache/iceberg

bmorck · 2026-08-03T18:35:09Z

This change adds a new spark.sql.iceberg.view.schema-binding-mode property, used in ResolveViews in Spark 4.1. Prior to this property, ResolveViews always wraps each output column in an UpCast that only does widening type coercion. So if the stored schema of the view has a narrower type than what the SQL actually produces, we get an error like the following:

[CANNOT_UP_CAST_DATATYPE] Cannot up cast id from "DOUBLE" to "BIGINT"

The new spark.sql.iceberg.view.schema-binding-mode introduces 2 modes:

value	coercion
`BINDING`	`UpCast(col, storedType)` — the current behaviour, and the default
`COMPENSATION`	`Cast(col, storedType, ansiEnabled = true)`, so a narrowing type change resolves to the stored type

Both names are taken from Spark's ViewSchemaMode, which applies these same two coercions to v1 views. BINDING remains the default because COMPENSATION can lose precision, and can fail at runtime on overflow.

Spark has the spark.sql.legacy.viewSchemaBindingMode and spark.sql.legacy.viewSchemaCompensation confs to relax this behavior on the v1 SessionCatalog view path, but these don't apply to v2 views.

This change was adapted from: #17453

ResolveViews rebuilds a view's output from the stored schema, by position, wrapping each column in an UpCast. UpCast only widens, so a view whose stored type is narrower than what its SQL produces cannot be read at all, and there is no way to relax it. Add spark.sql.iceberg.view.schema-binding-mode, taking its mode names and coercions from Spark's ViewSchemaMode: BINDING (UpCast, the default and current behaviour), COMPENSATION (an ANSI cast, allowing narrowing) and TYPE_EVOLUTION (no cast, so the view reports the types its SQL produces). All three keep the stored column name and metadata. When the conf is unset, Spark's spark.sql.legacy.viewSchemaBindingMode and viewSchemaCompensation are honored instead, reproducing how SessionCatalog.castColToType treats SchemaUnsupported.

bryanck · 2026-08-11T00:42:10Z

+ // COMPENSATION permits any ANSI cast, which can truncate values or fail at runtime.
+ // TYPE_EVOLUTION applies no cast, so the view reports the types its SQL produces.
+ // When unset, Spark's spark.sql.legacy.viewSchemaBindingMode and
+ // spark.sql.legacy.viewSchemaCompensation are honored instead; neither can select TYPE_EVOLUTION.


Consider breaking this comment up and moving the mode comments next to the constant below

bryanck · 2026-08-11T00:49:49Z

+ // VIEW_SCHEMA_COMPENSATION. Referenced by name because they were added in Spark 4.0 and this
+ // rule is also compiled against Spark 3.5. Both default to true.
+ private val sparkViewSchemaBindingMode = "spark.sql.legacy.viewSchemaBindingMode"
+ private val sparkViewSchemaCompensation = "spark.sql.legacy.viewSchemaCompensation"


These are available in Spark 4.0 and 4.1, can we remove them here and only add them for 3.5?

Ended up removing these from the diff, see my reply below

bryanck · 2026-08-11T00:54:59Z

+ // Mirror SessionCatalog.castColToType: turning binding mode off selects SchemaUnsupported,
+ // which compensates with an ANSI cast unless compensation is turned off as well. Neither conf
+ // can select TYPE_EVOLUTION: in Spark that mode is requested per view, with
+ // CREATE or ALTER VIEW ... WITH SCHEMA TYPE EVOLUTION, and stored on the view itself.


Some of the comments are pretty verbose, are there opportunities to make them more concise?

bryanck · 2026-08-11T00:56:12Z

+ // When unset, Spark's spark.sql.legacy.viewSchemaBindingMode and
+ // spark.sql.legacy.viewSchemaCompensation are honored instead; neither can select TYPE_EVOLUTION.
+ public static final String VIEW_SCHEMA_BINDING_MODE =
+ "spark.sql.iceberg.view.schema-binding-mode";


This property should be added to the docs.

Also, it seems this feature is global to the session, is there value in having it per view?

Also, it seems this feature is global to the session, is there value in having it per view?
Do you mean something like spark.sql.iceberg.view.<qualifiedVIewName>.schema-binding-mode? If so I'm thinking perhaps we do that in a follow-up PR.
I think it is useful to have a conf that applies to the session globally. This allows for different default behavior, for example without knowing which views are being used by a particular job.

bryanck · 2026-08-11T01:00:17Z

+ // CREATE or ALTER VIEW ... WITH SCHEMA TYPE EVOLUTION, and stored on the view itself.
+ if (isExplicitlyFalse(sparkViewSchemaBindingMode) &&
+ !isExplicitlyFalse(sparkViewSchemaCompensation)) {
+ SparkSQLProperties.VIEW_SCHEMA_MODE_COMPENSATION


Will this be backwards compatible if someone had previously set spark.sql.legacy.viewSchemaBindingMode=false?

Thanks for catching this. This wouldn't be backwards compatible since this would override the default spark.sql.iceberg.view.schema-binding-mode=BINDING, and spark.sql.legacy.viewSchemaBindingMode currently has no effect on V2 views. So if someone had been using spark.sql.legacy.viewSchemaBindingMode for their v1 session catalog views in spark and was expecting it to have no impact on their v2 views, this would break that assumption.
After thinking a bit more about this, I think we should remove spark.sql.legacy.viewSchemaBindingMode and spark.sql.legacy.viewSchemaCompensation since they were originally intended only for Spark's V1 session catalog views and this would overload their original intent.
Went ahead and removed them

bryanck · 2026-08-11T01:05:37Z

+ if (mode == SparkSQLProperties.VIEW_SCHEMA_MODE_COMPENSATION) {
+ Cast(attr, expected.dataType, ansiEnabled = true)
+ } else if (mode == SparkSQLProperties.VIEW_SCHEMA_MODE_TYPE_EVOLUTION) {
+ attr


If we remove UpCast will we still be checking Iceberg field IDs? e.g. if a field was dropped and added back with the same name.

Another thing, will the type being reported (say with describe view) mismatch what is returned?

I did a deep dive with claude and it doesn't look like field IDs are consulted on this path as is. So this shouldn't change that. If a column is dropped and re-added under the same name, the view's SQL resolves against the table by name so it would bind to the new column before and after this change.
Looks like the relevant path is:
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java#L80-L86
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkSchemaUtil.java#L96-L98
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java#L76-L77
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java#L197-L202
SparkView.schema() (80) calls SparkSchemaUtil.convert (96), which visits with TypeToSparkType, where each field's metadata comes from fieldMetadata(field.fieldId()) (76) — and that returns Metadata.empty() for anything that isn't a metadata column

On your other question, when we enable TYPE_EVOLUTION mode that will create a mismatch between what is returned and what the describe view has. For the default BINDING mode this wouldn't be the case
Let me know what you think. I'm thinking that since TYPE_EVOLUTION is not the default and is up to the user's discretion to set, it may not be an issue if the type reported from the schema mismatches. The user in this case would set TYPE_EVOLUTION likely understanding that it will create drift here. And that this mode was the primary motivation for this change

The existing TYPE EVOLUTION feature in Spark's V1 Session catalog views also can create the same drift between stored schema and output type, so there is precedent at least. But open to dissent here

My understanding was that the evolution option would actually evolve the type once a mismatch is detected, and store it via the catalog.

Perhaps we name "evolution" to something else?

bryanck · 2026-08-11T01:12:56Z

+ * Read on every resolution rather than cached, so that SET takes effect within a session.
+ */
+ private def viewSchemaMode: String = {
+ spark.conf.getOption(SparkSQLProperties.VIEW_SCHEMA_BINDING_MODE) match {


I think we want to use the rule's conf instead of reading the session conf?

Falling back to spark.sql.legacy.viewSchemaBindingMode and viewSchemaCompensation would change behaviour on upgrade for anyone who had set them. Iceberg views are strict today regardless of those confs, and the fallback would silently switch them to an ANSI cast. Those confs are also internal and are documented as controlling the WITH SCHEMA clause for view DDL, which Iceberg views do not implement; relaxing the cast is a side effect of disabling that feature rather than the confs' stated purpose. The Iceberg conf now defaults to BINDING when unset, matching current behaviour exactly. This removes the two conf names, the fallback branch, the isExplicitlyFalse helper and the tests that covered them.

Drops the SessionCatalog.castColToType cross-reference, which is in the PR description, and a scaladoc line that restated the method name. Also drops the note about Preconditions and Scala 2.12: that applies to spark/v3.5, which is cross-built against 2.12, not to this file.

Rule extends SQLConfHelper, so conf is the settings the surrounding analysis is running under. getConfString also takes the default, which removes the separate fallback branch.

…figuration table

COMPENSATION was the only mode that returned different data, silently, for every view in the session. NO_CAST returns the types the view SQL produces, and does not evolve the stored schema, so the Spark name did not fit.

# Conflicts: # docs/docs/spark-configuration.md

bryanck · 2026-08-24T20:35:32Z

For posterity, a view's stored schema can drift away from its underlying sources when the source data types change. For example, say a view selects from a table with an int column. Then the table is dropped and recreated to make that column a double. The stored schema in the view will still be int and now will throw a error attempting to cast the double to an int.

Another case is if a view is created in an engine that resolves data types differently. Say a view is created in Trino with a column that divides two bigints, which in Trino is a bigint. In Spark dividing two bigints is a double, so the view schema will not match the query result.

bryanck · 2026-08-24T21:43:21Z

- Some(expected.metadata))
+ val coerced =
+ if (mode == SparkSQLProperties.VIEW_SCHEMA_MODE_COMPENSATION) {
+ Cast(attr, expected.dataType, ansiEnabled = true)


Thinking this through a little bit more, I feel we should keep "compensation", as it aligns with Spark 4's default behavior if I understand it right. And remove the "no cast" option until we have true evolution.

bryanck · 2026-08-24T22:16:35Z

In sum, we have 3+ options for view data types drifting from source. First is upcasting to the view schema (current Iceberg behavior). Second is casting to the view schema, and allowing any compatible cast including downcasts (the default for Spark 4 views IIUC). Third is rely on the source data types and evolve the view schema to match as needed. A variation of #3 is to rely on the source and do not update anything, i.e. ignore the view data types.

Here I feel we should support the first 2 options, and consider the 3rd (with metadata update) as a follow-up, as that has more considerations like calling the catalog.

Replace the NO_CAST mode with COMPENSATION, which applies an ANSI cast to the view's stored type. Both mode names now match Spark's ViewSchemaMode.

One minor nit

jzhuge · 2026-08-26T22:33:16Z

COMPENSATION brings over Spark's cast but not the check Spark pairs it with. Spark's v1 view path resolves columns by name and ordinal (GetViewColumnByNameAndOrdinal); ResolveViews resolves purely by position. Today UpCast incidentally catches column-order drift and fails loudly — an ANSI cast won't, so two mutually-castable columns can quietly swap values. Not a blocker. Either add a name check under COMPENSATION, or add doc/comment that this mode trusts column position?

jzhuge · 2026-08-26T22:33:23Z

Could you add tests for the two behaviors the config comment warns about — silent truncation (1.5 -> 1) and the runtime failure on overflow/NaN? The current test only covers 1.0/2.0/3.0, where neither happens.

jzhuge

Non-blocking comments

bryanck · 2026-08-27T15:40:26Z

LGTM, thanks for the contribution @bmorck !

jzhuge · 2026-08-27T16:48:13Z

 | spark.sql.iceberg.executor-cache.max-total-size | 134217728 (128MB) | Max total executor cache size (bytes) |
 | spark.sql.iceberg.executor-cache.locality.enabled | false | Enables locality-aware executor cache usage |
 | spark.sql.iceberg.merge-schema | false | Enables modifying the table schema to match the write schema. Only adds missing columns |
+| spark.sql.iceberg.view.schema-binding-mode | BINDING | Coercion applied to view columns: `BINDING` (widening only), `COMPENSATION` (any ANSI cast) |


nit:
`COMPENSATION` (Permits any ANSI cast, which can truncate values or fail at runtime)

github-actionsBot added the spark label Aug 3, 2026

bmorck marked this pull request as ready for review August 3, 2026 18:51

bmorck changed the title ~~Spark 4.1: Make the view stored-schema coercion configurable~~Spark: Make the view stored-schema coercion configurableAug 3, 2026

bryanck reviewed Aug 11, 2026
View reviewed changes

Bobby Morck added 5 commits August 10, 2026 23:35

Spark 4.1: Read the schema-binding mode from the rule's conf
7fdebb8
Rule extends SQLConfHelper, so conf is the settings the surrounding analysis is running under. getConfString also takes the default, which removes the separate fallback branch.

Spark 4.1: Move the mode comments next to their constants
361996b

Docs: Add spark.sql.iceberg.view.schema-binding-mode to the Spark con…
2388e74
…figuration table

github-actionsBot added the docs label Aug 11, 2026

bmorck requested a review from bryanck August 11, 2026 05:04

Bobby Morck added 2 commits August 24, 2026 16:04

Spark 4.1: Remove COMPENSATION and rename TYPE_EVOLUTION to NO_CAST
ac96558
COMPENSATION was the only mode that returned different data, silently, for every view in the session. NO_CAST returns the types the view SQL produces, and does not evolve the stored schema, so the Spark name did not fit.

Merge remote-tracking branch 'origin/main' into merge-main-41
f9da6fe
# Conflicts: # docs/docs/spark-configuration.md

bryanck reviewed Aug 24, 2026
View reviewed changes

Bobby Morck added 2 commits August 26, 2026 16:46

Spark 4.1: use COMPENSATION instead of NO_CAST
d646886
Replace the NO_CAST mode with COMPENSATION, which applies an ANSI cast to the view's stored type. Both mode names now match Spark's ViewSchemaMode.

Add braces to satisfy scalastyle
b0e9a6c

bryanck reviewed Aug 26, 2026
View reviewed changes

Comment thread....1/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveViews.scala Outdated

bryanck previously approved these changes Aug 26, 2026
View reviewed changes

Rename coerced to cast
309b68f

bryanck approved these changes Aug 26, 2026
View reviewed changes

bmorck mentioned this pull request Aug 26, 2026
Spark: Backport view schema binding mode to Spark 3.5 and 4.0 #17838
Merged

jzhuge approved these changes Aug 26, 2026
View reviewed changes

bryanck changed the title ~~Spark: Make the view stored-schema coercion configurable~~Spark: Make the view stored-schema cast configurableAug 27, 2026

bryanck merged commit 16e84cb into apache:mainAug 27, 2026
35 checks passed

jzhuge reviewed Aug 27, 2026
View reviewed changes

Uh oh!

Conversation

bmorck commented Aug 3, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmorckAug 11, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryanckAug 24, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryanckAug 24, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryanck commented Aug 24, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryanck commented Aug 24, 2026

Uh oh!

Uh oh!

jzhuge commented Aug 26, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jzhuge commented Aug 26, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jzhuge left a comment

Choose a reason for hiding this comment

Uh oh!

bryanck commented Aug 27, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bmorck commented Aug 3, 2026•
edited
Loading

bmorckAug 11, 2026•
edited
Loading

bryanckAug 24, 2026•
edited
Loading

bryanckAug 24, 2026•
edited
Loading

bryanck commented Aug 24, 2026•
edited
Loading

jzhuge commented Aug 26, 2026•
edited
Loading

jzhuge commented Aug 26, 2026•
edited
Loading