Uh oh!
There was an error while loading. Please reload this page.
Spark: Backport view schema binding mode to Spark 3.5 and 4.0 - #17838
Conversation
bryanck
left a comment
There was a problem hiding this comment.
LGTM matches the original PR for Spark 4.1.
| val cast = if (compensate) { | ||
| Cast(attr, expected.dataType, ansiEnabled = true) | ||
| } else { | ||
| UpCast(attr, expected.dataType) | ||
| } | ||
| Alias(cast, expected.name)(explicitMetadata = Some(expected.metadata)) |
There was a problem hiding this comment.
[doubt] why didn't we add the full view schema compatibilty which spark supports ?
https://github.com/apache/spark/pull/46267/changes#diff-9dd0899e5406230aeff96654432da54f35255f6dc60eecb87264a5c508a8c826R930
There was a problem hiding this comment.
Type evolution has much bigger scope, so the thought here is to start with compensation mode and follow up with additional support as needed.
There was a problem hiding this comment.
+1 to Bryan's comment. This was left for follow on work, since type evolution would require overwriting the stored schema as well
Uh oh!
There was an error while loading. Please reload this page.
Backports
spark.sql.iceberg.view.schema-binding-modeto Spark 3.5 and 4.0. The change is identical to the Spark 4.1 version in #17499.BINDINGUpCast(col, storedType)— the current behaviour, and the defaultCOMPENSATIONCast(col, storedType, ansiEnabled = true), so a narrowing type change resolves to the stored typeThe docs entry is in #17499, since it isn't version-specific.