Skip to content

[SPARK-48031][SQL] Support view schema evolution #46267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
srielau wants to merge 1 commit into from
Closed

[SPARK-48031][SQL] Support view schema evolution #46267

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions common/utils/src/main/resources/error/error-conditions.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1339,6 +1339,12 @@
],
"sqlState" : "54001"
},
"FEATURE_NOT_ENABLED" : {
"message" : [
"The feature <featureName> is not enabled. Consider setting the config <configKey> to <configValue> to enable this capability."
],
"sqlState" : "56038"
},
"FIELD_ALREADY_EXISTS" : {
"message" : [
"Cannot <op> column, because <fieldNames> already exists in <struct>."
Expand Down Expand Up @@ -4495,6 +4501,11 @@
"Table <tableName> does not support <operation>. Please check the current catalog and namespace to make sure the qualified table name is expected, and also check the catalog implementation which is configured by \"spark.sql.catalog\"."
]
},
"TEMPORARY_VIEW_WITH_SCHEMA_BINDING_MODE" : {
"message" : [
"Temporary views cannot be created with the WITH SCHEMA clause. Recreate the temporary view when the underlying schema changes, or use a persisted view."
]
},
"TIME_TRAVEL" : {
"message" : [
"Time travel on the relation: <relationId>."
Expand Down
2 changes: 1 addition & 1 deletion common/utils/src/main/resources/error/error-states.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4572,7 +4572,7 @@
"usedBy": ["Spark"]
},
"42K0E": {
"description": "An expression is not valid in teh context it is used",
"description": "An expression is not valid in the context it is used",
"origin": "Spark",
"standard": "N",
"usedBy": ["Spark"]
Expand Down
1 change: 1 addition & 0 deletions docs/sql-migration-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ license: |
- Since Spark 4.0, The default value for `spark.sql.legacy.ctePrecedencePolicy` has been changed from `EXCEPTION` to `CORRECTED`. Instead of raising an error, inner CTE definitions take precedence over outer definitions.
- Since Spark 4.0, The default value for `spark.sql.legacy.timeParserPolicy` has been changed from `EXCEPTION` to `CORRECTED`. Instead of raising an `INCONSISTENT_BEHAVIOR_CROSS_VERSION` error, `CANNOT_PARSE_TIMESTAMP` will be raised if ANSI mode is enable. `NULL` will be returned if ANSI mode is disabled. See [Datetime Patterns for Formatting and Parsing](sql-ref-datetime-pattern.html).
- Since Spark 4.0, A bug falsely allowing `!` instead of `NOT` when `!` is not a prefix operator has been fixed. Clauses such as `expr ! IN (...)`, `expr ! BETWEEN ...`, or `col ! NULL` now raise syntax errors. To restore the previous behavior, set `spark.sql.legacy.bangEqualsNot` to `true`.
- Since Spark 4.0, Views allow control over how they react to underlying query changes. By default views tolerate column type changes in the query and compensate with casts. To restore the previous behavior, allowing up-casts only, set `spark.sql.viewSchemaBindingMode` to `DISABLED`. This disables the feature and also disallows the `WITH SCHEMA` clause.

## Upgrading from Spark SQL 3.5.1 to 3.5.2

Expand Down
2 changes: 2 additions & 0 deletions docs/sql-ref-ansi-compliance.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,6 +416,7 @@ Below is a list of all the keywords in Spark SQL.
|BETWEEN|non-reserved|non-reserved|reserved|
|BIGINT|non-reserved|non-reserved|reserved|
|BINARY|non-reserved|non-reserved|reserved|
|BINDING|non-reserved|non-reserved|non-reserved|
|BOOLEAN|non-reserved|non-reserved|reserved|
|BOTH|reserved|non-reserved|reserved|
|BUCKET|non-reserved|non-reserved|non-reserved|
Expand Down Expand Up @@ -445,6 +446,7 @@ Below is a list of all the keywords in Spark SQL.
|COMMIT|non-reserved|non-reserved|reserved|
|COMPACT|non-reserved|non-reserved|non-reserved|
|COMPACTIONS|non-reserved|non-reserved|non-reserved|
|COMPENSATION|non-reserved|non-reserved|non-reserved|
|COMPUTE|non-reserved|non-reserved|non-reserved|
|CONCATENATE|non-reserved|non-reserved|non-reserved|
|CONSTRAINT|reserved|non-reserved|reserved|
Expand Down
44 changes: 44 additions & 0 deletions docs/sql-ref-syntax-ddl-alter-view.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,32 @@ Note that `ALTER VIEW` statement does not support `SET SERDE` or `SET SERDEPROPE

Specifies the definition of the view. Check [select_statement](sql-ref-syntax-qry-select.html) for details.

#### ALTER View WITH SCHEMA

Changes the view's schema binding behavior.

If the view is cached, the command clears cached data of the view and all its dependents that refer to it. View's cache will be lazily filled when the next time the view is accessed. The command leaves view's dependents as uncached.

This statement is not supported for `TEMPORARY` views.

#### Syntax
```sql
ALTER VIEW view_identifier WITH SCHEMA { BINDING | COMPENSATION | [ TYPE ] EVOLUTION }
```

#### Parameters
* **view_identifier**

Specifies a view name, which may be optionally qualified with a database name.

**Syntax:** `[ database_name. ] view_name`

* **BINDING** - The view can tolerate only type changes in the underlying schema requiring safe up-casts.
* **COMPENSATION** - The view can tolerate type changes in the underlying schema requiring casts. Runtime casting errors may occur.
* **TYPE EVOLUTION** - The view will adapt to any type changes in the underlying schema.
* **EVOLUTION** - For views defined without a column lists any schema changes are adapted by the view, including, for queries with `SELECT *` dropped or added columns.
If the view is defined with a column list, the clause is interpreted as `TYPE EVOLUTION`.

### Examples

```sql
Expand Down Expand Up @@ -196,6 +222,24 @@ DESC TABLE EXTENDED tempdb1.v2;
| View Text| select * from tempdb1.v1| |
| View Original Text| select * from tempdb1.v1| |
+----------------------------+---------------------------+-------+

CREATE OR REPLACE VIEW open_orders AS SELECT * FROM orders WHERE status = 'open';
ALTER VIEW open_orders WITH SCHEMA EVOLUTION;
DESC TABLE EXTENDED open_orders;
+----------------------------+---------------------------+-------+
| col_name| data_type|comment|
+----------------------------+---------------------------+-------+
| order_no| int| null|
| order_date| date| null|
| | | |
|# Detailed Table Information| | |
| Database| mydb| |
| Table| open_orders| |
| Type| VIEW| |
| View Text| select * from orders| |
| View Original Text| select * from orders| |
| View Schema Mode | EVOLUTION| |
+----------------------------+---------------------------+-------+
```

### Related Statements
Expand Down
15 changes: 15 additions & 0 deletions docs/sql-ref-syntax-ddl-create-view.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,17 @@ CREATE [ OR REPLACE ] [ [ GLOBAL ] TEMPORARY ] VIEW [ IF NOT EXISTS ] view_ident
* `[ ( column_name [ COMMENT column_comment ], ... ) ]` to specify column-level comments.
* `[ COMMENT view_comment ]` to specify view-level comments.
* `[ TBLPROPERTIES ( property_name = property_value [ , ... ] ) ]` to add metadata key-value pairs.
* `[ WITH SCHEMA { BINDING | COMPENSATION | [ TYPE ] EVOLUTION } ]` to specify how the view reacts to schema changes

This clause is not supported for `TEMPORARY` views.
Comment thread
srielau marked this conversation as resolved.
Outdated

* **BINDING** - The view can tolerate only type changes in the underlying schema requiring safe up-casts.
* **COMPENSATION** - The view can tolerate type changes in the underlying schema requiring casts. Runtime casting errors may occur.
* **TYPE EVOLUTION** - The view will adapt to any type changes in the underlying schema.
* **EVOLUTION** - For views defined without a column lists any schema changes are adapted by the view, including, for queries with `SELECT *` dropped or added columns.
If the view is defined with a column list, the clause is interpreted as `TYPE EVOLUTION`.

The default is `WITH SCHEMA COMPENSATION`.

* **query**
A [SELECT](sql-ref-syntax-qry-select.html) statement that constructs the view from base tables or other views.
Expand All @@ -80,6 +91,10 @@ CREATE GLOBAL TEMPORARY VIEW IF NOT EXISTS subscribed_movies
AS SELECT mo.member_id, mb.full_name, mo.movie_title
FROM movies AS mo INNER JOIN members AS mb
ON mo.member_id = mb.id;

-- Create a view filtering the `orders` table which will adjust to schema changes in `orders`.
CREATE OR REPLACE VIEW open_orders WITH SCHEMA EVOLUTION
AS SELECT * FROM orders WHERE status = 'open';
```

### Related Statements
Expand Down
2 changes: 2 additions & 0 deletions sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseLexer.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ AUTHORIZATION: 'AUTHORIZATION';
BETWEEN: 'BETWEEN';
BIGINT: 'BIGINT';
BINARY: 'BINARY';
BINDING: 'BINDING';
BOOLEAN: 'BOOLEAN';
BOTH: 'BOTH';
BUCKET: 'BUCKET';
Expand Down Expand Up @@ -137,6 +138,7 @@ COMMENT: 'COMMENT';
COMMIT: 'COMMIT';
COMPACT: 'COMPACT';
COMPACTIONS: 'COMPACTIONS';
COMPENSATION: 'COMPENSATION';
COMPUTE: 'COMPUTE';
CONCATENATE: 'CONCATENATE';
CONSTRAINT: 'CONSTRAINT';
Expand Down
10 changes: 10 additions & 0 deletions sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,13 +156,15 @@ statement
VIEW (IF errorCapturingNot EXISTS)? identifierReference
identifierCommentList?
(commentSpec |
schemaBinding |
(PARTITIONED ON identifierList) |
(TBLPROPERTIES propertyList))*
AS query #createView
| CREATE (OR REPLACE)? GLOBAL? TEMPORARY VIEW
tableIdentifier (LEFT_PAREN colTypeList RIGHT_PAREN)? tableProvider
(OPTIONS propertyList)? #createTempViewUsing
| ALTER VIEW identifierReference AS? query #alterViewQuery
| ALTER VIEW identifierReference schemaBinding #alterViewSchemaBinding
| CREATE (OR REPLACE)? TEMPORARY? FUNCTION (IF errorCapturingNot EXISTS)?
identifierReference AS className=stringLit
(USING resource (COMMA resource)*)? #createFunction
Expand Down Expand Up @@ -342,6 +344,10 @@ locationSpec
: LOCATION stringLit
;

schemaBinding
: WITH SCHEMA (BINDING | COMPENSATION | EVOLUTION | TYPE EVOLUTION)
;

commentSpec
: COMMENT stringLit
;
Expand Down Expand Up @@ -1350,6 +1356,7 @@ ansiNonReserved
| BIGINT
| BINARY
| BINARY_HEX
| BINDING
| BOOLEAN
| BUCKET
| BUCKETS
Expand All @@ -1372,6 +1379,7 @@ ansiNonReserved
| COMMIT
| COMPACT
| COMPACTIONS
| COMPENSATION
| COMPUTE
| CONCATENATE
| COST
Expand Down Expand Up @@ -1650,6 +1658,7 @@ nonReserved
| BIGINT
| BINARY
| BINARY_HEX
| BINDING
| BOOLEAN
| BOTH
| BUCKET
Expand Down Expand Up @@ -1679,6 +1688,7 @@ nonReserved
| COMMIT
| COMPACT
| COMPACTIONS
| COMPENSATION
| COMPUTE
| CONCATENATE
| CONSTRAINT
Expand Down
6 changes: 6 additions & 0 deletions sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -546,6 +546,12 @@ private[sql] object QueryParsingErrors extends DataTypeErrorsBase {
new ParseException(errorClass = "_LEGACY_ERROR_TEMP_0052", ctx)
}

def temporaryViewWithSchemaBindingMode(ctx: StatementContext): Throwable = {
new ParseException(errorClass = "UNSUPPORTED_FEATURE.TEMPORARY_VIEW_WITH_SCHEMA_BINDING_MODE",
messageParameters = Map.empty,
ctx)
}

def parameterMarkerNotAllowed(statement: String, origin: Origin): Throwable = {
new ParseException(
command = origin.sqlText,
Expand Down
48 changes: 48 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/view.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,54 @@ object EliminateView extends Rule[LogicalPlan] with CastSupport {
}
}

/**
* ViewBindingMode is used to specify the expected schema binding mode when we want to create or
* replace a view in [[CreateViewStatement]].
*/
sealed trait ViewSchemaMode

/**
* SchemaBinding means the view only tolerates minimal changes to the underlying schema.
* It can tolerate extra columns in SELECT * and upcast to more generic types.
*/
object SchemaBinding extends ViewSchemaMode {
override val toString: String = "BINDING"
}

/**
* SchemaCompensation means the view only tolerates moderate changes to the underlying schema.
* It can tolerate extra columns in SELECT * and explicit casts between view body and view columns.
*/
object SchemaCompensation extends ViewSchemaMode {
override val toString: String = "COMPENSATION"
}

/**
* SchemaTypeEvolution means the view will adopt changed column types.
* In this mode the view will refresh its metastore data on reference to keep it up to day.
*/
object SchemaTypeEvolution extends ViewSchemaMode {
override val toString: String = "TYPE EVOLUTION"
}

/**
* SchemaUnsupported means the feature is not enabled.
* This mode is only transient and not persisted
*/
object SchemaUnsupported extends ViewSchemaMode {
override val toString: String = "UNSUPPORTED"
}

/**
* SchemaEvolution means the view will adopt changed column types and number of columns.
* This is a result of not having a column list and WITH EVOLUTION.
* Without an explicit column list the will also adopt changes to column names.
* In this mode the view will refresh its metastore data on reference to keep it up to day.
*/
object SchemaEvolution extends ViewSchemaMode {
override val toString: String = "EVOLUTION"
}

/**
* ViewType is used to specify the expected view type when we want to create or replace a view in
* [[CreateViewStatement]].
Expand Down
Loading