[SPARK-48031][SQL] Support view schema evolution

[srielau]

What changes were proposed in this pull request?

Add the following syntax to CREATE VIEW... WITH SCHEMA ...:

CREATE [ OR REPLACE ] [ TEMPORARY ] VIEW [ IF NOT EXISTS ] view_name
    [ column_list ]
    [ schema_binding ] 
    [ COMMENT view_comment ]
    [ TBLPROPERTIES clause ]
    AS query

schema_binding
   WITH SCHEMA { BINDING | [TYPE] EVOLUTION | COMPENSATION }

column_list
   ( { column_alias [ COMMENT column_comment ] } [, ...] )

Allow changing of the schema binding in:

ALTER VIEW view_name  schema_binding

The semantic is:

Schema_binding
Optionally specifies how the view adapts to changes to the schema of the query due to changes in the underlying object definitions.
This clause is not supported for temporary views.

• BINDING
  The view will become invalid if the query column-list changes except for the following conditions:
  The column-list includes a star clause, and there are additional columns. These additional columns are ignored.
  The type of one or more columns changed in a way that allows them to be safely cast to the types using implicit casting rules.

• COMPENSATION
  The view will become invalid if the query column list changes except for the following conditions:
  The column-list includes a star clause, and there are additional columns. These additional columns are ignored.
  The type of one or more columns changed in a way that allows them to be cast using explicit cast rules.
  This is the default behavior.

• TYPE EVOLUTION
  The view will adopt any changes to types in the query column list into is own definition when such a change is detected upon reference of the view.

• EVOLUTION
  Behaves like TYPE EVOLUTION and also adopts changes in column names or added and dropped columns if the view does not include an explicit column list.
  The view will only be invalidated if the query cannot be parsed anymore, or the optional view column_list does not match the number of expressions in the query select-list anymore.

We also introduce a new SQL Config:
spark.sql.defaultViewSchemaBinding
Control the default behavior of views when the underlying schema changes.
Valid values are:

• COMPENSATION - Any supported casts.
• DISABLED - Disable the feature.

Why are the changes needed?

Schema changes are a frequent occurrence, especially when ingesting data.
In the course of it most frequently:

• Types may need to be widened
• Columns are added to tables
• Fields are added to structs

The traditional SCHEMA BINDING behavior makes schema evolution very hard since it invalidates views agressively.
This causes erros, requiring user intervention.
Allowing views to be created to "roll with the punches" or tolerate changes in the underlying schema improves uptime.

Does this PR introduce any user-facing change?

Yes, this is a new feature with new grammar and a new config to influence its default.

How was this patch tested?

New tests are added

Was this patch authored or co-authored using generative AI tooling?

No

[srielau]

@cloud-fan @gengliangwang This is ready for an initial review.
Other than docs, the main open issue is NOT NULL behavior and COMMENTs.

[srielau]

@cloud-fan @gengliangwang This is ready now. Please review.
Docs are included in this PR.

[gengliangwang]

BINDING
The view will become invalid if the query column-list changes except for the following conditions:
The column-list includes a star clause, and there are additional columns. These additional columns are ignored.
The type of one or more columns changed in a way that allows them to be safely cast to the types using implicit casting rules.
This is the default behavior.

@srielau The PR description seems wrong. The default is COMPENSATION, right?

[mridulm]

+CC @shardulm94, @robreeves

[srielau]

@cloud-fan @gengliangwang I have addressed all comments (except the "override" on the, which intelij and Wenchen appear to be of different opinions).
The testsuite has been split up into:

• Default behavior depending on config.
• 1 file per mode.

I don't see a good way to have one file with 4 distinct modes since they have sufficient syntax and test differences that I think "spaghetti" is more readable than trying to maximize shared text.

[srielau]

@cloud-fan @gengliangwang All of @cloud-fan comments have been accepted and addressed.
Shall we merge it?

[cloud-fan]

the k8s test failure is unrelated, thanks, merging to master!

[HyukjinKwon]

Seems like this breaks non-ANSI build. Made a followup: #46614