[SPARK-38335][SQL] Implement parser support for DEFAULT column values

[dtenedor]

What changes were proposed in this pull request?

Implement parser changes needed to support for DEFAULT column values as tracked in https://issues.apache.org/jira/browse/SPARK-38334.

Note that these are the parser changes only. Analysis support will take place in a following PR.

Background: in the future, CREATE TABLE and ALTER TABLE invocations will support setting column default values for later operations. Following INSERT, UPDATE, MERGE statements may then reference the value using the DEFAULT keyword as needed.

Examples:

CREATE TABLE T(a INT, b INT NOT NULL);

-- The default default is NULL
INSERT INTO T VALUES (DEFAULT, 0);
INSERT INTO T(b)  VALUES (1);
SELECT * FROM T;
(NULL, 0)
(NULL, 1)

-- Adding a default to a table with rows, sets the values for the
-- existing rows (exist default) and new rows (current default).
ALTER TABLE T ADD COLUMN c INT DEFAULT 5;
INSERT INTO T VALUES (1, 2, DEFAULT);
SELECT * FROM T;
(NULL, 0, 5)
(NULL, 1, 5)
(1, 2, 5) 

Why are the changes needed?

This new API helps users run DDL and DML statements to add new values to tables and scan existing values from tables more easily.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added unit test coverage in DDLParserSuite.scala

[gengliangwang]

LGTM except minor comments

[AmplabJenkins]

Can one of the admins verify this patch?

[dtenedor]

jenkins merge

[gengliangwang]

Supporting default column values is very common among DBMS. However, this will be a breaking change for Spark SQL
Currently Spark SQL

> create table t(i int, j int);
> insert into t values(1);
Error in query: `default`.`t` requires that the data to be inserted have the same number of columns as the target table: target table has 2 column(s) but the inserted data has 1 column(s), including 0 partition column(s) having constant value(s).

After supporting default column value:

> create table t(i int, j int);
> insert into t values(1);
> select * from t;
1	NULL

> create table t2(i int, j int default 0);
> insert into t2 values(1);
> select * from t2;
1	0

I am +1 with the change.
Before merging this PR, I would like to collect the opinions of more committers. We can send SPIP for voting if necessary.
cc @cloud-fan @dongjoon-hyun @viirya @dbtsai @huaxingao @maropu @zsxwing @wangyum @yaooqinn WDYT?

[dongjoon-hyun]

Thank you for pinging me, @gengliangwang .

[dongjoon-hyun]

cc @aokolnychyi , @RussellSpitzer , @rdblue , too.

[viirya]

Thanks for the work.

For the parser change itself, looks okay. As this is a breaking change, I'd like to see some clarification on why this is necessary to have. What issue we have if we don't have this (because we don't have the default value for long time) and do we have any workaround now?

[dongjoon-hyun]

This PR doesn't seem to have the full body yet, what is your release target for this, @dtenedor and @gengliangwang ? I'm curious about the general error handling.

• Creating NULL default value for NOT NULL column
• Type mismatch between default value literal and column type.
• Upcasting or not in case of type mismatch

[dongjoon-hyun]

cc @MaxGekk too since he is a release manager for Apache Spark 3.3 who need to cut branch-3.3.

[dtenedor]

This is ready for another review round @gengliangwang @viirya @wangyum @HyukjinKwon @dongjoon-hyun :)

[dtenedor]

(Note the prior merge conflict in the lexer has now been resolved.)

[HyukjinKwon]

I'm good w/ this change.

[cloud-fan]

This is a parser-only change and the feature is not implemented yet, so definitely not a breaking change. But I'd like to confirm that, is every new SQL feature a breaking change? e.g. adding a new SQL function means that a query failed with "function not found" before, now succeeds. This doesn't seem like a breaking change to me. The same applies to accepting more parameters in a SQL function, accepting more parameter types, etc.

The code change itself LGTM.

[gengliangwang]

@dtenedor Thanks for the first contribution!
@HyukjinKwon @dongjoon-hyun @viirya @wangyum @cloud-fan thanks for the inputs! I am merging this parser-only PR to unblock @dtenedor's works in this feature.

[dongjoon-hyun]

@gengliangwang and @cloud-fan . Why do we have Spark 3.4 patch in master branch for Apache Spark 3.3?

@dongjoon-hyun This is for Spark 3.4

Are we going to revert this from branch-3.3? cc @MaxGekk

[dongjoon-hyun]

As I mentioned #35690 (review), I assumed we were going to wait until @MaxGekk cut a branch-3.3.

[gengliangwang]

@dongjoon-hyun I am merging this one to unblock @dtenedor's work on the actual changes in catalogs.
I will:

• revert this one on branch-3.3
• make sure no new related PRs merged to master until branch-3.3 is cut

[MaxGekk]

Are we going to revert this from branch-3.3? cc @MaxGekk

If there is a risk that it can hurt stability. Let's revert it. I will open a blocker for 3.3 than to not forget this.

[dongjoon-hyun]

Thank you for your confirmations, @gengliangwang and @MaxGekk !

[dongjoon-hyun]

Hey, @MaxGekk . Did you make a block issue?
To @gengliangwang and @HyukjinKwon , I saw the JIRA was resolved as 3.3 because we want to avoid our merge script shows 3.4 as a new version.

Since today is the feature freeze day, I recover it to 3.4.

[HyukjinKwon]

👍

[MaxGekk]

Hey, @MaxGekk . Did you make a block issue?

Here is the blocker SPARK-38566 and the PR which reverts the commit #35875