[SPARK-14954][SQL] Add PARTITIONED BY and CLUSTERED BY clause for data source CTAS syntax

[liancheng]

What changes were proposed in this pull request?

Currently, we can only create persisted partitioned and/or bucketed data source tables using the Dataset API but not using SQL DDL. This PR implements the following syntax to add partitioning and bucketing support to the SQL DDL:

CREATE TABLE <table-name>
USING <provider> [OPTIONS (<key1> <value1>, <key2> <value2>, ...)]
[PARTITIONED BY (col1, col2, ...)]
[CLUSTERED BY (col1, col2, ...) [SORTED BY (col1, col2, ...)] INTO <n> BUCKETS]
AS SELECT ...

How was this patch tested?

Test cases are added in MetastoreDataSourcesSuite to check the newly added syntax.

[SparkQA]

Test build #57127 has finished for PR 12734 at commit af973d6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #57129 has finished for PR 12734 at commit a193faf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

For DataFrameWriter, can we do sortBy without using bucketBy?

UPDATE: DataFrameWriter's sortBy does require bucketBy

[yhuai]

If the command is not CTAS statement, seems we should throw exceptions if users define any of PARTITIONED BY, SORTED BY, or CLUSTERED BY clause?

[yhuai]

One thing that is not very related to this pr. I always find that the keyword CLUSTERED BY is very confusing, because there is a CLUSTER BY keyword (, which is DISTRIBUTE BY + SORT BY). But, we do not need to change it right now.

[yhuai]

I am going to add the check for this else branch and add some tests.

[yhuai]

oh, sorry. PARTITIONED BY and CLUSTERED BY are both associated with CREATE TABLE USING AS SELECT rule. So, for CREATE TABLE USING, if PARTITIONED BY or CLUSTERED PY is provided, we already throw an exception.

[jodersky]

Does this pr fix a ticket? In that case it would be useful to change the title to include the [SPARK-] prefix so that the JIRA status gets updated

[yhuai]

Yea. https://issues.apache.org/jira/browse/SPARK-14954 is the jira.

[jodersky]

Could you change the title to [SPARK-14954] (current title)?

[yhuai]

oh, I cannot change it. @liancheng will change the title after he gets up :)

[yhuai]

@liancheng The last commit adds a new test.

[yhuai]

Changes look good to me.

[SparkQA]

Test build #57161 has finished for PR 12734 at commit 442265e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

I fixed the title while merging to master.

[liancheng]

@jodersky Oh sorry, pasted the JIRA ticket summary to the PR title but forgot to add the tags. Updated!