[SPARK-28002][SQL] Support WITH clause column aliases by peter-toth · Pull Request #24842 · apache/spark

peter-toth · 2019-06-11T15:10:02Z

What changes were proposed in this pull request?

This PR adds support of column aliasing in a CTE so this query becomes valid:

WITH t(x) AS (SELECT 1)
SELECT * FROM t WHERE x = 1

How was this patch tested?

Added new UTs.

peter-toth · 2019-06-11T17:10:40Z

@dongjoon-hyun, @gatorsmile this is another feature that PostgreSQL does support.

liancheng · 2019-06-11T17:56:09Z

ok to test

liancheng · 2019-06-11T18:06:16Z

LGTM pending Jenkins, thanks!

liancheng · 2019-06-11T18:12:55Z

In fact, I was thinking, instead of adding test cases in the SQLQuerySuite, maybe it's better to add test cases in the PlanParserSuite and AnalysisSuite.

The changes in this PR are purely about the parsing phase. Running full-blown SQL queries for testing parsing changes is slow and unnecessary. The Spark PR builder already takes a long time to finish, would be better to cut the cost whenever possible.

SparkQA · 2019-06-11T20:59:34Z

Test build #106393 has finished for PR 24842 at commit 8a7e7e0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

peter-toth · 2019-06-12T06:57:19Z

Thanks @liancheng for the review. I've moved analysis error test cases to AnalysisSuite as you suggested. But I left the positive test case in SQLQuerySuite as I think the value of x column should be tested. Please let me know if you disagree.

BTW there is another improvement PR regarding WITH clause I opened recently: #24831
Any comment is very welcome if you have some time to review it.

SparkQA · 2019-06-12T07:05:03Z

Test build #106405 has finished for PR 24842 at commit 92cc5d2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

hvanhovell · 2019-06-12T16:47:31Z

@peter-toth can you still add a test to the plan parser suite?

dongjoon-hyun · 2019-06-12T17:15:23Z

Thank you for ping me, @peter-toth . +1 for the above test case comments. And, this PR also looks good to me.

dongjoon-hyun · 2019-06-12T17:16:50Z

Oh, please move this to cte.sql. That is a perfect place for this.
cc @gatorsmile

peter-toth · 2019-06-12T19:27:27Z

@peter-toth can you still add a test to the plan parser suite?

@hvanhovell sure, I added a test to PlanParserSuite

SparkQA · 2019-06-12T22:38:59Z

Test build #106438 has finished for PR 24842 at commit ddb0554.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-06-13T06:28:36Z

Test build #106456 has finished for PR 24842 at commit 510eee1.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-06-13T07:05:02Z

Test build #106458 has finished for PR 24842 at commit 0a00a03.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

peter-toth · 2019-06-13T07:08:44Z

retest this please

gatorsmile · 2019-06-13T08:59:02Z

 CROSS JOIN CTE1 t2;

+-- CTE with column alias
+WITH t(x) AS (SELECT 1)


Could you add more test cases? For example, can a with clause in a subquery shadow a with clause in an enclosing query with the same name? Another example, use with clauses in a subquery expression?

I wanted to focus on column aliases in this PR, but I have another open here which focuses on nested WITH clauses: #24831

Yep. For the nested WITH, #24831 would be a better place to add them.
After merging this, let's proceed as @gatorsmile suggested, @peter-toth .

dongjoon-hyun · 2019-06-13T17:21:03Z

 interceptParseException(parsePlan)(sqlCommand, messages: _*)

- private def cte(plan: LogicalPlan, namedPlans: (String, LogicalPlan)*): With = {
+ private def cte(plan: LogicalPlan, namedPlans: (String, (LogicalPlan, Seq[String]))*): With = {


Thank you for consolidating both cte functions into one!

dongjoon-hyun

+1, LGTM.

@gatorsmile . Could you do the final review and sign-off this? For the nested WITH, I hope we can do that in his another PRs.

#24831 is adding nested WITH.
#24860 is adding WITH test cases.

dongjoon-hyun · 2019-06-14T16:51:05Z

Retest this please.

SparkQA · 2019-06-14T20:04:45Z

Test build #106523 has finished for PR 24842 at commit 0a00a03.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-06-14T23:05:43Z

Retest this please.

SparkQA · 2019-06-15T02:12:39Z

Test build #106533 has finished for PR 24842 at commit 0a00a03.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-06-15T03:46:53Z

Merged to master.

Thank you so much, @peter-toth , @liancheng, @hvanhovell , @gatorsmile !

@peter-toth . Please proceed to #24831 and #24860 .

## What changes were proposed in this pull request? This PR adds support of column aliasing in a CTE so this query becomes valid: ``` WITH t(x) AS (SELECT 1) SELECT * FROM t WHERE x = 1 ``` ## How was this patch tested? Added new UTs. Closesapache#24842 from peter-toth/SPARK-28002. Authored-by: Peter Toth <peter.toth@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

peter-toth · 2019-06-15T11:03:36Z

Thanks @dongjoon-hyun @liancheng @gatorsmile @hvanhovell for the review.

I have prepared #24831 for review.
Unfortunately I can't work on #24860 next week but you can expect updates on it after that.

dongjoon-hyun · 2019-06-15T17:23:18Z

Thanks. No problem, @peter-toth !

gatorsmile · 2019-06-23T23:20:44Z


 namedQuery
- : name=identifier AS? '(' query ')'
+ : name=identifier (columnAliases=identifierList)? AS? '(' query ')'


Duplicate names within a single CTE definition are not allowed.

gatorsmile · 2019-06-23T23:23:47Z

 -- !query 7
-DROP VIEW IF EXISTS t
+WITH t(x) AS (SELECT 1)
+SELECT * FROM t WHERE x = 1


Almost all the test cases are using one column in the CTE definition. Can you try your best to improve the current test coverage in this new syntax?

gatorsmile · 2019-06-23T23:24:36Z

Sorry for my late response. @peter-toth @dongjoon-hyun Could you submit a follow-up PR to improve it?

peter-toth · 2019-06-24T08:09:33Z

@gatorsmile @dongjoon-hyun I opened #24949 to add some new test cases. Please let me know if you want more cases.

Please note that I'm also working on #24860 and it will add many new tests that cover WITH column aliases.