Skip to content

[SPARK-20311][SQL] Support aliases for table value functions - #17928

Closed
maropu wants to merge 4 commits into
apache:masterfrom
maropu:SPARK-20311-3
Closed

[SPARK-20311][SQL] Support aliases for table value functions#17928
maropu wants to merge 4 commits into
apache:masterfrom
maropu:SPARK-20311-3

Conversation

@maropu

@maropumaropu commented May 10, 2017

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This pr added parsing rules to support aliases in table value functions.
The previous pr (#17666) has been reverted because of the regression. This new pr fixed the regression and add tests in SQLQueryTestSuite.

How was this patch tested?

Added tests in PlanParserSuite and SQLQueryTestSuite.

cloud-fan
cloud-fan reviewed May 10, 2017
View reviewed changes
Comment threadsql/core/src/test/resources/sql-tests/results/subquery/exists-subquery/exists-cte.sql.out Outdated

@cloud-fancloud-fanMay 10, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we breaking existing queries?

@maropumaropuMay 10, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, my bad. I'll check again.

@maropumaropuMay 10, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@SparkQA

SparkQA commented May 10, 2017

Copy link
Copy Markdown

Test build #76712 has finished for PR 17928 at commit 47b923b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class UnresolvedTableValuedFunction(

@SparkQA

SparkQA commented May 10, 2017

Copy link
Copy Markdown

Test build #76716 has finished for PR 17928 at commit 0904fc9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class UnresolvedTableValuedFunction(

@maropu

maropu commented May 10, 2017

Copy link
Copy Markdown
MemberAuthor

@cloud-fan ok, could you check again? Thanks!

@cloud-fan

cloud-fan commented May 10, 2017

Copy link
Copy Markdown
Contributor

LGTM, cc @gatorsmile to take another look

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
Comment threadsql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
;

tableAlias
: (AS? strictIdentifier identifierList?)?

@gatorsmilegatorsmileMay 10, 2017•
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also hits another bug in inline tables. Maybe you also can include the following query in the test case inline-table.sql?

sql("SELECT * FROM VALUES (\"one\", 1), (\"three\", null) CROSS JOIN VALUES (\"one\", 1), (\"three\", null)")

@maropumaropuMay 10, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added.

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
Comment threadsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala Outdated

test("SPARK-20311 range(N) as alias") {
assertEqual(
"select * from range(10) AS t",

@gatorsmilegatorsmileMay 10, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: SELECT * FROM range(10) AS t
BTW, we prefer to use upper case for the SQL keywords.

@gatorsmilegatorsmileMay 10, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You also can update the similar issues in your test cases.

@maropumaropuMay 10, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@gatorsmile

gatorsmile commented May 10, 2017

Copy link
Copy Markdown
Member

LGTM

@SparkQA

SparkQA commented May 10, 2017

Copy link
Copy Markdown

Test build #76737 has finished for PR 17928 at commit 29281b1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
Comment threadsql/core/src/test/resources/sql-tests/inputs/table-valued-functions.sql Outdated
EXPLAIN select * from RaNgE(2);

-- cross-join table valued functions
SET spark.sql.crossJoin.enabled=true;

@gatorsmilegatorsmileMay 10, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remove this line? If we specify CROSS JOIN in the query, no need to set this parm.

@maropumaropuMay 11, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@SparkQA

SparkQA commented May 11, 2017

Copy link
Copy Markdown

Test build #76770 has finished for PR 17928 at commit 54a05da.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed May 11, 2017
View reviewed changes
Comment threadsql/core/src/test/resources/sql-tests/inputs/inline-table.sql Outdated
select * from values (timestamp('1991-12-06 00:00:00.0'), array(timestamp('1991-12-06 01:00:00.0'), timestamp('1991-12-06 12:00:00.0'))) as data(a, b);

-- cross-join inline tables
SELECT * FROM VALUES ('one', 1), ('three', null) CROSS JOIN VALUES ('one', 1), ('three', null);

@cloud-fancloud-fanMay 11, 2017•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this expose the bug? If we treat CROSS as an alias, we still get the same result. how about we run EXPLAIN?

@maropumaropuMay 11, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aha, ok

@SparkQA

SparkQA commented May 11, 2017

Copy link
Copy Markdown

Test build #76784 has started for PR 17928 at commit a7a732b.

@cloud-fan

cloud-fan commented May 11, 2017

Copy link
Copy Markdown
Contributor

retest this please

@SparkQA

SparkQA commented May 11, 2017

Copy link
Copy Markdown

Test build #76791 has finished for PR 17928 at commit a7a732b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile

gatorsmile commented May 11, 2017

Copy link
Copy Markdown
Member

Thanks! Merging to master.

lycplus pushed a commit to lycplus/spark that referenced this pull request May 24, 2017
@maropu@lycplus
[SPARK-20311][SQL] Support aliases for table value functions
10e528a
## What changes were proposed in this pull request?
This pr added parsing rules to support aliases in table value functions.
The previous pr (apache#17666) has been reverted because of the regression. This new pr fixed the regression and add tests in `SQLQueryTestSuite`.
## How was this patch tested?
Added tests in `PlanParserSuite` and `SQLQueryTestSuite`.
Author: Takeshi Yamamuro <yamamuro@apache.org>
Closesapache#17928 from maropu/SPARK-20311-3.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fancloud-fancloud-fan left review comments
@gatorsmilegatorsmilegatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@maropu@SparkQA@cloud-fan@gatorsmile