[SPARK-16457] [SQL] Fix Wrong Messages when CTAS with a Partition By Clause by gatorsmile · Pull Request #14113 · apache/spark

gatorsmile · 2016-07-09T07:38:15Z

What changes were proposed in this pull request?

When doing a CTAS with a Partition By clause, we got a wrong error message.

For example,

CREATETABLEgen__tmp
PARTITIONED BY (key string)
ASSELECT key, value FROM mytable1

The error message we get now is like

Operation not allowed: Schema may not be specified in a Create Table As Select (CTAS) statement(line 2, pos 0)

However, based on the code, the message we should get is like

Operation not allowed: A Create Table As Select (CTAS) statement is not allowed to create a partitioned table using Hive's file formats. Please use the syntax of "CREATE TABLE tableName USING dataSource OPTIONS (...) PARTITIONED BY ...\" to create a partitioned table through a CTAS statement.(line 2, pos 0)

Currently, partitioning columns is part of the schema. This PR fixes the bug by changing the detection orders.

How was this patch tested?

Added test cases.

gatorsmile · 2016-07-09T07:38:43Z

- ctx)
- }
 // Hive does not allow to use a CTAS statement to create a partitioned table.
 if (tableDesc.partitionColumnNames.nonEmpty) {


This is a dead code if we do not change the order.

SparkQA · 2016-07-09T08:54:17Z

Test build #62013 has finished for PR 14113 at commit 55b1a8c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-07-09T14:41:10Z

retest this please

SparkQA · 2016-07-09T16:29:18Z

Test build #62026 has finished for PR 14113 at commit 55b1a8c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-07-10T07:11:50Z

wrong branch... Let me revert it back

SparkQA · 2016-07-10T09:04:49Z

Test build #62046 has finished for PR 14113 at commit 26bddfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-10T09:08:21Z

Test build #62045 has finished for PR 14113 at commit 568f133.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-07-12T17:24:10Z

retest this please

SparkQA · 2016-07-12T19:39:48Z

Test build #62181 has finished for PR 14113 at commit 26bddfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2016-07-12T20:58:24Z

cc @cloud-fan

gatorsmile · 2016-08-04T19:35:50Z

retest this please

gatorsmile · 2016-08-04T19:36:38Z

cc @cloud-fan This is not contained in #14482. Should I leave it open? or you will merge it into the PR?

SparkQA · 2016-08-04T19:44:02Z

Test build #63237 has finished for PR 14113 at commit 26bddfd.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-08-08T03:13:39Z

can you update it? thanks!

gatorsmile · 2016-08-08T04:17:10Z

Will update it tonight. Thanks!

cloud-fan · 2016-08-08T05:40:31Z

LGTM, pending jenkins

SparkQA · 2016-08-08T07:14:38Z

Test build #63343 has finished for PR 14113 at commit a518218.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-08-08T14:27:45Z

thanks, merging to master and 2.0!

…lause #### What changes were proposed in this pull request? When doing a CTAS with a Partition By clause, we got a wrong error message. For example, ```SQL CREATE TABLE gen__tmp PARTITIONED BY (key string) AS SELECT key, value FROM mytable1 ``` The error message we get now is like ``` Operation not allowed: Schema may not be specified in a Create Table As Select (CTAS) statement(line 2, pos 0) ``` However, based on the code, the message we should get is like ``` Operation not allowed: A Create Table As Select (CTAS) statement is not allowed to create a partitioned table using Hive's file formats. Please use the syntax of "CREATE TABLE tableName USING dataSource OPTIONS (...) PARTITIONED BY ...\" to create a partitioned table through a CTAS statement.(line 2, pos 0) ``` Currently, partitioning columns is part of the schema. This PR fixes the bug by changing the detection orders. #### How was this patch tested? Added test cases. Author: gatorsmile <gatorsmile@gmail.com> Closes#14113 from gatorsmile/ctas. (cherry picked from commit ab12690) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

gatorsmile reviewed Jul 9, 2016
View reviewed changes

gatorsmile force-pushed the ctas branch from 568f133 to 26bddfdCompare July 10, 2016 07:16

fix
a518218

gatorsmile force-pushed the ctas branch from 26bddfd to a518218Compare August 8, 2016 05:25

asfgit closed this in ab12690Aug 8, 2016

Uh oh!

Conversation

gatorsmile commented Jul 9, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

gatorsmileJul 9, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 9, 2016

Uh oh!

gatorsmile commented Jul 9, 2016

Uh oh!

SparkQA commented Jul 9, 2016

Uh oh!

gatorsmile commented Jul 10, 2016

Uh oh!

SparkQA commented Jul 10, 2016

Uh oh!

SparkQA commented Jul 10, 2016

Uh oh!

gatorsmile commented Jul 12, 2016

Uh oh!

SparkQA commented Jul 12, 2016

Uh oh!

gatorsmile commented Jul 12, 2016

Uh oh!

gatorsmile commented Aug 4, 2016

Uh oh!

gatorsmile commented Aug 4, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Aug 4, 2016

Uh oh!

cloud-fan commented Aug 8, 2016

Uh oh!

gatorsmile commented Aug 8, 2016

Uh oh!

cloud-fan commented Aug 8, 2016

Uh oh!

SparkQA commented Aug 8, 2016

Uh oh!

cloud-fan commented Aug 8, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gatorsmile commented Aug 4, 2016•
edited
Loading