[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception by windpiger · Pull Request #17272 · apache/spark

windpiger · 2017-03-13T07:26:34Z

What changes were proposed in this pull request?

This JIRA is a follow up work after SPARK-19583

As we discussed in that PR

The following DDL for a managed table with an existed default location should throw an exception:

CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
CREATE TABLE ... (PARTITIONED BY ...)

Currently there are some situations which are not consist with above logic:

CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location
situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog)
CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ...
situation: hive table succeed with an existed default location

This PR is going to make above two situations consist with the logic that it should throw an exception
with an existed default location.

How was this patch tested?

unit test added

…e path should throw an exception

windpiger · 2017-03-13T07:27:22Z

 val db = tableDefinition.identifier.database.get
 requireDbExists(db)
 val table = tableDefinition.identifier.table
- if (tableExists(db, table)) {


we have check if the table exists in SessionCatalog.createtable

SparkQA · 2017-03-13T07:28:59Z

Test build #74436 has finished for PR 17272 at commit a0ba419.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-13T07:29:00Z

 assert(actual.tableType === CatalogTableType.EXTERNAL)
 }

- test("create table when the table already exists") {


since in InMemoryCatalog and HiveExternalCatalog does not check if the table exists, here move the test case to SessionCatalogSuite(using InMemoryCatalog) and HiveExternalCatalogSuite

windpiger · 2017-03-13T07:30:28Z

 table.storage.locationUri
 }
+
+ sparkSession.sessionState.catalog.checkTableOrPathExists(table, ignoreIfExists = false)


because in following saveDataIntoTable will create the table path and store data in it, we should not check the path in following createTable, we check it here before saveDataIntoTable

SparkQA · 2017-03-13T08:48:40Z

Test build #74437 has finished for PR 17272 at commit 65d7ea9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-13T09:21:08Z

Test build #74441 has finished for PR 17272 at commit 0e753cc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-13T11:31:10Z

retest this please

SparkQA · 2017-03-13T19:03:53Z

Test build #74445 has finished for PR 17272 at commit c8d9b77.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-13T23:27:39Z

retest this please

SparkQA · 2017-03-14T01:16:56Z

Test build #74468 has finished for PR 17272 at commit c8d9b77.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T02:19:07Z

Test build #74472 has finished for PR 17272 at commit cd4a091.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T06:17:34Z

Test build #74493 has started for PR 17272 at commit 2ac70b4.

windpiger · 2017-03-14T08:10:02Z

retest this please

SparkQA · 2017-03-14T09:59:52Z

Test build #74499 has finished for PR 17272 at commit 2ac70b4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-14T12:10:12Z

python test failed
the location('file:/home/jenkins/workspace/SparkPullRequestBuilder/spark-warehouse/tab1') of table('`default`.`tab1`') already exists.;
the location of tab1 does not deleted , I have tried to found out in which test case it forget to delete it(search all test cases containing tab1 and run it), but they are all ok to delete the location of tab1, so here we change the table name to work around it.

SparkQA · 2017-03-14T13:46:22Z

Test build #74516 has finished for PR 17272 at commit 516c4e4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-14T13:52:52Z

retest this please

SparkQA · 2017-03-14T15:41:36Z

Test build #74524 has finished for PR 17272 at commit 516c4e4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-14T17:48:02Z

Test build #74535 has finished for PR 17272 at commit 739f207.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-15T14:34:56Z

Test build #74604 has finished for PR 17272 at commit 4351cd7.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-05-11T14:46:35Z

@windpiger is this still active?

gatorsmile · 2017-06-19T22:18:30Z

cc @ltnwgl Could you please take this over? Thanks!

gatorsmile · 2018-03-19T19:04:20Z

cc @dongjoon-hyun Are you interested in this PR? Just take it over?

gengliangwang · 2018-03-20T04:01:14Z

@gatorsmile I will take it over :)

…le should throw an exception ## What changes were proposed in this pull request? This PR is to finish #17272 This JIRA is a follow up work after SPARK-19583 As we discussed in that PR The following DDL for a managed table with an existed default location should throw an exception: CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... CREATE TABLE ... (PARTITIONED BY ...) Currently there are some situations which are not consist with above logic: CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog) CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... situation: hive table succeed with an existed default location This PR is going to make above two situations consist with the logic that it should throw an exception with an existed default location. ## How was this patch tested? unit test added Author: Gengliang Wang <gengliang.wang@databricks.com> Closes#20886 from gengliangwang/pr-17272.

…le should throw an exception ## What changes were proposed in this pull request? This PR is to finish apache#17272 This JIRA is a follow up work after SPARK-19583 As we discussed in that PR The following DDL for a managed table with an existed default location should throw an exception: CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... CREATE TABLE ... (PARTITIONED BY ...) Currently there are some situations which are not consist with above logic: CREATE TABLE ... (PARTITIONED BY ...) succeed with an existed default location situation: for both hive/datasource(with HiveExternalCatalog/InMemoryCatalog) CREATE TABLE ... (PARTITIONED BY ...) AS SELECT ... situation: hive table succeed with an existed default location This PR is going to make above two situations consist with the logic that it should throw an exception with an existed default location. ## How was this patch tested? unit test added Author: Gengliang Wang <gengliang.wang@databricks.com> Closesapache#20886 from gengliangwang/pr-17272.

## What changes were proposed in this pull request? This PR proposes to close PRs ... - inactive to the review comments more than a month - WIP and inactive more than a month - with Jenkins build failure but inactive more than a month - suggested to be closed and no comment against that - obviously looking inappropriate (e.g., Branch 0.5) To make sure, I left a comment for each PR about a week ago and I could not have a response back from the author in these PRs below: Closesapache#11129Closesapache#12085Closesapache#12162Closesapache#12419Closesapache#12420Closesapache#12491Closesapache#13762Closesapache#13837Closesapache#13851Closesapache#13881Closesapache#13891Closesapache#13959Closesapache#14091Closesapache#14481Closesapache#14547Closesapache#14557Closesapache#14686Closesapache#15594Closesapache#15652Closesapache#15850Closesapache#15914Closesapache#15918Closesapache#16285Closesapache#16389Closesapache#16652Closesapache#16743Closesapache#16893Closesapache#16975Closesapache#17001Closesapache#17088Closesapache#17119Closesapache#17272Closesapache#17971 Added: Closesapache#17778Closesapache#17303Closesapache#17872 ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18017 from HyukjinKwon/close-inactive-prs.

[SPARK-19724][SQL]create a managed table with an existed default tabl…
a0ba419
…e path should throw an exception

windpiger commented Mar 13, 2017
View reviewed changes

windpiger added 2 commits March 13, 2017 15:31

remove empty lines
65d7ea9

fix comment
0e753cc

fix test failed
c8d9b77

fix test failed
cd4a091

windpiger force-pushed the managedTableLocationExists branch from 2ac70b4 to cd4a091Compare March 14, 2017 12:08

windpiger commented Mar 14, 2017
View reviewed changes

reset catalog in pyspark tests
739f207

windpiger force-pushed the managedTableLocationExists branch from 516c4e4 to 739f207Compare March 14, 2017 16:00

to find why jenkins failed
4351cd7

HyukjinKwon mentioned this pull request May 17, 2017
[INFRA] Close stale PRs #18017
Closed

asfgit closed this in 5d2750aMay 18, 2017

gengliangwang mentioned this pull request Mar 22, 2018
[SPARK-19724][SQL]create a managed table with an existed default table should throw an exception #20886
Closed

Uh oh!

Conversation

windpiger commented Mar 13, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

windpigerMar 13, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 13, 2017

Uh oh!

windpigerMar 13, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

windpigerMar 13, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 13, 2017

Uh oh!

SparkQA commented Mar 13, 2017

Uh oh!

windpiger commented Mar 13, 2017

Uh oh!

SparkQA commented Mar 13, 2017

Uh oh!

windpiger commented Mar 13, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

windpiger commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

windpigerMar 14, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

windpiger commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 14, 2017

Uh oh!

SparkQA commented Mar 15, 2017

Uh oh!

HyukjinKwon commented May 11, 2017

Uh oh!

gatorsmile commented Jun 19, 2017

Uh oh!

gatorsmile commented Mar 19, 2018

Uh oh!

gengliangwang commented Mar 20, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

windpigerMar 13, 2017•
edited
Loading