[SPARK-19723][SQL]create datasource table with an non-existent location should work by windpiger · Pull Request #17055 · apache/spark

windpiger · 2017-02-24T07:35:01Z

What changes were proposed in this pull request?

This JIRA is a follow up work after SPARK-19583

As we discussed in that PR

The following DDL for datasource table with an non-existent location should work:

CREATE TABLE ... (PARTITIONED BY ...) LOCATION path

Currently it will throw exception that path not exists for datasource table for datasource table

How was this patch tested?

unit test added

…on should work

SparkQA · 2017-02-24T07:37:41Z

Test build #73408 has started for PR 17055 at commit 89eb03a.

windpiger · 2017-02-24T08:35:36Z

retest this please

SparkQA · 2017-02-24T10:10:40Z

Test build #73415 has finished for PR 17055 at commit 89eb03a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-02-24T10:28:35Z

+ withTable("t", "t1") {
+ withTempDir {
+ dir =>
+ dir.delete()


there are lots of dir existed test case in DDLSuit or HiveDDLSuit, so we just add non-existent test cases

windpiger · 2017-02-24T12:28:38Z

retest this please

SparkQA · 2017-02-24T14:36:02Z

Test build #73427 has finished for PR 17055 at commit 89eb03a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-02-26T01:41:49Z

cc @gatorsmile @cloud-fan I will appreciate that you could help to review this pr~

windpiger · 2017-02-28T06:19:14Z

ping @gatorsmile

gatorsmile · 2017-02-28T17:28:30Z

I will review this PR in the next few days. Thanks!

windpiger · 2017-03-01T01:20:27Z

ok, thanks very much~

SparkQA · 2017-03-02T13:52:51Z

Test build #73762 has finished for PR 17055 at commit 0e09b92.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-03-06T06:38:54Z

+ withTable("t", "t1") {
+ withTempDir {
+ dir =>
+ dir.delete()


withTempPath?

gatorsmile · 2017-03-06T06:39:23Z

+ test("create datasource table with a non-existing location") {
+ withTable("t", "t1") {
+ withTempDir {
+ dir =>


Nit: style issue. the above two lines can be combined together.

gatorsmile · 2017-03-06T06:39:59Z

+ |CREATE TABLE t(a int, b int)
+ |USING parquet
+ |LOCATION '$dir'
+ """.stripMargin)


The test case is already pretty long. Please reduce the sql statement to a single line.

gatorsmile · 2017-03-06T06:42:29Z

 }
 }

+ test("create datasource table with a non-existing location") {


Let me try to combine the HiveDDLSuite.scala and DDLSuite.scala. Otherwise, the test cases need to be duplicated in every PR.

SparkQA · 2017-03-06T12:34:55Z

Test build #74003 has finished for PR 17055 at commit d4378dd.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-06T14:49:15Z

Test build #74010 has finished for PR 17055 at commit 4c82dfb.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-06T17:00:56Z

Test build #74015 has finished for PR 17055 at commit 3ea3e9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…hNotExists

windpiger · 2017-03-08T11:47:46Z

@gatorsmile this pr could be merged?

SparkQA · 2017-03-09T12:09:56Z

Test build #74261 has finished for PR 17055 at commit bc7f08d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

windpiger · 2017-03-09T12:14:44Z

@gatorsmile I have merged with master，if it is ok, could you help to merge it?

SparkQA · 2017-03-09T12:49:36Z

Test build #74263 has finished for PR 17055 at commit 207448d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-09T12:53:51Z

Test build #74264 has finished for PR 17055 at commit 4024d7c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-03-09T16:17:05Z

I will review it today. Thanks!

windpiger · 2017-03-09T23:32:23Z

OK thanks a lot～

cloud-fan · 2017-03-10T19:39:52Z

 Seq(true, false).foreach { shouldDelete =>
- val tcName = if (shouldDelete) "non-existent" else "existed"
+ val tcName = if (shouldDelete) "non-existing" else "existed"
 test(s"CTAS for external data source table with a $tcName location") {


now it's duplicated with https://github.com/apache/spark/pull/17055/files#diff-2d45592564aa2eea4e85ffd000d8a14eR2054

oh, it is. after we also qualified the location path for datasource table , the code for equal two location are the same with HiveExternalCatalog. thanks~

gatorsmile · 2017-03-10T21:16:39Z

+ withTempDir { dir =>
+ if (shouldDelete) {
+ dir.delete()
+ }


if (shouldDelete) dir.delete()

ok thanks~~

gatorsmile · 2017-03-10T21:19:07Z

+ spark.sql(
+ s"""
+ |CREATE TABLE t1(a int, b int) USING parquet PARTITIONED BY(a) LOCATION '$dir'
+ """.stripMargin)


Nit: one line?

SparkQA · 2017-03-11T01:34:29Z

Test build #74360 has finished for PR 17055 at commit e1c891e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-11T03:44:23Z

Test build #74361 has finished for PR 17055 at commit ada275c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-03-11T04:59:54Z

thanks, merging to master!

[SPARK-19723][SQL]create datasource table with an non-existent locati…
89eb03a
…on should work

windpiger commented Feb 24, 2017
View reviewed changes

merge with master
0e09b92

gatorsmile reviewed Mar 6, 2017
View reviewed changes

optimize some code
d4378dd

fix code style
4c82dfb

fix code style
3ea3e9d

Merge branch 'master' of github.com:apache/spark into CTDataSourcePat…
b26cce1
…hNotExists

windpiger added 5 commits March 9, 2017 16:48

Merge branch 'master' into CTDataSourcePathNotExists
6f7045c

Merge branch 'master' into CTDataSourcePathNotExists
d53f7ab

merge with master
3ae5b2a

fix a code
bc7f08d

fix code style
5699059

windpiger added 2 commits March 9, 2017 18:41

fix code style
207448d

fix code style
4024d7c

cloud-fan reviewed Mar 10, 2017
View reviewed changes

gatorsmile reviewed Mar 10, 2017
View reviewed changes

windpiger added 2 commits March 11, 2017 09:29

fix some code review
e1c891e

fix some code review
ada275c

asfgit closed this in f6fdf92Mar 11, 2017

Uh oh!

Conversation

windpiger commented Feb 24, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Feb 24, 2017

Uh oh!

windpiger commented Feb 24, 2017

Uh oh!

SparkQA commented Feb 24, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

windpiger commented Feb 24, 2017

Uh oh!

SparkQA commented Feb 24, 2017

Uh oh!

windpiger commented Feb 26, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

windpiger commented Feb 28, 2017

Uh oh!

gatorsmile commented Feb 28, 2017

Uh oh!

windpiger commented Mar 1, 2017

Uh oh!

SparkQA commented Mar 2, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 6, 2017

Uh oh!

SparkQA commented Mar 6, 2017

Uh oh!

SparkQA commented Mar 6, 2017

Uh oh!

windpiger commented Mar 8, 2017

Uh oh!

SparkQA commented Mar 9, 2017

Uh oh!

windpiger commented Mar 9, 2017

Uh oh!

SparkQA commented Mar 9, 2017

Uh oh!

SparkQA commented Mar 9, 2017

Uh oh!

gatorsmile commented Mar 9, 2017

Uh oh!

windpiger commented Mar 9, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

windpigerMar 11, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 11, 2017

Uh oh!

SparkQA commented Mar 11, 2017

Uh oh!

cloud-fan commented Mar 11, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

windpiger commented Feb 24, 2017•
edited
Loading

windpiger commented Feb 26, 2017•
edited
Loading

windpigerMar 11, 2017•
edited
Loading