[SPARK-19229] [SQL] Disallow Creating Hive Source Tables when Hive Support is Not Enabled by gatorsmile · Pull Request #16587 · apache/spark

gatorsmile · 2017-01-15T19:12:37Z

What changes were proposed in this pull request?

It is weird to create Hive source tables when using InMemoryCatalog. We are unable to operate it. This PR is to block users to create Hive source tables.

How was this patch tested?

Fixed the test cases

gatorsmile · 2017-01-15T19:17:54Z

- |Hive support is required to insert into the following tables:
- |${s.catalogTable.identifier}
- """.stripMargin)
-


This is not necessary when we blocking users to create Hive source tables. When the table is data source tables, SimpleCatalogRelation will be replaced by LogicalRelation in the rule FindDataSourceTable.

gatorsmile · 2017-01-15T19:19:13Z


- test("alter table: set serde partition") {
+ // TODO: move this test to HiveDDLSuite.scala
+ ignore("alter table: set serde partition") {


These two test cases should be moved to HiveDDLSuite. However, it is ugly to copy the codes. Thus, my next plan is to improve the DDLSuite and HiveDDLSuite by creating an abstract class.

SparkQA · 2017-01-15T21:05:51Z

Test build #71401 has finished for PR 16587 at commit 0d2d9f2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-01-15T22:47:23Z

The following test cases failed:

change-column.sql
describe.sql
show-tables.sql
show_columns.sql

All these test cases are creating hive serde tables. We might need to create another SQLQueryTestSuite in the hive package.

gatorsmile · 2017-01-15T22:47:49Z

cc @cloud-fan @yhuai @hvanhovell Any comment?

SparkQA · 2017-01-16T01:09:39Z

Test build #71407 has finished for PR 16587 at commit 5e0cd26.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-01-16T01:57:10Z

+ withTable("my_ext_tab") {
+ (("a", "b") :: Nil).toDF().write.parquet(tempDir.getCanonicalPath)
+ (1 to 10).map { i => (i, i) }.toDF("a", "b").createTempView("my_temp_tab")
+ sql(s"CREATE TABLE my_ext_tab using parquet LOCATION '$tempDir'")


Ah, this is why you asked me #16586 (comment). I just ran a test on Windows to help.
- truncate table - external table, temporary table, view (not allowed) *** FAILED *** (188 milliseconds) org.apache.spark.sql.AnalysisException: Path does not exist: file:/C:projectsspark arget mpspark-9e70280d-56dc-4063-8f40-8e62fec18394; at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:382) at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$14.apply(DataSource.scala:370) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241) at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
Maybe, it'd be okay to just use toURI if this test is not supposed to test path variants.

cloud-fan · 2017-01-16T02:52:29Z

 val db = tableDefinition.identifier.database.get
 requireDbExists(db)
 val table = tableDefinition.identifier.table
+ if (tableDefinition.provider.isDefined && tableDefinition.provider.get.toLowerCase == "hive") {


shall we put this in HiveOnlyCheck?

cloud-fan · 2017-01-16T02:53:24Z

 .add("HelLo", "int", nullable = false)
 .add("WoRLd", "int", nullable = true),
- provider = Some("hive"),
+ provider = Some("parquet"),


shall we also use defaultProvider here?

SparkQA · 2017-01-17T07:23:36Z

Test build #71485 has started for PR 16587 at commit 49e6e81.

gatorsmile · 2017-01-17T08:32:16Z

retest this please

SparkQA · 2017-01-17T10:42:28Z

Test build #71495 has finished for PR 16587 at commit 49e6e81.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-17T22:54:24Z

Test build #71528 has finished for PR 16587 at commit c6d6a24.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-01-22T05:27:20Z

retest this please

SparkQA · 2017-01-22T05:33:45Z

Test build #71797 has started for PR 16587 at commit c6d6a24.

gatorsmile · 2017-01-22T08:50:28Z

retest this please

SparkQA · 2017-01-22T13:29:46Z

Test build #3545 has finished for PR 16587 at commit c6d6a24.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-01-23T04:28:26Z

LGTM

gatorsmile · 2017-01-23T04:47:51Z

Thanks! Merging to master.

…port is Not Enabled ### What changes were proposed in this pull request? It is weird to create Hive source tables when using InMemoryCatalog. We are unable to operate it. This PR is to block users to create Hive source tables. ### How was this patch tested? Fixed the test cases Author: gatorsmile <gatorsmile@gmail.com> Closesapache#16587 from gatorsmile/blockHiveTable.

gatorsmile added 3 commits January 15, 2017 10:59

fix.
3b718ed

Merge remote-tracking branch 'upstream/master' into blockHiveTable
1f20c65

fix.
0d2d9f2

gatorsmile commented Jan 15, 2017
View reviewed changes

fix test cases.
5e0cd26

HyukjinKwon reviewed Jan 16, 2017
View reviewed changes

cloud-fan reviewed Jan 16, 2017
View reviewed changes

gatorsmile mentioned this pull request Jan 16, 2017
[SPARK-19235] [SQL] [TESTS] Enable Test Cases in DDLSuite with Hive Metastore #16592
Closed

address comments.
49e6e81

fix.
c6d6a24

asfgit closed this in 772035eJan 23, 2017

Uh oh!

Conversation

gatorsmile commented Jan 15, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

gatorsmileJan 15, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmileJan 15, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 15, 2017

Uh oh!

gatorsmile commented Jan 15, 2017

Uh oh!

gatorsmile commented Jan 15, 2017

Uh oh!

SparkQA commented Jan 16, 2017

Uh oh!

HyukjinKwonJan 16, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fanJan 16, 2017

Choose a reason for hiding this comment

Uh oh!

cloud-fanJan 16, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 17, 2017

Uh oh!

gatorsmile commented Jan 17, 2017

Uh oh!

SparkQA commented Jan 17, 2017

Uh oh!

SparkQA commented Jan 17, 2017

Uh oh!

gatorsmile commented Jan 22, 2017

Uh oh!

SparkQA commented Jan 22, 2017

Uh oh!

gatorsmile commented Jan 22, 2017

Uh oh!

SparkQA commented Jan 22, 2017

Uh oh!

cloud-fan commented Jan 23, 2017

Uh oh!

gatorsmile commented Jan 23, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HyukjinKwonJan 16, 2017•
edited
Loading