[SPARK-17051][SQL] we should use hadoopConf in InsertIntoHiveTable

[cloud-fan]

What changes were proposed in this pull request?

Hive confs in hive-site.xml will be loaded in hadoopConf, so we should use hadoopConf in InsertIntoHiveTable instead of SessionState.conf

How was this patch tested?

N/A

[cloud-fan]

cc @yhuai

[SparkQA]

Test build #63746 has finished for PR 14634 at commit 64268f3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

Great! It resolves my original concern. Thanks!

[gatorsmile]

hive.exec.dynamic.partition also impacts our regular writing paths (DataFrameWriter APIs). I remember DataFrameWriter APIs always assume this conf is true, right?

If it is controlled by hadoopConf, users might hit strange errors when using our DataFrameWriter APIs.

[cloud-fan]

hive.exec.dynamic.partition also impacts our regular writing paths

I think it's hive only conf? Normal data source relation should not read this conf.

[gatorsmile]

If users want to use the DataFrameWriter's insertInto API for partitioned Hive table, they have to set hive.exec.dynamic.partition to true and hive.exec.dynamic.partition.mode to nonstrict. Otherwise, it does not work. Users are unable to specify the partition values in the insertInto APIs. See the code.

Let me use a test case to show the issue.

 withTempDir { tmpDir =>valbasePath= tmpDir.getCanonicalPath
valexternalTab="extTable_with_partitions"
withTable(externalTab) {
assert(tmpDir.listFiles.isEmpty)
sql(
s""" |CREATE EXTERNAL TABLE $externalTab (key INT, value STRING) |PARTITIONED BY (ds STRING, hr STRING) |stored as Parquet |LOCATION '$basePath'""".stripMargin)
// SQL interface
withSQLConf("hive.exec.dynamic.partition"->"false") {
for (ds <-Seq("2008-04-08", "2008-04-09"); hr <-Seq("11", "12")) {
sql(
s""" |INSERT OVERWRITE TABLE $externalTab
 |partition (ds='$ds',hr='$hr') |SELECT 1, 'a'""".stripMargin)
}
}
// DataFrameWriter interface
withSQLConf("hive.exec.dynamic.partition"->"true",
"hive.exec.dynamic.partition.mode"->"nonstrict") {
Seq((1, "2", "2008-04-09", "12")).toDF("key", "value", "ds", "hr").write
.insertInto(externalTab)
}
}
}

[gatorsmile]

Sorry, let me rephrase the potential issue.

• insertInto API forces users to set hive.exec.dynamic.partition to true and hive.exec.dynamic.partition.mode to nonstrict. This might not be convinient. The default value of hive.exec.dynamic.partition.mode is strict.
• If we always read the setting from hadoopConf, does that mean users are unable to control these settings for different queries? Is that possible users can change the setting values in hadoopConf at runtime?

[cloud-fan]

1. hadoopConf contains all the confs from SQL conf, see SessionState.newHadoopConf
2. users can change hadoopConf at runtime.
3. the default value is from hive, I'm ok to not follow it.

[gatorsmile]

uh, I see. Thank you! No more question. : )

[yhuai]

Sorry. What's the necessity to make this change?

[gatorsmile]

Based on my understanding, after this PR, we will respect the conf values of hive.exec.dynamic.partition, hive.exec.dynamic.partition.mode and hive.exec.compress.output that are specified in hive-site.xml?

[JoshRosen]

@yhuai, @cloud-fan What's the status of this issue? Should this still be targeted for 2.0.1 (which it is currently in JIRA)?

[cloud-fan]

cc @yhuai , In InsertIntoHiveTable we already called newHadoopConf, I think it's safer to get these hive confs from hadoopConf instead of sqlConf, to respect hive confs in hive-site.xml. We can check other places one by one later.

[cloud-fan]

retest this please

[cloud-fan]

@JoshRosen this is a regression in 2.0(it works in 1.6), so I think we should target it to 2.0

[SparkQA]

Test build #65620 has finished for PR 14634 at commit 64268f3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

This change looks good. Let's add a regression test.

[SparkQA]

Test build #65629 has finished for PR 14634 at commit 90fbe4e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class HiveQuerySuite extends HiveComparisonTest with SQLTestUtils with BeforeAndAfter

[yhuai]

LGTM. Merging to master and branch 2.0.