Uh oh!
There was an error while loading. Please reload this page.
[SPARK-20831] [SQL] Fix INSERT OVERWRITE data source tables with IF NOT EXISTS - #18050
[SPARK-20831] [SQL] Fix INSERT OVERWRITE data source tables with IF NOT EXISTS#18050gatorsmile wants to merge 3 commits into
Conversation
SparkQA
commented
May 21, 2017
Test build #77153 has finished for PR 18050 at commit
|
| """.stripMargin) | ||
| checkAnswer(sql(selQuery), Row(5, 2, 3, 6)) | ||
| // If the partition already exists, the insert will overwrite the data |
There was a problem hiding this comment.
Seems IF NOT EXISTS works previously with INSERT OVERWRITE? It doesn't overwrite the existing data.
There was a problem hiding this comment.
So seems we should also add a test for IF NOT EXISTS with INSERT OVERWRITE for datasource table?
There was a problem hiding this comment.
That is why I used testPartitionedTable to do the test. It tests both data source and hive serde tables.
gatorsmile
commented
May 22, 2017
cc @cloud-fan |
viirya
commented
May 22, 2017
LGTM |
| * @param query the logical plan representing data to write to. | ||
| * @param overwrite overwrite existing table or partitions. | ||
| * @param ifNotExists If true, only write if the table or partition does not exist. | ||
| * @param ifStaticPartitionNotExists If true, only write if the partition does not exist. |
There was a problem hiding this comment.
this name is a little verbose, how about ifPartitionNotExists?
| checkAnswer( | ||
| sql(selQuery), | ||
| Row("blarr2", "a", "b")) | ||
| testPartitionedTable("INSERT OVERWRITE - partition IF NOT EXISTS") { tableName => |
There was a problem hiding this comment.
it's a little weird to test data source table insertion in InsertIntoHiveTableSuite...
There was a problem hiding this comment.
Create and move all these tests to a new suite InsertIntoTableSuite?
There was a problem hiding this comment.
If it is not too verbose, I'd like to have them separated, instead of mixing together in one test suite under hive.
There was a problem hiding this comment.
We can move the test cases to a new one in /core and let InsertIntoHiveTableSuite extends that one.
There was a problem hiding this comment.
Yeah, we can do it as a test-only PR.
SparkQA
commented
May 22, 2017
Test build #77173 has finished for PR 18050 at commit
|
…T EXISTS ### What changes were proposed in this pull request? Currently, we have a bug when we specify `IF NOT EXISTS` in `INSERT OVERWRITE` data source tables. For example, given a query: ```SQL INSERT OVERWRITE TABLE $tableName partition (b=2, c=3) IF NOT EXISTS SELECT 9, 10 ``` we will get the following error: ``` unresolved operator 'InsertIntoTable Relation[a#425,d#426,b#427,c#428] parquet, Map(b -> Some(2), c -> Some(3)), true, true;; 'InsertIntoTable Relation[a#425,d#426,b#427,c#428] parquet, Map(b -> Some(2), c -> Some(3)), true, true +- Project [cast(9#423 as int) AS a#429, cast(10#424 as int) AS d#430] +- Project [9 AS 9#423, 10 AS 10#424] +- OneRowRelation$ ``` This PR is to fix the issue to follow the behavior of Hive serde tables > INSERT OVERWRITE will overwrite any existing data in the table or partition unless IF NOT EXISTS is provided for a partition ### How was this patch tested? Modified an existing test case Author: gatorsmile <gatorsmile@gmail.com> Closes#18050 from gatorsmile/insertPartitionIfNotExists. (cherry picked from commit f3ed62a) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
cloud-fan
commented
May 22, 2017
thanks, merging to master/2.2! |
…T EXISTS ### What changes were proposed in this pull request? Currently, we have a bug when we specify `IF NOT EXISTS` in `INSERT OVERWRITE` data source tables. For example, given a query: ```SQL INSERT OVERWRITE TABLE $tableName partition (b=2, c=3) IF NOT EXISTS SELECT 9, 10 ``` we will get the following error: ``` unresolved operator 'InsertIntoTable Relation[a#425,d#426,b#427,c#428] parquet, Map(b -> Some(2), c -> Some(3)), true, true;; 'InsertIntoTable Relation[a#425,d#426,b#427,c#428] parquet, Map(b -> Some(2), c -> Some(3)), true, true +- Project [cast(9#423 as int) AS a#429, cast(10#424 as int) AS d#430] +- Project [9 AS 9#423, 10 AS 10#424] +- OneRowRelation$ ``` This PR is to fix the issue to follow the behavior of Hive serde tables > INSERT OVERWRITE will overwrite any existing data in the table or partition unless IF NOT EXISTS is provided for a partition ### How was this patch tested? Modified an existing test case Author: gatorsmile <gatorsmile@gmail.com> Closesapache#18050 from gatorsmile/insertPartitionIfNotExists.
What changes were proposed in this pull request?
Currently, we have a bug when we specify
IF NOT EXISTSinINSERT OVERWRITEdata source tables. For example, given a query:we will get the following error:
This PR is to fix the issue to follow the behavior of Hive serde tables
How was this patch tested?
Modified an existing test case