[SPARK-20460][SQL] Make it more consistent to handle column name duplication by maropu · Pull Request #17758 · apache/spark

maropu · 2017-04-25T15:37:18Z

What changes were proposed in this pull request?

This pr made it more consistent to handle column name duplication. In the current master, error handling is different when hitting column name duplication:

// json
scala> val schema = StructType(StructField("a", IntegerType) :: StructField("a", IntegerType) :: Nil)
scala> Seq("""{"a":1, "a":1}"""""").toDF().coalesce(1).write.mode("overwrite").text("/tmp/data")
scala> spark.read.format("json").schema(schema).load("/tmp/data").show
org.apache.spark.sql.AnalysisException: Reference 'a' is ambiguous, could be: a#12, a#13.;
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:287)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:181)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolve$1.apply(LogicalPlan.scala:153)
scala> spark.read.format("json").load("/tmp/data").show
org.apache.spark.sql.AnalysisException: Duplicate column(s) : "a" found, cannot save to JSON format;
at org.apache.spark.sql.execution.datasources.json.JsonDataSource.checkConstraints(JsonDataSource.scala:81)
at org.apache.spark.sql.execution.datasources.json.JsonDataSource.inferSchema(JsonDataSource.scala:63)
at org.apache.spark.sql.execution.datasources.json.JsonFileFormat.inferSchema(JsonFileFormat.scala:57)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:176)
at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$7.apply(DataSource.scala:176)
// csv
scala> val schema = StructType(StructField("a", IntegerType) :: StructField("a", IntegerType) :: Nil)
scala> Seq("a,a", "1,1").toDF().coalesce(1).write.mode("overwrite").text("/tmp/data")
scala> spark.read.format("csv").schema(schema).option("header", false).load("/tmp/data").show
org.apache.spark.sql.AnalysisException: Reference 'a' is ambiguous, could be: a#41, a#42.;
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:287)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:181)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolve$1.apply(LogicalPlan.scala:153)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolve$1.apply(LogicalPlan.scala:152)
// If `inferSchema` is true, a CSV format is duplicate-safe (See SPARK-16896)
scala> spark.read.format("csv").option("header", true).load("/tmp/data").show
+---+---+
| a0| a1|
+---+---+
| 1| 1|
+---+---+
// parquet
scala> val schema = StructType(StructField("a", IntegerType) :: StructField("a", IntegerType) :: Nil)
scala> Seq((1, 1)).toDF("a", "b").coalesce(1).write.mode("overwrite").parquet("/tmp/data")
scala> spark.read.format("parquet").schema(schema).option("header", false).load("/tmp/data").show
org.apache.spark.sql.AnalysisException: Reference 'a' is ambiguous, could be: a#110, a#111.;
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:287)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolve(LogicalPlan.scala:181)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolve$1.apply(LogicalPlan.scala:153)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolve$1.apply(LogicalPlan.scala:152)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)

When this patch applied, the results change to;


// json
scala> val schema = StructType(StructField("a", IntegerType) :: StructField("a", IntegerType) :: Nil)
scala> Seq("""{"a":1, "a":1}"""""").toDF().coalesce(1).write.mode("overwrite").text("/tmp/data")
scala> spark.read.format("json").schema(schema).load("/tmp/data").show
org.apache.spark.sql.AnalysisException: Found duplicate column(s) in datasource: "a";
at org.apache.spark.sql.util.SchemaUtils$.checkColumnNameDuplication(SchemaUtil.scala:47)
at org.apache.spark.sql.util.SchemaUtils$.checkSchemaColumnNameDuplication(SchemaUtil.scala:33)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:186)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:368)
scala> spark.read.format("json").load("/tmp/data").show
org.apache.spark.sql.AnalysisException: Found duplicate column(s) in datasource: "a";
at org.apache.spark.sql.util.SchemaUtils$.checkColumnNameDuplication(SchemaUtil.scala:47)
at org.apache.spark.sql.util.SchemaUtils$.checkSchemaColumnNameDuplication(SchemaUtil.scala:33)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:186)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:368)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:156)
// csv
scala> val schema = StructType(StructField("a", IntegerType) :: StructField("a", IntegerType) :: Nil)
scala> Seq("a,a", "1,1").toDF().coalesce(1).write.mode("overwrite").text("/tmp/data")
scala> spark.read.format("csv").schema(schema).option("header", false).load("/tmp/data").show
org.apache.spark.sql.AnalysisException: Found duplicate column(s) in datasource: "a";
at org.apache.spark.sql.util.SchemaUtils$.checkColumnNameDuplication(SchemaUtil.scala:47)
at org.apache.spark.sql.util.SchemaUtils$.checkSchemaColumnNameDuplication(SchemaUtil.scala:33)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:186)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:368)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
scala> spark.read.format("csv").option("header", true).load("/tmp/data").show
+---+---+
| a0| a1|
+---+---+
| 1| 1|
+---+---+
// parquet
scala> val schema = StructType(StructField("a", IntegerType) :: StructField("a", IntegerType) :: Nil)
scala> Seq((1, 1)).toDF("a", "b").coalesce(1).write.mode("overwrite").parquet("/tmp/data")
scala> spark.read.format("parquet").schema(schema).option("header", false).load("/tmp/data").show
org.apache.spark.sql.AnalysisException: Found duplicate column(s) in datasource: "a";
at org.apache.spark.sql.util.SchemaUtils$.checkColumnNameDuplication(SchemaUtil.scala:47)
at org.apache.spark.sql.util.SchemaUtils$.checkSchemaColumnNameDuplication(SchemaUtil.scala:33)
at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:186)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:368)

How was this patch tested?

Added tests in DataFrameReaderWriterSuite and SQLQueryTestSuite.

SparkQA · 2017-04-25T15:44:11Z

Test build #76137 has finished for PR 17758 at commit 861a074.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-25T17:12:12Z

Test build #76138 has finished for PR 17758 at commit c222eae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-26T03:11:10Z

Test build #76165 has finished for PR 17758 at commit de11a5b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-04-26T05:54:10Z

Test build #76167 has finished for PR 17758 at commit 05a7a61.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-12T12:18:58Z

Test build #76864 has finished for PR 17758 at commit 42f3fc9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2017-05-13T02:22:44Z

@gatorsmile Could you check this and give me advise on this? Thanks!

maropu · 2017-05-16T19:26:44Z

ping

gatorsmile · 2017-05-16T23:13:19Z

Will review it tomorrow. Ping me if I forgot it.

maropu · 2017-05-17T00:47:47Z

Thanks alot!

maropu · 2017-05-20T14:47:31Z

@gatorsmile ping

maropu · 2017-05-23T15:42:52Z

ping

gatorsmile · 2017-06-10T23:57:44Z

cc @wzhfy

wzhfy · 2017-06-11T18:09:03Z

Rename the file (SchemaUtil) also as SchemaUtils?

wzhfy · 2017-06-11T18:35:38Z

nit: duplicate """

wzhfy · 2017-06-11T18:36:01Z

Is header option necessary?

wzhfy · 2017-06-11T18:43:03Z

We also need to add tests for case sensitivity.

wzhfy · 2017-06-11T18:43:56Z

In addition to end-to-end tests, we can also add a suite for SchemaUtils and put the case sensitivity cases in it.

Added SchemaUtilsSuite.

maropu · 2017-06-11T21:34:27Z

Thanks! I'll update in a day

wzhfy · 2017-06-12T05:39:26Z

Hi @maropu , I just did some simple search, and found other places also related to duplicate columns. e.g. InsertIntoHadoopFsRelationCommand, PartitioningUtils.normalizePartitionSpec, SessionCatalog.alterTableSchema. Can you do a more comprehensive search to find if there are other places missed? Let's make all of them consistent as best as we can.

maropu · 2017-06-12T05:57:10Z

ok, I'll also check again. Thanks!

SparkQA · 2017-06-13T06:34:34Z

Test build #77971 has finished for PR 17758 at commit ac3dcc3.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-13T06:44:35Z

Test build #77973 has finished for PR 17758 at commit 36f6849.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-13T11:58:47Z

Test build #77981 has finished for PR 17758 at commit 0bd161e.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-13T12:44:39Z

Test build #77986 has finished for PR 17758 at commit 1438b23.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-13T17:23:33Z

Test build #77991 has finished for PR 17758 at commit 46a3f30.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

This reverts commit 12159c403955f54066ed8c532ed991f829edfc1f.

SparkQA · 2017-07-06T15:37:05Z

Test build #79284 has finished for PR 17758 at commit 5c29a75.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-07-07T05:47:34Z

+-- Catch case-sensitive name duplication
+SET spark.sql.caseSensitive=true;
+
+CREATE TABLE t(c0 STRING, c1 INT, c1 DOUBLE, c0 INT) USING parquet;


We didn't have test cases for create table before?

In DDLSuite, we already have simple tests for duplicate columns. we better moving these tests there?

We should keep them in one place. For now I think we still need to put them in DDLSuite because we need to run it with and without hive support. Can we pick some typical test cases here and move them to DDLSuite?

ok, will update

I moved some tests to DDLSuite and removed this file.

SparkQA · 2017-07-07T09:29:47Z

Test build #79320 has finished for PR 17758 at commit 8cbfbd0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-07T09:44:59Z

Test build #79323 has finished for PR 17758 at commit 5ed2c0d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-07-07T23:18:04Z

LGTM

cloud-fan · 2017-07-10T07:41:53Z

+ }
+ }
+
+ val errorMsg = intercept[AnalysisException] {


why is this test related to streaming?

If we have streaming metadata logs, we have the different code path from non-streaming cases in Dataset. So, the original motivation of this tests is to check the duplication even in this path.

cloud-fan · 2017-07-10T07:59:53Z

thanks, merging to master!

maropu · 2017-07-10T08:18:00Z

@cloud-fan Thanks! BTW, we better handle other duplication (e.g., local temporary tables)?;

scala> Seq((1, 1)).toDF("a", "a").createOrReplaceTempView("t")
scala> sql("SELECT * FROM t").show
+---+---+
| a| a|
+---+---+
| 1| 1|
+---+---+

This changes the existing behaviour though.

cloud-fan · 2017-07-10T13:16:02Z

We can create a ticket first and discuss there.

maropu · 2017-07-10T13:17:30Z

ok, I'll file a ticket first. Thanks.

shaulre · 2017-11-05T09:42:04Z

Still when you load the json file through Dataset[String] by doing spark.read.json(spark.read.textFile("json.file"), Spark does not trow any error and you get DataFrame with duplicate columns. Is that an expected behaviour and a feature or it's actually a bug?

maropu · 2017-11-05T10:24:03Z

yea, I think that is an expected behaviour, actually, some operations of DataFrame/Dataset accepts the duplication. e.g.,

scala> Seq((1, 2)).toDF("a", "a").show
+---+---+
| a| a|
+---+---+
| 1| 2|
+---+---+

Also, the original motivation of this pr does not intend to change existing behaviour.

shaulre · 2017-11-05T10:37:44Z

Why it's ok to have duplicate columns when you read from RDD/DS and not when you read directly from file? Maybe it's should be configurable option?
When you have duplicate columns, how can you deal with columns selection or renaming?

maropu · 2017-11-05T13:59:29Z

Because it is a design of DataFrame and join results could have duplicate column names. Anyway, we couldn't change this behaviour in minor releases.

…lt datasources ### What changes were proposed in this pull request? When `spark.sql.caseSensitive` is `false` (by default), check that there are not duplicate column names on the same level (top level or nested levels) in reading from in-built datasources Parquet, ORC, Avro and JSON. If such duplicate columns exist, throw the exception: ``` org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the data schema: ``` ### Why are the changes needed? To make handling of duplicate nested columns is similar to handling of duplicate top-level columns i. e. output the same error when `spark.sql.caseSensitive` is `false`: ```Scala org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the data schema: `camelcase` ``` Checking of top-level duplicates was introduced by #17758. ### Does this PR introduce _any_ user-facing change? Yes. For the example from SPARK-32431: ORC: ```scala java.io.IOException: Error reading file: file:/private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/spark-c02c2f9a-0cdc-4859-94fc-b9c809ca58b1/part-00001-63e8c3f0-7131-4ec9-be02-30b3fdd276f4-c000.snappy.orc at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1329) at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:78) ... Caused by: java.io.EOFException: Read past end of RLE integer from compressed stream Stream for column 3 kind DATA position: 6 length: 6 range: 0 offset: 12 limit: 12 range 0 = 0 to 6 uncompressed: 3 to 3 at org.apache.orc.impl.RunLengthIntegerReaderV2.readValues(RunLengthIntegerReaderV2.java:61) at org.apache.orc.impl.RunLengthIntegerReaderV2.next(RunLengthIntegerReaderV2.java:323) ``` JSON: ```scala +------------+ |StructColumn| +------------+ | [,,]| +------------+ ``` Parquet: ```scala +------------+ |StructColumn| +------------+ | [0,, 1]| +------------+ ``` Avro: ```scala +------------+ |StructColumn| +------------+ | [,,]| +------------+ ``` After the changes, Parquet, ORC, JSON and Avro output the same error: ```scala Found duplicate column(s) in the data schema: `camelcase`; org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the data schema: `camelcase`; at org.apache.spark.sql.util.SchemaUtils$.checkColumnNameDuplication(SchemaUtils.scala:112) at org.apache.spark.sql.util.SchemaUtils$.checkSchemaColumnNameDuplication(SchemaUtils.scala:51) at org.apache.spark.sql.util.SchemaUtils$.checkSchemaColumnNameDuplication(SchemaUtils.scala:67) ``` ### How was this patch tested? Run modified test suites: ``` $ build/sbt "sql/test:testOnly org.apache.spark.sql.FileBasedDataSourceSuite" $ build/sbt "avro/test:testOnly org.apache.spark.sql.avro.*" ``` and added new UT to `SchemaUtilsSuite`. Closes#29234 from MaxGekk/nested-case-insensitive-column. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

…atasource ### What changes were proposed in this pull request? Check that there are not duplicate column names on the same level (top level or nested levels) in reading from JDBC datasource. If such duplicate columns exist, throw the exception: ``` org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the customSchema option value: ``` The check takes into account the SQL config `spark.sql.caseSensitive` (`false` by default). ### Why are the changes needed? To make handling of duplicate nested columns is similar to handling of duplicate top-level columns i. e. output the same error: ```Scala org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the customSchema option value: `camelcase` ``` Checking of top-level duplicates was introduced by #17758, and duplicates in nested structures by #29234. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? Added new test suite `JdbcNestedDataSourceSuite`. Closes#29317 from MaxGekk/jdbc-dup-nested-columns. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

maropu force-pushed the SPARK-20460 branch from 861a074 to c222eaeCompare April 25, 2017 15:47

maropu force-pushed the SPARK-20460 branch from c222eae to de11a5bCompare April 26, 2017 01:38

maropu force-pushed the SPARK-20460 branch from de11a5b to 05a7a61Compare April 26, 2017 03:32

maropu force-pushed the SPARK-20460 branch from 05a7a61 to 42f3fc9Compare May 12, 2017 10:00

wzhfy reviewed Jun 11, 2017
View reviewed changes

maropu force-pushed the SPARK-20460 branch from 42f3fc9 to ac3dcc3Compare June 13, 2017 06:27

maropu force-pushed the SPARK-20460 branch from ac3dcc3 to 36f6849Compare June 13, 2017 06:37

maropu force-pushed the SPARK-20460 branch from 36f6849 to 0bd161eCompare June 13, 2017 07:46

maropu force-pushed the SPARK-20460 branch from 1438b23 to 46a3f30Compare June 13, 2017 13:10

maropu force-pushed the SPARK-20460 branch from 46a3f30 to da91ad1Compare June 13, 2017 23:55

maropu added 5 commits July 6, 2017 22:07

Revert "Brush up code"
be20127
This reverts commit 12159c403955f54066ed8c532ed991f829edfc1f.

Fix more issues
f41bf80

Revert DataSourceValidator
0526391

Add the check for external relation providers
9e199bc

[WIP] Handle DataSource name duplication in one place
1ae132d

maropu force-pushed the SPARK-20460 branch from 6b9aceb to c6b1952Compare July 6, 2017 13:08

Fix more
5c29a75

maropu force-pushed the SPARK-20460 branch from c6b1952 to 5c29a75Compare July 6, 2017 13:10

cloud-fan reviewed Jul 7, 2017
View reviewed changes

Move some tests to DDLSuite
5ed2c0d

maropu force-pushed the SPARK-20460 branch from 8cbfbd0 to 5ed2c0dCompare July 7, 2017 07:13

cloud-fan reviewed Jul 10, 2017
View reviewed changes

asfgit closed this in 647963aJul 10, 2017

MaxGekk mentioned this pull request Jul 28, 2020
[SPARK-32431][SQL] Check duplicate nested columns in read from in-built datasources #29234
Closed

MaxGekk mentioned this pull request Jul 31, 2020
[SPARK-32510][SQL] Check duplicate nested columns in read from JDBC datasource #29317
Closed

Uh oh!

Conversation

maropu commented Apr 25, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Apr 25, 2017

Uh oh!

SparkQA commented Apr 25, 2017

Uh oh!

SparkQA commented Apr 26, 2017

Uh oh!

SparkQA commented Apr 26, 2017

Uh oh!

SparkQA commented May 12, 2017

Uh oh!

maropu commented May 13, 2017

Uh oh!

maropu commented May 16, 2017

Uh oh!

gatorsmile commented May 16, 2017

Uh oh!

maropu commented May 17, 2017

Uh oh!

maropu commented May 20, 2017

Uh oh!

maropu commented May 23, 2017

Uh oh!

gatorsmile commented Jun 10, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wzhfyJun 11, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wzhfyJun 11, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maropu commented Jun 11, 2017

Uh oh!

wzhfy commented Jun 12, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maropu commented Jun 12, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

SparkQA commented Jun 13, 2017

Uh oh!

SparkQA commented Jul 6, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 7, 2017

Uh oh!

SparkQA commented Jul 7, 2017

Uh oh!

gatorsmile commented Jul 7, 2017

wzhfyJun 11, 2017•
edited
Loading

wzhfyJun 11, 2017•
edited
Loading

wzhfy commented Jun 12, 2017•
edited
Loading