[SPARK-2890][SQL] Allow reading of data when case insensitive resolution could cause possible ambiguity.

[marmbrus]

Throwing an error in the constructor makes it possible to run queries, even when there is no actual ambiguity. Remove this check in favor of throwing an error in analysis when they query is actually is ambiguous.

Also took the opportunity to add test cases that would have caught a subtle bug in my first attempt at fixing this and refactor some other test code.

[SparkQA]

QA tests have started for PR 2209 at commit a703ff4.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2209 at commit a703ff4.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class LowerCaseSchema(child: LogicalPlan) extends UnaryNode with Logging

[yhuai]

Reading parquet files in HiveContext triggers the problem? If we have two columns c1, C1, we will not be able to read C1 when we are using case insensitive resolution, right?

[yhuai]

Provide more information on which column (with the original column name) we will keep in the lowerCaseSchema?

[marmbrus]

I actually encountered the error with a jsonRDD, but yeah it could happen with parquet files as well. Your comment about joins though makes me think that we should just get rid of this check entirely. We can throw an error when your query is invalid, but throwing an exception just because at some point in a query something could be ambiguous seems overly restrictive.

[yhuai]

Sounds good. I was not sure how to correctly query those results with ambiguous schemas when I added that check. Seems an more informative logging entry is better than an exception.

[SparkQA]

QA tests have started for PR 2209 at commit a703ff4.

• This patch merges cleanly.

[SparkQA]

Tests timed out after a configured wait of 120m.

[marmbrus]

Jenkins, test this please.

[marmbrus]

Ahh, this reorders the schema and breaks things. Props to @andyk.

[SparkQA]

QA tests have started for PR 2209 at commit a703ff4.

• This patch does not merge cleanly!

[SparkQA]

QA tests have finished for PR 2209 at commit a703ff4.

• This patch fails unit tests.
• This patch does not merge cleanly!

[SparkQA]

QA tests have started for PR 2209 at commit a703ff4.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2209 at commit a703ff4.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class LowerCaseSchema(child: LogicalPlan) extends UnaryNode with Logging

[SparkQA]

QA tests have started for PR 2209 at commit 729cca4.

• This patch merges cleanly.

[SparkQA]

Tests timed out after a configured wait of 120m.

[JoshRosen]

Jenkins will actually show you how long the tests took, which can be helpful in narrowing down why we're seeing these timeouts. In this case, it looks like the majority of the time is spent in certain Hive compatibility tests:

https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/90/testReport/org.apache.spark.sql.hive.execution/HiveCompatibilitySuite/

[marmbrus]

@JoshRosen I am hoping that #2164 will fix the test time outs.

[SparkQA]

QA tests have started for PR 2209 at commit 729cca4.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2209 at commit 729cca4.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
  • class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
  • class Encoder[T <: NativeType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
  • class Encoder extends compression.Encoder[IntegerType.type]
  • class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[IntegerType.type])
  • class Encoder extends compression.Encoder[LongType.type]
  • class Decoder(buffer: ByteBuffer, columnType: NativeColumnType[LongType.type])

[marmbrus]

Merged to master. Thanks for looking this over!