[SPARK-12293][SQL] Support UnsafeRow in LocalTableScan

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-12293

Make LocalTableScan support UnsafeRow.

[SparkQA]

Test build #47628 has finished for PR 10283 at commit ca4326a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class LocalRelation(output: Seq[Attribute], data: Seq[UnsafeRow] = Nil)\n

[SparkQA]

Test build #47634 has finished for PR 10283 at commit a0a991a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class LocalRelation(output: Seq[Attribute], data: Seq[UnsafeRow] = Nil)\n

[cloud-fan]

The return type of UnsafeProjection.apply is UnsafeRow already, looks like we don't need the asInstanceOf here?

[viirya]

I think the commits to fix these test errors are worth having their PR. I will create them later.

[cloud-fan]

Normally Decimal should only be used inside spark SQL as the internal representation of decimal type, and we don't need to catch it here. Do we break it in tests?

[viirya]

Yes. constructorFor will call dataTypeFor to determine if a type is ObjectType or not. If there is not case for Decimal, it will be recognized as ObjectType and causes bug.

[SparkQA]

Test build #47651 has finished for PR 10283 at commit f0e6ac0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class LocalRelation(output: Seq[Attribute], data: Seq[UnsafeRow] = Nil)\n

[cloud-fan]

If the unwrapped is null, the Invoke will return null, see https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects.scala#L172-L175

I think we don't need this extra If, did I miss something here?

[viirya]

I think you are right. I add this to make sure it is not possibly creating wrong values when I debug. I will remove these If.

[SparkQA]

Test build #47829 has finished for PR 10283 at commit 97a81c3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class LocalRelation(output: Seq[Attribute], data: Seq[UnsafeRow] = Nil)\n

[SparkQA]

Test build #47888 has finished for PR 10283 at commit 97d390f.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

retest this please.

[SparkQA]

Test build #47911 has finished for PR 10283 at commit 97d390f.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class LocalRelation(output: Seq[Attribute], data: Seq[UnsafeRow] = Nil)\n

[viirya]

retest this please.

[SparkQA]

Test build #47920 has finished for PR 10283 at commit 97d390f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class LocalRelation(output: Seq[Attribute], data: Seq[UnsafeRow] = Nil)\n

[cloud-fan]

we can use encoderFor[T] here

[viirya]

SQLUserDefinedType support for encoder is proposed in another pr #10390.

[viirya]

@cloud-fan I think I have addressed all your comments. These bugs found in implementing UnsafeRow support in LocalTableScan are submitted as other PRs with their tests, so you can review them better.

[SparkQA]

Test build #48094 has finished for PR 10283 at commit 2500de3.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * case class LocalRelation(output: Seq[Attribute], data: Seq[UnsafeRow] = Nil)\n

[viirya]

retest this please.