[SPARK-8432] [SQL] fix hashCode() and equals() of BinaryType in Row

[davies]

Also added more tests in LiteralExpressionSuite

[SparkQA]

Test build #35112 has finished for PR 6876 at commit 53c38b1.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

Is it same with java.util.Arrays.hashCode?

[davies]

Good idea, we should use that.

[SparkQA]

Test build #35114 has finished for PR 6876 at commit 0fff25d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35144 has finished for PR 6876 at commit 5819d33.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35146 has finished for PR 6876 at commit 6ad2a90.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

@marmbrus Could you help to review this one?

[marmbrus]

Existing: can you add some javadoc to this class to explain what its used for and why its in Java?

[davies]

Because Row is a trait, UnsafeRow and SpecificRow are both in Java, they can not inherit some default implementations from Row, so created BaseRow in Java for them. Right now, we have InternalRow, will be clean these in another PR.

[marmbrus]

@davies, thanks for working on this! I'm okay with this approach, but did you consider the alternative, where we instead change the internal type of BytesType to be a wrapper class instead of Array[Byte], and define a good equality function for that wrapper?

[davies]

@marmbrus We're working to have more efficient representation in catalyst, putting Array[Byte] inside a wrapper sounds not in the same direction. I'd like to go this approach.

[marmbrus]

I think using a wrapper might be necessary for efficiency. For example, we will want to reuse the same byte array when reading from something like parquet, instead of needing to allocate one of the exact size each time (think java.nio.ByteBuffer). However, I'm fine merging this first.

[SparkQA]

Test build #35189 timed out for PR 6876 at commit d96929b after a configured wait of 175m.

[cloud-fan]

Will we change it to isInstanceOf[InternalRow] after #6869?

[davies]

yes

[rxin]

this is a pretty weird way of indenting. you can do

List(0, 1, Int.MinValue, Int.MaxValue).foreach { d =>
...
}

or

for (d <-List(0, 1, Int.MinValue, Int.MaxValue)) {
...
}

[SparkQA]

Test build #35225 has finished for PR 6876 at commit 41caec6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35283 has finished for PR 6876 at commit bd20780.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #937 has finished for PR 6876 at commit a0626ed.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35284 has finished for PR 6876 at commit a0626ed.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

test this please

[SparkQA]

Test build #939 has finished for PR 6876 at commit a0626ed.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class SerializableConfiguration(@transient var value: Configuration) extends Serializable
  • class SerializableJobConf(@transient var value: JobConf) extends Serializable

[SparkQA]

Test build #35307 has finished for PR 6876 at commit a0626ed.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35322 has finished for PR 6876 at commit 32d9811.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

Should we also update equals for generated code?

[davies]

That's already done

[cloud-fan]

Ah I see, genEqual has already handled BinaryType.

[SparkQA]

Test build #946 has finished for PR 6876 at commit 32d9811.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35483 has finished for PR 6876 at commit 429c2c0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class UnresolvedAlias(child: Expression) extends NamedExpression
  • abstract class ExtractValueWithStruct extends ExtractValue

[marmbrus]

Thanks, merging to master.