[SPARK-8610] [SQL] Separate Row and InternalRow (part 2)

[davies]

Currently, we use GenericRow both for Row and InternalRow, which is confusing because it could contain Scala type also Catalyst types.

This PR changes to use GenericInternalRow for InternalRow (contains catalyst types), GenericRow for Row (contains Scala types).

Also fixes some incorrect use of InternalRow or Row.

[SparkQA]

Test build #35740 has finished for PR 7003 at commit defe931.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35737 timed out for PR 7003 at commit 288b31f after a configured wait of 175m.

[SparkQA]

Test build #960 has finished for PR 7003 at commit defe931.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

should we add final to make sure all InternalRow implementations can not define these functions?

[davies]

will do

[SparkQA]

Test build #35758 timed out for PR 7003 at commit 6f99a97 after a configured wait of 175m.

[SparkQA]

Test build #35901 has finished for PR 7003 at commit bd4e99c.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35903 timed out for PR 7003 at commit d2ebd72 after a configured wait of 175m.

[SparkQA]

Test build #35908 has finished for PR 7003 at commit 87b13cf.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35916 has finished for PR 7003 at commit efd0b25.

• This patch fails some tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[JoshRosen]

(That test failure is my fault; already hotfixed)

[SparkQA]

Test build #969 has finished for PR 7003 at commit efd0b25.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35921 has finished for PR 7003 at commit efd0b25.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #970 has finished for PR 7003 at commit efd0b25.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35934 has finished for PR 7003 at commit d05866c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

LGTM.

For data source API, I understand we need an efficient way to cast between InternalRow and Row. But I'm still feel we should make Row and InternalRow siblings so that we can ensure that we don't forget to do the row transformation at compile time. I'll look into it.

[cloud-fan]

Do we need SparkSqlSerializer to serialize GenericRow?

[davies]

I haven't looked the details, just follow the old code.

[davies]

In order to reduce the conflicts, I'm merging this into master.