[SPARK-13530][SQL] Add ShortType support to UnsafeRowParquetRecordReader

[viirya]

JIRA: https://issues.apache.org/jira/browse/SPARK-13530

What changes were proposed in this pull request?

By enabling vectorized parquet scanner by default, the unit test ParquetHadoopFsRelationSuite based on HadoopFsRelationTest will be failed due to the lack of short type support in UnsafeRowParquetRecordReader. We should fix it.

The error exception:

[info] ParquetHadoopFsRelationSuite:
[info] - test all data types - StringType (499 milliseconds)
[info] - test all data types - BinaryType (447 milliseconds)
[info] - test all data types - BooleanType (520 milliseconds)
[info] - test all data types - ByteType (418 milliseconds)
00:22:58.920 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 124.0 (TID 1949)
org.apache.commons.lang.NotImplementedException: Unimplemented type: ShortType
at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader$ColumnReader.readIntBatch(UnsafeRowParquetRecordReader.java:769)
at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader$ColumnReader.readBatch(UnsafeRowParquetRecordReader.java:640)
at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader$ColumnReader.access$000(UnsafeRowParquetRecordReader.java:461)
at org.apache.spark.sql.execution.datasources.parquet.UnsafeRowParquetRecordReader.nextBatch(UnsafeRowParquetRecordReader.java:224)

How was this patch tested?

The unit test ParquetHadoopFsRelationSuite based on HadoopFsRelationTest will be failed due to the lack of short type support in UnsafeRowParquetRecordReader. By adding this support, the test can be passed.

[SparkQA]

Test build #52117 has finished for PR 11412 at commit e923f7d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viirya]

cc @nongli@rxin

[nongli]

lgtm

[rxin]

Thanks - merging this in master. It would've been better if we could have a unit test for this module, rather than relying on some integration tests.

[JoshRosen]

It looks like this may now be failing tests in master? Take a look at https://spark-tests.appspot.com/tests/org.apache.spark.sql.sources.ParquetHadoopFsRelationSuite/test%20all%20data%20types%20-%20ByteType

[viirya]

@JoshRosen looks like at the same module but a different problem. I will look at it.

[nongli]

@viirya I fixed it with this patch:
#11414

[viirya]

@nongli Got it. Thanks!