[SPARK-6016][SQL] Cannot read the parquet table after overwriting the existing table when spark.sql.parquet.cacheMetadata=true

[yhuai]

Please see JIRA (https://issues.apache.org/jira/browse/SPARK-6016) for details of the bug.

[yhuai]

@liancheng Can we remove FilteringParquetRowInputFormat since task side split is in parquet? If you think it is good, we can do it in another PR.

[SparkQA]

Test build #27966 has finished for PR 4775 at commit 1541554.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

@yhuai As long as we decide to completely deprecate client side metadata reading, it's OK to remove FilteringParquetInputFormat.

[liancheng]

@yhuai Actually, as we discussed offline, FilteringParquetRowInputFormat is still necessary, as we have to do schema merging in getSplits to prevent the exception thrown in SPARK-6010

[liancheng]

This LGTM, please help rebasing it, then I can merge it.

[SparkQA]

Test build #28006 has finished for PR 4775 at commit 78787b1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

Merging into master and branch-1.3, thanks!

[karthikgolagani]

@liancheng
Hi lian, if you are using sparkContext(sc), you can set ("parquet.enable.summary-metadata", "false") like below:

sc.("parquet.enable.summary-metadata", "false"). This fixed my issue instantly .
I did it in my spark streaming application.

WARN ParquetOutputCommitter: could not write summary file for hdfs://localhost/user/hive/warehouse java.lang.NullPointerException