[SPARK-10339] [SPARK-10334] [SPARK-10301] [SQL]Partitioned table scan can OOM driver and throw a better error message when users need to enable parquet schema merging

[yhuai]

This fixes the problem that scanning partitioned table causes driver have a high memory pressure and takes down the cluster. Also, with this fix, we will be able to correctly show the query plan of a query consuming partitioned tables.

https://issues.apache.org/jira/browse/SPARK-10339
https://issues.apache.org/jira/browse/SPARK-10334

Finally, this PR squeeze in a "quick fix" for SPARK-10301. It is not a real fix, but it just throw a better error message to let user know what to do.

[yhuai]

@liancheng@marmbrus I add the check to make sure the number of fields of a parquet file's struct have the same number of the corresponding struct in the table schema. If this check fails, we will ask users to enable mergeSchema.

[marmbrus]

test this please

[liancheng]

Just a note: this is a quick fix version of #8509.

[SparkQA]

Test build #41770 has finished for PR 8515 at commit b509bee.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #41771 has finished for PR 8515 at commit b509bee.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[liancheng]

Would be nice to have the following test case in ParquetHadoopFsRelationSuite:

 test("SPARK-10334 Projections and filters should be kept in physical plan") {
withTempPath { dir =>valpath= dir.getCanonicalPath
sqlContext.range(2).select('id as 'a, 'id as 'b).write.partitionBy("b").parquet(path)
valdf= sqlContext.read.parquet(path).filter('a===0).select('b)
valphysicalPlan= df.queryExecution.executedPlan
assert(physicalPlan.collect { casep: execution.Project=> p }.length ===1)
assert(physicalPlan.collect { casep: execution.Filter=> p }.length ===1)
}
}

And probably add [SPARK-10301] in the PR title, since this PR also delivers a quick fix for it.

Otherwise LGTM. Verified locally that filter push-down and column pruning both work properly.

[SparkQA]

Test build #41786 has finished for PR 8515 at commit 8ecd3a6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[marmbrus]

Thanks, merging to master and branch-1.5.