SPARK-12619 Combine small files in a hadoop directory into single split

[navis]

When a directory contains too many (small) files, whole spark cluster will be exhausted scheduling tasks created for each file. Custom input format can handle that but if you're using hive metastore, it could hardly be an option.

[SparkQA]

Test build #48661 has finished for PR 10572 at commit 055f613.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class SimpleCombiner<K, V> implements InputFormat<K, V>
  • public static class InputSplits implements InputSplit, Configurable

[SparkQA]

Test build #48804 has finished for PR 10572 at commit e056332.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • public class SimpleCombiner<K, V> implements InputFormat<K, V>
  • public static class InputSplits implements InputSplit, Configurable

[HyukjinKwon]

Maybe we might have to correct the title just like the others, [SPARK-XXXX][SQL] (this is described in https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark).

[davies]

This is fixed in 2.0, could you close this PR?

[cerisier]

@davies do you have the commit that fixes this in 2.0 ?

[HyukjinKwon]

Is that #12095?

[jinxing64]

@HyukjinKwon
To merge small files, should I tune spark.sql.files.maxPartitionBytes? But IIUC it only works for FileSourceScanExec. So when I select from hive table, it doesn't work.