[SPARK-11225]Prevent generate empty file

[viper-kun]

If no data will be written into the bucket, it will be generate empty files. So open() must be called in the first write(key,value).

[JoshRosen]

Jenkins, this is ok to test.

[JoshRosen]

Hey @viper-kun,

I played around with this optimization myself at one point. If I remember correctly, I think that you might have to update other parts of the code to account for the fact that empty partitions' files will now be missing instead of empty.

Do you have performance benchmarking results that motivated this change? Just curious to know how much of a speedup / benefit this gives.

[SparkQA]

Test build #44033 has finished for PR 9191 at commit 9887ef1.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[viper-kun]

Thanks @JoshRosen

Sorry, I don't do performacne test. As I know, it will reduce number of open file. When there too much empty file, it will get some benefit.

[SparkQA]

Test build #44053 has finished for PR 9191 at commit 2dd4d1e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class BinaryClassificationEvaluator @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * class MulticlassClassificationEvaluator @Since(\"1.5.0\") (@Since(\"1.5.0\") override val uid: String)\n * final class RegressionEvaluator @Since(\"1.4.0\") (@Since(\"1.4.0\") override val uid: String)\n * abstract class ColumnarIterator extends Iterator[InternalRow]\n * class SpecificColumnarIterator extends $\n

[viper-kun]

@JoshRosen
In my test environment, it do not have this error. Pls retest it.Thanks!

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #44118 has finished for PR 9191 at commit 2dd4d1e.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viper-kun]

Hi @davies,
Sorry, I do not understand Python. Can you help me fix it?

[davies]

This test is flaky, just re-test it.

[SparkQA]

Test build #1941 has finished for PR 9191 at commit 2dd4d1e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viper-kun]

@davies This test is flaky, pls re-test it.

[SparkQA]

Test build #1949 has finished for PR 9191 at commit 2dd4d1e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[viper-kun]

@JoshRosen

Is it ok? If it doesn't work, I will close this pr.

[srowen]

Note this duplicates #5622

[viper-kun]

ok. close it

[srowen]

OK, Josh also closed the other one. One JIRA is open now. At least have <= 1 PR.