[SPARK-21236] Make the threshold of using HighlyCompressedStatus configurable.

[jinxing64]

What changes were proposed in this pull request?

Currently the threshold of using HighlyCompressedMapStatus is hardcoded 2000.
We could make this configurable. Thus users having enough memory on driver can configure the threshold to be larger thus to save the size of blocks more accurately in CompressedMapStatus.

[jiangxb1987]

This change LGTM, should we also add a new test case in MapStatusSuite ？

[jinxing64]

Sure, thanks for review :)

[SparkQA]

Test build #78761 has finished for PR 18446 at commit e219690.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[jinxing64]

Jenkins, retest this please.

[srowen]

It's probably OK, but what's the use case for configuring it? when would a caller know to set it higher or lower? just trying to figure out if this is a meaningful knob.

[jiangxb1987]

I guess the size of blocks are not very accurately stored in HighlyCompressedMapStatus, so if we have enough memory in the driver, we prefer to store them in CompressedMapStatus .

[jinxing64]

Yes, this is discussed ever in #16989 . Only average size of blocks are stored in HighlyCompressedMapStatus. If driver has enough memory, we can store more accurately in CompressedMapStatus. This is pretty useful for example controlling shuffle blocks.

[SparkQA]

Test build #78768 has finished for PR 18446 at commit 7b78df5.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #78774 has finished for PR 18446 at commit 7b78df5.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jinxing64]

Jenkins, retest this please.

[SparkQA]

Test build #78790 has finished for PR 18446 at commit 7b78df5.

• This patch fails due to an unknown error code, -10.
• This patch merges cleanly.
• This patch adds no public classes.

[jinxing64]

Jenkins, retest this please.

[SparkQA]

Test build #78801 has finished for PR 18446 at commit 7b78df5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

is this still useful after we have #18031 ?

I think users can just set spark.shuffle.accurateBlockThreshold to a lower value if they want to esitimate shuffle block size more accurate.

[jinxing64]

True. I just try to make it more complete and refine the hardcode.
spark.shuffle.accurateBlockThreshold is useful, but sometimes it's hard for user to tune it("how low is enough? and at the same time I want to control the size of MapStatus within a fixed bound). This is corner case anyway.
I can close this pr if you think it too trivial.

[cloud-fan]

every new config comes with a cost that users have to learn about it. For this case users already have a config(spark.shuffle.accurateBlockThreshold) to tune it, and this config is also hard to tune IMO.

Let's close it first. If we get a real use case that needs this config, we can reopen.

[jinxing64]

Sure :)