Uh oh!
There was an error while loading. Please reload this page.
[SPARK-19659][Core]Disable spark.reducer.maxReqSizeShuffleToMem - #18467
[SPARK-19659][Core]Disable spark.reducer.maxReqSizeShuffleToMem#18467zsxwing wants to merge 4 commits into
Conversation
zsxwing
commented
Jun 29, 2017
cc @JoshRosen As it's impossible to safely revert #16989, I just changed the default value to Long.MaxValue. |
| By allowing it to limit the number of fetch requests, this scenario can be mitigated. | ||
| </td> | ||
| </tr> | ||
| <tr> |
There was a problem hiding this comment.
don't document it as it's not a safe feature now.
SparkQA
commented
Jun 29, 2017
Test build #78924 has finished for PR 18467 at commit
|
| .doc("The blocks of a shuffle request will be fetched to disk when size of the request is " + | ||
| "above this threshold. This is to avoid a giant request takes too much memory.") | ||
| .bytesConf(ByteUnit.BYTE) | ||
| .createWithDefaultString("200m") |
There was a problem hiding this comment.
Might want to put a .internal() here.
JoshRosen
left a comment
There was a problem hiding this comment.
LGTM overall. This seems like the cleanest fix for now to unblock the RC.
I do think this feature is valuable so we should see if there's a way to support it without requiring any backend changes to the shuffle service.
Let's make sure to update the "Fix Version" on the JIRA which added this feature so that it's clear that we've not shipped it in 2.2.x.
cloud-fan
commented
Jun 30, 2017
thanks, merging to master/2.2! |
Disable spark.reducer.maxReqSizeShuffleToMem because it breaks the old shuffle service. Credits to wangyum Closes#18466 Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Author: Yuming Wang <wgyumg@gmail.com> Closes#18467 from zsxwing/SPARK-21253. (cherry picked from commit 80f7ac3) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
cloud-fan
commented
Jun 30, 2017
@JoshRosen I'd argue that this feature is shipped in 2.2.x, as users can change the config value to enable it. Though this is kind of an experimental feature that breaks old shuffle service. |
cloud-fan
commented
Jun 30, 2017
Oops, @wangyum sorry I forgot to set the author name... will be more careful next time :) |
SparkQA
commented
Jun 30, 2017
Test build #78943 has finished for PR 18467 at commit
|
What changes were proposed in this pull request?
Disable spark.reducer.maxReqSizeShuffleToMem because it breaks the old shuffle service.
Credits to @wangyum
Closes#18466
How was this patch tested?
Jenkins