Uh oh!
There was an error while loading. Please reload this page.
[SPARK-29816][MLLIB] Missing persist in mllib.evaluation.BinaryClassificationMetrics.recallByThreshold() - #26447
[SPARK-29816][MLLIB] Missing persist in mllib.evaluation.BinaryClassificationMetrics.recallByThreshold()#26447amanomer wants to merge 2 commits into
Conversation
AmplabJenkins
commented
Nov 9, 2019
Can one of the admins verify this patch? |
amanomer
commented
Nov 9, 2019
@MaxGekk Kindly review this PR. |
srowen
commented
Nov 9, 2019
There's no reason this has to be like 5 JIRAs and PRs. Just combine them into one, please. |
amanomer
commented
Nov 9, 2019
Okay. |
| mergeCombiners = (c1: BinaryLabelCounter, c2: BinaryLabelCounter) => c1 += c2 | ||
| ).sortByKey(ascending = false) | ||
| ) | ||
| binnedWeights.persist() |
There was a problem hiding this comment.
Could you explain why do you persist binnedWeights but not counts? As I can see binnedWeights is used only once ... or I missed something?
There was a problem hiding this comment.
Rdd scoreAndLabels.combineByKey is found to be used more than one time, as mentioned in SPARK-29816.
I think, counts rdd is constructed over scoreAndLabels.combineByKey after action sortByKey. counts is indirectly dependent on scoreAndLabels.combineByKey i.e, binnedWeights?
srowen
commented
Nov 10, 2019
Let's please merge this to #26454 after discussing some initial principles there. |
What changes were proposed in this pull request?
Use
persiston RDD which is used for more than one action.Why are the changes needed?
The rdd
scoreAndLabels.combineByKeyis used by two actions:sortByKeyandcount, so it needs to be persisted.Does this PR introduce any user-facing change?
No
How was this patch tested?
Tested manually.