Uh oh!
There was an error while loading. Please reload this page.
[SPARK-13371] [CORE] TaskSetManager.dequeueSpeculativeTask compares Option[String] and String directly. - #11253
[SPARK-13371] [CORE] TaskSetManager.dequeueSpeculativeTask compares Option[String] and String directly.#11253srowen wants to merge 2 commits into
Conversation
… in at least one case a likely bug (TaskSetManager)
| // Test whether dependencies have been changed from its earlier parent RDD | ||
| assert(operatedRDD.dependencies.head.rdd != parentRDD) | ||
| assert(operatedRDD.dependencies.head != parentRDD) |
There was a problem hiding this comment.
Please fix the original misnomer that encouraged this bad comparison: parentRDD is not an RDD, but rather a Dependency.
andrewor14
commented
Feb 18, 2016
What are the implications of this bug? We never schedule anything rack local? |
srowen
commented
Feb 18, 2016
Good call, will make that change. Eh, yeah I'm not sure what the implication is but it doesn't sound great. Well, fix going in for sure. |
| @@ -338,7 +338,7 @@ private[spark] class TaskSetManager( | |||
| if (TaskLocality.isAllowed(locality, TaskLocality.RACK_LOCAL)) { | |||
| for (rack <- sched.getRackForHost(host)) { | |||
| for (index <- speculatableTasks if canRunOnHost(index)) { | |||
There was a problem hiding this comment.
nit, but I'd actually refactor this as a single for:
for {
rack <- sched.getRackForHost(host)
index <- speculatableTasks if canRunOnHost(index)
} {
valracks= tasks(index).preferredLocations.map(_.host).flatMap(sched.getRackForHost)
if (racks.contains(rack)) {
speculatableTasks -= index
returnSome((index, TaskLocality.RACK_LOCAL))
}
}There was a problem hiding this comment.
not a big deal but I think that can happen outside of this patch since it's just cleanup
There was a problem hiding this comment.
yup, it's style, not substance
SparkQA
commented
Feb 18, 2016
Test build #51488 has finished for PR 11253 at commit
|
andrewor14
commented
Feb 18, 2016
LGTM |
…pares Option and String directly. ## What changes were proposed in this pull request? Fix some comparisons between unequal types that cause IJ warnings and in at least one case a likely bug (TaskSetManager) ## How was the this patch tested? Running Jenkins tests Author: Sean Owen <sowen@cloudera.com> Closes#11253 from srowen/SPARK-13371. (cherry picked from commit 7856253) Signed-off-by: Andrew Or <andrew@databricks.com>
andrewor14
commented
Feb 18, 2016
merged into master 1.6 |
SparkQA
commented
Feb 18, 2016
Test build #51494 has finished for PR 11253 at commit
|
What changes were proposed in this pull request?
Fix some comparisons between unequal types that cause IJ warnings and in at least one case a likely bug (TaskSetManager)
How was the this patch tested?
Running Jenkins tests