Uh oh!
There was an error while loading. Please reload this page.
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up when unregisterShuffle. - #31648
[SPARK-34541][CORE] Fixed an issue where data could not be cleaned up when unregisterShuffle.#31648yikf wants to merge 3 commits into
Conversation
| def testConcat(inputs: String*): Unit = { | ||
| val expected = if (inputs.contains(null)) null else inputs.mkString | ||
| checkEvaluation(Concat(inputs.map(Literal.create(_, StringType))), expected, EmptyRow) | ||
| checkEvaluation(Concat(inputs.map(Literal.create(_, StringType))), expected) |
There was a problem hiding this comment.
For reviewer : clean up unused code for code simplifications.
Method signature for checkEvaluation as follow:
checkEvaluation(expression: => Expression, expected: Any, inputRow: InternalRow = EmptyRow)
There was a problem hiding this comment.
Is this test change related to the code change?
| metrics: ShuffleWriteMetricsReporter): ShuffleWriter[K, V] = { | ||
| val mapTaskIds = taskIdMapsForShuffle.computeIfAbsent( | ||
| handle.shuffleId, _ => new OpenHashSet[Long](16)) | ||
| mapTaskIds.synchronized { mapTaskIds.add(context.taskAttemptId()) } |
There was a problem hiding this comment.
For more convenient review:
ShuffleMapTask#runTaskval mapId = if (SparkEnv.get.conf.get(config.SHUFFLE_USE_OLD_FETCH_PROTOCOL)) { partitionId } else context.taskAttemptId()
SortShuffleManager#getWriterval mapTaskIds = taskIdMapsForShuffle.computeIfAbsent( handle.shuffleId, _ => new OpenHashSet[Long](16)) mapTaskIds.synchronized { mapTaskIds.add(context.taskAttemptId()) }
SortShuffleManager#unregisterShuffleOption(taskIdMapsForShuffle.remove(shuffleId)).foreach { mapTaskIds => mapTaskIds.iterator.foreach { mapTaskId => shuffleBlockResolver.removeDataByMap(shuffleId, mapTaskId) } }
There was a problem hiding this comment.
@xuanyuanking do you have an opinion here? I think you wrote this piece of the code.
yikf
commented
Feb 25, 2021
gentle ping @srowen , thanks for taking a look. |
AmplabJenkins
commented
Feb 25, 2021
Can one of the admins verify this patch? |
srowen
left a comment
There was a problem hiding this comment.
This is open vs branch-3.0 - needs to be master
| metrics: ShuffleWriteMetricsReporter): ShuffleWriter[K, V] = { | ||
| val mapTaskIds = taskIdMapsForShuffle.computeIfAbsent( | ||
| handle.shuffleId, _ => new OpenHashSet[Long](16)) | ||
| mapTaskIds.synchronized { mapTaskIds.add(context.taskAttemptId()) } |
There was a problem hiding this comment.
@xuanyuanking do you have an opinion here? I think you wrote this piece of the code.
mridulm
commented
Feb 25, 2021
The existing tests are not catching this issue - can you add something to ensure we test for this problem ? |
mridulm
commented
Feb 25, 2021
+CC @otterc |
| def testConcat(inputs: String*): Unit = { | ||
| val expected = if (inputs.contains(null)) null else inputs.mkString | ||
| checkEvaluation(Concat(inputs.map(Literal.create(_, StringType))), expected, EmptyRow) | ||
| checkEvaluation(Concat(inputs.map(Literal.create(_, StringType))), expected) |
There was a problem hiding this comment.
Is this test change related to the code change?
yikf
commented
Feb 26, 2021
The change in |
yikf
commented
Feb 26, 2021
i will add the UT for this problem. |
HyukjinKwon
commented
Feb 26, 2021
@yikf, can you open a PR against master branch? |
open a PR #31664 against master branch, close this PR for branch 3.0 |
What changes were proposed in this pull request?
Fixed an issue where data could not be cleaned up when unregisterShuffle.
Why are the changes needed?
While we use the old shuffle fetch protocol, we use partitionId as mapId in the ShuffleBlockId construction,but we use
context.taskAttemptId()as mapId that it is cached intaskIdMapsForShufflewhen wegetWriter[K, V].where data could not be cleaned up when unregisterShuffle ,because we remove a shuffle's metadata from the
taskIdMapsForShuffle's mapIds, the mapId iscontext.taskAttemptId()instead of partitionId.Does this PR introduce any user-facing change?
yes
How was this patch tested?
exist test.