[SPARK-11974][CORE]Not all the temp dirs had been deleted when the JVM exits

[pzzs]

deleting the temp dir like that

scala> import scala.collection.mutable
import scala.collection.mutable
scala> val a = mutable.Set(1,2,3,4,7,0,8,98,9)
a: scala.collection.mutable.Set[Int] = Set(0, 9, 1, 2, 3, 7, 4, 8, 98)
scala> a.foreach(x => {a.remove(x) })
scala> a.foreach(println(_))
98

You may not modify a collection while traversing or iterating over it.This can not delete all element of the collection

[rxin]

What's the problem with the original approach?

[pzzs]

It can not delete all element of shutdownDeletePaths.
Like the example above, this method can not delete all element of a.
@rxin

[rxin]

The part I don't get is --- where are we deleting elements of shutdownDeletePaths?

[pzzs]

shutdownDeletePaths.foreach { dirPath =>
try {
logInfo("Deleting directory " + dirPath)
Utils.deleteRecursively(new File(dirPath))
} catch {
case e: Exception => logError(s"Exception while deleting Spark temp dir: $dirPath", e)
}
}

Utils.deleteRecursively(new File(dirPath) call ShutdownHookManager.removeShutdownDeleteDir(file)

def deleteRecursively(file: File) {
...
ShutdownHookManager.removeShutdownDeleteDir(file)
...
}

ShutdownHookManager.removeShutdownDeleteDir(file)will deleting elements of shutdownDeletePaths

 def removeShutdownDeleteDir(file: File) {
val absolutePath = file.getAbsolutePath()
shutdownDeletePaths.synchronized {
shutdownDeletePaths.remove(absolutePath)
}
}

@rxin

[rxin]

Ah ok - can you update the change to add a line of comment saying we need to materialize the paths to delete because deleteRecursively removes items from shutdownDeletePaths as we are traversing through it?

[pzzs]

Ok, get it @rxin

[rxin]

need a space after // - otherwise this will fail the style check

[pzzs]

Is it OK? @rxin

[SparkQA]

Test build #2108 has finished for PR 9951 at commit a3e7cb8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * public final class SortedIterator extends UnsafeSorterIterator\n * case class GetStructField(child: Expression, ordinal: Int, name: Option[String] = None)\n * case class PrettyAttribute(name: String, dataType: DataType = NullType)\n

[rxin]

I've merged this. Thanks.