[SPARK-20741][Spark Submit] Added cleanup of JARs archive generated by SparkSubmit

[liorregev]

What changes were proposed in this pull request?

Deleted generated JARs archive after distribution to HDFS

How was this patch tested?

Please review http://spark.apache.org/contributing.html before opening a pull request.

[srowen]

That seems OK to me. It might be a good time to address similar issues elsewhere. For instance, look at Client.createConfArchive. The one place it's called, I think the file can be deleted after it's uploaded. There are a few other potential situations like this we could clean up.

[vanzin]

I don't think this is really necessary. These files are created in Utils.getLocalDir, which on the launcher side is a temporary directory (see Utils.getOrCreateLocalRootDirsImpl). Meaning that as soon as the launcher exits, these files will be deleted.

If you really want to fix this instance, it may be better to follow Sean's suggestion and fix all instances, creating an explicit temporary directory where the files are stored. All this is going to do, though, is to delete the files earlier - they'd still be deleted when the process exits.

[liorregev]

Actually I ran into a problem with this not getting cleaned up.
After your explanation I can understand why it wasn't deleted.
I am running spark on EMR and the easiest way to programmatically submit applications to the cluster was to create an HTTP service that accepts the application details and programmatically calls SparkSubmit.main so the process never really exits.
I managed to solve this with spark.yarn.archive so I don't need this implemented, I just figured it would have been a better solution.

[vanzin]

I just figured it would have been a better solution.

It might be a good idea to do it, but then you can't just add this one line, you have to look at all the temp files that Client.scala generates.

[srowen]

@liorregev if you'll take care of a couple other cases like this here, it looks OK to merge. Proactively cleaning up seems reasonable.

[SparkQA]

Test build #3751 has finished for PR 17986 at commit cb03d8a.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #3753 has finished for PR 17986 at commit cb03d8a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master/2.2. It's a win and on second look it wasn't obvious that there's another instance of this that can safely be cleaned up.