[SPARK-8414] Ensure context cleaner periodic cleanups

[andrewor14]

Garbage collection triggers cleanups. If the driver JVM is huge and there is little memory pressure, we may never clean up shuffle files on executors. This is a problem for long-running applications (e.g. streaming).

[andrewor14]

@tdas@JoshRosen

[SparkQA]

Test build #46976 has finished for PR 10070 at commit c96d5e2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * * For example, we build an encoder forcase class Data(a: Int, b: String)and the real type\n * case class UpCast(child: Expression, dataType: DataType, walkedTypePath: Seq[String])\n * case class InitializeJavaBean(beanInstance: Expression, setters: Map[String, Expression])\n

[andrewor14]

retest this please

[SparkQA]

Test build #46993 has finished for PR 10070 at commit c96d5e2.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

retest this please

[JoshRosen]

After this change, do you think that there's any chance that we could deprecate the somewhat unsafe spark.cleaner.ttl parameter? Should we update the docs to reflect this new parameter and suggest that spark.cleaner.ttl should no longer need to be set? Also fine to do that separately, but just wondering.

[andrewor14]

Let's do that separately

[JoshRosen]

LGTM

[SparkQA]

Test build #47015 has finished for PR 10070 at commit c96d5e2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2143 has finished for PR 10070 at commit c96d5e2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

LGTM, so I'm going to merge this into master and branch-1.6.

[SparkQA]

Test build #2145 has finished for PR 10070 at commit c96d5e2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #2144 has finished for PR 10070 at commit c96d5e2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.