Uh oh!
There was an error while loading. Please reload this page.
[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0 - #10534
[SPARK-7689] Remove TTL-based metadata cleaning in Spark 2.0#10534JoshRosen wants to merge 13 commits into
Conversation
jerryshao
commented
Dec 31, 2015
Good to remove it :). It has been obsolete for a long time, and will potentially lead to some unexpected behaviors especially in Streaming (like block not found). |
SparkQA
commented
Dec 31, 2015
Test build #48528 has finished for PR 10534 at commit
|
rxin
commented
Jan 1, 2016
LGTM. We should let @tdas take a look at this though. |
SparkQA
commented
Jan 2, 2016
Test build #48592 has finished for PR 10534 at commit
|
SparkQA
commented
Jan 2, 2016
Test build #48590 has finished for PR 10534 at commit
|
SparkQA
commented
Jan 3, 2016
Test build #48601 has finished for PR 10534 at commit
|
JoshRosen
commented
Jan 3, 2016
Jenkins, retest this please. |
gatorsmile
commented
Jan 3, 2016
All the recent test builds failed due to the same issue. |
SparkQA
commented
Jan 3, 2016
Test build #48616 has finished for PR 10534 at commit
|
JoshRosen
commented
Jan 4, 2016
Jenkins, retest this please. |
SparkQA
commented
Jan 4, 2016
Test build #48645 has finished for PR 10534 at commit
|
andrewor14
commented
Jan 4, 2016
+1 |
There was a problem hiding this comment.
Isnt it simpler to keep the Scala map interface? Will minimize changes in rest of code in this class.
There was a problem hiding this comment.
I was worried about the pitfalls of .getOrElseUpdate not being atomic on ConcurrentHashMaps that had been wrapped into Scala maps.
SparkQA
commented
Jan 4, 2016
Test build #48693 has finished for PR 10534 at commit
|
0868fab to
8b7ca1cCompareThere was a problem hiding this comment.
Whoops, thanks for catching that.
tdas
commented
Jan 6, 2016
LGTM, pending tests. |
SparkQA
commented
Jan 7, 2016
Test build #48878 has finished for PR 10534 at commit
|
JoshRosen
commented
Jan 7, 2016
Jenkins, retest this please. |
SparkQA
commented
Jan 7, 2016
Test build #48889 has finished for PR 10534 at commit
|
rxin
commented
Jan 7, 2016
Merging. Thanks. |
JoshRosen
commented
Apr 17, 2020
Cross-references to related PRs (to aid other code archaeologists):
|
This PR removes
spark.cleaner.ttland the associated TTL-based metadata cleaning code.Now that we have the
ContextCleanerand a timer to trigger periodic GCs, I don't think thatspark.cleaner.ttlis necessary anymore. The TTL-based cleaning isn't enabled by default, isn't included in our end-to-end tests, and has been a source of user confusion when it is misconfigured. If the TTL is set too low, data which is still being used may be evicted / deleted, leading to hard to diagnose bugs.For all of these reasons, I think that we should remove this functionality in Spark 2.0. Additional benefits of doing this include marginally reduced memory usage, since we no longer need to store timetsamps in hashmaps, and a handful fewer threads.