Uh oh!
There was an error while loading. Please reload this page.
[SPARK-43576][CORE] Remove unused declarations from Core module - #41218
[SPARK-43576][CORE] Remove unused declarations from Core module#41218panbingkun wants to merge 2 commits into
Conversation
| protected val jobConfCacheKey: String = "rdd_%d_job_conf".format(id) | ||
| protected val inputFormatCacheKey: String = "rdd_%d_input_format".format(id) |
There was a problem hiding this comment.
This is a part of developer API. I don't think we can just remove this out.
There was a problem hiding this comment.
Ok, Let's reserve it.
HyukjinKwon
commented
May 18, 2023
@panbingkun how did you find them? FWIW, I thought we added a compilation feature to disallow unused variables IIRC, @LuciferYang . |
panbingkun
commented
May 19, 2023
1.When I review the HadoopRDD code, I noticed some inspection prompts, eg: 2.So I wonder if there are other cases in the core module, then I export 3.execute command: 4.The declared unused variable results in all 5.Open it by IDE. 6.Some inspect results may be inaccurate and ultimately require manual reconfirmation. |
LuciferYang
commented
May 19, 2023
Currently, we have only added compilation checks on Line 2864 in 411bcd2 For the current version of Scala 2.12, we can further try to add more checks, such as |
| (Math.min(totalGCTime / totalDuration + 0.5, 1)) : 1; | ||
| } | ||
| // When GCTimePercent is edited change ToolTips.TASK_TIME to match |
There was a problem hiding this comment.
hmm, why do you propose to remove this?
There was a problem hiding this comment.
ToolTips.TASK_TIME - Shaded red when garbage collection (GC) time is over 10% of task time
Adding this variable for the first time
CommitID: c037d25
JIRA: https://issues.apache.org/jira/browse/SPARK-12149
[SPARK-12149][WEB UI] Executor UI improvement suggestions - Color UI
PR: [SPARK-12149] [Web UI] Executor UI improvement suggestions - Color UI #10154
First declaration：https://github.com/apache/spark/pull/10154/files#diff-91f8d288f7aa0726fc067ad9eb67aff41c474ada4d702e5b393bd81ae5aee7b5R91-R92
First use：https://github.com/apache/spark/pull/10154/files#diff-3bb7f9962a6343cbc09be474b85f2f9ed2e80047f4863d447faa407d925f425cR82The variable content is directly hard coding in
executorspage-template.htmlfile
CommitID: b9bab4d
JIRA: https://issues.apache.org/jira/browse/SPARK-15951
[SPARK-15951] Change Executors Page to use datatables to support sorting columns and searching
PR: [SPARK-15951] Change Executors Page to use datatables to support sorting columns and searching #13670
The variable content is directly hard coding: https://github.com/apache/spark/pull/13670/files#diff-cc6993f75c03b53bb41d8d7113ed3d5157580e583672da68b15cbe4d9f374e15R36
Delete: https://github.com/apache/spark/pull/13670/files#diff-3bb7f9962a6343cbc09be474b85f2f9ed2e80047f4863d447faa407d925f425cL94
So I think it needs to be removed; Meanwhile, I did not find any usage in the spark code base using the following command: find . -name "*" -type file -print0 | xargs -0 grep "TASK_TIME"
MaxGekk
commented
May 26, 2023
+1, LGTM. Merging to master. |
### What changes were proposed in this pull request? The pr aims to remove unused declarations from `Core` module ### Why are the changes needed? Make code clean. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Closesapache#41218 from panbingkun/remove_unused_declaration_core. Authored-by: panbingkun <pbk1982@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
What changes were proposed in this pull request?
The pr aims to remove unused declarations from
CoremoduleWhy are the changes needed?
Make code clean.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.