Uh oh!
There was an error while loading. Please reload this page.
[SPARK-25550][WEBUI]Environment Page of Spark Job History UI showing wrong value for spark.ui.retainedJobs - #22567
[SPARK-25550][WEBUI]Environment Page of Spark Job History UI showing wrong value for spark.ui.retainedJobs#22567shivusondur wants to merge 3 commits into
Conversation
…value for spark.ui.retainedJobs
379493e to
5c401c4Compareshivusondur
commented
Sep 28, 2018
@vanzin Please review this patch. |
vanzin
commented
Oct 4, 2018
Hmmm, not sure about this. That page is supposed to show the application's configuration, not the history server configuration. If you want to add a new page to show SHS-specific configs, then that would be a better change. Otherwise this can be misleading. e.g. knowing the original config value can help someone debug things if the app ran into OOM issues. Otherwise I don't think this should go in. |
…wrong value for spark.ui.retainedJobs
@vanzin History UI
Only in the history UI, History Properties will be displayed. In live UI history properties won't be there Live UI |
vanzin
commented
Oct 15, 2018
That has the same problem as before. This page is for the application, not for the history server. It should not show different things just because you're looking at it in the history server. If you want to expose the SHS config you have to do it in a separate page that is not part of the application-specific UI. And then you have to be careful with what you show there, since the SHS config may have sensitive information that users should not see. |
AmplabJenkins
commented
Oct 22, 2018
Can one of the admins verify this patch? |
Closesapache#22567Closesapache#18457Closesapache#21517Closesapache#21858Closesapache#22383Closesapache#19219Closesapache#22401Closesapache#22811Closesapache#20405Closesapache#21933Closesapache#22819 from srowen/ClosePRs. Authored-by: Sean Owen <sean.owen@databricks.com> Signed-off-by: Sean Owen <sean.owen@databricks.com>
What changes were proposed in this pull request?
Currently, in History spark UI, the number of Jobs in the jobs page and "spark.ui.retainedJobs" value in
the Environment page is not matching.
So we need to override the spark properties from the History spark-default.conf properties, which will match the number of jobs showing in jobs page and value of "spark.ui.retainedJobs" in the Environment Page.
Same applies to other History specific parameters such as "spark.ui.retainedTasks" etc.
How was this patch tested?
Tested in master branch
"spark-shell --master yarn --conf spark.ui.retainedJobs=100"
Attached the test result screenshot
spark.ui.retainedJobs is showing specific to historypage
Now the total jobs showing and the spark.ui,retainedJobs value in the Environment page are matching.