[SPARK-12299][CORE] Remove history serving functionality from Master by BryanCutler · Pull Request #10991 · apache/spark

BryanCutler · 2016-01-29T22:50:02Z

Remove history server functionality from standalone Master. Previously, the Master process rebuilt a SparkUI once the application was completed which sometimes caused problems, such as OOM, when the application event log is large (see SPARK-6270). Keeping this functionality out of the Master will help to simplify the process and increase stability.

Testing for this change included running core unit tests and manually running an application on a standalone cluster to verify that it completed successfully and that the Master UI functioned correctly. Also added 2 unit tests to verify killing an application and driver from MasterWebUI makes the correct request to the Master.

…elated WebUI and JSON ApiRootResources

SparkQA · 2016-01-29T22:58:47Z

Test build #50415 has finished for PR 10991 at commit 6918f0b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2016-01-29T23:13:38Z

Removing the history serving from Master is pretty straightforward, but leaves some gaps in the UI and serving JSON requests. I am marking this as WIP to get some discussion on how to best handle a few of these points.

Currently, the Master page has a the application name as a link to the Application UI Page. While the app is running, it has a link to the live application server, then when complete the Master replaces that link with one for the rebuilt UI.
Now, since there is no rebuilt UI, I left the live link, then once complete the application name is static and does not provide a link
Similarly, the Master Application page has a link to "Application Detail" to open up the Application UI Page. While the app is running, it has a link to the live application server, then when complete the Master replaces that link with one for the rebuilt UI.
Now, the live link is still there while the application is running and once complete the "Application Detail" is removed entirely.
Currently, the Master follows the REST API to serve JSON data and the MasterPage extends the trait UIRoot to allow requests for ApiRootResources.
Since a SparkUI with application info is required to serve these resources, now MasterPage no longer extends this trait.
Although the Master can no long serve ApiRootResources, it still can list applications and also redirect resource requests to any currently running application.
I put in some functionality that redirects requests, but I stopped short of serving the application list (the "/applications" endpoint) because it was getting a little messy. Is this something that is worth having in? Not going to do this, since there are probably not many people depending on this API

BryanCutler · 2016-02-01T20:21:45Z

cc @andrewor14, @squito for thoughts on above

andrewor14 · 2016-02-26T01:16:38Z

Ah, that's too bad. When I first proposed to do this I did not think there would be a REST API. Overall I'm still in favor of removing the functionality from Master. I don't have strong opinions about what we should do about the REST API but I believe that's less important; I don't think many people depend on the specific get SparkUI call from the standalone Master.

I believe @JoshRosen has some opinions about how we should move forward in Spark 2.0.

BryanCutler · 2016-03-04T22:02:36Z

Hi @JoshRosen , what are your thoughts on the Master REST API if history is no longer available?

JoshRosen · 2016-03-07T19:02:30Z

The Master REST API functionality which existed prior to the unification with the Spark driver REST API should still work after this change, so we shouldn't break or remove those methods.

I think that there are a few ways to handle this JIRA / task, some more complex than others. The only "must have" requirement for 2.0 is removing the HistoryServer functionality; other cleanup and interface refactoring can be deferred for later. So, I wouldn't try to do any major refactoring or cleanup; just remove the feature so we get the API / functionality change in and address the other pieces later.

BryanCutler · 2016-03-16T22:35:49Z

@JoshRosen @andrewor14 , just to be clear what needs to be done here - we want to keep the ability for Master to rebuild a SparkUI to be able to respond to REST API calls, but just not automatically rebuild every SparkUI when complete? Also, need to remove UI links to completed apps, etc.

JoshRosen · 2016-03-16T23:36:12Z

I don't think that the Master should have any event-log consumption logic whatsoever.

BryanCutler · 2016-03-17T18:01:04Z

Maybe I'm missing something then. Most of the REST API defined in ApiRootResource depends on having a SparkUI object which the Master currently creates by playing back event-log files. How can we keep this API in the Master and remove event-log consumption without some major refactoring? Is there some other way for the Master to create a SparkUI on the fly?

steveloughran · 2016-03-28T19:00:17Z

There's a more fundamental issue which the history server has too: log replay is too expensive for long-lived applications. replay time is O(jobs)+O(stages), even if GC keeps those numbers in the main UI down.

I'm not proposing a solution other than to observe that it's essentially some form of ETL operation, one that may be possible to do prepare for before building the UI up, just running through the events and discarding old/surplus ones. Has anyone played with something like that ?

andrewor14 · 2016-03-28T20:01:43Z

@BryanCutler Actually, for simplicity I would recommend simply removing the REST API related to the Master serving history UIs. Then you can just remove the actual history serving functionality itself easily. I don't see any reason we'll possibly add it back after Spark 2.0 so it doesn't make sense to keep around a REST API that does not have a backing functionality.

Also I haven't heard of anyone relying on this obscure REST API so I don't think it matters a lot. We should never have added this API in the first place.

BryanCutler · 2016-03-29T01:02:36Z

Thanks @andrewor14 , that sounds fine to me also - I just wanted to make sure I didn't remove some functionality that some people relied on. I already had everything removed, just need to clean up and merge with latest. I'm on vacation now, but I should be able to finish this up next week.

andrewor14 · 2016-03-29T01:06:40Z

Sure, thanks for your work!

andrewor14 · 2016-03-29T21:44:40Z

Also, please update docs/monitoring.md as well

This reverts commit 6918f0b.

…ster-SPARK-12299 Conflicts: core/src/main/scala/org/apache/spark/deploy/master/ApplicationInfo.scala core/src/main/scala/org/apache/spark/deploy/master/Master.scala core/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala core/src/main/scala/org/apache/spark/status/api/v1/ApplicationListResource.scala

…stead of listing applications, which is no longer supported

SparkQA · 2016-04-08T01:12:23Z

Test build #55264 has finished for PR 10991 at commit fadd392.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2016-04-08T17:04:31Z

jenkins retest this please

SparkQA · 2016-04-08T17:08:48Z

Test build #55365 has finished for PR 10991 at commit fadd392.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2016-04-08T17:16:56Z

jenkins retest this please

SparkQA · 2016-04-08T17:18:52Z

Test build #55366 has finished for PR 10991 at commit fadd392.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2016-04-08T19:12:44Z

jenkins retest this please

SparkQA · 2016-04-08T21:13:28Z

Test build #55382 has finished for PR 10991 at commit fadd392.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2016-04-08T21:23:08Z

@andrewor14 or @JoshRosen , this is ready for review whenever you get a chance, thanks!

…ster-SPARK-12299 Conflicts: core/src/main/scala/org/apache/spark/deploy/master/Master.scala

SparkQA · 2016-04-19T20:19:30Z

Test build #56243 has finished for PR 10991 at commit f9c09a1.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2016-04-19T20:20:02Z

Test build #56252 has finished for PR 10991 at commit d1814a3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2016-04-19T20:36:04Z

jenkins retest this please

SparkQA · 2016-04-19T22:14:14Z

Test build #56262 has finished for PR 10991 at commit d1814a3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2016-04-19T22:58:35Z

jenkins retest this please

SparkQA · 2016-04-20T00:55:07Z

Test build #56285 has finished for PR 10991 at commit d1814a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

BryanCutler · 2016-04-20T01:10:52Z

Ping @andrewor14 , please review. Is this still slated for 2.0 release? Thanks!

andrewor14 · 2016-05-04T00:26:37Z

LGTM! @JoshRosen should have a look too. Also retest this please because this hasn't run tests in a while.

SparkQA · 2016-05-04T02:16:27Z

Test build #57697 has finished for PR 10991 at commit d1814a3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2016-05-04T21:29:39Z

Merging into master 2.0. Thanks for your work @BryanCutler.

Remove history server functionality from standalone Master. Previously, the Master process rebuilt a SparkUI once the application was completed which sometimes caused problems, such as OOM, when the application event log is large (see SPARK-6270). Keeping this functionality out of the Master will help to simplify the process and increase stability. Testing for this change included running core unit tests and manually running an application on a standalone cluster to verify that it completed successfully and that the Master UI functioned correctly. Also added 2 unit tests to verify killing an application and driver from MasterWebUI makes the correct request to the Master. Author: Bryan Cutler <cutlerb@gmail.com> Closes#10991 from BryanCutler/remove-history-master-SPARK-12299. (cherry picked from commit cf2e9da) Signed-off-by: Andrew Or <andrew@databricks.com>

BryanCutler · 2016-05-04T21:50:05Z

Great, thanks!

PR apache#10991 removed the built-in history view from Spark Standalone, so the history server is no longer useful to Yarn or Mesos only.

Hello PR #10991 removed the built-in history view from Spark Standalone, so the history server is no longer useful to Yarn or Mesos only. Author: Hervé <dud225@users.noreply.github.com> Closes#17709 from dud225/patch-1. (cherry picked from commit 3476799) Signed-off-by: Sean Owen <sowen@cloudera.com>

Hello PR #10991 removed the built-in history view from Spark Standalone, so the history server is no longer useful to Yarn or Mesos only. Author: Hervé <dud225@users.noreply.github.com> Closes#17709 from dud225/patch-1.

stanzhai · 2017-05-15T12:40:37Z

We've just upgraded our Spark cluster from 1.6.x to 2.x, I found that the REST APIs from Spark MasterUI is unavailable.

It's important for us to use the REST APIs to monitor our Applications. I believe that some other people would rely on this function too.

Right now, the only way to get them is using the Spark Master WebUI, it's too bad.

It would be great that we have some REST APIs to access Master、Workers and Applications information from Master.

@BryanCutler @andrewor14 @JoshRosen

squito · 2017-05-30T15:47:08Z

@stanzhai I think you have a good point. I don't think removing the REST API for access to basic app info in the standalone master was really necessary for the original goals of SPARK-12299. But I'd open another jira, for restoring an api with the same info in the master UI, and move discussion there.

squito · 2017-05-30T15:47:57Z

whoops, now I see this is already present as https://issues.apache.org/jira/browse/SPARK-18683

Hello PR apache#10991 removed the built-in history view from Spark Standalone, so the history server is no longer useful to Yarn or Mesos only. Author: Hervé <dud225@users.noreply.github.com> Closesapache#17709 from dud225/patch-1.

BryanCutler added 2 commits January 27, 2016 16:35

[SPARK-12299] Removed history server functionality from Master with r…
21cb506
…elated WebUI and JSON ApiRootResources

Added UIRoot redirection from Master to Application
6918f0b

BryanCutler added 5 commits April 6, 2016 11:34

Revert "Added UIRoot redirection from Master to Application"
ce655d8
This reverts commit 6918f0b.

changed MasterWebUISuite to test killing of driver and application in…
d915850
…stead of listing applications, which is no longer supported

removed custom MasterPage from MasterWebUI, no longer needed for testing
ba8b962

updated docs to remove references to Master history serving
fadd392

BryanCutler changed the title ~~[SPARK-12299][CORE][WIP] Remove history serving functionality from Master~~[SPARK-12299][CORE] Remove history serving functionality from MasterApr 7, 2016

BryanCutler added 2 commits April 19, 2016 11:05

Merge remote-tracking branch 'upstream/master' into remove-history-ma…
f9c09a1
…ster-SPARK-12299 Conflicts: core/src/main/scala/org/apache/spark/deploy/master/Master.scala

Merge remote-tracking branch 'upstream/master' into remove-history-ma…
d1814a3
…ster-SPARK-12299 Conflicts: core/src/main/scala/org/apache/spark/deploy/master/Master.scala

asfgit closed this in cf2e9daMay 4, 2016

BryanCutler deleted the remove-history-master-SPARK-12299 branch January 3, 2017 22:28

dud225 added a commit to dud225/spark that referenced this pull request Apr 20, 2017
Small rewording
2560555
PR apache#10991 removed the built-in history view from Spark Standalone, so the history server is no longer useful to Yarn or Mesos only.

dud225 mentioned this pull request Apr 20, 2017
[DOC] Small rewording about history server use case #17709
Closed

Uh oh!

Conversation

BryanCutler commented Jan 29, 2016

Uh oh!

SparkQA commented Jan 29, 2016

Uh oh!

BryanCutler commented Jan 29, 2016

Uh oh!

BryanCutler commented Feb 1, 2016

Uh oh!

andrewor14 commented Feb 26, 2016

Uh oh!

BryanCutler commented Mar 4, 2016

Uh oh!

JoshRosen commented Mar 7, 2016

Uh oh!

BryanCutler commented Mar 16, 2016

Uh oh!

JoshRosen commented Mar 16, 2016

Uh oh!

BryanCutler commented Mar 17, 2016

Uh oh!

steveloughran commented Mar 28, 2016

Uh oh!

andrewor14 commented Mar 28, 2016

Uh oh!

BryanCutler commented Mar 29, 2016

Uh oh!

andrewor14 commented Mar 29, 2016

Uh oh!

andrewor14 commented Mar 29, 2016

Uh oh!

SparkQA commented Apr 8, 2016

Uh oh!

BryanCutler commented Apr 8, 2016

Uh oh!

SparkQA commented Apr 8, 2016

Uh oh!

BryanCutler commented Apr 8, 2016

Uh oh!

SparkQA commented Apr 8, 2016

Uh oh!

BryanCutler commented Apr 8, 2016

Uh oh!

SparkQA commented Apr 8, 2016

Uh oh!

BryanCutler commented Apr 8, 2016

Uh oh!

SparkQA commented Apr 19, 2016

Uh oh!

SparkQA commented Apr 19, 2016

Uh oh!

BryanCutler commented Apr 19, 2016

Uh oh!

SparkQA commented Apr 19, 2016

Uh oh!

BryanCutler commented Apr 19, 2016

Uh oh!

SparkQA commented Apr 20, 2016

Uh oh!

BryanCutler commented Apr 20, 2016

Uh oh!

andrewor14 commented May 4, 2016

Uh oh!

SparkQA commented May 4, 2016

Uh oh!

andrewor14 commented May 4, 2016

Uh oh!

BryanCutler commented May 4, 2016

Uh oh!

stanzhai commented May 15, 2017

Uh oh!

squito commented May 30, 2017

Uh oh!

squito commented May 30, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects