[SPARK-20242][Web UI] Add spark.ui.stopDelay

[barnardb]

What changes were proposed in this pull request?

Adds a spark.ui.stopDelay configuration property that can be used to keep the UI running when an application has finished. This is very useful for debugging, especially when the driver application is running remotely.

How was this patch tested?

This patch was tested manually. E.g. here's a screenshot from bin/spark-submit run-example --conf spark.ui.stopDelay=30s SparkPi 100:

[srowen]

I would oppose this change. It is what the history server is for. Or also profiling tools, or simple debugging mechanisms you allude to. It doesn't belong as yet another flag in Spark

[AmplabJenkins]

Can one of the admins verify this patch?

[jerryshao]

Agree with @srowen , the proposed solution overlaps the key functionality of history server. Usually we should stop the app and release the resources as soon as application finished. This solution unnecessarily block the whole stop chain and potentially introduce some issues when trying to visit objects that have already stopped.

[ajbozarth]

I agree with @srowen and @jerryshao this is what the history server is for

[barnardb]

Our use case involves jobs running in a remote cluster without a Spark master. I agree that the history server is the better to solve this, but we'd like to get a solution that doesn't depend on a distributed filesystem. Would you be willing to consider a pull request for https://issues.apache.org/jira/browse/SPARK-19802 (sending events to a remote history server) instead?

[vanzin]

Our use case involves jobs running in a remote cluster without a Spark master.

It's still running your code, right? Why can't you add a configuration to your own code that tells it to wait some time before shutting down the SparkContext? That would achieve the same goal without changing Spark.

[barnardb]

It's still running your code, right? Why can't you add a configuration to your own code that tells it to wait some time before shutting down the SparkContext?

We're trying to support arbitrary jobs running on the cluster, to make it easy for users to inspect the jobs that they run there. This was a quick way to achieve that, but I agree with the other commenters that this quite hacky, and that the history server would be a nicer solution. Our problem with the history server right now is that while the current driver-side EventLoggingListener + history-server-side FsHistoryProvider implementations are great for environments with HDFS, they're much less convenient in a cluster without a distributed filesystem. I'd propose that I close this PR, and work on an RPC-based listener-provider combination to use with the history server.

[jerryshao]

@barnardb only in Spark standalone mode HistoryServer is embedded into Master process for convenience IIRC. You can always start a standalone HistoryServer process.

Also FsHistoryProvider is not bound to HDFS, other Hadoop compatible FS could be supported, like wasb, s3 and other object stores that has Hadoop FS compatible layer. I would think even in your cluster environment (k8s), you probably have a object store. And at least you could implement a customized ApplicationHistoryProvider.