[SPARK-2750] support https in spark web ui by scwf · Pull Request #1980 · apache/spark

scwf · 2014-08-16T03:15:50Z

https://issues.apache.org/jira/browse/SPARK-2750

enable spark web ui to support https, also compatible with old web ui(http).

user can switch between https and http by configure "spark.http.policy".

SparkQA · 2014-09-05T23:43:06Z

Can one of the admins verify this patch?

andrewor14 · 2014-09-11T22:52:18Z

ok to test

andrewor14 · 2014-09-11T22:53:35Z

Looks like this is a duplicate of #1714, which is now closed, however. @WangTaoTheTonic you originally filed the issue to support this. Is there a particular motivation? How is this related to the view ACLs we already have (http://spark.apache.org/docs/latest/security.html)?

SparkQA · 2014-09-11T22:59:15Z

QA tests have started for PR 1980 at commit 8f7cc96.

This patch merges cleanly.

SparkQA · 2014-09-11T23:59:14Z

QA tests have finished for PR 1980 at commit 8f7cc96.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

WangTaoTheTonic · 2014-09-12T02:53:35Z

I did not make very much stuty of SecurityManager, but it only does authentication while not the encryption in communication.
I am not sure if it is appropriate using https to ensure the security in Spark ui communication, and not considering consistency of encryption in different underlying mechanisms.
BTW, @scwf is my workmate. We wanna encrypt the Web ui communiation now, maybe other protocols in future.

JoshRosen · 2014-09-15T21:20:53Z

It looks like this publicAddress field is never read anymore, so we should remove it.

JoshRosen · 2014-09-15T21:35:06Z

Is spark.http.policy the best name for this configuration option? Do you think that this can be a boolean option, or are there cases for wanting to have values other than http and https? It looks like Hadoop has a dfs.http.policy option that accepts three values, HTTP_ONLY, HTTPS_ONLY, and HTTP_AND_HTTPS.

Whatever configuration options we end up using, we should add documentation for them to the Spark Security and Configuration guides.

JoshRosen · 2014-09-15T21:38:13Z

This should be spark.worker.ui.port.

In fact, spark.worker.ui.port is already read on line 51, so this is unnecessary.

JoshRosen · 2014-09-15T21:41:38Z

Also, it looks like this adds configuration options under several (new) namespaces:

spark.http.policy
spark.client.https.need-auth
spark.ssl.server.keystore.*
spark.ssl.server.truststore.*

Are these the best names? What if we decide to add HTTPS / SSL to other components?

What is spark.client?

JoshRosen · 2014-09-15T21:55:04Z

What happens if I've configured the web UI to use https then attempt to browse to the http URL? Is it easy to set up an automatic redirect?

SparkQA · 2014-09-21T14:54:21Z

QA tests have started for PR 1980 at commit c90d84e.

This patch merges cleanly.

SparkQA · 2014-09-21T14:55:20Z

QA tests have finished for PR 1980 at commit c90d84e.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

scwf · 2014-09-21T14:57:39Z

@JoshRosen, here i named the configuration options refer to how hadoop does(actually just use spark instead hadoop). spark.client is the browser which connect to UIServer.
Now browse to httpui when configure http will get error, and i will check that how to set up automatic redirect(maybe configure this in jetty )

SparkQA · 2014-09-21T15:04:25Z

QA tests have started for PR 1980 at commit de8d1bd.

This patch merges cleanly.

SparkQA · 2014-09-21T15:34:30Z

QA tests have started for PR 1980 at commit 35074fd.

This patch merges cleanly.

SparkQA · 2014-09-21T15:37:09Z

QA tests have finished for PR 1980 at commit de8d1bd.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-21T16:10:11Z

QA tests have finished for PR 1980 at commit 35074fd.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-21T16:24:19Z

QA tests have started for PR 1980 at commit 967950f.

This patch merges cleanly.

vanzin · 2014-10-06T23:53:10Z

Might be a good idea to call response.encodeRedirectURL() first.

vanzin · 2014-10-06T23:55:30Z

For some reason I downloaded the patch but it doesn't apply cleanly to my copy of the repo... I'll try again later.

SparkQA · 2014-10-07T02:34:33Z

QA tests have started for PR 1980 at commit 8b32853.

This patch merges cleanly.

SparkQA · 2014-10-07T03:37:46Z

QA tests have finished for PR 1980 at commit 8b32853.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2014-10-07T16:50:54Z

Hi @scwf,
The reason I asked for a comment is that it seems like the method is doing a little more than just that. For example, L238 seems to be doing some sort of parsing of the server string, so it's more than just concatenating the different arguments into a URL.
It would be nice if the comment explained exactly what the relationship between the input and the output is. A unit test wouldn't hurt either.

Hi @vanzin, actually here i just refer to the code of jetty 9 see(https://github.com/eclipse/jetty.project/blob/master/jetty-util/src/main/java/org/eclipse/jetty/util/URIUtil.java#L726-L733) since there is no newURImethod in spark jetty version(spark use jetty 8).
And L238 is for the case to handle IPv6 address, here we can remove it if unnecessary.

Could you put that reference in the code itself, so we know where it comes from? Thanks!

Ok, updated.

SparkQA · 2014-10-10T08:09:43Z

QA tests have started for PR 1980 at commit 3b01d3a.

This patch merges cleanly.

SparkQA · 2014-10-10T10:09:44Z

Tests timed out for PR 1980 at commit 3b01d3a after a configured wait of 120m.

SparkQA · 2014-10-10T15:34:45Z

QA tests have started for PR 1980 at commit 4ae834b.

This patch merges cleanly.

SparkQA · 2014-10-10T16:40:15Z

QA tests have finished for PR 1980 at commit 4ae834b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jacek-lewandowski · 2014-10-22T23:51:07Z

@scwf can you take a look at #2739? I created PR which aim is to add SSL support for Akka based connections and for HttpServer which in turn bases on Jetty. I use SSLOptions object to configure SSL. All the settings are being kept in a separate file ssl.conf. You can use different name spaces to separate configurations for say UI, control and data. Would you take a look? It would be nice to have a single place to configure SSL.

jacek-lewandowski · 2014-10-22T23:54:50Z

Btw. I was trying to do SSL for UI as well. One thing I'm afraid of is that this solution would break any recovery information for Spark Master (different signature of WorkerInfo). Also, it doesn't allow to run the cluster in mixed mode during upgrade - some nodes running older Spark and some nodes running newer Spark, because it changes the signature of messages. Anyway, this is very minor I think.

scwf · 2014-10-23T01:17:04Z

@jacek-lewandowski, you mean in ssl.conf to set ssl configs for ui and akka and httpserver? then ssl.con f is the single place to configure ssl, right? if so, i think it's ok.

For the second one "some nodes running older Spark and some nodes running newer Spark", i don't think there is this situation.

@JoshRosen @vanzin is this ok to go? I think we can merge this first then in #2739 to unify the ssl configs.

JoshRosen · 2015-02-03T01:31:29Z

Hi @scwf,

I just merged #3571, which added SSL support for Akka and the HTTPServer. That PR added a common configuration mechanism for SSL, plus some useful utilities for testing this functionality. If you're interested, it would be great to revisit HTTPS for the web UI now that we've merged that other patch.

scwf · 2015-02-03T02:16:41Z

@JoshRosen in that PR now no https support for web ui right? if so i will rework this to support it.

JoshRosen · 2015-02-03T02:18:46Z

@scwf Yep, that PR only added SSL support to Akka and the HTTP FileServer, not the web UI or the shuffle service.

scwf · 2015-04-23T13:02:48Z

i am closing this in favor of #5664

pritpalm · 2016-12-15T16:06:09Z

I want to enable https on spark UI. I added following config to spark-defaults.config, but when we access spark ui via https::/:8080 or https://:443 or https://:8480, it's not able to connect.

spark.ui.https.enabled true
spark.ssl.keyPassword abcd
spark.ssl.keyStore rtmqa-clientid.jks
spark.ssl.keyStorePassword changeit

scwf added 2 commits August 16, 2014 08:20

support https in spark web ui
e8256e5

add unit test
8f7cc96

JoshRosen reviewed Sep 15, 2014
View reviewed changes

fix according to comments
c90d84e

fix scalastyle
de8d1bd

fix workerinfo in JsonProtocolSuite
35074fd

Merge branch 'master' of github.com:apache/spark into https
967950f

vanzin reviewed Oct 6, 2014
View reviewed changes

scwf added 2 commits October 7, 2014 10:22

address vanzin's comments
2dadb2f

Merge branch 'master' ofhttps://github.com/apache/sparkinto https
8b32853

vanzin reviewed Oct 7, 2014
View reviewed changes

add reference to method newURI
3b01d3a

Merge branch 'master' ofhttps://github.com/apache/sparkinto https
4ae834b

JoshRosen mentioned this pull request Oct 13, 2014
SPARK-3883 SSL support for HttpServer and Akka #2739
Closed

jacek-lewandowski mentioned this pull request Jan 19, 2015
Spark 3883: SSL support for HttpServer and Akka #3571
Closed

WangTaoTheTonic mentioned this pull request Apr 23, 2015
[SPARK-2750][WEB UI]Add Https support for Web UI #5664
Closed

scwf closed this Apr 23, 2015

WangTaoTheTonic added a commit to WangTaoTheTonic/spark that referenced this pull request Apr 23, 2015
rebase based onapache#1980
d80f7e9

Uh oh!

Conversation

scwf commented Aug 16, 2014

Uh oh!

SparkQA commented Sep 5, 2014

Uh oh!

andrewor14 commented Sep 11, 2014

Uh oh!

andrewor14 commented Sep 11, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

SparkQA commented Sep 11, 2014

Uh oh!

WangTaoTheTonic commented Sep 12, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoshRosen commented Sep 15, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoshRosen commented Sep 15, 2014

Uh oh!

JoshRosen commented Sep 15, 2014

Uh oh!

SparkQA commented Sep 21, 2014

Uh oh!

SparkQA commented Sep 21, 2014

Uh oh!

scwf commented Sep 21, 2014

Uh oh!

SparkQA commented Sep 21, 2014

Uh oh!

SparkQA commented Sep 21, 2014

Uh oh!

SparkQA commented Sep 21, 2014

Uh oh!

SparkQA commented Sep 21, 2014

Uh oh!

SparkQA commented Sep 21, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vanzin commented Oct 6, 2014

Uh oh!

SparkQA commented Oct 7, 2014

Uh oh!

SparkQA commented Oct 7, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

SparkQA commented Oct 10, 2014

Uh oh!

jacek-lewandowski commented Oct 22, 2014

Uh oh!

jacek-lewandowski commented Oct 22, 2014

Uh oh!

scwf commented Oct 23, 2014

Uh oh!

JoshRosen commented Feb 3, 2015

Uh oh!

scwf commented Feb 3, 2015

Uh oh!

JoshRosen commented Feb 3, 2015

Uh oh!