[SPARK-2750][WEB UI]Add Https support for Web UI

[WangTaoTheTonic]

Based on #1980, as @scwf is busy doing sql stuff, I will finish this patch.

[SparkQA]

Test build #30839 has finished for PR 5664 at commit fef5003.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[scwf]

revert this

[SparkQA]

Test build #30841 has finished for PR 5664 at commit d80f7e9.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected[sql] abstract class AtomicType extends DataType
  • abstract class NumericType extends AtomicType
  • class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
  • class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
  • class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
  • class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
  • class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
  • class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
• This patch does not change any dependencies.

[SparkQA]

Test build #31749 has finished for PR 5664 at commit 18982b4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[WangTaoTheTonic]

@vanzin
Thanks for you advise. After importing my cert into JVM ResourceManager use, I can jump to the driver UI page from RM's link.

After taking some look at the docs and codes, I rename the configs to spark.ssl.ui.* so that it can be set via spark.ssl.* by default.

@jacek-lewandowski Could you help to review yet?

[vanzin]

Akka?

[SparkQA]

Test build #31824 has finished for PR 5664 at commit dfbe1d6.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[WangTaoTheTonic]

Jenkins, retest this please.

[WangTaoTheTonic]

@vanzin Not sure if we should modify createAkkaConfig, please give some reference.

[SparkQA]

Test build #31831 has finished for PR 5664 at commit dfbe1d6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

I'm not really familiar with Akka's SSL configuration, don't even know if they support those options. But if not modifying it, you should mention in the docs that the options don't apply to Akka.

[WangTaoTheTonic]

@CodingCat@jacek-lewandowski We add some configs for SSLOptions (keystoreType, needClientAuth, trustStoreType), they are supported in Jetty SSL, but we have no idea if akka support them too. Could you guys give some thought?

@pwendell@andrewor14@srowen Do you know anybody else who is familiar with Akka'\s SSL configurations?

[andrewor14]

@vanzin@JoshRosen is this already resolved by #3571?

[vanzin]

I don't think so; from what I remember, that PR only added SSL to the HTTP server that serves jars and files to executors, not to the server that serves the UI.

[jacek-lewandowski]

@vanzin@andrewor14 I may take a look at this and finish or create a separate pr with my approach if you like.

[steveloughran]

I'm not sure that YARN likes custom SSL certificates at all. Even if you do set up the RM proxy up for them, there's the issue that as they are host-wide, so every web/REST endpoint on the host can/should use the same certificate. That said, there's probably no explicit reason why the RM Proxy doesn't handle SSL except it's not been done —that'd be the place to add the code.

Note of course that web/REST clients of AM web pages redirected via the Am filter will have their communications direct with that RM proxy, using SPNEGO and any https support of the AM proxy; you can lock down the long-haul bit of the communications without adding HTTPS. It's the in-cluster TCP connections that HTTPS -in-AM would address

[jacek-lewandowski]

@WangTaoTheTonic can you rebase and squash?

[WangTaoTheTonic]

@jacek-lewandowski Okay I will rebase this in a week or so.

[jacek-lewandowski]

@WangTaoTheTonic if you don't have time, would you mind if i take your commits, rebase, squash them and add few changes?

[WangTaoTheTonic]

@jacek-lewandowski Sure. I'm glad for this.

[andrewor14]

@WangTaoTheTonic would you mind closing this patch for now? We can always re-open it later if there is interest or if @jacek-lewandowski takes over.

[jacek-lewandowski]

I rebased this pr on the current master with a little pain 😄
Will create a new PR soon.

[WangTaoTheTonic]

ok once jacky raise the PR, I will close this one.

[pritpalm]

I want to enable https on spark UI. I added following config to spark-defaults.config, but when we access spark ui via https::/:8080 or https://:443 or https://:8480, it's not able to connect.

spark.ui.https.enabled true
spark.ssl.keyPassword abcd
spark.ssl.keyStore rtmqa-clientid.jks
spark.ssl.keyStorePassword changeit