[SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to run in Kubernetes Cluster mode by ozzieba · Pull Request #20272 · apache/spark

ozzieba · 2018-01-15T19:06:27Z

What changes were proposed in this pull request?

allow Spark Thrift Server to run in Kubernetes Cluster mode

How was this patch tested?

Edit: see apache-spark-on-k8s/spark-integration#38

foxish · 2018-01-15T23:14:51Z

@ozzieba, can we add a test to our integration test set to ensure this works? It's rather late in the Spark 2.3 release, and I'd be apprehensive about adding things that haven't been extensively tested.

ozzieba · 2018-01-16T03:52:32Z

I'm getting stuck on https://github.com/apache-spark-on-k8s/spark-integration/blob/master/integration-test/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala#L106, will look again tomorrow

felixcheung

please add [K8s] to the PR title.
we really need to test this...

felixcheung · 2018-01-17T08:12:31Z

 printErrorAndExit("Cluster deploy mode is not applicable to Spark SQL shell.")
- case (_, CLUSTER) if isThriftServer(args.mainClass) =>
+ case (_, CLUSTER) if (clusterManager != KUBERNETES) && isThriftServer(args.mainClass) =>
 printErrorAndExit("Cluster deploy mode is not applicable to Spark Thrift server.")


I haven't dug through but this might break if there has been any assumption that thrift is not running in cluster mode?

Can you elaborate on what might go wrong? I have been using it successfully to run queries, is there anything else that should be tested?

ozzieba · 2018-01-17T16:32:23Z

@foxish @felixcheung I wrote a test, but I'm having trouble with Minikube on Windows, and I can't run on a remote Kubernetes cluster. I'll try to get minikube running in a different way and keep you updated

ozzieba · 2018-01-19T00:07:29Z

@foxish @felixcheung Now verified my test runs successfully apache-spark-on-k8s/spark-integration#38

jiangxb1987 · 2018-01-23T19:43:51Z

cc @liyinan926 Do you have some time to verify this?

foxish · 2018-01-24T10:59:02Z

@jiangxb1987 Is there any specific owner of the thrift server that we can ping here? The testing looks good - so, all we're waiting for is confirmation from them on the original intent behind disallowing the thrift server in cluster mode; and if this is safe.

ozzieba · 2018-01-24T14:17:31Z

@foxish per SPARK-5176 and the associated PR, it seems there was a technical issue with spark-internal that wouldn't allow Thrift Server to run on YARN cluster mode. It was deemed a minor fix at the time, so perhaps the changes were made in the meantime. Regardless, it does now work at least on Kubernetes

ozzieba · 2018-01-24T14:27:42Z

@andrewor14 @liancheng can you chime in?

jiangxb1987 · 2018-01-29T18:22:04Z

IIUC there was a issue in launching Thrift Server on YARN cluster mode, and I'm not sure whether it has been fixed (maybe @jerryshao can kindly check that?) Anyway that is not a problem on Spark side, therefore should not affect the Kubernetes cluster mode.

foxish · 2018-01-29T18:24:18Z

Makes sense. The change LGTM.

…

On Jan 29, 2018 10:23 AM, "Jiang Xingbo" ***@***.***> wrote: IIUC there was a issue in launching Thrift Server on YARN cluster mode, and I'm not sure whether it has been fixed (maybe @jerryshao <https://github.com/jerryshao> can kindly check that?) Anyway that is not a problem on Spark side, therefore should not affect the Kubernetes cluster mode. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20272 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA3U51U4wlskTcUiDRn3JSJUIg2Dx7vfks5tPgyDgaJpZM4Re1NU> .

jerryshao · 2018-01-30T01:01:57Z

IIUC there was a issue in launching Thrift Server on YARN cluster mode, and I'm not sure whether it has been fixed (maybe @jerryshao can kindly check that?)

Sorry I cannot remember the issue. Yarn cluster mode doesn't support thriftserver.

vanzin · 2018-02-01T22:20:40Z

So how are you expected to contact the thrift server after it's up, and even figure out where it started? Isn't the container unreachable from the outside world unless you do some port mapping on the host?

I see part of that is explained in the bug, but sounds like this change needs better user documentation.

ozzieba · 2018-02-02T01:00:57Z

@vanzin In general Kubernetes makes this super easy:

The most basic workflow is to use the driver pod name (which is in the output of Spark Submit, or can be found with kubectl get pods), and run kubectl port-forward spark-app-driver-podname 31416:10000, which will automatically forward localhost:31416 to the pod's port 10000. Any jdbc client can then be used to query jdbc:hive2://localhost:31416. This is the approach I took in the integration tests linked above.
Alternatively, any other application on the cluster can simply use spark-app-driver-podname:10000, which will be resolved by kube-dns.
For persistent external access one can run kubectl expose pod spark-app-driver-podname --type=NodePort --port 10000 to create a Kubernetes Service which will accept connections on a particular port of every node on the cluster and send them to the driver's port 10000.
On a cloud environment, using type=LoadBalancer in the above will create a global load balancer that can be used to access Thrift from across the Internet.
One can also define a service that automatically selects the driver using a user-specified label (eg bin/spark-submit.sh --conf spark.kubernetes.driver.label.mythriftserver=true followed by kubectl expose pod --selector=mythriftserver=true)

Perhaps the key point is that these are all core Kubernetes features, and not specific to Spark in any way. Users familiar with Kubernetes should be able to find the approach that works best for their environment and use case.

vanzin · 2018-02-02T01:54:45Z

Great, how about explaining that in some place more visible to Spark users than a PR on github.

ozzieba · 2018-02-02T02:40:23Z

Ultimately I see this as being in the realm of Kubernetes knowledge, rather than Spark knowledge. These features are well-documented in Kubernetes documentation, and I am not sure there's a need to replicate that here. For a user new to both Spark and Kubernetes a combined tutorial could certainly be helpful, but that should likely be a more comprehensive undertaking.

vanzin · 2018-02-02T05:11:35Z

It's really not that much work to write a couple of sentences about this in the running-on-k8s docs, even if you're just pointing people to the kubernetes documentation. Is that really so controversial?

felixcheung

note to self: Jenkins hasn't run on this.

ah- · 2018-05-21T14:56:19Z

Is anyone still looking at this? It seems like it should just work?

nrchakradhar · 2018-05-22T07:51:54Z

Its working for us in our test environment. No issues so far. Only thing to take care is the clean-up of the driver manually when the thrift server is re-installed or started due to any reason.

AmplabJenkins · 2018-06-09T00:15:42Z

Can one of the admins verify this patch?

felixcheung · 2018-06-10T19:18:24Z

Is this aligned with the "in cluster client"? @foxish @mccheah

liyinan926 · 2018-07-20T22:43:43Z

@felixcheung I think yes and with #21748, users should be able to run the Thrift server in a pod.

srowen · 2018-07-27T17:10:09Z

If this can get a rebase, and maybe a few sentences in the k8s docs, I'll merge.

felixcheung · 2018-07-28T04:49:34Z

still need to run tests #20272 (review)

deveshk0 · 2018-08-01T07:54:30Z

I have built spark with the same changes for thrift server. It is running fine for me.

deveshk0 · 2018-09-10T18:46:01Z

is this coming with spark 2.4 ?

suryag10 · 2018-09-17T15:44:15Z

If this can get a rebase, and maybe a few sentences in the k8s docs, I'll merge.

Hi, I had rebased this patch to the latest master and this patch was missing another fix to run the STS on K8S cluster mode. Following is the PR

#22433

Can you please review this once?

vanzin · 2018-12-21T19:01:40Z

I'm closing this for now since there's another PR, and also I haven't seen an answer to my questions on the bug.

piyush9194 · 2019-06-27T08:39:32Z

Is this feature available in spark 2.4?

allow Spark Thrift Server to run in Kubernetes Cluster mode
62ae925

felixcheung reviewed Jan 17, 2018
View reviewed changes

ozzieba mentioned this pull request Jan 19, 2018
[SPARK-23078] add tests for Spark Thrift Server on Kubernetes apache-spark-on-k8s/spark-integration#38
Open

ozzieba changed the title ~~[SPARK-23078] [CORE] allow Spark Thrift Server to run in Kubernetes Cluster mode~~[SPARK-23078] [CORE] [k8s] allow Spark Thrift Server to run in Kubernetes Cluster modeJan 19, 2018

ozzieba changed the title ~~[SPARK-23078] [CORE] [k8s] allow Spark Thrift Server to run in Kubernetes Cluster mode~~[SPARK-23078] [CORE] [K8s] allow Spark Thrift Server to run in Kubernetes Cluster modeJan 19, 2018

felixcheung requested changes Mar 30, 2018
View reviewed changes

nrchakradhar mentioned this pull request Sep 19, 2018
[SPARK-25442][SQL][K8S] Support STS to run in k8s deployments with spark deployment mode as cluster. #22433
Closed

liyinan926 mentioned this pull request Sep 24, 2018
[SPARK-25516][k8s]Utilities needed for spark-history and spark-shuffle-service #22538
Closed

vanzin closed this Dec 21, 2018

Uh oh!

Conversation

ozzieba commented Jan 15, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

foxish commented Jan 15, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ozzieba commented Jan 16, 2018

Uh oh!

felixcheung left a comment

Choose a reason for hiding this comment

Uh oh!

felixcheungJan 17, 2018

Choose a reason for hiding this comment

Uh oh!

ozziebaJan 17, 2018

Choose a reason for hiding this comment

Uh oh!

ozzieba commented Jan 17, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ozzieba commented Jan 19, 2018

Uh oh!

jiangxb1987 commented Jan 23, 2018

Uh oh!

foxish commented Jan 24, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ozzieba commented Jan 24, 2018

Uh oh!

ozzieba commented Jan 24, 2018

Uh oh!

jiangxb1987 commented Jan 29, 2018

Uh oh!

foxish commented Jan 29, 2018 via email

Uh oh!

jerryshao commented Jan 30, 2018

Uh oh!

vanzin commented Feb 1, 2018

Uh oh!

ozzieba commented Feb 2, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vanzin commented Feb 2, 2018

Uh oh!

ozzieba commented Feb 2, 2018

Uh oh!

vanzin commented Feb 2, 2018

Uh oh!

felixcheung left a comment

Choose a reason for hiding this comment

Uh oh!

ah- commented May 21, 2018

Uh oh!

nrchakradhar commented May 22, 2018

Uh oh!

AmplabJenkins commented Jun 9, 2018

Uh oh!

felixcheung commented Jun 10, 2018

Uh oh!

liyinan926 commented Jul 20, 2018

Uh oh!

srowen commented Jul 27, 2018

Uh oh!

felixcheung commented Jul 28, 2018

Uh oh!

deveshk0 commented Aug 1, 2018

Uh oh!

deveshk0 commented Sep 10, 2018

Uh oh!

suryag10 commented Sep 17, 2018

Uh oh!

vanzin commented Dec 21, 2018

Uh oh!

piyush9194 commented Jun 27, 2019

Uh oh!

Reviewers

ozzieba commented Jan 15, 2018•
edited
Loading

foxish commented Jan 15, 2018•
edited
Loading

ozzieba commented Jan 17, 2018•
edited
Loading

foxish commented Jan 24, 2018•
edited
Loading

ozzieba commented Feb 2, 2018•
edited
Loading