[SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment by zwangsheng · Pull Request #3527 · apache/kyuubi

zwangsheng · 2022-09-20T08:55:03Z

Why are the changes needed?

On case spark on kubernetes, spark using env SPARK_USER_NAME as user name.
So kyuubi should build spark engine with this env when proxy user or using keytab.
This conf only affect on kubernetes case.

Ref: apache/spark#23017

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request

zwangsheng · 2022-09-20T08:55:41Z

plz help check @pan3793 @turboFei @wForget

codecov-commenter · 2022-09-21T01:50:36Z

Codecov Report

Merging #3527 (9596372) into master (73612b7) will increase coverage by 0.00%.
The diff coverage is 88.88%.

@@ Coverage Diff @@## master #3527 +/- ##
=========================================
Coverage 51.67% 51.67% Complexity 13 13 =========================================
Files 482 482 Lines 26933 26943 +10 Branches 3760 3760 =========================================
+ Hits 13917 13923 +6 - Misses 11661 11664 +3 - Partials 1355 1356 +1

Impacted Files	Coverage Δ
...ache/kyuubi/engine/spark/SparkProcessBuilder.scala	`86.66% <88.88%> (+0.41%)`	⬆️
...ache/kyuubi/operation/KyuubiOperationManager.scala	`79.45% <0.00%> (-2.74%)`	⬇️
...che/kyuubi/server/KyuubiTHttpFrontendService.scala	`64.23% <0.00%> (-0.73%)`	⬇️
...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala	`80.74% <0.00%> (-0.63%)`	⬇️
...g/apache/kyuubi/operation/BatchJobSubmission.scala	`74.64% <0.00%> (ø)`
...a/org/apache/kyuubi/service/TFrontendService.scala	`91.47% <0.00%> (+0.29%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

yaooqinn · 2022-09-21T02:34:30Z

+ // For spark on kubernetes, spark pod using env SPARK_USER_NAME as current user
+ def setSparkUserName(userName: String): Unit = {
+ clusterManager().foreach(cm => {
+ if (cm.startsWith("k8s://")) {


there is no harm to set for all cluster managers？

spark.kubernetes.driverEnv. only affect on kubernetes case. We are only setting up this On Kubernetes for now.
As for SPARK_USER_NAME this env, it's affect in spark/kubernetes/entrypoint.sh

pan3793 · 2022-09-23T04:31:50Z

Thanks, merging to master/1.6

…deployment ### _Why are the changes needed?_ On case spark on kubernetes, spark using env `SPARK_USER_NAME` as user name. So kyuubi should build spark engine with this env when proxy user or using keytab. This conf only affect on kubernetes case. Ref: apache/spark#23017 ### _How was this patch tested?_ - [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes#3527 from zwangsheng/feature/add_spark_user_name. Closes#3527 9596372 [zwangsheng] only k8s case ddd713f [zwangsheng] fix 48b9b22 [zwangsheng] add Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit 3738512) Signed-off-by: Cheng Pan <chengpan@apache.org>

pan3793 · 2022-09-29T15:50:23Z

@zwangshengspark.kubernetes.executorEnv.[EnvironmentVariableName] does not take effect, the correct key is spark.executorEnv.[EnvironmentVariableName]
https://spark.apache.org/docs/latest/configuration.html

zwangsheng · 2022-10-08T01:50:03Z

@zwangshengspark.kubernetes.executorEnv.[EnvironmentVariableName] does not take effect, the correct key is spark.executorEnv.[EnvironmentVariableName]https://spark.apache.org/docs/latest/configuration.html

😭sry, i will make follow PR.

…e spark.executorEnv ### _Why are the changes needed?_ Fix bug in #3527 Modify `spark.kubernetes.executorEnv` to `spark.executorEnv` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request - [x] Unit Test Closes#3591 from zwangsheng/follow/SPARK_USER_NAME. Closes#3527 7254fe9 [zwangsheng] fix executorEnv Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

…e spark.executorEnv ### _Why are the changes needed?_ Fix bug in #3527 Modify `spark.kubernetes.executorEnv` to `spark.executorEnv` ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request - [x] Unit Test Closes#3591 from zwangsheng/follow/SPARK_USER_NAME. Closes#3527 7254fe9 [zwangsheng] fix executorEnv Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit fb6516a) Signed-off-by: Cheng Pan <chengpan@apache.org>

…AME on submitting Spark batch job to K8s ### _Why are the changes needed?_ Batch job should also submit with `SPARK_USER_NAME`. See more in pre pr #3527 . ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes#3664 from zwangsheng/feature/batch_with_user_name. Closes#3664 76dee3e [zwangsheng] style 0d40508 [zwangsheng] fix a1fcf7f [zwangsheng] fix Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org>

…AME on submitting Spark batch job to K8s ### _Why are the changes needed?_ Batch job should also submit with `SPARK_USER_NAME`. See more in pre pr #3527 . ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes#3664 from zwangsheng/feature/batch_with_user_name. Closes#3664 76dee3e [zwangsheng] style 0d40508 [zwangsheng] fix a1fcf7f [zwangsheng] fix Authored-by: zwangsheng <2213335496@qq.com> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit 5f8b57b) Signed-off-by: Cheng Pan <chengpan@apache.org>

add
48b9b22

github-actionsBot added the module:server label Sep 20, 2022

pan3793 changed the title ~~[FEATURE][SPARK] Build Spark engine with env SPARK_USER_NAME using proxyName or keytab shortUserName~~[FEATURE][SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deploymentSep 20, 2022

pan3793 changed the title ~~[FEATURE][SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment~~[SPARK] Build Spark engine with env SPARK_USER_NAME using proxyName or keytab shortUserNameSep 20, 2022

pan3793 reviewed Sep 20, 2022
View reviewed changes

Comment threadkyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala Outdated

pan3793 reviewed Sep 20, 2022
View reviewed changes

Comment threadkyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala Outdated

pan3793 changed the title ~~[SPARK] Build Spark engine with env SPARK_USER_NAME using proxyName or keytab shortUserName~~[SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deploymentSep 20, 2022

fix
ddd713f

pan3793 approved these changes Sep 20, 2022
View reviewed changes

pan3793 added this to the v1.6.1 milestone Sep 20, 2022

pan3793 assigned zwangsheng Sep 20, 2022

turboFei approved these changes Sep 20, 2022
View reviewed changes

turboFei reviewed Sep 20, 2022
View reviewed changes

Comment threadkyuubi-server/src/main/scala/org/apache/kyuubi/engine/spark/SparkProcessBuilder.scala Outdated

only k8s case
9596372

yaooqinn reviewed Sep 21, 2022
View reviewed changes

pan3793 closed this in 3738512Sep 23, 2022

zwangsheng mentioned this pull request Oct 8, 2022
[KYUUBI #3527][SPARK][FOLLOWUP] spark.kubernetes.executorEnv should be spark.executorEnv #3591
Closed
4 tasks

zwangsheng mentioned this pull request Oct 19, 2022
[SPARK][K8S][BATCH] Kyuubi should set env SPARK_USER_NAME on submitting Spark batch job to K8s #3664
Closed
3 tasks

Navigation Menu

Uh oh!

[SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment - #3527

[SPARK] Kyuubi should set env SPARK_USER_NAME for K8s deployment#3527
zwangsheng wants to merge 3 commits into
apache:masterfrom
zwangsheng:feature/add_spark_user_name

zwangsheng commented Sep 20, 2022•
edited by pan3793
Loading

Uh oh!

zwangsheng commented Sep 20, 2022•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Sep 21, 2022•
edited
Loading

Uh oh!

yaooqinnSep 21, 2022

Uh oh!

zwangshengSep 21, 2022

Uh oh!

pan3793 commented Sep 23, 2022

Uh oh!

pan3793 commented Sep 29, 2022

Uh oh!

zwangsheng commented Oct 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

zwangsheng commented Sep 20, 2022• edited by pan3793 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are the changes needed?

How was this patch tested?

Uh oh!

zwangsheng commented Sep 20, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Sep 21, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yaooqinnSep 21, 2022

Choose a reason for hiding this comment

Uh oh!

zwangshengSep 21, 2022

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Sep 23, 2022

Uh oh!

pan3793 commented Sep 29, 2022

Uh oh!

zwangsheng commented Oct 8, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zwangsheng commented Sep 20, 2022•
edited by pan3793
Loading

zwangsheng commented Sep 20, 2022•
edited
Loading

codecov-commenter commented Sep 21, 2022•
edited
Loading