[SPARK-36061][K8S] Add support for PodGroup by Yikun · Pull Request #34456 · apache/spark

Yikun · 2021-11-01T09:55:51Z

What changes were proposed in this pull request?

PodGroup is a group of pods with strong association and is mainly used in batch scheduling, is of a Custom Resource Definition (CRD) type in Kubernetes, PodGroup concept which was approved by Kuberentes community in KEP-583 Coscheduling.

This patch adds the PodGroup support for Kuberentes:

Add PodGroup configuration: Introduce configurations to enable PodGroup support: spark.kubernetes.enablePodGroup, and also adds two configurations (spark.kubernetes.podgroup.min.[cpu|memory]) to helps user specifing min CPU and min Memory for a PodGroup.
Add Volcano implementaions: if user specify the spark k8s scheduler as volcano, will create the PodGroup with minReousrce requirement in Volcano automically, If available resources in the cluster cannot satisfy the requirement, no pod in the PodGroup will be scheduled.
Driver/Executor pod would be labeled with scheduling.k8s.io/group-name key and value s"${kubernetesConf.resourceNamePrefix}-podgroup".

Such as, user can use below configuration to request a group of pods with 4 CPU/ 8G Mem as min requirement, the volcano will help user create these pods if the meet the min requirement (4 CPUU, 8G Mem), If available resources in the cluster cannot satisfy the requirement, no pod in the PodGroup will be scheduled.

 --conf spark.kubernetes.driver.scheduler.name=volcano \
--conf spark.kubernetes.enablePodGroup=true \
--conf spark.kubernetes.podgroup.min.cpu=4 \
--conf spark.kubernetes.podgroup.min.memory=8G \

Why are the changes needed?

Provide feature to request minimum resources before scheduling jobs.

Does this PR introduce any user-facing change?

Yes, add podgroup related configuration.

How was this patch tested?

UT
e2e test:

# Setup K8S
minikube start --cpus 3 --memory 4096
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=spark:spark --namespace=spark
# Setup Volcano
kubectl apply -f https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml
# Submit job
bin/spark-submit \
--master k8s://https://127.0.0.1:6443 \
--deploy-mode cluster \
--conf spark.kubernetes.driver.scheduler.name=volcano \
--conf spark.kubernetes.enablePodGroup=true \
--conf spark.executor.instances=1 \
--conf spark.kubernetes.namespace=default \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.container.image=spark:latest \
--class org.apache.spark.examples.SparkPi \
--name spark-pi \
local:///opt/spark/examples/jars/spark-examples_2.12-3.3.0-SNAPSHOT.jar

Yikun · 2021-11-01T09:57:00Z

cc @holdenk @dongjoon-hyun @attilapiros

SparkQA · 2021-11-01T10:45:56Z

Test build #144808 has finished for PR 34456 at commit c066dd2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class PodGroup extends CustomResource[PodGroupSpec, PodGroupStatus] with Namespaced
class PodGroupSpec extends KubernetesResource
class PodGroupStatus extends KubernetesResource

SparkQA · 2021-11-01T11:12:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49278/

SparkQA · 2021-11-01T12:12:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49278/

SparkQA · 2021-11-10T14:22:20Z

Test build #145065 has finished for PR 34456 at commit c066dd2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class PodGroup extends CustomResource[PodGroupSpec, PodGroupStatus] with Namespaced
class PodGroupSpec extends KubernetesResource
class PodGroupStatus extends KubernetesResource

Yikun · 2021-11-15T09:40:27Z

I moved the PodGroup related API to a proper place, as a extension of k8s-client: fabric8io/kubernetes-client#3580 .

Yikun · 2021-11-27T10:38:02Z

 <type>test-jar</type>
 </dependency>
-
+ <dependency>


Volcano support in k8s-cli would be released at kubernetes-client v5.11
fabric8io/kubernetes-client#3580
TODO: neet to bump kubernetes-client version to latest when it publised.
spark/pom.xml
Line 207 in 7b50cf0
<kubernetes-client.version>5.10.1</kubernetes-client.version>

The new dependencies are available since 5.11.0 - e.g. https://search.maven.org/artifact/io.fabric8/volcano-model-v1beta1
Latest available is 5.12.1 - https://search.maven.org/artifact/io.fabric8/volcano-model-v1beta1/5.12.1/bundle

Yikun · 2021-11-27T10:40:48Z

+ * Pod creation.
 */
 def getAdditionalKubernetesResources(): Seq[HasMetadata] = Seq.empty
+


This is submited as a separated PR in #34599 .(1st and 2nd commits)
You could only see the 3rd commit for more clearly understand, that is, we only add pod feature step in this PR.

codecov-commenter · 2021-11-27T10:54:02Z

Codecov Report

Merging #34456 (7b50cf0) into master (d57f1bb) will decrease coverage by 7.98%.
The diff coverage is 74.35%.

❗ Current head 7b50cf0 differs from pull request most recent head c6a6688. Consider uploading reports for the commit c6a6688 to get more accurate results

@@ Coverage Diff @@## master #34456 +/- ##
==========================================
- Coverage 90.15% 82.17% -7.99% 
==========================================
Files 290 251 -39 Lines 62515 56994 -5521 Branches 9104 9281 +177 ==========================================
- Hits 56362 46833 -9529 - Misses 4784 8947 +4163 + Partials 1369 1214 -155

Flag	Coverage Δ
unittests	`82.15% <74.35%> (-7.99%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
python/pyspark/context.py	`57.86% <ø> (-27.67%)`	⬇️
python/pyspark/shuffle.py	`18.73% <0.00%> (-53.68%)`	⬇️
python/pyspark/rdd.py	`40.40% <33.33%> (-52.10%)`	⬇️
python/pyspark/cloudpickle/cloudpickle.py	`50.78% <39.68%> (-4.60%)`	⬇️
python/pyspark/ml/common.py	`51.35% <40.90%> (-24.37%)`	⬇️
python/pyspark/cloudpickle/cloudpickle_fast.py	`66.25% <61.90%> (-3.24%)`	⬇️
python/pyspark/pandas/sql_formatter.py	`89.02% <89.02%> (ø)`
python/pyspark/serializers.py	`60.86% <91.11%> (-20.85%)`	⬇️
python/pyspark/mllib/common.py	`87.20% <92.85%> (-1.82%)`	⬇️
python/pyspark/__init__.py	`90.69% <100.00%> (-4.66%)`	⬇️
... and 99 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04671bd...c6a6688. Read the comment docs.

SparkQA · 2021-11-27T11:23:06Z

Test build #145675 has finished for PR 34456 at commit c6a6688.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-11-27T11:31:51Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50145/

dongjoon-hyun

I close this PR since the artifact doesn't exist for now. Please reopen this when you are ready.

[error] sbt.librarymanagement.ResolveException: Error downloading io.fabric8:volcano-model-v1beta1:5.10.1
[error] Not found
[error] Not found
[error] not found: https://maven-central.storage-download.googleapis.com/maven2/io/fabric8/volcano-model-v1beta1/5.10.1/volcano-model-v1beta1-5.10.1.pom
[error] not found: https://repo1.maven.org/maven2/io/fabric8/volcano-model-v1beta1/5.10.1/volcano-model-v1beta1-5.10.1.pom
[error] not found: /home/jenkins/sparkivy/per-executor-caches/11/.m2/repository/io/fabric8/volcano-model-v1beta1/5.10.1/volcano-model-v1beta1-5.10.1.pom
[error] not found: /home/jenkins/sparkivy/per-executor-caches/11/.ivy2/localio.fabric8/volcano-model-v1beta1/5.10.1/ivys/ivy.xml
[error] Error downloading io.fabric8:volcano-client:5.10.1
[error] Not found
[error] Not found
[error] not found: https://maven-central.storage-download.googleapis.com/maven2/io/fabric8/volcano-client/5.10.1/volcano-client-5.10.1.pom
[error] not found: https://repo1.maven.org/maven2/io/fabric8/volcano-client/5.10.1/volcano-client-5.10.1.pom
[error] not found: /home/jenkins/sparkivy/per-executor-caches/11/.m2/repository/io/fabric8/volcano-client/5.10.1/volcano-client-5.10.1.pom
[error] not found: /home/jenkins/sparkivy/per-executor-caches/11/.ivy2/localio.fabric8/volcano-client/5.10.1/ivys/ivy.xml

martin-g

I see that it is/will be possible to setup Driver and/or Executor configurations with their own PodGroups.
What about a PodGroup that reserves resources (pods) for all actors (driver + executors) ?

martin-g · 2022-02-07T06:31:26Z

 <type>test-jar</type>
 </dependency>
-
+ <dependency>


The new dependencies are available since 5.11.0 - e.g. https://search.maven.org/artifact/io.fabric8/volcano-model-v1beta1
Latest available is 5.12.1 - https://search.maven.org/artifact/io.fabric8/volcano-model-v1beta1/5.12.1/bundle

Yikun · 2022-02-07T09:48:15Z

@martin-g Thanks for review, because the implementation it has a big different than before, so I just replace this by #35422 .

github-actionsBot added the KUBERNETES label Nov 1, 2021

Yikun commented Nov 1, 2021
View reviewed changes

Comment thread...tes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala Outdated
Comment thread...tes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala Outdated

Yikun commented Nov 15, 2021
View reviewed changes

Comment thread...tes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala Outdated

Yikun added 3 commits November 25, 2021 11:46

Add the ability to creating resources before driver pod
d3ee26c

Address nits
b7fe35b

Add PodGroup (using k8s client)
c6a6688

Yikun force-pushed the SPARK-36061 branch from c066dd2 to c6a6688Compare November 27, 2021 10:34

github-actionsBot added the BUILD label Nov 27, 2021

Yikun marked this pull request as draft November 27, 2021 10:35

Yikun changed the title ~~[WIP][SPARK-36061][K8S] Add support for PodGroup~~ [SPARK-36061][K8S] Add support for PodGroup Nov 27, 2021

Yikun commented Nov 27, 2021
View reviewed changes

dongjoon-hyun reviewed Dec 14, 2021
View reviewed changes

dongjoon-hyun closed this Dec 14, 2021

martin-g reviewed Feb 7, 2022
View reviewed changes

Uh oh!

Conversation

Yikun commented Nov 1, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Yikun commented Nov 1, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Nov 1, 2021

Uh oh!

SparkQA commented Nov 1, 2021

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Nov 1, 2021

Uh oh!

SparkQA commented Nov 10, 2021

Uh oh!

Yikun commented Nov 15, 2021

Uh oh!

Uh oh!

YikunNov 27, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martin-gFeb 7, 2022

Choose a reason for hiding this comment

Uh oh!

YikunNov 27, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Nov 27, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

SparkQA commented Nov 27, 2021

Uh oh!

SparkQA commented Nov 27, 2021

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

martin-g left a comment

Choose a reason for hiding this comment

Uh oh!

martin-gFeb 7, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Yikun commented Feb 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Yikun commented Nov 1, 2021•
edited
Loading

Yikun commented Nov 1, 2021•
edited
Loading

YikunNov 27, 2021•
edited
Loading

YikunNov 27, 2021•
edited
Loading

codecov-commenter commented Nov 27, 2021•
edited
Loading