Skip to content

[SPARK-49703] Publish Java 21 Docker image for preview1 - #69

Closed
dongjoon-hyun wants to merge 3 commits into
apache:masterfrom
dongjoon-hyun:SPARK-49703
Closed

[SPARK-49703] Publish Java 21 Docker image for preview1#69
dongjoon-hyun wants to merge 3 commits into
apache:masterfrom
dongjoon-hyun:SPARK-49703

Conversation

@dongjoon-hyun

@dongjoon-hyundongjoon-hyun commented Sep 18, 2024•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR aims to publish Java 21 Docker image for preview1 and will be extended for preview2.

Why are the changes needed?

Apache Spark supports Java 21 via SPARK-43831.

Does this PR introduce any user-facing change?

No, this is a new image.

How was this patch tested?

Pass the CIs.

HyukjinKwon
HyukjinKwon approved these changes Sep 19, 2024
View reviewed changes

@HyukjinKwonHyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @Yikun

@dongjoon-hyun

dongjoon-hyun commented Sep 19, 2024

Copy link
Copy Markdown
MemberAuthor

Thank you, @HyukjinKwon .

viirya
viirya reviewed Sep 19, 2024
View reviewed changes
Comment threadREADME.md Outdated
@dongjoon-hyun

dongjoon-hyun commented Sep 19, 2024

Copy link
Copy Markdown
MemberAuthor

For the record, while preparing 4.0.0-preview2 release, I'm making this repo up-to-date by re-implementing the Apache Spark patches like the following. The following three are published to https://hub.docker.com/r/apache/spark/ .

@dongjoon-hyun@viirya
Update README.md
494f5b0
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
@dongjoon-hyun

dongjoon-hyun commented Sep 19, 2024

Copy link
Copy Markdown
MemberAuthor

Thank you, @viirya .

Yikun
Yikun reviewed Sep 19, 2024
View reviewed changes
Comment threadversions.json Outdated
Comment threadversions.json Outdated
@dongjoon-hyun

dongjoon-hyun commented Sep 19, 2024

Copy link
Copy Markdown
MemberAuthor

Thank you, @Yikun .

@dongjoon-hyun

dongjoon-hyun commented Sep 19, 2024

Copy link
Copy Markdown
MemberAuthor

Thank you, @HyukjinKwon , @viirya , @Yikun .
Merged to master.

This was referenced Sep 19, 2024
Closed
Closed
dongjoon-hyun added a commit to apache/spark-kubernetes-operator that referenced this pull request Sep 19, 2024
@dongjoon-hyun
[SPARK-49714] AddJava 21-basedSparkPiexample
815cefe
### What changes were proposed in this pull request?
This PR aims to add `Java 21`-based SparkPi example.
### Why are the changes needed?
Apache Spark starts to publish Java 21-based image from Today (2024-09-19). This new example will illustrate how to use it.
- apache/spark-docker#69
### Does this PR introduce _any_ user-facing change?
No, this is a new example.
### How was this patch tested?
Manual review.
```
$ kubectl apply -f examples/pi-java21.yaml
```
```
$ kubectl get sparkapp
NAME CURRENT STATE AGE
pi-java21 ResourceReleased 28s
```
```
$ kubectl describe sparkapp pi-java21
Name: pi-java21
Namespace: default
Labels: <none>
Annotations: <none>
API Version: spark.apache.org/v1alpha1
Kind: SparkApplication
Metadata:
Creation Timestamp: 2024-09-19T04:08:16Z
Finalizers:
sparkapplications.spark.apache.org/finalizer
Generation: 2
Resource Version: 95294
UID: 2bc46e8d-6339-4867-9a28-6552c6c471e5
Spec:
Application Tolerations:
Application Timeout Config:
Driver Ready Timeout Millis: 300000
Driver Start Timeout Millis: 300000
Executor Start Timeout Millis: 300000
Force Termination Grace Period Millis: 300000
Termination Requeue Period Millis: 2000
Instance Config:
Init Executors: 0
Max Executors: 0
Min Executors: 0
Resource Retain Policy: OnFailure
Restart Config:
Max Restart Attempts: 3
Restart Backoff Millis: 30000
Restart Policy: Never
Deployment Mode: ClusterMode
Driver Args:
Jars: local:///opt/spark/examples/jars/spark-examples.jar
Main Class: org.apache.spark.examples.SparkPi
Runtime Versions:
Scala Version: 2.13
Spark Version: 4.0.0-preview1
Spark Conf:
spark.dynamicAllocation.enabled: true
spark.dynamicAllocation.maxExecutors: 3
spark.dynamicAllocation.shuffleTracking.enabled: true
spark.kubernetes.authenticate.driver.serviceAccountName: spark
spark.kubernetes.container.image: apache/spark:4.0.0-preview1-java21-scala
spark.log.structuredLogging.enabled: false
Status:
Current Attempt Summary:
Attempt Info:
Id: 0
Current State:
Current State Summary: ResourceReleased
Last Transition Time: 2024-09-19T04:08:33.316041381Z
State Transition History:
0:
Current State Summary: Submitted
Last Transition Time: 2024-09-19T04:08:16.584629470Z
Message: Spark application has been created on Kubernetes Cluster.
1:
Current State Summary: DriverRequested
Last Transition Time: 2024-09-19T04:08:17.269457304Z
Message: Requested driver from resource scheduler.
2:
Current State Summary: DriverStarted
Last Transition Time: 2024-09-19T04:08:17.809898304Z
Message: Driver has started running.
3:
Current State Summary: DriverReady
Last Transition Time: 2024-09-19T04:08:17.810393971Z
Message: Driver has reached ready state.
4:
Current State Summary: RunningHealthy
Last Transition Time: 2024-09-19T04:08:17.828526471Z
Message: Application is running healthy.
5:
Current State Summary: Succeeded
Last Transition Time: 2024-09-19T04:08:33.241514089Z
Message: Spark application completed successfully.
6:
Current State Summary: ResourceReleased
Last Transition Time: 2024-09-19T04:08:33.316041381Z
Events: <none>
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#129 from dongjoon-hyun/SPARK-49714.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
dongjoon-hyun added a commit to apache/spark-kubernetes-operator that referenced this pull request Sep 19, 2024
@dongjoon-hyun
[SPARK-49715] AddJava 21-basedSparkClusterexample
b187764
### What changes were proposed in this pull request?
This PR aims to add `Java 21`-based `SparkCluster` example.
### Why are the changes needed?
Apache Spark starts to publish Java 21-based image from Today (2024-09-19). This new example will illustrate how to use it.
- apache/spark-docker#69
### Does this PR introduce _any_ user-facing change?
No, this is a new example.
### How was this patch tested?
Manual review.
```
$ kubectl apply -f cluster-java21.yaml
```
```
$ kubectl get sparkcluster
NAME CURRENT STATE AGE
cluster-java21 RunningHealthy 9s
```
```
$ kubectl describe sparkcluster cluster-java21
Name: cluster-java21
Namespace: default
Labels: <none>
Annotations: <none>
API Version: spark.apache.org/v1alpha1
Kind: SparkCluster
Metadata:
Creation Timestamp: 2024-09-19T04:25:30Z
Finalizers:
sparkclusters.spark.apache.org/finalizer
Generation: 2
Resource Version: 96663
UID: 9421c957-380d-4a26-a266-379304bf83ee
Spec:
Cluster Tolerations:
Instance Config:
Init Workers: 3
Max Workers: 3
Min Workers: 3
Master Spec:
Runtime Versions:
Spark Version: 4.0.0-preview1
Spark Conf:
spark.kubernetes.container.image: apache/spark:4.0.0-preview1-java21
spark.master.rest.enabled: true
spark.master.rest.host: 0.0.0.0
spark.master.ui.title: Prod Spark Cluster (Java 21)
spark.ui.reverseProxy: true
Worker Spec:
Status:
Current Attempt Summary:
Attempt Info:
Id: 0
Current State:
Current State Summary: RunningHealthy
Last Transition Time: 2024-09-19T04:25:30.665095088Z
Message: Cluster has reached ready state.
State Transition History:
0:
Current State Summary: Submitted
Last Transition Time: 2024-09-19T04:25:30.640072963Z
Message: Spark cluster has been submitted to Kubernetes Cluster.
1:
Current State Summary: RunningHealthy
Last Transition Time: 2024-09-19T04:25:30.665095088Z
Message: Cluster has reached ready state.
Events: <none>
```
```
$ k get pod
NAME READY STATUS RESTARTS AGE
cluster-java21-master-0 1/1 Running 0 3m20s
cluster-java21-worker-0 1/1 Running 0 3m20s
cluster-java21-worker-1 1/1 Running 0 3m20s
cluster-java21-worker-2 1/1 Running 0 3m20s
spark-kubernetes-operator-778b9bbdc6-fqks9 1/1 Running 0 20m
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#130 from dongjoon-hyun/SPARK-49715.
Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@dongjoon-hyun

dongjoon-hyun commented Sep 19, 2024

Copy link
Copy Markdown
MemberAuthor

For the record, Java 21 images are published successfully.

@dongjoon-hyun
dongjoon-hyun deleted the SPARK-49703 branch September 19, 2024 05:47
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@YikunYikunYikun left review comments
@viiryaviiryaviirya approved these changes
@HyukjinKwonHyukjinKwonHyukjinKwon approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dongjoon-hyun@viirya@Yikun@HyukjinKwon