[SPARK-44690][BUILD][3.5] Downgrade Scala to 2.13.8 by LuciferYang · Pull Request #42362 · apache/spark

LuciferYang · 2023-08-06T02:11:32Z

What changes were proposed in this pull request?

The aim of this PR is to downgrade the Scala 2.13 dependency in Apache Spark 3.5 to 2.13.8, to ensure that Apache Spark 3.5 behaves the same as Apache Spark 3.4.x in terms of Maven build and testing: it can be build with -target:jvm-1.8, and tested with Java 11/17.

Why are the changes needed?

As reported in SPARK-44376, there are issues when maven build and test using Java 11/17 with -target:jvm-1.8:

run build/mvn clean install -Pscala-2.13 with Java 17

[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) @ spark-core_2.13 ---
[INFO] Compiler bridge file: /Users/yangjie01/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.11__61.0-1.8.0_20221110T195421.jar
[INFO] compiling 602 Scala sources and 77 Java sources to /Users/yangjie01/SourceCode/git/spark-mine-13/core/target/scala-2.13/classes ...
[WARNING] [Warn] : [deprecation @ | origin= | version=] -target is deprecated: Use -release instead to compile against the correct platform API.
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: not found: value sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210: not found: type Unsafe
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212: not found: type Unsafe
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452: not found: value sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type SignalHandler
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: type SignalHandler
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: Unused import
[WARNING] one warning found
[ERROR] 23 errors found

run build/mvn clean install -Pscala-2.13 -Djava.version=17 with Java 17

[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) @ spark-tags_2.13 ---
[INFO] Compiler bridge file: /Users/yangjie01/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.11__61.0-1.8.0_20221110T195421.jar
[INFO] compiling 2 Scala sources and 8 Java sources to /Users/yangjie01/SourceCode/git/spark-mine-13/common/tags/target/scala-2.13/classes ...
[WARNING] [Warn] : [deprecation @ | origin= | version=] -target is deprecated: Use -release instead to compile against the correct platform API.
[ERROR] [Error] : target platform version 8 is older than the release version 17
[WARNING] one warning found
[ERROR] one error found

run build/mvn clean package -Pscala-2.13 -DskipTests or build/mvn clean install -Pscala-2.13 -DskipTests with Java 8 first, then run build/mvn test -Pscala-2.13 with Java 17

[INFO] --- scala-maven-plugin:4.8.0:compile (scala-compile-first) @ spark-core_2.13 ---
[INFO] Compiler bridge file: /Users/yangjie01/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.13-1.8.0-bin_2.13.11__61.0-1.8.0_20221110T195421.jar
[INFO] compiling 602 Scala sources and 77 Java sources to /Users/yangjie01/SourceCode/git/spark-mine-13/core/target/scala-2.13/classes ...
[WARNING] [Warn] : [deprecation @ | origin= | version=] -target is deprecated: Use -release instead to compile against the correct platform API.
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: not found: value sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:210: not found: type Unsafe
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:212: not found: type Unsafe
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:26: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:452: not found: value sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: not found: object sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type SignalHandler
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:99: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:83: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: type SignalHandler
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:108: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:114: not found: type Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:116: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:128: not found: value Signal
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: Unused import
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/util/SignalUtils.scala:26: Unused import
[WARNING] one warning found
[ERROR] 23 errors found

This is inconsistent with the behavior of Apache Spark 3.4.x, so we need to use the previous Scala 2.13 version supporting this behavior in Apache Spark 3.5.0.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GitHub Actions
Manual checked, the above command can run normally after this pr

HyukjinKwon · 2023-08-06T07:00:10Z

Should we downgrade in the master branch too for now?

LuciferYang · 2023-08-06T07:46:59Z

Mistaken operation...

LuciferYang · 2023-08-06T07:49:21Z

Should we downgrade in the master branch too for now?

If we cannot accept the solution proposed in #41943 for Spark 4.0, then we should temporarily downgrade in master too.

srowen · 2023-08-06T13:53:44Z

Yes let's downgrade master too for consistency. Later, we either resolve this another way or just drop Java 8 support in 4.0

LuciferYang · 2023-08-06T14:12:13Z

Okay, let me submit a new PR to revert it in master

LuciferYang · 2023-08-07T02:25:25Z

close this one due to #42364 merged

LuciferYang · 2023-08-08T18:04:16Z

The test results are not as we expected, more investigation is needed to solve this problem.

Scenario 1：If the minimum support is Java 11, build with Java 11 and test with Java 17 and Java 21-ea.

Modify pom.xml

Change java.version to 11
Change -target:jvm-1.8 to -target:11.

Build & Test

run build/mvn clean install -DskipTests -Pscala-2.13 with Java 11
run build/mvn test -Pscala-2.13 with Java 17

failed:

[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: object security is not a member of package sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: object nio is not a member of package sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: Unused import
[WARNING] one warning found
[ERROR] 7 errors found

3.run build/mvn test -Pscala-2.13 with Java 21

failed:

[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: object security is not a member of package sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: object nio is not a member of package sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: Unused import
[WARNING] one warning found
[ERROR] 7 errors found

Scenario 2：If the minimum support is Java 17, build with Java 17 and test with Java 21-ea

Modify pom.xml

Change java.version to 17
Change -target:jvm-1.8 to -target:17.

Build & Test

run build/mvn clean install -DskipTests -Pscala-2.13 with Java 17
run build/mvn test -Pscala-2.13 with Java 21-ea

failed:

[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/serializer/SerializationDebugger.scala:71: object security is not a member of package sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: object nio is not a member of package sun
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:206: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:213: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:216: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:236: not found: type DirectBuffer
[ERROR] [Error] /Users/yangjie01/SourceCode/git/spark-mine-13/core/src/main/scala/org/apache/spark/storage/StorageUtils.scala:27: Unused import
[WARNING] one warning found
[ERROR] 7 errors found

srowen · 2023-08-08T18:15:32Z

Actually, for that particular problem, I think the answer is straightforward: those internal classes really just don't exist in later Java versions :) However, from looking at StorageUtils, all that is referenced only to try to bridge Java 8 and 9 support. A lot of that hacky code should go away along with the problem. I'm confident that much is fixable as part of updating past Java 8. There may be other issues though!

LuciferYang · 2023-08-09T03:38:58Z

Okay, let's address these issues after we stop supporting Java 8. I believe they can definitely be solved too.

LuciferYang · 2023-08-09T06:31:07Z

@srowen You're right, after making some changes to the core module's code, it is now possible to build Spark with Java 11(-target:11) and test with Java 17. Indeed, this task will need to be accomplished once we've discontinued support for Java 8. Happy ~

downgrade scala from 2.13.11 to 2.13.8
cc70f53

LuciferYang marked this pull request as draft August 6, 2023 02:11

github-actionsBot added the BUILD label Aug 6, 2023

LuciferYang changed the title ~~[BUILD][3.5] Downgrade Scala from 2.13.11 to 2.13.8~~[BUILD][3.5] Downgrade Scala to 2.13.8Aug 6, 2023

LuciferYang mentioned this pull request Aug 6, 2023
[SPARK-44376][BUILD] Fix maven build using scala 2.13 and Java 11 or later #41943
Closed

LuciferYang changed the title ~~[BUILD][3.5] Downgrade Scala to 2.13.8~~[SPARK-44690][BUILD][3.5] Downgrade Scala to 2.13.8Aug 6, 2023

LuciferYang marked this pull request as ready for review August 6, 2023 04:48

HyukjinKwon approved these changes Aug 6, 2023
View reviewed changes

LuciferYang closed this Aug 6, 2023

LuciferYang reopened this Aug 6, 2023

LuciferYang marked this pull request as draft August 6, 2023 14:12

LuciferYang mentioned this pull request Aug 6, 2023
[SPARK-44690][SPARK-44376][BUILD] Downgrade Scala to 2.13.8 #42364
Closed

LuciferYang closed this Aug 7, 2023

LuciferYang deleted the downgrade-scala-branch35 branch March 30, 2026 08:09

Navigation Menu

Uh oh!

[SPARK-44690][BUILD][3.5] Downgrade Scala to 2.13.8 - #42362

[SPARK-44690][BUILD][3.5] Downgrade Scala to 2.13.8#42362
LuciferYang wants to merge 1 commit into
apache:branch-3.5from
LuciferYang:downgrade-scala-branch35

LuciferYang commented Aug 6, 2023•
edited
Loading

Uh oh!

HyukjinKwon commented Aug 6, 2023

Uh oh!

LuciferYang commented Aug 6, 2023

Uh oh!

LuciferYang commented Aug 6, 2023•
edited
Loading

Uh oh!

srowen commented Aug 6, 2023

Uh oh!

LuciferYang commented Aug 6, 2023

Uh oh!

LuciferYang commented Aug 7, 2023

Uh oh!

LuciferYang commented Aug 8, 2023

Uh oh!

srowen commented Aug 8, 2023

Uh oh!

LuciferYang commented Aug 9, 2023•
edited
Loading

Uh oh!

LuciferYang commented Aug 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

LuciferYang commented Aug 6, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HyukjinKwon commented Aug 6, 2023

Uh oh!

LuciferYang commented Aug 6, 2023

Uh oh!

LuciferYang commented Aug 6, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Aug 6, 2023

Uh oh!

LuciferYang commented Aug 6, 2023

Uh oh!

LuciferYang commented Aug 7, 2023

Uh oh!

LuciferYang commented Aug 8, 2023

Uh oh!

srowen commented Aug 8, 2023

Uh oh!

LuciferYang commented Aug 9, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang commented Aug 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LuciferYang commented Aug 6, 2023•
edited
Loading

LuciferYang commented Aug 6, 2023•
edited
Loading

LuciferYang commented Aug 9, 2023•
edited
Loading