Skip to content

[ZEPPELIN-4950]. Support for manually specifying the Java version of Spark Interpreter Scala REPL#3852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
xiejiajun wants to merge 10 commits into from
Closed

[ZEPPELIN-4950]. Support for manually specifying the Java version of Spark Interpreter Scala REPL #3852

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
9b3c744
added timeout for getting Thrift client to avoid situations where the…
xiejiajun Mar 20, 2020
97d2714
Merge pull request #2 from apache/branch-0.9
xiejiajun Mar 23, 2020
9bc5605
bug fix: when we removing a paragraph , an interpreter process refus…
xiejiajun Mar 31, 2020
9bb7341
Merge remote-tracking branch 'origin/branch-0.9' into branch-0.9
xiejiajun Mar 31, 2020
00e14dd
Merge pull request #5 from apache/branch-0.9
xiejiajun May 26, 2020
076668a
Merge pull request #7 from apache/branch-0.9
xiejiajun Jun 18, 2020
6a2cf04
Merge branch 'branch-0.9' of https://github.com/apache/zeppelin into …
xiejiajun Jul 13, 2020
6a05b4f
Support for manually specifying the Java version of Spark Interpreter…
xiejiajun Jul 13, 2020
6872054
add spark.repl.target to spark.md docs
xiejiajun Jul 13, 2020
d131f19
spark.md docs fix
xiejiajun Jul 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/interpreter/spark.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -199,6 +199,10 @@ You can also set other Spark properties which are not listed in the table. For a
<td>false</td>
<td>whether use yarn proxy url as spark weburl, e.g. http://localhost:8088/proxy/application_1583396598068_0004</td>
</tr>
<td>spark.repl.target</td>
<td>jvm-1.8</td>
<td>Manually specifying the Java version of Spark Interpreter Scala REPL,Available options:[jvm-1.5, jvm-1.6, jvm-1.7, jvm-1.8] </td>
</tr>
</table>

Without any configuration, Spark interpreter works out of box in local mode. But if you want to connect to your Spark cluster, you'll need to follow below two simple steps.
Expand Down
3 changes: 3 additions & 0 deletions spark/scala-2.10/src/main/scala/org/apache/zeppelin/spark/SparkScala210Interpreter.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -67,10 +67,13 @@ class SparkScala210Interpreter(override val conf: SparkConf,
sparkHttpServer = server
conf.set("spark.repl.class.uri", uri)
}
val target = conf.get("spark.repl.target", "jvm-1.8")

val settings = new Settings()
settings.embeddedDefaults(sparkInterpreterClassLoader)
settings.usejavacp.value = true
settings.target.value = target

this.userJars = getUserJars()
LOGGER.info("UserJars: " + userJars.mkString(File.pathSeparator))
settings.classpath.value = userJars.mkString(File.pathSeparator)
Expand Down
2 changes: 2 additions & 0 deletions spark/scala-2.11/src/main/scala/org/apache/zeppelin/spark/SparkScala211Interpreter.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -66,12 +66,14 @@ class SparkScala211Interpreter(override val conf: SparkConf,
sparkHttpServer = server
conf.set("spark.repl.class.uri", uri)
}
val target = conf.get("spark.repl.target", "jvm-1.8")

val settings = new Settings()
settings.processArguments(List("-Yrepl-class-based",
"-Yrepl-outdir", s"${outputDir.getAbsolutePath}"), true)
settings.embeddedDefaults(sparkInterpreterClassLoader)
settings.usejavacp.value = true
settings.target.value = target

this.userJars = getUserJars()
LOGGER.info("UserJars: " + userJars.mkString(File.pathSeparator))
Expand Down
3 changes: 3 additions & 0 deletions spark/scala-2.12/src/main/scala/org/apache/zeppelin/spark/SparkScala212Interpreter.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -60,12 +60,15 @@ class SparkScala212Interpreter(override val conf: SparkConf,
LOGGER.info("Scala shell repl output dir: " + outputDir.getAbsolutePath)
outputDir.deleteOnExit()
conf.set("spark.repl.class.outputDir", outputDir.getAbsolutePath)
val target = conf.get("spark.repl.target", "jvm-1.8")

val settings = new Settings()
settings.processArguments(List("-Yrepl-class-based",
"-Yrepl-outdir", s"${outputDir.getAbsolutePath}"), true)
settings.embeddedDefaults(sparkInterpreterClassLoader)
settings.usejavacp.value = true
settings.target.value = target

this.userJars = getUserJars()
LOGGER.info("UserJars: " + userJars.mkString(File.pathSeparator))
settings.classpath.value = userJars.mkString(File.pathSeparator)
Expand Down