ZEPPELIN-3552. Support Scala 2.12 of SparkInterpreter by zjffdu · Pull Request #3034 · apache/zeppelin

zjffdu · 2018-06-19T08:29:58Z

What is this PR for?

This PR add support for scala 2.12 of SparkInterpreter. In this PR, I did some refactoring of whole spark modules. Each scala version interrpeter will be loaded dynamically via URLClassLoad, so that we can just write code once and compile it multiple times via different scala version and load it dynamically based on the current scala version.

What type of PR is it?

[Feature | Refactoring]

Todos

- Task

What is the Jira issue?

https://jira.apache.org/jira/browse/ZEPPELIN-3552

How should this be tested?

CI passed, UT of scala 2.12 is added and passed

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

felixcheung · 2018-06-27T06:06:11Z

hmm, Spark doesn't support Scala 2.12?

zjffdu · 2018-06-27T09:35:30Z

The first scala supported spark is 2.4.0 IIUC

felixcheung

Given that there are a few unresolved subtasks that no one is working on at the moment, I'm not sure if this would be resolved by Spark 2.4.0 release. So my thought would be not yet for now.

https://issues.apache.org/jira/browse/SPARK-14220

antonkulaga · 2018-08-20T19:14:03Z

Spark 2.4 will officially support Scala 2.12, so it will be great if Zeppelin will support it together with Spark. And also, there are some libs that are Scala 2.12 only

zjffdu · 2018-08-21T00:20:52Z

We would try to support it after spark 2.4 release.

HyukjinKwon · 2018-10-17T15:09:02Z

+
+ sparkILoop.in = reader
+ sparkILoop.initializeSynchronous()
+ callMethod(sparkILoop, "scala$tools$nsc$interpreter$ILoop$$loopPostInit")


Hmmm .. weird. From my testing against Spark 2.4.0 RC, this failed to find the method because it became a nested function (https://github.com/scala/scala/blob/v2.12.6/src/repl/scala/tools/nsc/interpreter/ILoop.scala#L993). Let me try to manually test against this one.

HyukjinKwon · 2018-10-17T15:09:12Z

+import scala.tools.nsc.interpreter._
+
+/**
+ * SparkInterpreter for scala-2.11


scala-2.11 -> scala-2.12 :-)

HyukjinKwon · 2018-10-18T05:36:19Z

Spark 2.4 will officially support Scala 2.12, so it will be great if Zeppelin will support it together with Spark. And also, there are some libs that are Scala 2.12 only

The default distribution will still be with Scala 2.11 for Spark 2.4 if I am not mistaken. It is nice to support it but Spark 2.4 with 2.11 should be supported first as a higher priority. I can work on 2.4.0 with 2.12 support further after this one got merged.

HyukjinKwon · 2018-12-05T08:54:46Z

Hey Jeff, I can take over this one too if you're busy since I took a look for similar code paths.

conker84 · 2019-04-02T10:22:43Z

Hi @felixcheung @zjffdu any news on this? Spark 3.0 (which will be out late of summer) seems to remove the support to v 2.11 so it would be great merge this ASAP

HyukjinKwon · 2019-04-02T23:26:45Z

Sorry, actually I am stuck in some works .. I think I wouldn't be able to take over this.

zjffdu · 2019-04-03T02:48:40Z

Don't worry @HyukjinKwon let me continue this.

zjffdu · 2019-04-04T13:11:52Z

 <scala.version>2.10.5</scala.version>
 <scala.binary.version>2.10</scala.binary.version>
- <scalatest.version>2.2.4</scalatest.version>
+ <scalatest.version>3.0.7</scalatest.version>


3.0.7 support both scala 2.10/2.11/2.12

felixcheung · 2019-04-05T05:44:02Z

whoa long time

antonkulaga · 2019-04-07T01:25:42Z

Scala 2.12 is no longer experimental in Spark 2.4.1 so I suggest targeting 2.4.1

conker84 · 2019-04-17T22:40:11Z

Any ETA for this?

zjffdu · 2019-04-19T08:46:51Z

@conker84 I make some updates to this PR, and it basically works in my local environment. But I still need more time to refine this PR. I suppose this could be done in 0.9.

conker84 · 2019-04-19T10:29:24Z

Ok thanks!
Spark 3.0 will add native Property Graph & Cypher Support, so we need this to start to test how Zeppelin behaves with it.
Please ping me when it will be merged!

conker84 · 2019-05-16T11:03:31Z

Any news on this?

conker84 · 2019-05-31T13:44:37Z

Ping

zjffdu · 2019-05-31T13:56:54Z

@conker84 I am busy on other stuff recently, but make some progress on this. The plan is still completing it before 0.9 which is planed to be released this summer.

zjffdu · 2019-06-24T05:51:44Z

Folks, This PR is ready for review. Now scala 2.12 is supported.
@felixcheung @jongyoul @conker84 Could you help review or try this PR ? Thanks

felixcheung · 2019-06-25T04:23:20Z

LGTM

HyukjinKwon

LGTM too

zjffdu · 2019-06-26T01:52:10Z

Will merge if no more comments

zjffdu changed the title ~~ZEPPELIN-3552. Support Scala 2.12 of SparkInterpreter~~[WIP] ZEPPELIN-3552. Support Scala 2.12 of SparkInterpreterJun 19, 2018

felixcheung requested changes Jun 28, 2018
View reviewed changes

zjffdu mentioned this pull request Oct 17, 2018
[ZEPPELIN-3810] Support Spark 2.4 #3206
Closed
1 task

HyukjinKwon reviewed Oct 17, 2018
View reviewed changes

zjffdu force-pushed the ZEPPELIN-3552 branch 2 times, most recently from 03d3122 to 657a443Compare April 3, 2019 05:10

zjffdu changed the title ~~[WIP] ZEPPELIN-3552. Support Scala 2.12 of SparkInterpreter~~ZEPPELIN-3552. Support Scala 2.12 of SparkInterpreterApr 3, 2019

zjffdu force-pushed the ZEPPELIN-3552 branch 12 times, most recently from 7581580 to 7feee7bCompare April 4, 2019 09:14

zjffdu commented Apr 4, 2019
View reviewed changes

zjffdu force-pushed the ZEPPELIN-3552 branch from 7a11147 to 7d0a187Compare April 4, 2019 14:38

zjffdu force-pushed the ZEPPELIN-3552 branch 2 times, most recently from 608046f to ddcf7b7Compare April 27, 2019 13:35

zjffdu force-pushed the ZEPPELIN-3552 branch from 606d1e5 to 08b1af9Compare June 1, 2019 06:38

zjffdu mentioned this pull request Jun 4, 2019
ZEPPELIN-4176. Remove old spark interpreter #3375
Closed
1 task

zjffdu force-pushed the ZEPPELIN-3552 branch 2 times, most recently from 56e4c56 to ef2c6afCompare June 12, 2019 04:19

zjffdu force-pushed the ZEPPELIN-3552 branch from ef2c6af to 6698c26Compare June 24, 2019 03:09

[ZEPPELIN-3552]. Support Scala 2.12 of SparkInterpreter
fe9c9eb

zjffdu force-pushed the ZEPPELIN-3552 branch from 6698c26 to fe9c9ebCompare June 24, 2019 03:18

HyukjinKwon approved these changes Jun 25, 2019
View reviewed changes

asfgit closed this in f61bdddJun 26, 2019

Uh oh!

Conversation

zjffdu commented Jun 19, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

felixcheung commented Jun 27, 2018

Uh oh!

zjffdu commented Jun 27, 2018

Uh oh!

felixcheung left a comment

Choose a reason for hiding this comment

Uh oh!

antonkulaga commented Aug 20, 2018

Uh oh!

zjffdu commented Aug 21, 2018

Uh oh!

HyukjinKwonOct 17, 2018

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonOct 17, 2018

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Oct 18, 2018

Uh oh!

HyukjinKwon commented Dec 5, 2018

Uh oh!

conker84 commented Apr 2, 2019

Uh oh!

HyukjinKwon commented Apr 2, 2019

Uh oh!

zjffdu commented Apr 3, 2019

Uh oh!

zjffduApr 4, 2019

Choose a reason for hiding this comment

Uh oh!

felixcheung commented Apr 5, 2019

Uh oh!

antonkulaga commented Apr 7, 2019

Uh oh!

conker84 commented Apr 17, 2019

Uh oh!

zjffdu commented Apr 19, 2019

Uh oh!

conker84 commented Apr 19, 2019

Uh oh!

conker84 commented May 16, 2019

Uh oh!

conker84 commented May 31, 2019

Uh oh!

zjffdu commented May 31, 2019

Uh oh!

zjffdu commented Jun 24, 2019

Uh oh!

felixcheung commented Jun 25, 2019

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

zjffdu commented Jun 26, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zjffdu commented Jun 19, 2018•
edited
Loading