ZEPPELIN-598 ] Dynamic loading for Interpreter and API by cloverhearts · Pull Request #908 · apache/zeppelin

cloverhearts · 2016-05-22T05:50:22Z

What is this PR for?

Use of external libraries that are included in the Roadmap,
The interpreter needs related to dynamic loading capabilities.

What type of PR is it?

Feature

Todos

- Dynamic load
- make api
- Modified according to feedback (ZEPPELIN-598 ] Dynamic loading for Interpreter #631)

ScreenShots

Is there a relevant Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-598

How should this be tested?

By annotating the interpreter setting items below, try using the loadDynamicInterpreter method.
Or create a new interpreter loads created in the local Maven repository.

Step 1.
markdown interpreter to local distribute

mvn install -pl '!spark-dependencies,!spark,!flink,!ignite,!lens,!hive,!phoenix,!postgresql,!tajo,!cassandra,!kylin,!elasticsearch,!zeppelin-distribution' -DskipTests

Step 2.
Comment out the markdown
incubator-zeppelin/zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java

ZEPPELIN_INTERPRETERS("zeppelin.interpreters", "org.apache.zeppelin.spark.SparkInterpreter,"
+ "org.apache.zeppelin.spark.PySparkInterpreter,"
+ "org.apache.zeppelin.spark.SparkSqlInterpreter,"
+ "org.apache.zeppelin.spark.DepInterpreter,"// + "org.apache.zeppelin.markdown.Markdown,"
...

Step 3.
and build.
For example.

mvn clean package -Pspark-1.6 -Phadoop-2.4 -Pyarn -Ppyspark -DskipTests

Step 4.
remove to interpreter/md

rm -rf interpreter/md

option.
fixed conf/zeppelin-env.sh(cmd) download interpreter directory.

export ZEPPELIN_INTERPRETER_DOWNLOAD_DIR=MyZeppelinInterpreter

Step 5.
and API Call (load interpreter) example load markdown

Content-Type: application/json

POST http://127.0.0.1:8080/api/interpreter/load/md/markdown
{
"artifact": "org.apache.zeppelin:zeppelin-markdown:0.6.0-incubating-SNAPSHOT",
"className": "org.apache.zeppelin.markdown.Markdown",
"repository": {
"url": "http://dl.bintray.com/spark-packages/maven",
"snapshot" : true
}
}

Step 6.
In interperter menu for zeppelin web, you can new setting information for md.

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

astroshim · 2016-05-23T02:23:49Z

I tested this funtionality but I got failed.

[My test process]

I removed interpreter/md path
start zeppelin server
request following curl command

curl -H "Content-Type: application/json" -XPOST "http://127.0.0.1:8080/api/interpreter/load/md/markdown" -d '{
"artifact": "org.apache.zeppelin:zeppelin-markdown:0.6.0-incubating-SNAPSHOT",
"className": "org.apache.zeppelin.markdown.Markdown",
"repository": {
"url": "http://dl.bintray.com/spark-packages/maven",
"snapshot" : true
}
}'

but I got an exception and md is not loaded.
The exception is following

ERROR [2016-05-23 11:16:52,948] ({qtp1148546316-14} InterpreterFactory.java[loadDynamicInterpreter]:146) - Failed load dynamic interpreter : org.sonatype.aether.resolution.DependencyResolutionException: Could not find artifact org.apache.zeppelin:zeppelin-markdown:jar:0.6.0-incubating-SNAPSHOT in central (http://repo1.maven.org/maven2/)
at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:375)
at org.apache.zeppelin.dep.DependencyResolver.getArtifactsWithDep(DependencyResolver.java:170)
at org.apache.zeppelin.dep.DependencyResolver.loadFromMvn(DependencyResolver.java:123)
at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:81)
at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:94)
at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:68)
at org.apache.zeppelin.interpreter.InterpreterFactory.loadDynamicInterpreter(InterpreterFactory.java:143)
at org.apache.zeppelin.rest.InterpreterRestApi.loadDynamicInterpreter(InterpreterRestApi.java:265)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:192)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.apache.zeppelin.server.CorsFilter.doFilter(CorsFilter.java:72)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.sonatype.aether.resolution.ArtifactResolutionException: Could not find artifact org.apache.zeppelin:zeppelin-markdown:jar:0.6.0-incubating-SNAPSHOT in central (http://repo1.maven.org/maven2/)
at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:537)
at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:216)
at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:358)
... 63 more
Caused by: org.sonatype.aether.transfer.ArtifactNotFoundException: Could not find artifact org.apache.zeppelin:zeppelin-markdown:jar:0.6.0-incubating-SNAPSHOT in central (http://repo1.maven.org/maven2/)
at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:971)
at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:966)
at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.flush(WagonRepositoryConnector.java:707)
at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.flush(WagonRepositoryConnector.java:701)
at org.sonatype.aether.connector.wagon.WagonRepositoryConnector.get(WagonRepositoryConnector.java:452)
at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:456)
... 65 more

Did i something wrong?

AhyoungRyu · 2016-05-23T03:53:02Z

Hi @cloverhearts,
Just FYI, the documentation for this feature was already written in here : https://zeppelin.incubator.apache.org/docs/0.6.0-incubating-SNAPSHOT/manual/dynamicinterpreterload.html

I hope this docs can help any others who want to review this PR :)

cloverhearts · 2016-05-23T05:01:02Z

Thank you @AhyoungRyu fixed.

cloverhearts · 2016-05-23T06:22:03Z

done.
Please, to review.

cloverhearts · 2016-05-23T07:40:39Z

Thank you, for tested. @astroshim
Please, retry test.
I fixed it.

astroshim · 2016-05-24T01:53:37Z

@cloverhearts I tested and really nice working. Thanks.

felixcheung · 2016-05-25T04:01:15Z

cool - should we have some sort of access control for this later? loading a jar by POST request on a server seems rather open ;)

cloverhearts · 2016-05-25T04:37:28Z

@felixcheung Thank you for a good opinion.
For access control to the API...
In my opinion, currently it is possible through the "Shiro" or "Basic Access Authentication".

jongyoul · 2016-06-01T03:45:21Z

Ping. I make a new issue that may break this PR - https://issues.apache.org/jira/browse/ZEPPELIN-932 -. Could anyone please check the status of this PR?

Leemoonsoo · 2016-06-02T00:42:17Z

 # export ZEPPELIN_IDENT_STRING # A string representing this instance of zeppelin. $USER by default.
 # export ZEPPELIN_NICENESS # The scheduling priority for daemons. Defaults to 0.
 # export ZEPPELIN_INTERPRETER_LOCALREPO # Local repository for interpreter's additional dependency loading
+# export ZEPPELIN_INTERPRETER_DOWNLOAD_DIR # download interpreter for directory path


I think something like "Path for interpreter download" is more clear

@Leemoonsoo It is meant for Comments?

cloverhearts · 2016-06-02T03:46:57Z

@jongyoul
Thank you for your feedback.
The PR is intended the available plug-ins . For example, download interpreter that are not included in the Zeppelin.
I believe it will this feature is in the future as required.

felixcheung · 2016-06-02T03:57:24Z

+ try {
+ urls = recursiveBuildLibList(interpreterDir);
+ } catch (MalformedURLException e1) {
+ logger.error("Can't load jars ", e1);


should it abort when it fails? or is this more a warning?

@felixcheung
Thank you for your opinion.
i've changed.

Conflicts: conf/zeppelin-env.cmd.template conf/zeppelin-env.sh.template

cloverhearts · 2016-06-04T14:56:45Z

rebase and fix

cloverhearts · 2016-06-09T07:20:19Z

Merge master branch into this pr.

bzz · 2016-06-14T03:45:06Z

@cloverhearts exciting changes! This would help us to make convenience binary for a release smaller as well

ggdupont · 2016-06-22T20:21:44Z

No luck, it can't get to load my jar from my local repo (nexus on docker). My request :

{
"artifact": "org.apache.zeppelin:zeppelin-neo4j:0.6.0-SNAPSHOT",
"className": "org.apache.zeppelin.neo4j.Neo4JInterpreter",
"repository":
{
"url": "http://localhost:8081/service/local/repositories/releases/content/",
"snapshot": true
}
}

The java trace (extract at the end) shows that my local repo seems to be ignored and that of course, it can't find my jar in official repo ;-) Any advice?

INFO [2016-06-21 22:57:37,461]({qtp922151033-19} InterpreterRestApi.java[loadDynamicInterpreter]:262) - dynamic load interpreter interpreterGroupName [jdbc] name [neo4j]
INFO [2016-06-21 22:57:37,461]({qtp922151033-19} InterpreterFactory.java[loadDynamicInterpreter]:150) - interpreter path : /home/gdupont/git/cloverhearts/zeppelin/interpreter/jdbc/neo4j
ERROR [2016-06-21 22:57:37,871]({qtp922151033-19} InterpreterFactory.java[loadDynamicInterpreter]:154) - Failed load dynamic interpreter :
org.sonatype.aether.resolution.DependencyResolutionException: Could not find artifact org.apache.zeppelin:zeppelin-neo4j:jar:0.6.0-SNAPSHOT in central (http://repo1.maven.org/maven2/)
at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:375)
at org.apache.zeppelin.dep.DependencyResolver.getArtifactsWithDep(DependencyResolver.java:170)
at org.apache.zeppelin.dep.DependencyResolver.loadFromMvn(DependencyResolver.java:123)
at org.apache.zeppelin.dep.DependencyResolver.load(DependencyResolver.java:81)

### What is this PR for? This PR is to automate release publish to maven repository. We used to use maven-deploy-plugin and maven-release-plugin for release but somehow it didn't work well with Zeppelin so 0.5.5 and 0.5.6 haven't been published to maven repository. Publishing release to maven repository will eventually help zeppelin to reduce binary package size by leading users to use Dynamic interpreter loading(#908). Originally below modules were skipped for maven release - all interpreters(except spark) - zeppelin-display - zeppelin-server - zeppelin-distribution on the other hand this pr will skip only: - zeppelin-distribution ### What type of PR is it? Infra ### Todos - [x] Include SparkR/R interpreter in release - [x] Create common_release.sh to remove build configuration duplication - [x] Check curl networking failure ### What is the Jira issue? [ZEPPELIN-986](https://issues.apache.org/jira/browse/ZEPPELIN-986) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes, https://cwiki.apache.org/confluence/display/ZEPPELIN/Preparing+Zeppelin+Release will be updated accordingly once this pr is merged. Author: Mina Lee <minalee@apache.org> Closes#994 from minahlee/ZEPPELIN-986 and squashes the following commits: b0e8e67 [Mina Lee] Revert "Add geode, scalding profile in maven artifact build" cd4cbcd [Mina Lee] curl failure check c0ea07c [Mina Lee] Fix wrong indentation a88bc1d [Mina Lee] Add geode, scalding profile in maven artifact build 2cced61 [Mina Lee] Add r to binary package and maven build 903bc12 [Mina Lee] Move duplicate code to common_release.sh a3eb676 [Mina Lee] Include zeppelin-server module in publish artifiact 48d338f [Mina Lee] Rollback mistakenly removed plugin aafaf42 [Mina Lee] Follow google shell style guide 30dcc65 [Mina Lee] remove deploy plugin from pom since custom script will be used instead for deploy cd1f08c [Mina Lee] Refactor create release script e764f5f [Mina Lee] Add maven publish release script

### What is this PR for? This PR is to automate release publish to maven repository. We used to use maven-deploy-plugin and maven-release-plugin for release but somehow it didn't work well with Zeppelin so 0.5.5 and 0.5.6 haven't been published to maven repository. Publishing release to maven repository will eventually help zeppelin to reduce binary package size by leading users to use Dynamic interpreter loading(#908). Originally below modules were skipped for maven release - all interpreters(except spark) - zeppelin-display - zeppelin-server - zeppelin-distribution on the other hand this pr will skip only: - zeppelin-distribution ### What type of PR is it? Infra ### Todos - [x] Include SparkR/R interpreter in release - [x] Create common_release.sh to remove build configuration duplication - [x] Check curl networking failure ### What is the Jira issue? [ZEPPELIN-986](https://issues.apache.org/jira/browse/ZEPPELIN-986) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? Yes, https://cwiki.apache.org/confluence/display/ZEPPELIN/Preparing+Zeppelin+Release will be updated accordingly once this pr is merged. Author: Mina Lee <minalee@apache.org> Closes#994 from minahlee/ZEPPELIN-986 and squashes the following commits: b0e8e67 [Mina Lee] Revert "Add geode, scalding profile in maven artifact build" cd4cbcd [Mina Lee] curl failure check c0ea07c [Mina Lee] Fix wrong indentation a88bc1d [Mina Lee] Add geode, scalding profile in maven artifact build 2cced61 [Mina Lee] Add r to binary package and maven build 903bc12 [Mina Lee] Move duplicate code to common_release.sh a3eb676 [Mina Lee] Include zeppelin-server module in publish artifiact 48d338f [Mina Lee] Rollback mistakenly removed plugin aafaf42 [Mina Lee] Follow google shell style guide 30dcc65 [Mina Lee] remove deploy plugin from pom since custom script will be used instead for deploy cd1f08c [Mina Lee] Refactor create release script e764f5f [Mina Lee] Add maven publish release script (cherry picked from commit 85d7057) Signed-off-by: Mina Lee <minalee@apache.org>

cloverhearts · 2016-06-29T02:15:17Z

@ggdupont
Sorry. The first answer seems much too late.
OK, it seems to do the modification.
In a somehow given log, store, see a problem I think.
You can modify the repository maven setting.xml change by force.
(For example, you can replace the repo1 repository mirror set).
I'll check for more details.
Thank you for your feedback.

jongyoul · 2016-07-12T15:42:26Z

What's the status of this PR?

cloverhearts · 2016-07-13T02:29:45Z

@jongyoul Task is waiting.
It has been delayed due to other work in progress.

jongyoul · 2016-07-13T04:58:59Z

@cloverhearts I've got it. Thanks.

Implement feat. dynamic load interpreter class
3d0a9d4

fixed invalid result for dynamic interpreter
599639f

Leemoonsoo reviewed Jun 2, 2016
View reviewed changes

felixcheung reviewed Jun 2, 2016
View reviewed changes

cloverhearts closed this Jun 4, 2016

cloverhearts reopened this Jun 4, 2016

cloverhearts added 4 commits June 4, 2016 23:47

Merge branch 'master' into feat/ZEPPELIN-598
1b598b7
Conflicts: conf/zeppelin-env.cmd.template conf/zeppelin-env.sh.template

add config in zeppelin env - dynamic interperter
3965f74

add path for interpreter download path at doc/install/install.md
1bb3a0c

add interpreter download dir for zeppelin-site.xml.template
377393a

cloverhearts added 3 commits June 8, 2016 11:20

Merge branch 'master' into feat/ZEPPELIN-598
43ad199

add exception for dynamic loading
04d84d3

Merge branch 'master' into feat/ZEPPELIN-598
b10b64d

cloverhearts closed this Jun 9, 2016

cloverhearts reopened this Jun 9, 2016

cloverhearts closed this Jun 10, 2016

cloverhearts reopened this Jun 10, 2016

minahlee mentioned this pull request Jun 10, 2016
[ZEPPELIN-986] Create publish release script #994
Closed
3 tasks

echarles mentioned this pull request Jun 14, 2016
[ZEPPELIN-871] [WIP] spark 2.0 interpreter on scala 2.11 #980
Closed
4 tasks

cloverhearts closed this Jan 21, 2017

Uh oh!

Conversation

cloverhearts commented May 22, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

ScreenShots

Is there a relevant Jira issue?

How should this be tested?

Questions:

Uh oh!

astroshim commented May 23, 2016

Uh oh!

AhyoungRyu commented May 23, 2016

Uh oh!

cloverhearts commented May 23, 2016

Uh oh!

cloverhearts commented May 23, 2016

Uh oh!

cloverhearts commented May 23, 2016

Uh oh!

astroshim commented May 24, 2016

Uh oh!

felixcheung commented May 25, 2016

Uh oh!

cloverhearts commented May 25, 2016

Uh oh!

jongyoul commented Jun 1, 2016

Uh oh!

LeemoonsooJun 2, 2016

Choose a reason for hiding this comment

Uh oh!

cloverheartsJun 2, 2016

Choose a reason for hiding this comment

Uh oh!

cloverhearts commented Jun 2, 2016

Uh oh!

felixcheungJun 2, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloverheartsJun 9, 2016

Choose a reason for hiding this comment

Uh oh!

cloverhearts commented Jun 4, 2016

Uh oh!

cloverhearts commented Jun 9, 2016

Uh oh!

bzz commented Jun 14, 2016

Uh oh!

ggdupont commented Jun 22, 2016

Uh oh!

cloverhearts commented Jun 29, 2016

Uh oh!

jongyoul commented Jul 12, 2016

Uh oh!

cloverhearts commented Jul 13, 2016

Uh oh!

jongyoul commented Jul 13, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

cloverhearts commented May 22, 2016•
edited
Loading

felixcheungJun 2, 2016•
edited
Loading