Skip to content

[DRAFT] [ZEPPELIN-5722] Improve PluginManager#4355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Reamer wants to merge 9 commits into
base:master
Choose a base branch
from
Draft

[DRAFT] [ZEPPELIN-5722] Improve PluginManager #4355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3f4c090
Add pf4j for plugin loading
Reamer Apr 5, 2021
0888f35
fix tests
Reamer Apr 14, 2022
0e04dbc
Edit license file
Reamer Apr 21, 2022
270dcf7
Add some documentation
Reamer Apr 21, 2022
25af34b
Pass the PluginManager as a zengine object via a interface to the ext…
Reamer Apr 21, 2022
e52868b
use BasePlugin
Reamer May 13, 2022
7d3df8c
Remove duplicate pf4j version
Reamer May 13, 2022
5284020
cast to the returned class
Reamer May 13, 2022
4315bd8
Fix rat
Reamer May 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/_includes/themes/zeppelin/_navigation.html
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -166,6 +166,7 @@
<ul class="dropdown-menu scrollable-menu" style="right: 0; left: auto;">
<li class="title"><span>Extending Zeppelin</span></li>
<li><a href="{{BASE_PATH}}/development/writing_zeppelin_interpreter.html">Writing Zeppelin Interpreter</a></li>
<li><a href="{{BASE_PATH}}/development/writing_zeppelin_plugin.html">Writing Zeppelin Plugin</a></li>
<li role="separator" class="divider"></li>
<li class="title"><span>Helium (Experimental)</span></li>
<li><a href="{{BASE_PATH}}/development/helium/overview.html">Overview</a></li>
Expand Down
42 changes: 42 additions & 0 deletions docs/development/writing_zeppelin_plugin.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
---
layout: page
title: "Writing a New Zeppelin Plugin"
description: "Apache Zeppelin Plugin can extend the functionality of the Zeppelin Server."
group: development
---
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
{% include JB/setup %}

# Writing a New Zeppelin Plugin

<div id="toc"></div>

## What is Apache Zeppelin Plugin

Apache Zeppelin Plugin can extend the functionality of the Zeppelin zengine. Various extensions can be made to the Zeppelin zengine. For this purpose, an interface must first be implemented in the Zeppelin zengine. The biggest advantage of plugins is the [PluginClassLoader](https://pf4j.org/doc/class-loading.html), which ensures that different versions of the same Java classes do not mix between Zeppelin zengine and the plugins and therefore there are no side effects when running Zeppelin.

## Make your own Zeppelin Plugin

Creating a new plugin is quite simple. Just create a new Maven module and make `zengine-plugins-parent` as parent. Create a class with extends `org.pf4j.Plugin`. Plugins are started during Zeppelin Server Startup.
You can also create classes, which implements `org.pf4j.ExtensionPoint` in Zeppelin zengine.

Current ExtensionPoints:

- `InterpreterLauncher`
- `NotebookRepo`

Because Zeppelin does nothing magical you can find also more informations in the [official PF4J documentation](https://pf4j.org/).


1 change: 1 addition & 0 deletions docs/index.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -122,6 +122,7 @@ limitations under the License.
#### Developer Guide
* Extending Zeppelin
* [Writing Zeppelin Interpreter](./development/writing_zeppelin_interpreter.html)
* [Writing Zeppelin Plugin](./development/writing_zeppelin_plugin.html)
* [Helium: Overview](./development/helium/overview.html)
* [Helium: Writing Application](./development/helium/writing_application.html)
* [Helium: Writing Spell](./development/helium/writing_spell.html)
Expand Down
1 change: 1 addition & 0 deletions pom.xml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -134,6 +134,7 @@
<maven.version>3.6.3</maven.version>
<dropwizard.version>4.1.14</dropwizard.version>
<micrometer.version>1.6.0</micrometer.version>
<pf4j.version>3.6.0</pf4j.version>

<hadoop2.7.version>2.7.7</hadoop2.7.version>
<hadoop2.6.version>2.6.5</hadoop2.6.version>
Expand Down
2 changes: 2 additions & 0 deletions zeppelin-distribution/src/bin_license/LICENSE
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -227,6 +227,7 @@ The following components are provided under Apache License.
(Apache 2.0) Dropwizard Jackson Integration for Metrics (io.dropwizard.metrics:metrics-json:4.1.14) - https://github.com/dropwizard/metrics/blob/release/4.1.x/LICENSE
(Apache 2.0) Dropwizard Metrics Health Checks (io.dropwizard.metrics:metrics-healthchecks:4.1.14) - https://github.com/dropwizard/metrics/blob/release/4.1.x/LICENSE
(Apache 2.0) Dropwizard Metrics Integration with JMX (io.dropwizard.metrics:metrics-jmx:4.1.14) - https://github.com/dropwizard/metrics/blob/release/4.1.x/LICENSE
(Apache 2.0) Plugin Framework for Java (org.pf4j:pf4j:3.6.0) - https://github.com/pf4j/pf4j

========================================================================
MIT licenses
Expand DownExpand Up@@ -285,6 +286,7 @@ The text of each license is also included at licenses/LICENSE-[project]-[version
(The MIT License) ansi-up 2.0.2 (https://github.com/drudru/ansi_up) - https://github.com/drudru/ansi_up#license
(The MIT License) bcpkix-jdk15on 1.60 (org.bouncycastle:bcpkix-jdk15on:1.60 https://github.com/bcgit/bc-java) - https://github.com/bcgit/bc-java/blob/master/LICENSE.html
(The MIT License) influxdb-client-java 1.4.0 (com.influxdb:influxdb-client-java:1.4.0 https://github.com/influxdata/influxdb-client-java) - https://github.com/influxdata/influxdb-client-java/blob/master/LICENSE
(The MIT License) Java implementation of the SemVer Specification (com.github.zafarkhaja:java-semver:0.9.0) - https://github.com/zafarkhaja/jsemver

========================================================================
BSD-style licenses
Expand Down
6 changes: 5 additions & 1 deletion ...n-interpreter-integration/src/test/java/org/apache/zeppelin/integration/MiniZeppelin.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -24,6 +24,8 @@
import org.apache.zeppelin.interpreter.InterpreterFactory;
import org.apache.zeppelin.interpreter.InterpreterSettingManager;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcessListener;
import org.apache.zeppelin.plugin.IPluginManager;
import org.apache.zeppelin.plugin.ZPluginManager;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand DownExpand Up@@ -59,8 +61,10 @@ public void start(Class clazz) throws IOException {
System.setProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_NOTEBOOK_DIR.getVarName(), notebookDir.getAbsolutePath());
System.setProperty(ZeppelinConfiguration.ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT.getVarName(), "120000");
conf = ZeppelinConfiguration.create();
IPluginManager pluginManager = new ZPluginManager(conf);
pluginManager.loadAndStartPlugins();
interpreterSettingManager = new InterpreterSettingManager(conf,
mock(AngularObjectRegistryListener.class), mock(RemoteInterpreterProcessListener.class), mock(ApplicationEventListener.class));
mock(AngularObjectRegistryListener.class), mock(RemoteInterpreterProcessListener.class), mock(ApplicationEventListener.class), pluginManager);
interpreterFactory = new InterpreterFactory(interpreterSettingManager);
}

Expand Down
44 changes: 3 additions & 41 deletions zeppelin-plugins/launcher/cluster/pom.xml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -36,53 +36,15 @@

<properties>
<plugin.name>Launcher/ClusterInterpreterLauncher</plugin.name>
<plugin.class>org.apache.zeppelin.interpreter.launcher.ClusterPlugin</plugin.class>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.zeppelin</groupId>
<artifactId>launcher-docker</artifactId>
<version>0.11.0-SNAPSHOT</version>
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>

<build>
<testResources>
<testResource>
<directory>${project.basedir}/src/test/resources</directory>
</testResource>
<testResource>
<directory>${project.basedir}/src/main/resources</directory>
</testResource>
</testResources>
<plugins>
<plugin>
<artifactId>maven-enforcer-plugin</artifactId>
<executions>
<execution>
<id>enforce</id>
<phase>none</phase>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
<skip>false</skip>
</configuration>
</plugin>
</plugins>
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.*</include>
</includes>
</resource>
</resources>
</build>
</project>
23 changes: 17 additions & 6 deletions ...er/src/main/java/org/apache/zeppelin/interpreter/launcher/ClusterInterpreterLauncher.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -30,6 +30,9 @@
import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreterRunningProcess;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreterUtils;
import org.apache.zeppelin.plugin.ExtensionWithPluginManager;
import org.apache.zeppelin.plugin.IPluginManager;
import org.pf4j.Extension;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand All@@ -46,22 +49,30 @@
/**
* Interpreter Launcher which use cluster to launch the interpreter process.
*/
@Extension(points = InterpreterLauncher.class)
public class ClusterInterpreterLauncher extends StandardInterpreterLauncher
implements ClusterEventListener {
implements ClusterEventListener, ExtensionWithPluginManager {
private static final Logger LOGGER = LoggerFactory.getLogger(ClusterInterpreterLauncher.class);

private InterpreterLaunchContext context;
private ClusterManagerServer clusterServer;
public ClusterInterpreterLauncher(ZeppelinConfiguration zConf, RecoveryStorage recoveryStorage)
throws IOException {
super(zConf, recoveryStorage);
private IPluginManager pluginManager;

@Override
public void setPluginManager(IPluginManager pluginManager) {
this.pluginManager = pluginManager;
}

@Override
public void init(ZeppelinConfiguration zConf, RecoveryStorage recoveryStorage) {
super.init(zConf, recoveryStorage);
this.clusterServer = ClusterManagerServer.getInstance(zConf);
clusterServer.addClusterEventListeners(ClusterManagerServer.CLUSTER_INTP_EVENT_TOPIC, this);
}

@Override
public InterpreterClient launchDirectly(InterpreterLaunchContext context) throws IOException {
LOGGER.info("Launching Interpreter: " + context.getInterpreterSettingGroup());
LOGGER.info("Launching Interpreter: {}", context.getInterpreterSettingGroup());

this.context = context;
this.properties = context.getProperties();
Expand DownExpand Up@@ -224,7 +235,7 @@ private InterpreterClient createInterpreterProcess(InterpreterLaunchContext cont

InterpreterClient intpProcess = null;
if (isRunningOnDocker(zConf)) {
DockerInterpreterLauncher dockerIntpLauncher = new DockerInterpreterLauncher(zConf, null);
DockerInterpreterLauncher dockerIntpLauncher = (DockerInterpreterLauncher) pluginManager.createInterpreterLauncher("DockerInterpreterLauncher", null);
dockerIntpLauncher.setProperties(context.getProperties());
intpProcess = dockerIntpLauncher.launch(context);
} else {
Expand Down
28 changes: 28 additions & 0 deletions ...auncher/cluster/src/main/java/org/apache/zeppelin/interpreter/launcher/ClusterPlugin.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.zeppelin.interpreter.launcher;

import org.apache.zeppelin.plugin.BasePlugin;
import org.pf4j.PluginWrapper;

public class ClusterPlugin extends BasePlugin {

public ClusterPlugin(PluginWrapper wrapper) {
super(wrapper, "ClusterPlugin");
}
}
16 changes: 10 additions & 6 deletions ...rc/test/java/org/apache/zeppelin/interpreter/launcher/ClusterInterpreterLauncherTest.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -19,6 +19,8 @@
import org.apache.zeppelin.conf.ZeppelinConfiguration;
import org.apache.zeppelin.interpreter.InterpreterOption;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreterRunningProcess;
import org.apache.zeppelin.plugin.IPluginManager;
import org.apache.zeppelin.plugin.ZPluginManager;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
Expand All@@ -33,7 +35,8 @@
import static org.junit.Assert.assertTrue;

public class ClusterInterpreterLauncherTest extends ClusterMockTest {
private static Logger LOGGER = LoggerFactory.getLogger(ClusterInterpreterLauncherTest.class);
private static final Logger LOGGER = LoggerFactory.getLogger(ClusterInterpreterLauncherTest.class);
private IPluginManager pluginManager;

@BeforeClass
public static void startTest() throws IOException, InterruptedException {
Expand All@@ -50,6 +53,8 @@ public void setUp() {
for (final ZeppelinConfiguration.ConfVars confVar : ZeppelinConfiguration.ConfVars.values()) {
System.clearProperty(confVar.getVarName());
}
pluginManager = new ZPluginManager(zconf);
pluginManager.loadAndStartPlugins();
}

// TODO(zjffdu) disable this test because this is not a correct unit test,
Expand All@@ -58,8 +63,7 @@ public void setUp() {
public void testConnectExistOnlineIntpProcess() throws IOException {
mockIntpProcessMeta("intpGroupId", true);

ClusterInterpreterLauncher launcher
= new ClusterInterpreterLauncher(ClusterMockTest.zconf, null);
ClusterInterpreterLauncher launcher = (ClusterInterpreterLauncher) pluginManager.createInterpreterLauncher("ClusterInterpreterLauncher", null);
Properties properties = new Properties();
properties.setProperty(
ZeppelinConfiguration.ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT.getVarName(), "5000");
Expand All@@ -83,7 +87,7 @@ public void testConnectExistOfflineIntpProcess() throws IOException {
mockIntpProcessMeta("intpGroupId2", false);

ClusterInterpreterLauncher launcher
= new ClusterInterpreterLauncher(ClusterMockTest.zconf, null);
= (ClusterInterpreterLauncher) pluginManager.createInterpreterLauncher("ClusterInterpreterLauncher", null);;
Properties properties = new Properties();
properties.setProperty(
ZeppelinConfiguration.ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT.getVarName(), "5000");
Expand All@@ -110,7 +114,7 @@ public void testCreateIntpProcessDockerMode() throws IOException {
zconf.setRunMode(ZeppelinConfiguration.RUN_MODE.DOCKER);

ClusterInterpreterLauncher launcher
= new ClusterInterpreterLauncher(zconf, null);
= (ClusterInterpreterLauncher) pluginManager.createInterpreterLauncher("ClusterInterpreterLauncher", null);
Properties properties = new Properties();
properties.setProperty(
ZeppelinConfiguration.ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT.getVarName(), "1000");
Expand All@@ -129,7 +133,7 @@ public void testCreateIntpProcessLocalMode() throws IOException {
zconf.setRunMode(ZeppelinConfiguration.RUN_MODE.LOCAL);

ClusterInterpreterLauncher launcher
= new ClusterInterpreterLauncher(zconf, null);
= (ClusterInterpreterLauncher) pluginManager.createInterpreterLauncher("ClusterInterpreterLauncher", null);
Properties properties = new Properties();
properties.setProperty(
ZeppelinConfiguration.ConfVars.ZEPPELIN_INTERPRETER_CONNECT_TIMEOUT.getVarName(), "1000");
Expand Down
55 changes: 13 additions & 42 deletions zeppelin-plugins/launcher/docker/pom.xml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -36,6 +36,7 @@

<properties>
<plugin.name>Launcher/DockerInterpreterLauncher</plugin.name>
<plugin.class>org.apache.zeppelin.interpreter.launcher.DockerPlugin</plugin.class>
</properties>

<dependencies>
Expand All@@ -49,12 +50,24 @@
<groupId>com.hubspot.jinjava</groupId>
<artifactId>jinjava</artifactId>
<version>2.5.4</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.spotify</groupId>
<artifactId>docker-client</artifactId>
<classifier>shaded</classifier>
<version>8.15.2</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
Expand All@@ -64,52 +77,10 @@
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-api-mockito</artifactId>
<version>${powermock.version}</version>
</dependency>
<dependency>
<groupId>org.powermock</groupId>
<artifactId>powermock-module-junit4</artifactId>
<version>${powermock.version}</version>
</dependency>
</dependencies>

<build>
<testResources>
<testResource>
<directory>${project.basedir}/src/test/resources</directory>
</testResource>
<testResource>
<directory>${project.basedir}/src/main/resources</directory>
</testResource>
</testResources>
<plugins>
<plugin>
<artifactId>maven-enforcer-plugin</artifactId>
<executions>
<execution>
<id>enforce</id>
<phase>none</phase>
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<configuration>
<skip>false</skip>
</configuration>
</plugin>
</plugins>
<resources>
<resource>
<directory>src/main/resources</directory>
<includes>
<include>**/*.*</include>
</includes>
</resource>
</resources>
</build>
</project>
Loading