Presto Interpreter by danielhaviv · Pull Request #68 · apache/zeppelin

danielhaviv · 2015-05-11T06:23:42Z

Added an interpreter for Presto (http://prestodb.io)

Leemoonsoo · 2015-05-12T23:36:06Z

Thanks for the contribution! Great work!

One question. Is there any possibility to have a test using TestingPrestoServer.java? if it is not too difficult.

danielhaviv · 2015-05-13T03:42:38Z

Yes
I will work on it

Daniel

On 13 במאי 2015, at 02:36, Lee moon soo notifications@github.com wrote:
Thanks for the contribution! Great work!
One question. Is there any possibility to have a test using TestingPrestoServer.java? if it is not too difficult.
—
Reply to this email directly or view it on GitHub.

Leemoonsoo · 2015-05-13T05:25:01Z

version should be 0.5.0-incubating-SNAPSHOT

Leemoonsoo · 2015-05-31T00:26:58Z

Will there be some tests?

Leemoonsoo · 2015-06-19T14:35:58Z

@danielhaviv Do you mind if i make PR to your branch that adds some tests?

danielhaviv · 2015-06-19T15:03:34Z

Not at all.
My apologies for not having the time to take care of it.

Daniel

On 19 ביוני 2015, at 17:36, Lee moon soo notifications@github.com wrote:
@danielhaviv Do you mind if i make PR to your branch that adds some tests?
—
Reply to this email directly or view it on GitHub.

danielhaviv · 2015-07-04T12:36:25Z

Hi,
I see the interpreter is currently not included but there's a Jira for it, any reason why ?

Thanks,
Daniel

Leemoonsoo · 2015-07-09T14:34:05Z

Hi, @danielhaviv
Thanks for asking.

This interpreter code looks good but i think there should be some sort of test for it.
I recently i've been trying to implement some test for it, but i've stuck with JDK8 requirement.

It looks like version 0.87 and later requires JDK8 to use it's maven release artifact, while Zeppelin still stay with JDK7.

I'm trying to figure out the way, but any good idea about JDK8 requirement?

ashwinshankar77 · 2015-08-07T22:26:52Z

Instead of hard-coding default, can we first try to get default through environment variables and system properties.
For eg. please see SparkInterpreter.getSystemDefault()
This would be useful because admins can set a defaults and users can just start using presto and not worry about updating properties on the interpreter page.

ashwinshankar77 · 2015-08-07T22:29:48Z

We need to add this interpreter in ZeppelinConfiguration.java under ZEPPELIN_INTERPRETERS,
otherwise zeppelin doesn't automatically show presto as one of the available interpreters on the UI.
Also add "presto" module to zeppelin pom.xml.

martin-g · 2015-08-08T12:20:06Z

IMO the UI application should show only the discovered interpreters.
Each interpreter is in its own Maven module.
Zeppelin UI app should not load all interpreter modules with their transitive dependencies.
It should load only the ones me, as a developer, decided to put in the classpath.
Otherwise Zeppelin app soon will be ~1GB big and most users will use just the core and 1/2 interpreters, all the rest will be just extra unused stuff.

jongyoul · 2015-11-13T07:08:56Z

Linked to #361 because it has a same structure of a generic jdbc driver

jongyoul · 2016-02-25T06:36:39Z

@danielhaviv Do you mind closing this PR and updating doc for using Presto via JDBCInterpreter?

danielhaviv · 2016-02-28T13:41:56Z

No longer relevant as there is now a JDBCInterpreter.

…ron jobs takes long time or gets stuck ### What is this PR for? The cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck. I sometimes come across the issue that the cron scheduler stops working suddenly. According to the thread dump of ZeppelinServer, all of the DefaultQuartzScheduler_Worker threads were waiting for the job's completion and there was no thread to launch a new job. Here is the contents of the thread dump: ``` "DefaultQuartzScheduler_Worker-10" #76 prio=5 os_prio=0 tid=0x00007fb41d3b4000 nid=0x1b521 sleeping[0x00007fb3daef1000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0a7dbf0> (a java.lang.Object) Locked ownable synchronizers: - None "DefaultQuartzScheduler_Worker-9" #75 prio=5 os_prio=0 tid=0x00007fb41d3b2000 nid=0x1b520 waiting on condition [0x00007fb3daff2000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0a7a470> (a java.lang.Object) Locked ownable synchronizers: - None ... "DefaultQuartzScheduler_Worker-2" #68 prio=5 os_prio=0 tid=0x00007fb41d3c8800 nid=0x1b519 waiting on condition [0x00007fb3da473000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0a7a7b0> (a java.lang.Object) Locked ownable synchronizers: - None "DefaultQuartzScheduler_Worker-1" #67 prio=5 os_prio=0 tid=0x00007fb41d3cc800 nid=0x1b518 waiting on condition [0x00007fb3da372000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0a7dd90> (a java.lang.Object) Locked ownable synchronizers: - None ``` The above thread dump says that all of the worker threads get stuck at https://github.com/apache/zeppelin/blob/v0.7.3/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java#L889. One way to reproduce this kind of issue is creating a paragraph whose status is "READY" and "disable run". That makes the paragraph status "READY" permanently and `note.isTerminated()` never turns to `true`. To fix this issue, the following two improvements has been made at this PR: 1. Remove the unnecessary `while (!note.isTerminated()) { ... }` block because the execution of all of the paragraphs is finished after `note.runAll()`. 2. Skip the cron execution if there is a running or pending paragraph. That prevents the Zeppelin cron scheduler from getting stuck by the long running paragraph whose execution duration is greater than the cron execution cycle. ### What type of PR is it? [Bug] ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3077 ### How should this be tested? * Tested manually. 1. The cron scheduler does not get stuck if there is a paragraph whose status is "READY" and "disable run". 2. The following message is printed on the log file when the cron job is launched while the previous cron job still has been running. * `execution of the cron job is skipped because there is a running or pending paragraph (note id: XXXXXXXXX)` ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No. * Is there breaking changes for older versions? No. * Does this needs documentation? Yes. The behavior of the cron job was changed not to run if there is a running or pending paragraph by this PR. Thus, the documentation `docs/usage/other_features/cron_scheduler.md` was also added by this PR. Its layout is as follow: <img width="711" alt="screen shot 2017-11-28 at 18 30 54" src="https://user-images.githubusercontent.com/31149688/33312407-20664e02-d46b-11e7-9715-9e2562d5e064.png"> Author: Keiji Yoshida <kjmrknsn@gmail.com> Closes#2687 from kjmrknsn/ZEPPELIN-3077 and squashes the following commits: 81e7218 [Keiji Yoshida] [ZEPPELIN-3077] Cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck

…ron jobs takes long time or gets stuck ### What is this PR for? The cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck. I sometimes come across the issue that the cron scheduler stops working suddenly. According to the thread dump of ZeppelinServer, all of the DefaultQuartzScheduler_Worker threads were waiting for the job's completion and there was no thread to launch a new job. Here is the contents of the thread dump: ``` "DefaultQuartzScheduler_Worker-10" apache#76 prio=5 os_prio=0 tid=0x00007fb41d3b4000 nid=0x1b521 sleeping[0x00007fb3daef1000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0a7dbf0> (a java.lang.Object) Locked ownable synchronizers: - None "DefaultQuartzScheduler_Worker-9" apache#75 prio=5 os_prio=0 tid=0x00007fb41d3b2000 nid=0x1b520 waiting on condition [0x00007fb3daff2000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0a7a470> (a java.lang.Object) Locked ownable synchronizers: - None ... "DefaultQuartzScheduler_Worker-2" apache#68 prio=5 os_prio=0 tid=0x00007fb41d3c8800 nid=0x1b519 waiting on condition [0x00007fb3da473000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0a7a7b0> (a java.lang.Object) Locked ownable synchronizers: - None "DefaultQuartzScheduler_Worker-1" apache#67 prio=5 os_prio=0 tid=0x00007fb41d3cc800 nid=0x1b518 waiting on condition [0x00007fb3da372000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:889) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0a7dd90> (a java.lang.Object) Locked ownable synchronizers: - None ``` The above thread dump says that all of the worker threads get stuck at https://github.com/apache/zeppelin/blob/v0.7.3/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java#L889. One way to reproduce this kind of issue is creating a paragraph whose status is "READY" and "disable run". That makes the paragraph status "READY" permanently and `note.isTerminated()` never turns to `true`. To fix this issue, the following two improvements has been made at this PR: 1. Remove the unnecessary `while (!note.isTerminated()) { ... }` block because the execution of all of the paragraphs is finished after `note.runAll()`. 2. Skip the cron execution if there is a running or pending paragraph. That prevents the Zeppelin cron scheduler from getting stuck by the long running paragraph whose execution duration is greater than the cron execution cycle. ### What type of PR is it? [Bug] ### Todos ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-3077 ### How should this be tested? * Tested manually. 1. The cron scheduler does not get stuck if there is a paragraph whose status is "READY" and "disable run". 2. The following message is printed on the log file when the cron job is launched while the previous cron job still has been running. * `execution of the cron job is skipped because there is a running or pending paragraph (note id: XXXXXXXXX)` ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No. * Is there breaking changes for older versions? No. * Does this needs documentation? Yes. The behavior of the cron job was changed not to run if there is a running or pending paragraph by this PR. Thus, the documentation `docs/usage/other_features/cron_scheduler.md` was also added by this PR. Its layout is as follow: <img width="711" alt="screen shot 2017-11-28 at 18 30 54" src="https://user-images.githubusercontent.com/31149688/33312407-20664e02-d46b-11e7-9715-9e2562d5e064.png"> Author: Keiji Yoshida <kjmrknsn@gmail.com> Closesapache#2687 from kjmrknsn/ZEPPELIN-3077 and squashes the following commits: 81e7218 [Keiji Yoshida] [ZEPPELIN-3077] Cron scheduler is easy to get stuck when one of the cron jobs takes long time or gets stuck

…re.refactor.assetion_interface to V_1.0.0 * commit '63515cbfe88c785636fb545ae268e5d9a943f95e': [ZP-000] add assertion interface

Added a Presto interpreter
bbf10cd
Added an interpreter for Presto (http://prestodb.io)

Leemoonsoo reviewed May 13, 2015
View reviewed changes

Comment threadpresto/pom.xml Outdated
LeemoonsooMay 13, 2015
Copy link
Copy Markdown
Member
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
version should be 0.5.0-incubating-SNAPSHOT

Fixed pom to build as incubating
e2eb8f9

ashwinshankar77 reviewed Aug 7, 2015
View reviewed changes

danielhaviv closed this Feb 28, 2016

Uh oh!

Conversation

danielhaviv commented May 11, 2015

Uh oh!

Leemoonsoo commented May 12, 2015

Uh oh!

danielhaviv commented May 13, 2015

Uh oh!

LeemoonsooMay 13, 2015

Choose a reason for hiding this comment

Uh oh!

Leemoonsoo commented May 31, 2015

Uh oh!

Leemoonsoo commented Jun 19, 2015

Uh oh!

danielhaviv commented Jun 19, 2015

Uh oh!

danielhaviv commented Jul 4, 2015

Uh oh!

Leemoonsoo commented Jul 9, 2015

Uh oh!

ashwinshankar77Aug 7, 2015

Choose a reason for hiding this comment

Uh oh!

ashwinshankar77 commented Aug 7, 2015

Uh oh!

martin-g commented Aug 8, 2015

Uh oh!

jongyoul commented Nov 13, 2015

Uh oh!

jongyoul commented Feb 25, 2016

Uh oh!

danielhaviv commented Feb 28, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants