ZEPPELIN-885 Add option to send email in scheduler execution by vgmartinez · Pull Request #918 · apache/zeppelin

vgmartinez · 2016-05-25T21:50:29Z

What is this PR for?

Add the option to send email when a note is runing using the scheduler.
The option to send an email when you finish executing a note or when an error occurs in one paragraph is a good utility for the user.

What type of PR is it?

Improvement

Todos

What is the Jira issue?

ZEPPELIN-885

How should this be tested?

Configure smtp properties in zeppelin-site.xml
Set email in scheduler and add cron expresion to execute note.

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

cloverhearts · 2016-05-26T02:44:13Z

Thank you for good feature.
I like this feature.

but, i got following problem.

Zeppelin Server log.

ERROR [2016-05-26 11:27:01,752] ({DefaultQuartzScheduler_Worker-2} Notebook.java[sendEmail]:594) - Error: org.apache.commons.mail.EmailException: Sending the email to the following server failed : smtp.googlemail.com:465
at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1421)
at org.apache.commons.mail.Email.send(Email.java:1448)
at org.apache.zeppelin.notebook.Notebook.sendEmail(Notebook.java:591)
at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:513)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: javax.mail.SendFailedException: Send failure (javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8 https://support.google.com/mail/answer/14257 z12sm1446736pfj.1 - gsmtp
)
at javax.mail.Transport.send(Transport.java:163)
at javax.mail.Transport.send(Transport.java:48)
at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1411)
... 5 more
Caused by: javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8 https://support.google.com/mail/answer/14257 z12sm1446736pfj.1 - gsmtp
at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:892)
at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:814)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:728)
at javax.mail.Service.connect(Service.java:274)
at javax.mail.Service.connect(Service.java:91)
at javax.mail.Service.connect(Service.java:76)
at javax.mail.Transport.send(Transport.java:94)
... 7 more
ERROR [2016-05-26 11:27:03,392] ({DefaultQuartzScheduler_Worker-2} Notebook.java[sendEmail]:594) - Error: org.apache.commons.mail.EmailException: Sending the email to the following server failed : smtp.googlemail.com:465
at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1421)
at org.apache.commons.mail.Email.send(Email.java:1448)
at org.apache.zeppelin.notebook.Notebook.sendEmail(Notebook.java:591)
at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:539)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: javax.mail.SendFailedException: Send failure (javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8 https://support.google.com/mail/answer/14257 f9sm1425655pfd.56 - gsmtp
)
at javax.mail.Transport.send(Transport.java:163)
at javax.mail.Transport.send(Transport.java:48)
at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1411)
... 5 more
Caused by: javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8 https://support.google.com/mail/answer/14257 f9sm1425655pfd.56 - gsmtp
at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:892)
at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:814)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:728)
at javax.mail.Service.connect(Service.java:274)
at javax.mail.Service.connect(Service.java:91)
at javax.mail.Service.connect(Service.java:76)
at javax.mail.Transport.send(Transport.java:94)
... 7 more

My Environment : Mac OS elcapitan, chrome lastests.
my note context : "println("hellow");

cron time 1m;

email : onStart : myemail@gmail.com
email : onError : myemail@gmail.com
email : onSuccess : myemail@gmail.com

astroshim · 2016-05-26T04:03:19Z

I tested and working well.
@cloverhearts maybe you need to set up the smtp informations. you can set the variables in the "conf/zeppelin-site.xml".

felixcheung · 2016-05-26T06:38:27Z

cool - should we have the info in documentation?

cloverhearts · 2016-05-26T06:47:11Z

@astroshim Oh, I see. Thank you!

bzz · 2016-05-26T07:46:18Z

@vgmartinez Thank you for contribution!

Before a deep dive into the code, I have a question - could you pelase explain the use-case you have in mind for this feature?

I'm trying to understand in particular what is the benefit of introducing complexity of SNTP support in the Zeppelin codebase (not sure that Notebook.java is the best place for it either), versus just scheduling a notebook using existing scheduller, that just loads the SNTP library (using %dep) and then sends any template out.

See #8 from this notebook for an example of the data product, that uses this approach for sending out email notifications.

vgmartinez · 2016-05-26T08:02:59Z

@cloverhearts @astroshim @felixcheung thks, add more info and doc ASAP.

hi @bzz, I think having the option to send an email when you finish executing a note or when an error occurs in one paragraph is a good utility for the user.
e.g if you have cron expression to run every monday is useful to receive an email when it starts, when finished or when there is an error.
what do you think?

Leemoonsoo · 2016-05-26T22:09:41Z

@vgmartinez Thanks for contribution.

@vgmartinez and folks, how about having generalized event hook instead of just handling email after scheduler execution?

Zeppelin can generate events,

Scheduler execution (start, success, fail)
Paragraph execution (CRUD, status change)
Notebook (CRUD)
etc

And if there're some extensible mechanism that allow user plugin custom event handler. So possibly not only sending email, but also execute shell script or any other action will be possible for all type of events.

What do you think?

vgmartinez · 2016-05-29T20:21:27Z

Hi @Leemoonsoo, I totally agree with you. I'll think a bit about this topic, if you already have some idea we can debate.

thanks for the suggestion.

bzz · 2016-05-31T02:23:05Z

@vgmartinez thank you for explaining the use case, it makes perfect sense and is not covered by the case I was referring to. May it's worth putting it in JIRA\PR description as well, how do you think?

@Leemoonsoo suggestion sounds really nice indeed. Even something simpler and less general with just an email would work as a first step, but email facilities better be placed outside of the Notebook.java. I presume it may very well be some kind of 'client' of the Notebook.java API inside zengine.

Leemoonsoo · 2016-05-31T03:58:45Z

@vgmartinez I think move email facilities from Notebook.java to new class, like @bzz mentioned, would be the first step to generalize it. Doesn't need to generalize everything from the beginning, but i think it worth to show possibilities of implementing more events and more actions. And taking code out from Notebook.java will help.

vgmartinez · 2016-05-31T21:59:02Z

@bzz @Leemoonsoo thks for review.
@bzz I'm sorry I have not explained well. I've added more description to JIRA/PR
I created a new class for events outside notebook, to use for other types of event notifications. Please take a look.

Leemoonsoo · 2016-05-31T22:37:42Z

 private NotebookRepo notebookRepo;
 private SearchService notebookIndex;
 private NotebookAuthorization notebookAuthorization;
+ private static EventNotification notification;


are there any special reasons making notification and conf static?

Leemoonsoo · 2016-05-31T22:37:53Z

Do you mind add new configurations to https://github.com/apache/incubator-zeppelin/blob/master/docs/install/install.md#zeppelin-configuration as well??

vgmartinez · 2016-06-06T22:16:40Z

@Leemoonsoo sorry for delay. I added the configurations in the file install.md and delete the static modifier.

Leemoonsoo · 2016-06-07T18:50:03Z

@vgmartinez No worries. CI build fails with error

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:30 min
[INFO] Finished at: 2016-06-06T22:47:47+00:00
[INFO] Final Memory: 38M/471M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (verify.rat) on project zeppelin-zengine: Too many files with unapproved license: 1 See RAT report in: /home/travis/build/apache/incubator-zeppelin/zeppelin-zengine/target/rat.txt -> [Help 1]

Could you take a look?

vgmartinez · 2016-06-13T21:10:23Z

@Leemoonsoo CI pass...

Leemoonsoo · 2016-06-14T22:45:59Z

@vgmartinez Thanks for update.

I tried and found 'Cancel' button is not actually discarding chances in dialog. Check following screenshot

vgmartinez · 2016-11-28T09:59:49Z

Hi @Leemoonsoo,
I have refactored the code to be able to add more channels of notifications, e.g slack, SNS (in the next PR)...
Please take a look...

thks

Leemoonsoo reviewed May 31, 2016
View reviewed changes

vgmartinez force-pushed the email_option branch from b3cdefa to 128bb58Compare June 6, 2016 22:06

vgmartinez force-pushed the email_option branch 2 times, most recently from 92edccf to 8ab7fe6Compare June 13, 2016 19:28

vgmartinez force-pushed the email_option branch from 8ab7fe6 to 3dbdba2Compare November 11, 2016 14:00

vgmartinez added 4 commits November 27, 2016 22:43

rebase with master
93ed834

rebase master
8fbabfb

fix bug in angular copy
adae5e7

rebase with master
8d50f1a

vgmartinez force-pushed the email_option branch from 1469364 to 8d50f1aCompare November 27, 2016 21:56

victor added 2 commits November 27, 2016 23:02

back change in tutorial note
8fe4505

fix end line and add isError funtions
8bca02c

asfgit closed this in c38a0a0May 9, 2018

Uh oh!

Conversation

vgmartinez commented May 25, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

cloverhearts commented May 26, 2016

Uh oh!

astroshim commented May 26, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

felixcheung commented May 26, 2016

Uh oh!

cloverhearts commented May 26, 2016

Uh oh!

bzz commented May 26, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vgmartinez commented May 26, 2016

Uh oh!

Leemoonsoo commented May 26, 2016

Uh oh!

vgmartinez commented May 29, 2016

Uh oh!

bzz commented May 31, 2016

Uh oh!

Leemoonsoo commented May 31, 2016

Uh oh!

vgmartinez commented May 31, 2016

Uh oh!

LeemoonsooMay 31, 2016

Choose a reason for hiding this comment

Uh oh!

Leemoonsoo commented May 31, 2016

Uh oh!

vgmartinez commented Jun 6, 2016

Uh oh!

Leemoonsoo commented Jun 7, 2016

Uh oh!

vgmartinez commented Jun 13, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Leemoonsoo commented Jun 14, 2016

Uh oh!

vgmartinez commented Nov 28, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vgmartinez commented May 25, 2016•
edited
Loading

astroshim commented May 26, 2016•
edited
Loading

bzz commented May 26, 2016•
edited
Loading

vgmartinez commented Jun 13, 2016•
edited
Loading