ZEPPELIN-1692: Ability to access Spark jobs UI from the paragraph by karuppayya · Pull Request #1663 · apache/zeppelin

karuppayya · 2016-11-21T15:32:34Z

What is this PR for?

A paragraph execution may result in spark job(s).
Adding ability to access the spark job UI(corresponding to the job generated by the paragraph run), directly from the paragraph.

What type of PR is it?

Improvement

Todos

Write tests

What is the Jira issue?

ZEPPELIN-1692

How should this be tested?

Run paragraphs with spark code(scala, pyspark, sql, R).
The paragraph will display a button on the top right corner, which on click will open up the corresponding job UI

Screenshots (if appropriate)

Questions:

Does the licenses files need update? NA
Is there breaking changes for older versions? NA
Does this needs documentation? NA

cloverhearts · 2016-11-21T15:58:57Z

awsome!

karuppayya · 2016-11-21T16:11:22Z

 Date dateUpdated;
 private Map<String, Object> config; // paragraph configs like isOpen, colWidth, etc
 public final GUI settings; // form and parameter settings
+ private Map<String, Set<String>> runtimeInfos;


I dont want to persist this info in json.
When i add the transient modifier, the serailization to JSON(for broadcasting to client browser) doesnot happen.
I can add an Exclusion strategy for all the storage handlers. But wanted to check if there is an easier way to achieve this . @Leemoonsoo @cloverhearts

Hello,
I think there is no problem using transient.
If you think you have a problem with using transient, can you comment on reason?
Actually, my english is not good.
I am sorry if I have been rude to you.
Thank you.

@cloverhearts Your English is not bad 👍
If i add transient modifier, when Gson.toJson(Paragraph) is done to send message about note/paragraph, the runtimeInfos wont be serialized and will not be received by browser(runtimeInfos wont get persisted in note.json),
Whereas if i add the modifier(current implementation), it will get persisted in JSON. But we dont want it, since it a run time information, and we might not want to persist in the note.json
Is there a way where i can serailize this info(for sending to client browser) as well as not persist in json?
PS: runtimeInfos object has the paragraph url information.

karuppayya · 2016-11-23T01:54:17Z

Ready for review @Leemoonsoo

1ambda · 2016-11-23T05:42:02Z

Let me review this PR and then give you some feedbacks. Thanks @karup1990!

I think Spark job button might be located on the left of the FINISHED button to keep consistency with other interpreter results which are not using spark. Maybe we can put it on the top left corner of a paragraph and other interpreters can use the location to give their interpreter specific buttons, options.
What about using uppercase chars like SPARK JOB because we are already using uppercases in FINISHED
If when I clicked hide output button on a paragraph, Spark job button doesn't disappear. What do you think? Should it be removed?

karuppayya · 2016-11-23T18:50:24Z

Thanks @1ambda for looking into this PR and for the suggestions.
For the 3rd point,
I am not very sure of the use cases for hiding the output in a paragraph.
Do you see a case where the job link has to be hidden when output is hidden?

1ambda · 2016-11-23T22:42:26Z

@karup1990 sorry for misspelling It was clear output

zjffdu · 2016-11-23T23:38:46Z

 Method take;
- String jobGroup = "zeppelin-" + interpreterContext.getParagraphId();
+ String jobGroup = "zeppelin-" + interpreterContext.getNoteId() + "-"
+ + interpreterContext.getParagraphId();


I see several places of constructing jobGroupId from noteId/paragraphId and extract noteId/paragraphId from jobGroupId. It's better to put them together so that others can understand it easily and other places can reuse it.

I have added Utils. buildJobGroupId and using it across.

zjffdu · 2016-11-23T23:41:31Z

+ String id = interpreterGroup.getId();
+ int indexOfColon = id.indexOf(":");
+ return id.substring(0, indexOfColon);
+ }


This assume the interpreterGroup id format, seems fragile.

I am seeing that interpreter setting is the first prefix, then based on the interpreter mode we are appending ids. Please correct me if I am wrong .

I think it is better to put getInterpreterSettingId into Utils as well because it is about how we parse interpreterGroup

zjffdu · 2016-11-24T04:40:58Z

+ SparkInterpreter sparkInterpreter = getSparkInterpreter();
+ sparkInterpreter.populateSparkWebUrl(interpreterContext);
+
+ String jobGroup = sparkInterpreter.getJobGroup(interpreterContext);


Would it be better to rename it sparkInterpreter.buildJobGroupId as each time it would create a new jobGroupId IIUC.

zjffdu · 2016-11-24T04:44:38Z

I left some comments inline and could you add test for this feature ?

AhyoungRyu · 2016-11-24T06:01:41Z

+ <a href="{{paragraph.runtimeInfos.jobUrl[0]}}" target="_blank"><span class="fa fa-tasks"></span> Spark job </a>
+ </span>
+ <span class="dropdown" ng-show="paragraph.runtimeInfos.jobUrl.length > 1">
+ <span class="fa fa-tasks" style="cursor:pointer;color:#3071A9" tooltip-placement="top" tooltip="Run this paragraph (Shift+Enter)"


I think tooltip-placement="top" tooltip="Run this paragraph (Shift+Enter)" doesn't need in here :)

AhyoungRyu · 2016-11-24T06:22:48Z

Hi @karup1990, really nice and useful feature indeed.
While quickly looking through this, I have some minor feedbacks about UI :)

Spark Job and Spark jobs have different font like below.
And I think these block that you added in this PR should be below this line.

 <span ng-show="paragraph.runtimeInfos.jobUrl.length == 1">
<a href="{{paragraph.runtimeInfos.jobUrl[0]}}" target="_blank"><span class="fa fa-tasks"></span> Spark Job </a>
</span>
<span class="dropdown" ng-show="paragraph.runtimeInfos.jobUrl.length > 1">
<span class="fa fa-tasks" style="cursor:pointer;color:#3071A9" tooltip-placement="top" tooltip="Run this paragraph (Shift+Enter)"
data-toggle="dropdown"
type="button"> Spark Jobs
</span>
<ul class="dropdown-menu" role="menu" style="width:200px;z-index:1002">
<li ng-class="{'option-disabled': !noteOperationsAllowed()}" ng-repeat="url in paragraph.runtimeInfos.jobUrl">
<a href="{{url}}" target="_blank"><span class="fa fa-external-link-square"></span> Jobs #{{$index}}</a>
</li>
</ul>
</span>

current
after

AhyoungRyu · 2016-11-24T06:07:39Z

 ng-click="runParagraph(getEditorValue())"
 ng-show="paragraph.status!='RUNNING' && paragraph.status!='PENDING' && paragraph.config.enabled"></span>
+ <span ng-show="paragraph.runtimeInfos.jobUrl.length == 1">
+ <a href="{{paragraph.runtimeInfos.jobUrl[0]}}" target="_blank"><span class="fa fa-tasks"></span> Spark job </a>


Maybe it's nit pick, let's use capital letter for Spark job as below "Spark Jobs" does :)

How about display runtime info button before paragraph status and run button?
So all other other control buttons are stay in the same position whether runtimeInfo button is displayed or not.

And how about we generalize this feature little bit more, such as
runtimeInfos = { label: "Spark Job" jobUrl : [], }
So other interpreters can also leverage this feature.

@1ambda Clearing the links when we clear the output 👍
@AhyoungRyu Fixed font issue
@Leemoonsoo Added ParagraphRuntimeInfos class(https://github.com/apache/zeppelin/pull/1663/files#diff-a2ac97dfb3f07b2a3fd1533af701295e) to have a generic support to add more properties. Let me know how it looks.
@AhyoungRyu @Leemoonsoo @1ambda Fixed the alignment and capitalized the text(to access the jobs)

karuppayya · 2016-11-25T16:31:41Z

Thanks for the feedback. i will have them handled some time over the weekend.

karuppayya · 2016-11-26T04:29:26Z

Failure seems not related
04:09:12,112 ERROR org.apache.zeppelin.AbstractZeppelinIT:136 - Exception in ZeppelinIT while testAngularDisplay org.openqa.selenium.TimeoutException: Timed out after 60 seconds waiting for org.apache.zeppelin.AbstractZeppelinIT$1@4c7305e4 Build info: version: '2.48.2', revision: '41bccdd10cf2c0560f637404c2d96164b67d9d67', time: '2015-10-09 13:08:06' System info: host: 'testing-docker-bcffa7c6-8593-4893-a72f-07a719f105fa', ip: '172.17.0.6', os.name: 'Linux', os.arch: 'amd64', os.version: '4.8.7-040807-generic', java.version: '1.7.0_76' Driver info: driver.version: unknown

karuppayya · 2016-11-27T13:54:50Z

Updated the PR with test @zjffdu

karuppayya · 2016-11-30T15:42:23Z

Ready for review

zjffdu · 2016-12-01T00:59:02Z

+ int secondIndex = jobgroupId.indexOf("-", indexOf + 1);
+ return jobgroupId.substring(secondIndex + 1, jobgroupId.length());
+ }
+ };


Would it be better to put getNoteId and getParagraphId to Utils ? So that the constructing and parsing logic are in the same class.

zjffdu · 2016-12-01T01:03:20Z

+
+ public void addValue(String value) {
+ if (values == null) {
+ values = new ArrayList<>();


Should we create values in the constructor ? Otherwise at least this method is not thread-safe (we might create 2 values in 2 threads)

zjffdu · 2016-12-01T01:07:47Z

 Date dateUpdated;
 private Map<String, Object> config; // paragraph configs like isOpen, colWidth, etc
 public final GUI settings; // form and parameter settings
+ private Map<String, ParagraphRuntimeInfos> runtimeInfos;


Should we rename ParagraphRuntimeInfos to ParagraphRuntimeInfo ? Because it is a little confusing that runtimeInfos is also plural

zjffdu · 2016-12-01T01:09:55Z

 if (p.getDateFinished() != null && lastUpdatedDate.before(p.getDateFinished())) {
 lastUpdatedDate = p.getDateFinished();
 }
+ p.clearRuntimeInfo();


Why we need to clear RuntimeInfo when we load note if the runtimeInfo is not stored ?

I will nedd to remove this line.
The runtime infos is persisted currently. But I dont want to persist it.
If i add transient modifier to runtimeInfos, when Gson.toJson(Paragraph) is done to send message about note/paragraph, the runtimeInfos wont be serialized and will not be received by browser(runtimeInfos wont get persisted in note.json),
Whereas if i add the modifier(current implementation), it will get persisted in JSON. But we dont want it, since it a run time information, and we might not want to persist in the note.json
Is there a way where i can serialize this info(for sending to client browser) as well as not persist in json?
PS: runtimeInfos object has the paragraph url information.

zjffdu · 2016-12-01T01:22:38Z

+
+ String propertyName;
+ String label;
+ String group;


I think here the label and group are duplicated. I don't see you use label in frontend. Another approach I think of is that we always use interpreter group name as the prefix of propertyName as convention, (e.g. spark.jobUrl), so that we don't need label or group. @Leemoonsoo

The label is used here
zeppelin/zeppelin-web/src/app/notebook/paragraph/paragraph-control.html
Line 19 in 9612dd0
{{paragraph.runtimeInfos.jobUrl.label}}

I had added group , so that two interpreters having same property name can be rendered in a different manner in UI.

zjffdu · 2016-12-01T01:28:14Z

@@ -970,6 +1015,7 @@ public void populateSparkWebUrl(InterpreterContext ctx) {
 infos.put("url", sparkUrl);
 logger.info("Sending metainfos to Zeppelin server: {}", infos.toString());


Should we put the logging in the if block ? And what about else block ? Will that happen ?

zjffdu · 2016-12-01T01:49:41Z

+ }
+ }
+
+ public void clearParagraphRuntimeInfo(InterpreterSetting setting) {


I am afraid this method may some issues because the binding between InterpreterSetting and Note/Paragraph may change. e.g. At the beginning I use spark in p1, but later I use flink in p1, but the method will clear all the runtime info of p1 when I restart spark interpreter.

Right.. I have handled this now.

Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Signed-off-by: Karup <karuppayya@outlook.com>

Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Signed-off-by: karuppayya <karuppayyar@qubole.com>

Signed-off-by: Karup <karuppayya@outlook.com>

karuppayya · 2017-02-04T01:30:30Z

Thanks @Leemoonsoo for the suggestion and the pointer.
Looks like all tested have passed. Ready to merge.

Leemoonsoo · 2017-02-04T02:51:12Z

@karuppayya Great work!

LGTM and merge to master if no further discussions.

Tagar · 2017-03-01T21:33:37Z

Just upgraded to yesterday's master snapshot.

When I click on any of these links, link is leading to

http://hostname.domain.com:8088/proxy/application_1488384993892_0001/jobs/job/

and that error shows HTTP ERROR 400

Page reads

Problem accessing /jobs/job/.
Reason: requirement failed: Missing id parameter
Powered by Jetty://

Thanks.

karuppayya · 2017-03-03T12:56:00Z

@Tagar to access a specific job, the id has to be part of the url .
Something like http://hostname.domain.com:8088/proxy/application_1488384993892_0001/jobs/job?id=
I am not sure why the id is not part of the URL. Did you find anything suspicious in logs? Also were you able to access the jobs from Spark Web UI?
Is it consistent?

mindcrusher11 · 2017-03-03T13:58:26Z

not sure but http://sparkkafka:4041/streaming/batch/?id=1488549398000 from history server job ids can be accessed.

…

On Fri, Mar 3, 2017 at 12:56 PM, Karuppayya ***@***.***> wrote: [image: Boxbe] <https://www.boxbe.com/overview> This message is eligible for Automatic Cleanup! ***@***.***) Add cleanup rule <https://www.boxbe.com/popup?url=https%3A%2F%2Fwww.boxbe.com%2Fcleanup%3Fkey%3DhZ2jaDHPZTj0KuKoaNm7fK9e%252FzMLHTLqTv%252FFNfodeFk%253D%26token%3DaEdD%252F%252BeLmCv0tCOGsZpb4o%252BEK5LEtxCCSSsv8PPya3JSu%252F5lN7vTnmpP2Sg9oTH7ezUWL7vqbNgezPE0xVQ39d%252BN5cY1norzDv3iDnMds%252FGzIQhRKIfX4iLHJowWwPdmejdhDctIKx2WZ8%252F%252FVC6yVQ%253D%253D&tc_serial=29148464902&tc_rand=1062463868&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> | More info <http://blog.boxbe.com/general/boxbe-automatic-cleanup?tc_serial=29148464902&tc_rand=1062463868&utm_source=stf&utm_medium=email&utm_campaign=ANNO_CLEANUP_ADD&utm_content=001> @Tagar <https://github.com/Tagar> to access a specific job, the id has to be part of the url . Something like http://hostname.domain.com:8088/proxy/application_ 1488384993892_0001/jobs/job?id= I am not sure why the id is not part of the URL. Did you find anything suspicious in logs? Also were you able to access the jobs from Spark Web UI? Is it consistent? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1663 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACVt9jetc90A_mWk4WINCAQ4V3ekg8eMks5riA3igaJpZM4K4Vff> .

Tagar · 2017-03-03T17:51:23Z

@karuppayya thank you for the follow up.

A little bit more information - the link on paragprah leads to

http://10.20.32.57:28009/jobs/job?id=123

A Spark Driver UI.
Spark Driver web server then redirects to a link like I sent in my previous post,
and it misses job id in the redirected URL, i.e.

http://host.domain.com:8088/proxy/application_1488384993892_0044/jobs/job/

To your question on consistency - yes it happens in 100% cases, I never saw this new feature works for us.
Is this Spark version dependent or something? We're running CDH Spark 2.

Tagar · 2017-03-06T19:50:12Z

According to apache/spark#5947 URL format is different in YARN and non-YARN modes? Was PR-1663 for ZEPPELIN-1692 tested on both of these modes? Not sure what else might break those links. We do use Spark on YARN. Created https://issues.apache.org/jira/browse/ZEPPELIN-2221 to move discussion there. Thank you.

A paragraph execution may result in spark job(s). Adding ability to access the spark job UI(corresponding to the job generated by the paragraph run), directly from the paragraph. Improvement * [x] Write tests ZEPPELIN-1692 Run paragraphs with spark code(scala, pyspark, sql, R). The paragraph will display a button on the top right corner, which on click will open up the corresponding job UI ![spark_jobs](https://cloud.githubusercontent.com/assets/5082742/20488337/a07fe35a-b02c-11e6-9400-db9f1c10df90.gif) * Does the licenses files need update? NA * Is there breaking changes for older versions? NA * Does this needs documentation? NA Author: Karup <karuppayya@outlook.com> Author: karuppayya <karuppayya1990@gmail.com> Author: karuppayya <karuppayyar@qubole.com> Closesapache#1663 from karuppayya/ZEPPELIN-1692 and squashes the following commits: 4253d0b [Karup] Fix bad rebase d7eb3b6 [Karup] Fix paragraph.js 8e2cd85 [Karup] tryout: fix selenium tests based on moons suggstion 732b0a4 [karuppayya] Fix test 890107d [Karup] Fix test - tryout ed4685c [Karup] Fix tooltip d27221d [Karup] Adding license header 87214a7 [Karup] Fix incorrect rebase 19513a6 [Karup] Send para runtimeinfos via websocker, but dont persist in json 09fc0e2 [Karup] Fix compilation fc44d9b [Karup] Address review comments b837c6c [karuppayya] Fix incorrect variable used 42d92ac [karuppayya] Fix test d4e54e8 [karuppayya] Address review feedbacks 1a45284 [Karup] Fix test 717eedf [Karup] Add tests , refactor 25379aa [Karup] Clear job urls when we clear output 7383c0a [Karup] Address review comments e2cd4db [karuppayya] Fix NPE in tests 3d9a573 [karuppayya] Fix NPE and some refactoring 9b3a3e2 [karuppayya] Fix checkstyle f16422f [karuppayya] Ability to view spark job urls in each paragraph (cherry picked from commit e9caebc)

Tagar · 2017-05-16T18:34:19Z

Does this feature work for anyone who is using Spark on YARN?

It seems to be broken by https://issues.apache.org/jira/browse/SPARK-20772

I've updated https://issues.apache.org/jira/browse/ZEPPELIN-2221

karuppayya commented Nov 21, 2016
View reviewed changes

karuppayya force-pushed the ZEPPELIN-1692 branch from 2fc4dcf to ad50955Compare November 22, 2016 15:58

zjffdu reviewed Nov 23, 2016
View reviewed changes

zjffdu reviewed Nov 24, 2016
View reviewed changes

AhyoungRyu reviewed Nov 24, 2016
View reviewed changes

karuppayya closed this Nov 26, 2016

karuppayya reopened this Nov 26, 2016

karuppayya force-pushed the ZEPPELIN-1692 branch from 4fcc24e to d2d633eCompare November 26, 2016 04:30

karuppayya force-pushed the ZEPPELIN-1692 branch from e8fd1e0 to 9612dd0Compare November 30, 2016 11:58

zjffdu reviewed Dec 1, 2016
View reviewed changes

karuppayya force-pushed the ZEPPELIN-1692 branch from f85bfcc to d978c48Compare December 1, 2016 16:21

karuppayyaand others added 17 commits February 2, 2017 13:42

Fix NPE in tests
e2cd4db
Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Address review comments
7383c0a
Signed-off-by: Karup <karuppayya@outlook.com>

Clear job urls when we clear output
25379aa
Signed-off-by: Karup <karuppayya@outlook.com>

Add tests , refactor
717eedf
Signed-off-by: Karup <karuppayya@outlook.com>

Fix test
1a45284
Signed-off-by: Karup <karuppayya@outlook.com>

Address review feedbacks
d4e54e8
Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Fix test
42d92ac
Signed-off-by: karuppayya <karuppayya1990@gmail.com>

Fix incorrect variable used
b837c6c
Signed-off-by: karuppayya <karuppayyar@qubole.com>

Address review comments
fc44d9b
Signed-off-by: Karup <karuppayya@outlook.com>

Fix compilation
09fc0e2
Signed-off-by: Karup <karuppayya@outlook.com>

Send para runtimeinfos via websocker, but dont persist in json
19513a6

Fix incorrect rebase
87214a7

Adding license header
d27221d

Fix tooltip
ed4685c

Fix test - tryout
890107d

Fix test
732b0a4

tryout: fix selenium tests based on moons suggstion
8e2cd85

karuppayya force-pushed the ZEPPELIN-1692 branch from 27ca32c to 8e2cd85Compare February 2, 2017 08:15

Karup added 2 commits February 3, 2017 17:58

Fix paragraph.js
d7eb3b6

Fix bad rebase
4253d0b

asfgit closed this in e9caebcFeb 5, 2017

		@@ -970,6 +1015,7 @@ public void populateSparkWebUrl(InterpreterContext ctx) {
		infos.put("url", sparkUrl);
		logger.info("Sending metainfos to Zeppelin server: {}", infos.toString());

Uh oh!

Conversation

karuppayya commented Nov 21, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

cloverhearts commented Nov 21, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karuppayya commented Nov 23, 2016

Uh oh!

1ambda commented Nov 23, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

karuppayya commented Nov 23, 2016

Uh oh!

1ambda commented Nov 23, 2016

Uh oh!

zjffduNov 23, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karuppayyaNov 26, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjffdu commented Nov 24, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AhyoungRyu commented Nov 24, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karuppayya commented Nov 25, 2016

Uh oh!

karuppayya commented Nov 26, 2016

Uh oh!

karuppayya commented Nov 27, 2016

Uh oh!

karuppayya commented Nov 30, 2016

Uh oh!

zjffduDec 1, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjffduDec 1, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

karuppayya commented Nov 21, 2016•
edited
Loading

1ambda commented Nov 23, 2016•
edited
Loading

zjffduNov 23, 2016•
edited
Loading

karuppayyaNov 26, 2016•
edited
Loading

zjffduDec 1, 2016•
edited
Loading

zjffduDec 1, 2016•
edited
Loading

zjffduDec 1, 2016•
edited
Loading