[SPARK-17171][WEB UI] DAG will list all partitions in the graph by cenyuhai · Pull Request #14737 · apache/spark

cenyuhai · 2016-08-21T04:26:48Z

What changes were proposed in this pull request?

DAG will list all partitions in the graph, it is too slow and hard to see all graph.
Always we don't want to see all partitions，we just want to see the relations of DAG graph.
So I just show 2 root nodes for Rdds.

Before this PR, the DAG graph looks like dag1.png, dag3.png, after this PR, the DAG graph looks like dag2.png,dag4.png

merge latest code to my fork

SparkQA · 2016-08-21T06:33:14Z

Test build #64158 has finished for PR 14737 at commit 595453f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-08-21T10:15:53Z

+ if (parentIds.size == 0) {
+ rootNodeCount < retainedNodes
+ } else {
+ if (ids.size > 0) {


The whole if-else is just ids.isEmpty || parentIds.exists(id => ids.contains(id) || !dropRDDIds.contains(id)) ?

yes, you are right...

SparkQA · 2016-08-21T15:37:46Z

Test build #64167 has finished for PR 14737 at commit 770b3de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cenyuhai · 2016-08-21T16:32:28Z

@srowen

markhamstra · 2016-08-21T20:24:46Z

 val DEFAULT_POOL_NAME = "default"
 val DEFAULT_RETAINED_STAGES = 1000
 val DEFAULT_RETAINED_JOBS = 1000
+ val DEFAULT_RETAINED_NODES = 2


NODES, both here and in spark.ui.retainedNodes if far too ambiguous and non-specific for this configuration value -- "node" is already overloaded too many times in the existing Spark code and documentation; we don't need or want to add another overload.
Additionally, the default behavior should be the same as current behavior, since the change in behavior would be unexpected and it is far from clear to me that the overwhelming majority of users would prefer the proposed new behavior.

SparkQA · 2016-08-22T05:21:39Z

Test build #64173 has finished for PR 14737 at commit 4019842.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cenyuhai · 2016-08-25T09:42:18Z

@srowen please review the latest codes, thank you.

srowen · 2016-08-25T09:44:22Z

Pardon the dumb question, but your before and after pictures seem like different graphs. Is that really showing the before and after of one display?

cenyuhai · 2016-08-25T11:36:41Z

I am very sorry about, the first picture is for stage, the second picture is for job, but it is the same job "select count(1) from partitionedTables ". I upload new pictures. dag1.png vs dag4.png, dag3.png vs dag2.png.

srowen · 2016-08-25T11:45:15Z

I can see the value in this, but it also removes any indication that there are more elements to the graph that are not displayed. I wonder if there is any easy way to do that or do I misunderstand? Also why 2, just arbitrary?

cenyuhai · 2016-08-25T12:00:13Z

@srowen Why I set this value 2, because a "JOIN" action needs 2 elements.Users want to know the relations from DAG graphs, if there are too manys elements, it is meesy. I have changed the codes as @markhamstra said, it will not remove elements by default. Users can set this value as they like. But I recommend that 2 is better.

srowen · 2016-08-25T12:47:39Z

OK, another config eh. If there's no default change in behavior, and it's undocumented and it doesn't introduce much complexity, seems OK

cenyuhai · 2016-09-09T04:03:27Z

@srowen If it is ok，can you merge this pr to master？thank you.

srowen · 2016-09-09T12:33:36Z

+ dropRDDIds.add(rdd.id)
+ }
+
+ if (rdd.parentIds.size == 0) {


Minor stuff like:
if (rdd.parnetIds.isEmpty) { rootNoteCount += 1 }

SparkQA · 2016-09-09T16:13:59Z

Test build #65152 has finished for PR 14737 at commit 40def0f.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-10T05:51:14Z

Test build #65190 has finished for PR 14737 at commit 3c72dde.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-09-10T09:02:53Z

+ stage.rddInfos.sortBy(_.id).foreach { rdd =>
+ var isAllowed = true
+ val parentIds = rdd.parentIds
+ if (parentIds.isEmpty) {


Now we're just down to style stuff but I how about...
val isAllowed = if (parentId.isEmpty) { rootNodeCount += 1 rootNodeCount <= retainedNodes } else { parentIds.exists(...) }

SparkQA · 2016-09-10T13:58:57Z

Test build #65209 has finished for PR 14737 at commit 5163a51.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-09-12T10:52:11Z

I looked one more time and the only thing that crossed my mind is whether spark.ui.dagGraph.retainedRootRDDs is the rightest naming convention, but, it's a hidden internal property at the moment, and I don't have a better idea, so let's leave it.

srowen · 2016-09-12T10:53:04Z

Merged to master

cenyuhai · 2016-09-12T10:57:20Z

thank you!

## What changes were proposed in this pull request? DAG will list all partitions in the graph, it is too slow and hard to see all graph. Always we don't want to see all partitions，we just want to see the relations of DAG graph. So I just show 2 root nodes for Rdds. Before this PR, the DAG graph looks like [dag1.png](https://issues.apache.org/jira/secure/attachment/12824702/dag1.png), [dag3.png](https://issues.apache.org/jira/secure/attachment/12825456/dag3.png), after this PR, the DAG graph looks like [dag2.png](https://issues.apache.org/jira/secure/attachment/12824703/dag2.png),[dag4.png](https://issues.apache.org/jira/secure/attachment/12825457/dag4.png) Author: cenyuhai <cenyuhai@didichuxing.com> Author: 岑玉海 <261810726@qq.com> Closesapache#14737 from cenyuhai/SPARK-17171.

…pache#14737

cenyuhaiand others added 3 commits August 20, 2016 23:44

Just show 2 root partitions for a stage
7991d76

Merge pull request#8from apache/master
869eaaf
merge latest code to my fork

Merge remote-tracking branch 'remotes/origin/master' into SPARK-17171
595453f

srowen reviewed Aug 21, 2016
View reviewed changes

Remove unnecessary code
770b3de

cenyuhai changed the title ~~[Spark-17171][WEB UI] DAG will list all partitions in the graph~~[SPARK-17171][WEB UI] DAG will list all partitions in the graphAug 21, 2016

markhamstra reviewed Aug 21, 2016
View reviewed changes

change configuration name and default value
4019842

sadikovi mentioned this pull request Aug 22, 2016
RDD Graph (json-relay part) hammerlab/spark-json-relay#8
Closed

srowen reviewed Sep 9, 2016
View reviewed changes

move isAllow funtion inline
40def0f

Merge remote-tracking branch 'remotes/apache/master' into SPARK-17171
3c72dde

srowen reviewed Sep 10, 2016
View reviewed changes

simplifying codes
5163a51

asfgit closed this in cc87280Sep 12, 2016

zzcclp added a commit to zzcclp/spark that referenced this pull request Sep 20, 2016
[EXT][SPARK-17171][WEB UI] DAG will list all partitions in the grapha…
fd5190e
…pache#14737

Uh oh!

Conversation

cenyuhai commented Aug 21, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Uh oh!

SparkQA commented Aug 21, 2016

Uh oh!

srowenAug 21, 2016

Choose a reason for hiding this comment

Uh oh!

cenyuhaiAug 21, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 21, 2016

Uh oh!

cenyuhai commented Aug 21, 2016

Uh oh!

markhamstraAug 21, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 22, 2016

Uh oh!

cenyuhai commented Aug 25, 2016

Uh oh!

srowen commented Aug 25, 2016

Uh oh!

cenyuhai commented Aug 25, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Aug 25, 2016

Uh oh!

cenyuhai commented Aug 25, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Aug 25, 2016

Uh oh!

cenyuhai commented Sep 9, 2016

Uh oh!

srowenSep 9, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 9, 2016

Uh oh!

SparkQA commented Sep 10, 2016

Uh oh!

srowenSep 10, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 10, 2016

Uh oh!

srowen commented Sep 12, 2016

Uh oh!

srowen commented Sep 12, 2016

Uh oh!

cenyuhai commented Sep 12, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cenyuhai commented Aug 21, 2016•
edited
Loading

cenyuhai commented Aug 25, 2016•
edited
Loading

cenyuhai commented Aug 25, 2016•
edited
Loading