[SPARK-13415] [SQL] Visualize subquery in SQL web UI by davies · Pull Request #11417 · apache/spark

davies · 2016-02-28T07:16:21Z

What changes were proposed in this pull request?

This PR support visualization for subquery in SQL web UI, also improve the explain of subquery, especially when it's used together with whole stage codegen.

For example:

>>>sqlContext.range(100).registerTempTable("range")
>>>sqlContext.sql("select id / (select sum(id) from range) from range where id > (select id from range limit 1)").explain(True)
==ParsedLogicalPlan=='Project [unresolvedalias(('id/subquery#9), None)]
: +- 'SubqueryAliassubquery#9
: +-'Project [unresolvedalias('sum('id), None)]
: +- 'UnresolvedRelation`range`, None+-'Filter ('id>subquery#8)
: +- 'SubqueryAliassubquery#8
: +- 'GlobalLimit1
: +- 'LocalLimit1
: +-'Project [unresolvedalias('id, None)]
: +- 'UnresolvedRelation`range`, None+- 'UnresolvedRelation`range`, None==AnalyzedLogicalPlan==
(id/scalarsubquery()): doubleProject [(cast(id#0L as double) / cast(subquery#9 as double)) AS (id / scalarsubquery())#11]
: +-SubqueryAliassubquery#9
: +-Aggregate [(sum(id#0L),mode=Complete,isDistinct=false) AS sum(id)#10L]
: +-SubqueryAliasrange
: +-Range0, 100, 1, 4, [id#0L]+-Filter (id#0L > subquery#8)
: +-SubqueryAliassubquery#8
: +-GlobalLimit1
: +-LocalLimit1
: +-Project [id#0L]
: +-SubqueryAliasrange
: +-Range0, 100, 1, 4, [id#0L]+-SubqueryAliasrange+-Range0, 100, 1, 4, [id#0L]==OptimizedLogicalPlan==Project [(cast(id#0L as double) / cast(subquery#9 as double)) AS (id / scalarsubquery())#11]
: +-SubqueryAliassubquery#9
: +-Aggregate [(sum(id#0L),mode=Complete,isDistinct=false) AS sum(id)#10L]
: +-Range0, 100, 1, 4, [id#0L]+-Filter (id#0L > subquery#8)
: +-SubqueryAliassubquery#8
: +-GlobalLimit1
: +-LocalLimit1
: +-Project [id#0L]
: +-Range0, 100, 1, 4, [id#0L]+-Range0, 100, 1, 4, [id#0L]==PhysicalPlan==WholeStageCodegen
: +-Project [(cast(id#0L as double) / cast(subquery#9 as double)) AS (id / scalarsubquery())#11]
: : +-Subquerysubquery#9
: : +-WholeStageCodegen
: : : +-TungstenAggregate(key=[], functions=[(sum(id#0L),mode=Final,isDistinct=false)], output=[sum(id)#10L])
: : : +-INPUT
: : +-ExchangeSinglePartition, None
: : +-WholeStageCodegen
: : : +-TungstenAggregate(key=[], functions=[(sum(id#0L),mode=Partial,isDistinct=false)], output=[sum#14L])
: : : +-Range0, 1, 4, 100, [id#0L]
: +-Filter (id#0L > subquery#8)
: : +-Subquerysubquery#8
: : +-CollectLimit1
: : +-WholeStageCodegen
: : : +-Project [id#0L]
: : : +-Range0, 1, 4, 100, [id#0L]
: +-Range0, 1, 4, 100, [id#0L]

The web UI looks like:

This PR also change the tree structure of WholeStageCodegen to make it consistent than others. Before this change, Both WholeStageCodegen and InputAdapter hold a references to the same plans, those could be updated without notify another, causing problems, this is discovered by #11403 .

How was this patch tested?

Existing tests, also manual tests with the example query, check the explain and web UI.

rxin · 2016-02-28T07:21:50Z

Thanks for adding the screenshot and the explain output!

What's the "(id / scalarsubquery()): double" in Analyzed Logical Plan ?

davies · 2016-02-28T07:53:53Z

@rxin I think that the schema of the query. The sql string of ScalarSubquery is scalarsubquery

s"$prettyName($childrenSQL)"

SparkQA · 2016-02-28T10:10:25Z

Test build #52140 has finished for PR 11417 at commit 24757f1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class InputAdapter(child: SparkPlan) extends UnaryNode with CodegenSupport
- case class WholeStageCodegen(child: CodegenSupport) extends UnaryNode with CodegenSupport

SparkQA · 2016-02-28T10:10:58Z

Test build #52141 has finished for PR 11417 at commit bc2c66b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class WholeStageCodegen(child: SparkPlan) extends UnaryNode with CodegenSupport

SparkQA · 2016-02-28T19:22:10Z

Test build #2589 has finished for PR 11417 at commit bc2c66b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class WholeStageCodegen(child: SparkPlan) extends UnaryNode with CodegenSupport

nongli · 2016-02-29T21:24:59Z

 */
 protected def treeChildren: Seq[BaseType] = children

+ /**


All the nodes that are?
How is this different than children?

davies · 2016-03-01T01:28:04Z

@yhuai @nongli Had update the doc string of innerChildren and treeChildren to include a example.

SparkQA · 2016-03-01T01:41:44Z

Test build #52208 has finished for PR 11417 at commit 2a27cd8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-01T01:46:59Z

Test build #52209 has finished for PR 11417 at commit cff0871.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-01T20:53:02Z

Test build #52252 has finished for PR 11417 at commit 3293994.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-02T02:05:50Z

Test build #52268 has finished for PR 11417 at commit e69262a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-02T06:35:21Z

Test build #2601 has finished for PR 11417 at commit e69262a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2016-03-04T01:35:48Z

Thanks. Looks good. I am merging this to master. Let's think about how to improve the name and doc of treeChildren and innerChildren.

## What changes were proposed in this pull request? This PR support visualization for subquery in SQL web UI, also improve the explain of subquery, especially when it's used together with whole stage codegen. For example: ```python >>> sqlContext.range(100).registerTempTable("range") >>> sqlContext.sql("select id / (select sum(id) from range) from range where id > (select id from range limit 1)").explain(True) == Parsed Logical Plan == 'Project [unresolvedalias(('id / subquery#9), None)] : +- 'SubqueryAlias subquery#9 : +- 'Project [unresolvedalias('sum('id), None)] : +- 'UnresolvedRelation `range`, None +- 'Filter ('id > subquery#8) : +- 'SubqueryAlias subquery#8 : +- 'GlobalLimit 1 : +- 'LocalLimit 1 : +- 'Project [unresolvedalias('id, None)] : +- 'UnresolvedRelation `range`, None +- 'UnresolvedRelation `range`, None == Analyzed Logical Plan == (id / scalarsubquery()): double Project [(cast(id#0L as double) / cast(subquery#9 as double)) AS (id / scalarsubquery())apache#11] : +- SubqueryAlias subquery#9 : +- Aggregate [(sum(id#0L),mode=Complete,isDistinct=false) AS sum(id)#10L] : +- SubqueryAlias range : +- Range 0, 100, 1, 4, [id#0L] +- Filter (id#0L > subquery#8) : +- SubqueryAlias subquery#8 : +- GlobalLimit 1 : +- LocalLimit 1 : +- Project [id#0L] : +- SubqueryAlias range : +- Range 0, 100, 1, 4, [id#0L] +- SubqueryAlias range +- Range 0, 100, 1, 4, [id#0L] == Optimized Logical Plan == Project [(cast(id#0L as double) / cast(subquery#9 as double)) AS (id / scalarsubquery())apache#11] : +- SubqueryAlias subquery#9 : +- Aggregate [(sum(id#0L),mode=Complete,isDistinct=false) AS sum(id)#10L] : +- Range 0, 100, 1, 4, [id#0L] +- Filter (id#0L > subquery#8) : +- SubqueryAlias subquery#8 : +- GlobalLimit 1 : +- LocalLimit 1 : +- Project [id#0L] : +- Range 0, 100, 1, 4, [id#0L] +- Range 0, 100, 1, 4, [id#0L] == Physical Plan == WholeStageCodegen : +- Project [(cast(id#0L as double) / cast(subquery#9 as double)) AS (id / scalarsubquery())apache#11] : : +- Subquery subquery#9 : : +- WholeStageCodegen : : : +- TungstenAggregate(key=[], functions=[(sum(id#0L),mode=Final,isDistinct=false)], output=[sum(id)#10L]) : : : +- INPUT : : +- Exchange SinglePartition, None : : +- WholeStageCodegen : : : +- TungstenAggregate(key=[], functions=[(sum(id#0L),mode=Partial,isDistinct=false)], output=[sum#14L]) : : : +- Range 0, 1, 4, 100, [id#0L] : +- Filter (id#0L > subquery#8) : : +- Subquery subquery#8 : : +- CollectLimit 1 : : +- WholeStageCodegen : : : +- Project [id#0L] : : : +- Range 0, 1, 4, 100, [id#0L] : +- Range 0, 1, 4, 100, [id#0L] ``` The web UI looks like: ![subquery](https://cloud.githubusercontent.com/assets/40902/13377963/932bcbae-dda7-11e5-82f7-03c9be85d77c.png) This PR also change the tree structure of WholeStageCodegen to make it consistent than others. Before this change, Both WholeStageCodegen and InputAdapter hold a references to the same plans, those could be updated without notify another, causing problems, this is discovered by apache#11403 . ## How was this patch tested? Existing tests, also manual tests with the example query, check the explain and web UI. Author: Davies Liu <davies@databricks.com> Closesapache#11417 from davies/viz_subquery.

Visualize subquery in SQL web UI
24757f1

fix tests
bc2c66b

Merge branch 'master' of github.com:apache/spark into viz_subquery
faba044

nongli reviewed Feb 29, 2016
View reviewed changes

address comments
cff0871

davies force-pushed the viz_subquery branch from 2a27cd8 to cff0871Compare March 1, 2016 01:27

Davies Liu added 2 commits March 1, 2016 11:05

Merge branch 'master' of github.com:apache/spark into viz_subquery
43dba95

fix conflict
3293994

update doc string
e69262a

asfgit closed this in b373a88Mar 4, 2016

Navigation Menu

Uh oh!

[SPARK-13415] [SQL] Visualize subquery in SQL web UI - #11417

[SPARK-13415] [SQL] Visualize subquery in SQL web UI#11417
davies wants to merge 7 commits into
apache:masterfrom
davies:viz_subquery

davies commented Feb 28, 2016

Uh oh!

rxin commented Feb 28, 2016

Uh oh!

davies commented Feb 28, 2016

Uh oh!

SparkQA commented Feb 28, 2016

Uh oh!

SparkQA commented Feb 28, 2016

Uh oh!

SparkQA commented Feb 28, 2016

Uh oh!

nongliFeb 29, 2016

Uh oh!

davies commented Mar 1, 2016

Uh oh!

SparkQA commented Mar 1, 2016

Uh oh!

SparkQA commented Mar 1, 2016

Uh oh!

SparkQA commented Mar 1, 2016

Uh oh!

SparkQA commented Mar 2, 2016

Uh oh!

SparkQA commented Mar 2, 2016

Uh oh!

yhuai commented Mar 4, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

davies commented Feb 28, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

rxin commented Feb 28, 2016

Uh oh!

davies commented Feb 28, 2016

Uh oh!

SparkQA commented Feb 28, 2016

Uh oh!

SparkQA commented Feb 28, 2016

Uh oh!

SparkQA commented Feb 28, 2016

Uh oh!

nongliFeb 29, 2016

Choose a reason for hiding this comment

Uh oh!

davies commented Mar 1, 2016

Uh oh!

SparkQA commented Mar 1, 2016

Uh oh!

SparkQA commented Mar 1, 2016

Uh oh!

SparkQA commented Mar 1, 2016

Uh oh!

SparkQA commented Mar 2, 2016

Uh oh!

SparkQA commented Mar 2, 2016

Uh oh!

yhuai commented Mar 4, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants