[SPARK-15438] [SQL] improve explain of whole stage codegen by davies · Pull Request #13204 · apache/spark

davies · 2016-05-19T22:07:14Z

What changes were proposed in this pull request?

Currently, the explain of a query with whole-stage codegen looks like this

>>> df = sqlCtx.range(1000);df2 = sqlCtx.range(1000);df.join(pyspark.sql.functions.broadcast(df2), 'id').explain()
== Physical Plan ==
WholeStageCodegen
: +- Project [id#1L]
: +- BroadcastHashJoin [id#1L], [id#4L], Inner, BuildRight, None
: :- Range 0, 1, 4, 1000, [id#1L]
: +- INPUT
+- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint]))
+- WholeStageCodegen
: +- Range 0, 1, 4, 1000, [id#4L]

The problem is that the plan looks much different than logical plan, make us hard to understand the plan (especially when the logical plan is not showed together).

This PR will change it to:

>>> df = sqlCtx.range(1000);df2 = sqlCtx.range(1000);df.join(pyspark.sql.functions.broadcast(df2), 'id').explain()
== Physical Plan ==
*Project [id#0L]
+- *BroadcastHashJoin [id#0L], [id#3L], Inner, BuildRight, None
:- *Range 0, 1, 4, 1000, [id#0L]
+- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]))
+- *Range 0, 1, 4, 1000, [id#3L]

The *before the plan means that it's part of whole-stage codegen, it's easy to understand.

How was this patch tested?

Manually ran some queries and check the explain.

davies · 2016-05-19T22:07:25Z

cc @marmbrus @rxin

davies · 2016-05-19T22:12:17Z

Another proposal is to have a special prefix for the operators that are part of whole stage codegen.

>>> df = sqlCtx.range(1000);df2 = sqlCtx.range(1000);df.join(pyspark.sql.functions.broadcast(df2), 'id').explain()
== Physical Plan ==
*Project [id#0L]
+- *BroadcastHashJoin [id#0L], [id#3L], Inner, BuildRight, None
:- *Range 0, 1, 4, 1000, [id#0L]
+- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false]))
+- *Range 0, 1, 4, 1000, [id#3L]

SparkQA · 2016-05-19T23:33:51Z

Test build #58910 has finished for PR 13204 at commit db5ad62.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yucai · 2016-05-20T02:56:42Z

Hi Davis,

In your comments "Hopefully we do need to pay much attention on this", do you mean "we do not need to pay much attention..."?

davies · 2016-05-20T02:59:22Z

@yucai Yes, corrected, thanks!

kiszk · 2016-05-20T03:38:33Z

- override def simpleString: String = "INPUT"
-
- override def treeChildren: Seq[SparkPlan] = Nil
+ override def simpleString: String = "CodegenInput"


Would it be better to synchronize name "CodegenInput" in simpleString with the class name InputAdapter?

yhuai · 2016-05-20T06:10:12Z

I like 3. We can tell what operators are in a single WholeStageCodeGen operator and we can also know what are input operators of a WholeStageCodeGen.

davies · 2016-05-20T06:30:38Z

@yhuai @marmbrus Updated to #3.

SparkQA · 2016-05-20T06:40:22Z

Test build #58949 has finished for PR 13204 at commit bc2aed4.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-20T08:22:26Z

Test build #3003 has finished for PR 13204 at commit bc2aed4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-20T19:23:34Z

Test build #3005 has finished for PR 13204 at commit bc2aed4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-05-20T20:04:21Z

New explain lgtm. I didn't look at the code though.

yhuai · 2016-05-20T20:08:40Z

LGTM

rxin · 2016-05-20T20:21:46Z

Merging in master/2.0.

## What changes were proposed in this pull request? Currently, the explain of a query with whole-stage codegen looks like this ``` >>> df = sqlCtx.range(1000);df2 = sqlCtx.range(1000);df.join(pyspark.sql.functions.broadcast(df2), 'id').explain() == Physical Plan == WholeStageCodegen : +- Project [id#1L] : +- BroadcastHashJoin [id#1L], [id#4L], Inner, BuildRight, None : :- Range 0, 1, 4, 1000, [id#1L] : +- INPUT +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint])) +- WholeStageCodegen : +- Range 0, 1, 4, 1000, [id#4L] ``` The problem is that the plan looks much different than logical plan, make us hard to understand the plan (especially when the logical plan is not showed together). This PR will change it to: ``` >>> df = sqlCtx.range(1000);df2 = sqlCtx.range(1000);df.join(pyspark.sql.functions.broadcast(df2), 'id').explain() == Physical Plan == *Project [id#0L] +- *BroadcastHashJoin [id#0L], [id#3L], Inner, BuildRight, None :- *Range 0, 1, 4, 1000, [id#0L] +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, false])) +- *Range 0, 1, 4, 1000, [id#3L] ``` The `*`before the plan means that it's part of whole-stage codegen, it's easy to understand. ## How was this patch tested? Manually ran some queries and check the explain. Author: Davies Liu <davies@databricks.com> Closes#13204 from davies/explain_codegen. (cherry picked from commit 0e70fd6) Signed-off-by: Reynold Xin <rxin@databricks.com>

yucai · 2016-05-30T07:39:53Z

Dear @davies,

Sometimes, it could be hard for me to recognize the boundary of whole stage codegen,
for example, in below SMJ, there are 3 separate whole stage codegen actually, can we read this info from the physical plan output?

 +- Exchange hashpartitioning(sourceIP#0, 200), None
+- *TungstenAggregate(key=[sourceIP#0], ......
+- *Project [pagerank#30,sourceIP#0,adRevenue#3] +- *SortMergeJoin [pageURL#29], [destURL#1], Inner, None
:- *Sort [pageURL#29 ASC], false, 0
: +- Exchange hashpartitioning(pageURL#29, 200), None
: +- ......
+- *Sort [destURL#1 ASC], false, 0
+- Exchange hashpartitioning(destURL#1, 200), None
+- ......

It looks easier in this way:

 +- Exchange hashpartitioning(sourceIP#0, 200), None
+- WholeStageCodegen
+- TungstenAggregate(key=[sourceIP#0], ......
+- Project [pagerank#30,sourceIP#0,adRevenue#3]
+- SortMergeJoin [pageURL#29], [destURL#1], Inner, None
:- InputAdapter
: +- WholeStageCodegen
: +- Sort [pageURL#29 ASC], false, 0
: +- InputAdapter
: +- Exchange hashpartitioning(pageURL#29, 200), None
: +- ......
+- InputAdapter
+- WholeStageCodegen
+- Sort [destURL#1 ASC], false, 0
+- InputAdapter
+- Exchange hashpartitioning(destURL#1, 200), None
+- ......

How do you think?

davies · 2016-05-31T17:52:50Z

@yucai It's true that the case you posted is a little confusing, you can see the expected boundary on Spark UI. The other one is too verbose the see the plan (the logical parts), may not worth that.

explain of whole stage codegen
db5ad62

kiszk reviewed May 20, 2016
View reviewed changes

change the explain
bc2aed4

davies changed the title ~~explain of whole stage codegen~~[SPARK-15438] [SQL] improve explain of whole stage codegenMay 20, 2016

asfgit closed this in 0e70fd6May 20, 2016

Uh oh!

Conversation

davies commented May 19, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

davies commented May 19, 2016

Uh oh!

davies commented May 19, 2016

Uh oh!

SparkQA commented May 19, 2016

Uh oh!

yucai commented May 20, 2016

Uh oh!

davies commented May 20, 2016

Uh oh!

kiszkMay 20, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yhuai commented May 20, 2016

Uh oh!

davies commented May 20, 2016

Uh oh!

SparkQA commented May 20, 2016

Uh oh!

SparkQA commented May 20, 2016

Uh oh!

SparkQA commented May 20, 2016

Uh oh!

rxin commented May 20, 2016

Uh oh!

yhuai commented May 20, 2016

Uh oh!

rxin commented May 20, 2016

Uh oh!

yucai commented May 30, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davies commented May 31, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

davies commented May 19, 2016•
edited
Loading

kiszkMay 20, 2016•
edited
Loading

yucai commented May 30, 2016•
edited
Loading