[SPARK-28220][SQL] Improve PropagateEmptyRelation to support join with false condition by wangyum · Pull Request #31857 · apache/spark

wangyum · 2021-03-17T03:23:37Z

What changes were proposed in this pull request?

Improve PropagateEmptyRelation to support join with false condition. For example:

SELECT*FROM t1 LEFT JOIN t2 ON false

Before this pr:

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- BroadcastNestedLoopJoin BuildRight, LeftOuter, false
:- FileScan parquet default.t1[a#4L]
+- BroadcastExchange IdentityBroadcastMode, [id=#40]
+- FileScan parquet default.t2[b#5L]

After this pr:

== Physical Plan ==
*(1) Project [a#4L, null AS b#5L]
+- *(1) ColumnarToRow
+- FileScan parquet default.t1[a#4L]

Why are the changes needed?

Avoid BroadcastNestedLoopJoin to improve query performance.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test.

wangyum · 2021-03-17T06:26:26Z

cc @cloud-fan

SparkQA · 2021-03-19T09:45:27Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40832/

SparkQA · 2021-03-19T09:54:59Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40832/

SparkQA · 2021-03-19T13:41:10Z

Test build #136250 has finished for PR 31857 at commit 25689da.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-03-19T13:55:04Z


+ private def pushDownJoinConditions(conditions: Seq[Expression], plan: LogicalPlan) = {
+ conditions
+ .filterNot(_.semanticEquals(TrueLiteral)) // Push down true condition is useless.


shouldn't this be optimized by BooleanSimplification already? true And cond -> cond

It is another issue. Will be fixed by another PR.

SparkQA · 2021-03-19T15:42:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40841/

SparkQA · 2021-03-19T15:51:00Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40841/

# Conflicts: # sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala

SparkQA · 2021-03-19T17:49:10Z

Test build #136259 has finished for PR 31857 at commit debb094.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-19T18:28:15Z

Test build #136263 has finished for PR 31857 at commit 3202590.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-03-19T18:50:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40845/

SparkQA · 2021-03-19T19:00:27Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40845/

wangyum · 2021-03-19T23:58:41Z

retest this please.

SparkQA · 2021-03-20T00:43:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40854/

SparkQA · 2021-03-20T00:52:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40854/

SparkQA · 2021-03-20T04:29:01Z

Test build #136273 has finished for PR 31857 at commit 3202590.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

wangyum · 2021-03-20T14:58:21Z

Merged to master.

dongjoon-hyun

+1, late LGTM.
Thank you, @wangyum and @cloud-fan .

Push down the foldable predicate to both sides of Join
9ad1833

wangyum commented Mar 17, 2021
View reviewed changes

Comment threadsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala Outdated

github-actionsBot added the SQL label Mar 17, 2021

cloud-fan reviewed Mar 17, 2021
View reviewed changes

Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala Outdated

cloud-fan reviewed Mar 17, 2021
View reviewed changes

Comment threadsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala Outdated

wangyum added 4 commits March 17, 2021 20:34

fix
7997eee

Fix
5988d28

Fix
ee5b4eb

pushDownJoinConditions
9e913e9

cloud-fan reviewed Mar 18, 2021
View reviewed changes

Comment thread...catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala

fix
25689da

cloud-fan reviewed Mar 19, 2021
View reviewed changes

fix
debb094

wangyum changed the title ~~[SPARK-28220][SQL] Push down the foldable predicate to both sides of Join~~[SPARK-28220][SQL] Improve PropagateEmptyRelation to support join with false conditionMar 19, 2021

cloud-fan approved these changes Mar 19, 2021
View reviewed changes

Merge remote-tracking branch 'upstream/master' intoSPARK-28220
3202590
# Conflicts: # sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelationSuite.scala

wangyum closed this in 908318fMar 20, 2021

wangyum deleted the SPARK-28220 branch March 20, 2021 14:58

dongjoon-hyun reviewed Mar 20, 2021
View reviewed changes

Uh oh!

Conversation

wangyum commented Mar 17, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

wangyum commented Mar 17, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

cloud-fanMar 19, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wangyumMar 19, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 19, 2021

Uh oh!

wangyum commented Mar 19, 2021

Uh oh!

SparkQA commented Mar 20, 2021

Uh oh!

SparkQA commented Mar 20, 2021

Uh oh!

SparkQA commented Mar 20, 2021

Uh oh!

wangyum commented Mar 20, 2021

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wangyum commented Mar 17, 2021•
edited
Loading

cloud-fanMar 19, 2021•
edited
Loading