[SPARK-34533][SQL] Eliminate LEFT ANTI join to empty relation in AQE by c21 · Pull Request #31641 · apache/spark

c21 · 2021-02-25T05:14:19Z

What changes were proposed in this pull request?

I discovered from review discussion - #31630 (comment) , that we can eliminate LEFT ANTI join (with no join condition) to empty relation, if the right side is known to be non-empty. So with AQE, this is doable similar to #29484 .

Why are the changes needed?

This can help eliminate the join operator during logical plan optimization.
Before this PR, left side physical plan execute() will be called, so if left side is complicated (e.g. contain broadcast exchange operator), then some computation would happen. However after this PR, the join operator will be removed during logical plan, and nothing will be computed from left side. Potentially it can save resource for these kinds of query.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added unit tests for positive and negative queries in AdaptiveQueryExecSuite.scala.

c21 · 2021-02-25T05:14:47Z

cc @cloud-fan and @maropu to take a look if you have time, thanks.

SparkQA · 2021-02-25T05:52:43Z

Test build #135454 has finished for PR 31641 at commit ead65c8.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-02-25T05:57:32Z

Test build #135456 has finished for PR 31641 at commit e9e7b16.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-02-25T06:21:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40036/

SparkQA · 2021-02-25T06:31:17Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40034/

SparkQA · 2021-02-25T06:48:57Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40036/

SparkQA · 2021-02-25T07:07:48Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40034/

SparkQA · 2021-02-25T07:31:59Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40037/

SparkQA · 2021-02-25T08:01:07Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40037/

SparkQA · 2021-02-25T11:16:44Z

Test build #135457 has finished for PR 31641 at commit 255ee07.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-02-25T11:57:41Z

+
+ case j @ Join(_, _, LeftAnti, None, _) =>
+ val isNonEmptyBroadcastedRightSide = j.right match {
+ case LogicalQueryStage(_, stage: BroadcastQueryStageExec)


nit:
case LogicalQueryStage(_, stage: QueryStageExec) => stage.getRuntimeStatistics.rowCount.exists(_ > 0)

@cloud-fan - thanks, use row count is better, updated.

cloud-fan · 2021-02-25T11:58:51Z

 * at the first place.
+ *
+ * 3. Join is left anti join without condition, and broadcasted join right side is not empty.
+ * This applies to broadcast nested loop join only.


Why do we care about the physical plan? I think it's a logical optimization that left anti join without condition can be turned into empty relation if right side is non-empty.

@cloud-fan - I agree. Updated to remove this restriction. I was not thinking towards to use row count stats at the first place, so was checking the size of Array[InternalRow] for BroadcastNestedLoopJoinExec, so was having this restriction.

SparkQA · 2021-02-25T21:09:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40059/

SparkQA · 2021-02-25T21:40:35Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40059/

SparkQA · 2021-02-25T22:30:30Z

Test build #135479 has finished for PR 31641 at commit 8e5f35d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

c21 · 2021-02-25T22:33:54Z

retest this please

SparkQA · 2021-02-25T23:42:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40061/

SparkQA · 2021-02-26T00:16:46Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40061/

cloud-fan · 2021-02-26T03:14:50Z

+ ("SELECT /*+ broadcast(testData) */ * FROM testData LEFT ANTI JOIN testData3", false)
+ ).foreach { case (query, isEliminated) =>
+ val (plan, adaptivePlan) = runAdaptiveAndVerifyResult(query)
+ val bnlj = findTopLevelBroadcastNestedLoopJoin(plan)


Do we need to create findTopLevelBroadcastNestedLoopJoin? We just want to prove that, the non-AQE plan has join and the AQE one has no join. findTopLevelBaseJoin should be good enough here.

@cloud-fan - agree, updated.

SparkQA · 2021-02-26T03:27:19Z

Test build #135481 has finished for PR 31641 at commit 8e5f35d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-02-26T05:20:35Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40070/

SparkQA · 2021-02-26T05:25:27Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40070/

SparkQA · 2021-02-26T09:16:09Z

Test build #135489 has finished for PR 31641 at commit 934306e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-02-26T11:46:26Z

thanks, merging to master!

maropu · 2021-02-26T12:15:24Z

Thanks for the update, @c21 !

c21 · 2021-02-26T17:28:21Z

Thank you @cloud-fan and @maropu for review!

Eliminate LEFT ANTI join to empty relation in AQE
ead65c8

github-actionsBot added the SQL label Feb 25, 2021

Try to fix scala style
e9e7b16

Try to fix scala style after retry on laptop
255ee07

cloud-fan reviewed Feb 25, 2021
View reviewed changes

Address comment to use row count stats
8e5f35d

cloud-fan reviewed Feb 26, 2021
View reviewed changes

simplify test
934306e

cloud-fan approved these changes Feb 26, 2021
View reviewed changes

cloud-fan closed this in 7d5021fFeb 26, 2021

c21 deleted the left-anti-aqe branch February 26, 2021 17:28

Uh oh!

Conversation

c21 commented Feb 25, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

c21 commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

cloud-fanFeb 25, 2021

Choose a reason for hiding this comment

Uh oh!

c21Feb 25, 2021

Choose a reason for hiding this comment

Uh oh!

cloud-fanFeb 25, 2021

Choose a reason for hiding this comment

Uh oh!

c21Feb 25, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

c21 commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 25, 2021

Uh oh!

SparkQA commented Feb 26, 2021

Uh oh!

cloud-fanFeb 26, 2021

Choose a reason for hiding this comment

Uh oh!

c21Feb 26, 2021

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 26, 2021

Uh oh!

SparkQA commented Feb 26, 2021

Uh oh!

SparkQA commented Feb 26, 2021

Uh oh!

SparkQA commented Feb 26, 2021

Uh oh!

cloud-fan commented Feb 26, 2021

Uh oh!

maropu commented Feb 26, 2021

Uh oh!

c21 commented Feb 26, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants