[SPARK-33472][SQL] Adjust RemoveRedundantSorts rule order by allisonwang-db · Pull Request #30373 · apache/spark

allisonwang-db · 2020-11-13T23:58:18Z

What changes were proposed in this pull request?

This PR switched the order for the rule RemoveRedundantSorts and EnsureRequirements so that EnsureRequirements will be invoked before RemoveRedundantSorts to avoid IllegalArgumentException when instantiating PartitioningCollection.

Why are the changes needed?

RemoveRedundantSorts rule uses SparkPlan's outputPartitioning to check whether a sort node is redundant. Currently, it is added before EnsureRequirements. Since PartitioningCollection requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying EnsureRequirements, the rule can fail with the following exception:

IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit test

SparkQA · 2020-11-14T00:44:40Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35688/

SparkQA · 2020-11-14T01:07:47Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35688/

SparkQA · 2020-11-14T04:23:05Z

Test build #131085 has finished for PR 30373 at commit e6b0bb3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-11-14T13:43:57Z

 RemoveRedundantProjects,
- RemoveRedundantSorts,
 EnsureRequirements,
+ RemoveRedundantSorts,


Could you leave some comments here about why we need to put this rule after EnsureRequirements?

+1 for @maropu 's comment.

maropu · 2020-11-14T13:45:29Z

 }
 }
+
+ test("shuffled join with different left and right side partition numbers") {


nit: could you add the prefix: SPARK-33183:

maropu · 2020-11-14T13:47:47Z

+ (0 to 100).toDF("key").createOrReplaceTempView("t2")
+
+ // left side partitioning: RangePartitioning(key ASC, 2)
+ // right side partitioning: UnknownPartitioning(0)


Could you add assert to check if the query below has the output partitions above?

dongjoon-hyun

@allisonwang-db . Please split this PR.

Please create a new JIRA for RemoveRedundantSorts. This looks worth.
Version update PR can be a follow-up.

SparkQA · 2020-11-18T02:34:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35844/

SparkQA · 2020-11-18T02:59:07Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35844/

dongjoon-hyun · 2020-11-18T03:50:58Z

cc @cloud-fan

SparkQA · 2020-11-18T04:48:11Z

Test build #131240 has finished for PR 30373 at commit fa6050a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-11-18T05:55:34Z

retest this please

SparkQA · 2020-11-18T07:24:09Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35859/

SparkQA · 2020-11-18T07:48:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35859/

SparkQA · 2020-11-18T08:05:01Z

Test build #131256 has finished for PR 30373 at commit fa6050a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-11-18T13:33:47Z

retest this please

SparkQA · 2020-11-18T14:19:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35887/

SparkQA · 2020-11-18T14:49:00Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35887/

SparkQA · 2020-11-18T16:41:16Z

Test build #131282 has finished for PR 30373 at commit fa6050a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-19T05:39:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35923/

SparkQA · 2020-11-19T06:11:59Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35923/

SparkQA · 2020-11-19T08:05:02Z

Test build #131319 has finished for PR 30373 at commit 4e684df.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2020-11-19T13:30:02Z

thanks, merging to master!

cloud-fan · 2020-11-19T13:30:26Z

@allisonwang-db can you create backport PRs for 2.4 and 3.0?

This PR switched the order for the rule `RemoveRedundantSorts` and `EnsureRequirements` so that `EnsureRequirements` will be invoked before `RemoveRedundantSorts` to avoid IllegalArgumentException when instantiating PartitioningCollection. `RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to check whether a sort node is redundant. Currently, it is added before `EnsureRequirements`. Since `PartitioningCollection` requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying `EnsureRequirements`, the rule can fail with the following exception: ``` IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. ``` No Unit test Closesapache#30373 from allisonwang-db/sort-follow-up. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit a03c540) Signed-off-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com>

dongjoon-hyun · 2020-11-20T17:26:40Z

Thank you, @allisonwang-db , @maropu , @cloud-fan !

Backport #30373 for branch-3.0. ### What changes were proposed in this pull request? This PR switched the order for the rule `RemoveRedundantSorts` and `EnsureRequirements` so that `EnsureRequirements` will be invoked before `RemoveRedundantSorts` to avoid IllegalArgumentException when instantiating PartitioningCollection. ### Why are the changes needed? `RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to check whether a sort node is redundant. Currently, it is added before `EnsureRequirements`. Since `PartitioningCollection` requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying `EnsureRequirements`, the rule can fail with the following exception: ``` IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test Closes#30438 from allisonwang-db/spark-33472-3.0. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Backport #30373 for branch-2.4. ### What changes were proposed in this pull request? This PR switched the order for the rule `RemoveRedundantSorts` and `EnsureRequirements` so that `EnsureRequirements` will be invoked before `RemoveRedundantSorts` to avoid IllegalArgumentException when instantiating PartitioningCollection. ### Why are the changes needed? `RemoveRedundantSorts` rule uses SparkPlan's `outputPartitioning` to check whether a sort node is redundant. Currently, it is added before `EnsureRequirements`. Since `PartitioningCollection` requires left and right partitioning to have the same number of partitions, which is not necessarily true before applying `EnsureRequirements`, the rule can fail with the following exception: ``` IllegalArgumentException: requirement failed: PartitioningCollection requires all of its partitionings have the same numPartitions. ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit test Closes#30437 from allisonwang-db/spark-33472-2.4. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

gatorsmile · 2020-11-21T17:30:17Z

 RemoveRedundantProjects,
- RemoveRedundantSorts,
 EnsureRequirements,
+ // `RemoveRedundantSorts` needs to be added before `EnsureRequirements` to guarantee the same


before -> after ?

Ah, I missed it. @allisonwang-db could you fix it?

Thanks for catching it! Will create a fix.

### What changes were proposed in this pull request? This PR is a follow-up for #30373 that updates the comment for RemoveRedundantSorts in QueryExecution. ### Why are the changes needed? To update an incorrect comment. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A Closes#30584 from allisonwang-db/spark-33472-followup. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR is a follow-up for #30373 that updates the comment for RemoveRedundantSorts in QueryExecution. ### Why are the changes needed? To update an incorrect comment. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N/A Closes#30584 from allisonwang-db/spark-33472-followup. Authored-by: allisonwang-db <66282705+allisonwang-db@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 960d6af) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actionsBot added the SQL label Nov 13, 2020

maropu reviewed Nov 14, 2020
View reviewed changes

dongjoon-hyun requested changes Nov 17, 2020
View reviewed changes

allisonwang-db changed the title ~~[SPARK-33183][SQL][FOLLOW-UP] Adjust RemoveRedundantSorts rule order and update config version~~[SPARK-33472][SQL] Adjust RemoveRedundantSorts rule orderNov 18, 2020

allisonwang-db added 3 commits November 18, 2020 20:37

update
117dc6d

address comments
9d28f40

fix test
4e684df

allisonwang-db force-pushed the sort-follow-up branch from fa6050a to 4e684dfCompare November 19, 2020 04:50

cloud-fan closed this in a03c540Nov 19, 2020

This was referenced Nov 20, 2020
[SPARK-33472][SQL][2.4] Adjust RemoveRedundantSorts rule order #30437
Closed
[SPARK-33472][SQL][3.0] Adjust RemoveRedundantSorts rule order #30438
Closed

gatorsmile reviewed Nov 21, 2020
View reviewed changes

allisonwang-db mentioned this pull request Dec 3, 2020
[SPARK-33472][SQL][FOLLOW-UP] Update RemoveRedundantSorts comment #30584
Closed

allisonwang-db deleted the sort-follow-up branch January 19, 2024 01:21

Uh oh!

Conversation

allisonwang-db commented Nov 13, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Nov 14, 2020

Uh oh!

SparkQA commented Nov 14, 2020

Uh oh!

SparkQA commented Nov 14, 2020

Uh oh!

maropuNov 14, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunNov 17, 2020

Choose a reason for hiding this comment

Uh oh!

maropuNov 14, 2020

Choose a reason for hiding this comment

Uh oh!

maropuNov 14, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 18, 2020

Uh oh!

SparkQA commented Nov 18, 2020

Uh oh!

dongjoon-hyun commented Nov 18, 2020

Uh oh!

SparkQA commented Nov 18, 2020

Uh oh!

cloud-fan commented Nov 18, 2020

Uh oh!

SparkQA commented Nov 18, 2020

Uh oh!

SparkQA commented Nov 18, 2020

Uh oh!

SparkQA commented Nov 18, 2020

Uh oh!

cloud-fan commented Nov 18, 2020

Uh oh!

SparkQA commented Nov 18, 2020

Uh oh!

SparkQA commented Nov 18, 2020

Uh oh!

SparkQA commented Nov 18, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

SparkQA commented Nov 19, 2020

Uh oh!

cloud-fan commented Nov 19, 2020

Uh oh!

cloud-fan commented Nov 19, 2020

Uh oh!

dongjoon-hyun commented Nov 20, 2020

Uh oh!

gatorsmileNov 21, 2020

Choose a reason for hiding this comment

Uh oh!

maropuNov 22, 2020

Choose a reason for hiding this comment

Uh oh!

allisonwang-dbNov 24, 2020

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

allisonwang-db commented Nov 13, 2020•
edited
Loading

dongjoon-hyun left a comment •
edited
Loading