Skip to content

[SPARK-20636] Add the rule TransposeWindow to the optimization batch - #23222

Closed
gatorsmile wants to merge 5 commits into
apache:masterfrom
gatorsmile:followupSPARK-20636
Closed

[SPARK-20636] Add the rule TransposeWindow to the optimization batch#23222
gatorsmile wants to merge 5 commits into
apache:masterfrom
gatorsmile:followupSPARK-20636

Conversation

@gatorsmile

@gatorsmilegatorsmile commented Dec 5, 2018•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR is a follow-up of the PR #17899. It is to add the rule TransposeWindow the optimizer batch.

How was this patch tested?

The existing tests.

@gatorsmile

gatorsmile commented Dec 5, 2018

Copy link
Copy Markdown
MemberAuthor

cc @ptkool@jiangxb1987@cloud-fan

@cloud-fan

cloud-fan commented Dec 5, 2018

Copy link
Copy Markdown
Contributor

That PR also added an end-to-end test, does this mean that test is not valid?

@gatorsmile

gatorsmile commented Dec 5, 2018

Copy link
Copy Markdown
MemberAuthor

We can compare the plans and see whether the rule takes an effect.

@SparkQA

SparkQA commented Dec 5, 2018

Copy link
Copy Markdown

Test build #99698 has finished for PR 23222 at commit 1270e89.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun

dongjoon-hyun commented Dec 5, 2018

Copy link
Copy Markdown
Member

Retest this please.

@SparkQA

SparkQA commented Dec 5, 2018

Copy link
Copy Markdown

Test build #99701 has finished for PR 23222 at commit 1270e89.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

jiangxb1987
jiangxb1987 approved these changes Dec 5, 2018
View reviewed changes

@jiangxb1987jiangxb1987 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jiangxb1987

jiangxb1987 commented Dec 5, 2018

Copy link
Copy Markdown
Contributor

Shall we add a SQL tag to the title?

@cloud-fan

cloud-fan commented Dec 5, 2018•
edited
Loading

Copy link
Copy Markdown
Contributor

Jenkins passes, which means the previously added end-to-end test can't show the benefit of this rule. We should update it.

@ptkool

ptkool commented Dec 5, 2018

Copy link
Copy Markdown
Contributor

I'm not sure how this ended up being omitted. TransposeWindowSuite will be fine since it creates a simple optimizer from this rule and a few others. The new test added to DataFrameWindowFunctionsSuite should pass with or without this rule added to the optimizer batch.

@cloud-fan

cloud-fan commented Dec 17, 2018

Copy link
Copy Markdown
Contributor

LGTM

@SparkQA

SparkQA commented Dec 17, 2018

Copy link
Copy Markdown

Test build #100219 has finished for PR 23222 at commit 083c726.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Dec 17, 2018

Copy link
Copy Markdown

Test build #100215 has finished for PR 23222 at commit d540120.

  • This patch fails due to an unknown error code, -9.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@gatorsmile

gatorsmile commented Dec 17, 2018

Copy link
Copy Markdown
MemberAuthor

Thanks! Merged to master.

holdenk pushed a commit to holdenk/spark that referenced this pull request Jan 5, 2019
@gatorsmile@holdenk
[SPARK-20636] Add the rule TransposeWindow to the optimization batch
a1c97b5
## What changes were proposed in this pull request?
This PR is a follow-up of the PR apache#17899. It is to add the rule TransposeWindow the optimizer batch.
## How was this patch tested?
The existing tests.
Closesapache#23222 from gatorsmile/followupSPARK-20636.
Authored-by: gatorsmile <gatorsmile@gmail.com>
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
@gatorsmileJackey Lee
[SPARK-20636] Add the rule TransposeWindow to the optimization batch
84787a0
## What changes were proposed in this pull request?
This PR is a follow-up of the PR apache#17899. It is to add the rule TransposeWindow the optimizer batch.
## How was this patch tested?
The existing tests.
Closesapache#23222 from gatorsmile/followupSPARK-20636.
Authored-by: gatorsmile <gatorsmile@gmail.com>
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jiangxb1987jiangxb1987jiangxb1987 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@gatorsmile@cloud-fan@SparkQA@dongjoon-hyun@jiangxb1987@ptkool