Skip to content

[SPARK-34952][SQL][FOLLOW-UP] DSv2 aggregate push down follow-up - #33526

Closed
huaxingao wants to merge 5 commits into
apache:masterfrom
huaxingao:aggPD_followup
Closed

[SPARK-34952][SQL][FOLLOW-UP] DSv2 aggregate push down follow-up#33526
huaxingao wants to merge 5 commits into
apache:masterfrom
huaxingao:aggPD_followup

Conversation

@huaxingao

@huaxingaohuaxingao commented Jul 26, 2021•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

update java doc, JDBC data source doc, address follow up comments

Why are the changes needed?

update doc and address follow up comments

Does this PR introduce any user-facing change?

Yes, add the new JDBC option pushDownAggregate in JDBC data source doc.

How was this patch tested?

manually checked

@huaxingao

huaxingao commented Jul 26, 2021

Copy link
Copy Markdown
ContributorAuthor

cc @cloud-fan@viirya

viirya
viirya previously approved these changes Jul 26, 2021
View reviewed changes
@viiryaviirya mentioned this pull request Jul 26, 2021
Closed
@SparkQA

SparkQA commented Jul 26, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46179/

@SparkQA

SparkQA commented Jul 26, 2021

Copy link
Copy Markdown

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46179/

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Test build #141663 has finished for PR 33526 at commit 4a37da5.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@huaxingaohuaxingao changed the title [SPARK-34952][SQL][FOLLOW-UP] DSv2 aggregate push down: update doc[SPARK-34952][SQL][FOLLOW-UP] DSv2 aggregate push down follow-upJul 27, 2021
@huaxingao

huaxingao commented Jul 27, 2021

Copy link
Copy Markdown
ContributorAuthor

cc @sunchao

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46190/

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46190/

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46191/

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46191/

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Test build #141674 has finished for PR 33526 at commit 21ef1a1.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Test build #141675 has finished for PR 33526 at commit 6e73fb3.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
viirya self-requested a review July 27, 2021 06:53
@viirya
viirya dismissed their stale reviewJuly 27, 2021 06:54

there are new changes

cloud-fan
cloud-fan reviewed Jul 27, 2021
View reviewed changes
Comment threadsql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/Aggregation.java Outdated

@cloud-fancloud-fanJul 27, 2021•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we change the code style of groupByColumns as well?

@huaxingaohuaxingaoJul 27, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, the method was automatically collapsed into one single line in my IDE, I only saw one line there. I tried to be careful, but still missed this one.

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46225/

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46225/

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Test build #141712 has finished for PR 33526 at commit d204505.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

sunchao
sunchao approved these changes Jul 27, 2021
View reviewed changes

@sunchaosunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (non-binding) with one nit. Thanks @huaxingao !

Comment threadsql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownAggregates.java Outdated
* be: grouping columns, aggregate columns (in the same order as the aggregate functions in
* the given Aggregation).
*
* Returns true if the aggregation can be pushed down to datasource.

@sunchaosunchaoJul 27, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe "@return true if the aggregation can be pushed down to datasource. False otherwise."

@huaxingaohuaxingaoJul 27, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks!

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46248/

@SparkQA

SparkQA commented Jul 27, 2021

Copy link
Copy Markdown

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46248/

@SparkQA

SparkQA commented Jul 28, 2021

Copy link
Copy Markdown

Test build #141735 has finished for PR 33526 at commit 90356c8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented Jul 28, 2021

Copy link
Copy Markdown
Contributor

thanks, merging to master/3.2!

cloud-fan pushed a commit that referenced this pull request Jul 28, 2021
@huaxingao@cloud-fan
[SPARK-34952][SQL][FOLLOW-UP] DSv2 aggregate push down follow-up
33ef52e
### What changes were proposed in this pull request?
update java doc, JDBC data source doc, address follow up comments
### Why are the changes needed?
update doc and address follow up comments
### Does this PR introduce _any_ user-facing change?
Yes, add the new JDBC option `pushDownAggregate` in JDBC data source doc.
### How was this patch tested?
manually checked
Closes#33526 from huaxingao/aggPD_followup.
Authored-by: Huaxin Gao <huaxin_gao@apple.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit c8dd97d)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
@huaxingao
huaxingao deleted the aggPD_followup branch July 28, 2021 05:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sunchaosunchaosunchao approved these changes
@cloud-fancloud-fancloud-fan approved these changes
@viiryaviiryaAwaiting requested review from viirya

Assignees

No one assigned

Labels

DOCSSQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@huaxingao@SparkQA@cloud-fan@viirya@sunchao