Skip to content

[SPARK-16391][SQL] KeyValueGroupedDataset.reduceGroups should support partial aggregation - #14222

Closed
viirya wants to merge 3 commits into
apache:masterfrom
viirya:improve-reducegroups
Closed

[SPARK-16391][SQL] KeyValueGroupedDataset.reduceGroups should support partial aggregation#14222
viirya wants to merge 3 commits into
apache:masterfrom
viirya:improve-reducegroups

Conversation

@viirya

@viiryaviirya commented Jul 15, 2016

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

KeyValueGroupedDataset.reduceGroups is currently implemented via flatMapGroups, which does not support partial aggregation and so is very inefficient.

KeyValueGroupedDataset.reduceGroups should support partial aggregation. This PR implements it with Aggregator.

How was this patch tested?

Existing tests.

viirya
viirya reviewed Jul 15, 2016
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala Outdated

implicit val resultEncoder = ExpressionEncoder.tuple(kExprEnc, vExprEnc)
flatMapGroups(func)
def zero: (Int, V) = (0, null.asInstanceOf[V])

@viiryaviiryaJul 15, 2016

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One problem with Aggregator here is the zero value. This PR uses an Int (can be Boolean too) to indicate if the buffer is initialized.

@rxinrxinJul 15, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pull this out to be a generic ReduceAggregator and add unit test for it?

@viiryaviiryaJul 16, 2016

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. I will update this.

@SparkQA

SparkQA commented Jul 15, 2016

Copy link
Copy Markdown

Test build #62375 has finished for PR 14222 at commit 1135773.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@koertkuipers

koertkuipers commented Jul 18, 2016

Copy link
Copy Markdown
Contributor

there is a usefulness to this ReduceAggregator beyond .reduceGroups. basically you can take any Aggregator without a zero and turn it into a valid Aggregator, with the caveat being that the result is nullable and will be null if no inputs are provided to the Aggregator.

@viirya
viiryaforce-pushed the improve-reducegroups branch from 4ba124c to 7e8d8c1CompareJuly 18, 2016 08:55
@SparkQA

SparkQA commented Jul 18, 2016

Copy link
Copy Markdown

Test build #62454 has finished for PR 14222 at commit 4ba124c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Jul 18, 2016

Copy link
Copy Markdown

Test build #62456 has finished for PR 14222 at commit 7e8d8c1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya

viirya commented Jul 19, 2016•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

ping @rxin The change is ok for you? Please review this. Thanks.

rxin
rxin reviewed Jul 19, 2016
View reviewed changes
Comment threadsql/core/src/test/scala/org/apache/spark/sql/expressions/ReduceAggregatorSuite.scala Outdated
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.expressions.ReduceAggregator

class ReduceAggregatorSuite extends SparkFunSuite {

@rxinrxinJul 19, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just put this in DatasetAggregatorSuite

@viiryaviiryaJul 19, 2016

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I need to update this as you want to take over it?

@viiryaviiryaJul 20, 2016

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated it.

@rxin

rxin commented Jul 19, 2016

Copy link
Copy Markdown
Contributor

@viirya I'm going to take over the PR and play with the API a little bit.

@viirya

viirya commented Jul 19, 2016

Copy link
Copy Markdown
MemberAuthor

Ok.

@SparkQA

SparkQA commented Jul 20, 2016

Copy link
Copy Markdown

Test build #62594 has finished for PR 14222 at commit 6032325.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya

viirya commented Jul 21, 2016

Copy link
Copy Markdown
MemberAuthor

@rxin Any thing I need to update for this? Thanks.

@viirya

viirya commented Jul 25, 2016

Copy link
Copy Markdown
MemberAuthor

ping @rxin

@viirya

viirya commented Aug 9, 2016

Copy link
Copy Markdown
MemberAuthor

ping @rxin any thoughts on this? It is waiting for a while. Thanks.

rxin added a commit to rxin/spark that referenced this pull request Aug 10, 2016
@rxin
Merge pull requestapache#14222from viirya/improve-reducegroups
9de56ee
[SPARK-16391][SQL] KeyValueGroupedDataset.reduceGroups should support partial aggregation
@rxin

rxin commented Aug 10, 2016

Copy link
Copy Markdown
Contributor

I've created a pr here for discussion, based on my playing with the API: #14576

@viirya

viirya commented Aug 19, 2016

Copy link
Copy Markdown
MemberAuthor

Close this now since the pr #14576 is merged.

@viiryaviirya closed this Aug 19, 2016
@viirya
viirya deleted the improve-reducegroups branch December 27, 2023 18:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@viirya@SparkQA@koertkuipers@rxin