Skip to content

[SQL][WIP][Test] Supports object-based aggregation function which can store arbitrary objects in aggregation buffer. - #14723

Closed
clockfly wants to merge 2 commits into
apache:masterfrom
clockfly:object_aggregation_buffer_part1
Closed

[SQL][WIP][Test] Supports object-based aggregation function which can store arbitrary objects in aggregation buffer.#14723
clockfly wants to merge 2 commits into
apache:masterfrom
clockfly:object_aggregation_buffer_part1

Conversation

@clockfly

@clockflyclockfly commented Aug 19, 2016

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request

This PR allows user to define an AggregateFunction which can store arbitrary Java objects
in aggregation buffer, and use the Java object to do aggregation. Before this PR, user are only allowed to store a limited set of object type in aggregation buffer. Please see example usage at
class org.apache.spark.sql.AggregateWithObjectAggregateBufferSuite.MaxWithObjectAggregateBuffer

How was this patch tested?

Unit tests.

@clockfly
object aggregation buffer
520a17f
Revert: object aggregation buffer
@SparkQA

SparkQA commented Aug 19, 2016

Copy link
Copy Markdown

Test build #64103 has finished for PR 14723 at commit 520a17f.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • trait WithObjectAggregateBuffer

yhuai
yhuai reviewed Aug 20, 2016
View reviewed changes
Comment thread...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
* 1. Spark framework moves on to next group, until all groups have been processed.
*/
trait WithObjectAggregateBuffer {
this: ImperativeAggregate =>

@yhuaiyhuaiAug 20, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Semes we do not really need this line.

@yhuaiyhuaiAug 20, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess having this line will make this trait hard to be used in Java.

@yhuaiyhuaiAug 20, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, seems this trait will be still an java interface. But, I think in general, we do not really need to have this line.

@yhuai

yhuai commented Aug 21, 2016

Copy link
Copy Markdown
Contributor

Can you create a jira?

cloud-fan
cloud-fan reviewed Aug 22, 2016
View reviewed changes
Comment thread...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala Outdated
* object A with current inputRow. After updating, object A is stored back to mutableAggBuffer.
* 1. After processing all rows of current group, the framework will call method
* `serializeObjectAggregationBufferInPlace(aggregationBuffer: MutableRow)` to serialize object A
* to a serializable format in place.

@cloud-fancloud-fanAug 22, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to a Spark SQL internal format(mostly BinaryType) in place

@clockfly
clockflyforce-pushed the object_aggregation_buffer_part1 branch from 9b16f89 to 9ae648cCompareAugust 22, 2016 06:18
@SparkQA

SparkQA commented Aug 22, 2016

Copy link
Copy Markdown

Test build #64179 has finished for PR 14723 at commit 9b16f89.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Aug 22, 2016

Copy link
Copy Markdown

Test build #64180 has finished for PR 14723 at commit 9ae648c.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

liancheng
liancheng reviewed Aug 22, 2016
View reviewed changes
Comment thread...catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
}

/**
* This traits allows an AggregateFunction to store **arbitrary** Java objects in internal

@lianchenglianchengAug 22, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: traits => trait

@clockfly

clockfly commented Aug 22, 2016

Copy link
Copy Markdown
ContributorAuthor

@liancheng@cloud-fan
@yhuai@hvanhovell@gatorsmile
This PR is superceded by #14753, please review the new PR instead.

The motivation behind the change is that the aggregation function is also used by WindowExec, which may do continous update and eval. We have to override eval of ImperativeAggregate so that eval can accepts an aggregation buffer which contains generic Java object.

For example:

agg.update(buffer, row1)
agg.eval(buffer)
agg.update(buffer, row2)
agg.eal(buffer)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@clockfly@SparkQA@yhuai@liancheng@cloud-fan@hvanhovell@gatorsmile