Skip to content

[SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset - #24993

Closed
nooberfsh wants to merge 3 commits into
apache:masterfrom
nooberfsh:sqlagg
Closed

[SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset#24993
nooberfsh wants to merge 3 commits into
apache:masterfrom
nooberfsh:sqlagg

Conversation

@nooberfsh

@nooberfshnooberfsh commented Jun 28, 2019

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add 4 additional agg to KeyValueGroupedDataset

How was this patch tested?

New test in DatasetSuite for typed aggregation

@JoshRosen

JoshRosen commented Jun 29, 2019

Copy link
Copy Markdown
Contributor

jenkins this is ok to test

@SparkQA

SparkQA commented Jun 29, 2019

Copy link
Copy Markdown

Test build #107041 has finished for PR 24993 at commit 9e1aacb.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Jul 1, 2019

Copy link
Copy Markdown

Test build #107068 has finished for PR 24993 at commit e1a3442.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun reviewed Jul 5, 2019
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala Outdated
dongjoon-hyun
dongjoon-hyun reviewed Jul 5, 2019
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala Outdated
dongjoon-hyun
dongjoon-hyun reviewed Jul 5, 2019
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala Outdated
dongjoon-hyun
dongjoon-hyun reviewed Jul 5, 2019
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala Outdated
@SparkQA

SparkQA commented Jul 8, 2019

Copy link
Copy Markdown

Test build #107319 has finished for PR 24993 at commit 1fbe526.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Jul 8, 2019

Copy link
Copy Markdown

Test build #107320 has finished for PR 24993 at commit 3520138.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun

dongjoon-hyun commented Jul 16, 2019

Copy link
Copy Markdown
Member

Retest this please.

@dongjoon-hyun

dongjoon-hyun commented Jul 16, 2019

Copy link
Copy Markdown
Member

Hi, @gatorsmile, @cloud-fan , @JoshRosen .
Since this is a very old issue, I'm wondering if this has not been considered in order to keep the API concise. Can we have this in Apache Spark 3.0.0?

@SparkQA

SparkQA commented Jul 16, 2019

Copy link
Copy Markdown

Test build #107716 has finished for PR 24993 at commit 3520138.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun

dongjoon-hyun commented Jul 16, 2019

Copy link
Copy Markdown
Member

Retest this please.

@SparkQA

SparkQA commented Jul 16, 2019

Copy link
Copy Markdown

Test build #107744 has finished for PR 24993 at commit 3520138.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun approved these changes Jul 16, 2019
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Merged to master.

@dongjoon-hyun

dongjoon-hyun commented Jul 16, 2019

Copy link
Copy Markdown
Member

Thank you, @nooberfsh and @JoshRosen .

@dongjoon-hyun

dongjoon-hyun commented Jul 16, 2019

Copy link
Copy Markdown
Member

@nooberfsh . What is your Apache JIRA ID? I want to add you to the Apache Spark Contributor group and assign SPARK-18299 to you.

vinodkc pushed a commit to vinodkc/spark that referenced this pull request Jul 18, 2019
@nooberfsh@vinodkc
[SPARK-18299][SQL] Allow more aggregations on KeyValueGroupedDataset
c42e0a1
## What changes were proposed in this pull request?
Add 4 additional agg to KeyValueGroupedDataset
## How was this patch tested?
New test in DatasetSuite for typed aggregation
Closesapache#24993 from nooberfsh/sqlagg.
Authored-by: nooberfsh <nooberfsh@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
@zhouzach

zhouzach commented Aug 2, 2019

Copy link
Copy Markdown

+1, LGTM. Merged to master.

but it can not be used in 2.4.3, is it not be published?

@dongjoon-hyun

dongjoon-hyun commented Aug 2, 2019

Copy link
Copy Markdown
Member

Correct, @zhouzach . SPARK-18299 is Apache Spark 3.0.0.

FYI, we don't backport New features are Improvements.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dongjoon-hyundongjoon-hyundongjoon-hyun approved these changes

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@nooberfsh@JoshRosen@SparkQA@dongjoon-hyun@zhouzach