Uh oh!
There was an error while loading. Please reload this page.
[SQL] Minor: Introduce SchemaRDD#aggregate() for simple aggregations - #874
Closed
aarondav wants to merge 2 commits into
Closed
[SQL] Minor: Introduce SchemaRDD#aggregate() for simple aggregations#874aarondav wants to merge 2 commits into
aarondav wants to merge 2 commits into
Conversation
rdd.aggregate(Sum('val)) is just shorthand for
rdd.groupBy()(Sum('val)), but seems be more natural than
doing a groupBy with no grouping expressions when you
really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to
be lacking in several other methods like groupBy() already --
leaving that cleanup for future patches.ContributorAuthor
There was a problem hiding this comment.
This example doesn't compile with the \ in there.
AmplabJenkins
commented
May 25, 2014
Merged build triggered. |
AmplabJenkins
commented
May 25, 2014
Merged build started. |
Contributor
There was a problem hiding this comment.
Can you say in the scaladoc that this is equivalent to groupBy()(...) ?
rxin
commented
May 25, 2014
Contributor
LGTM other than the small addition to scaladoc. |
AmplabJenkins
commented
May 25, 2014
Merged build finished. All automated tests passed. |
AmplabJenkins
commented
May 25, 2014
All automated tests passed. |
aarondav
commented
May 25, 2014
ContributorAuthor
Added comment! |
AmplabJenkins
commented
May 25, 2014
Merged build triggered. |
AmplabJenkins
commented
May 25, 2014
Merged build started. |
rxin
commented
May 25, 2014
Contributor
LGTM |
AmplabJenkins
commented
May 26, 2014
Merged build finished. All automated tests passed. |
AmplabJenkins
commented
May 26, 2014
All automated tests passed. |
rxin
commented
May 26, 2014
Contributor
I've merged this into master & branch-1.0. |
asfgit pushed a commit
that referenced
this pull request
May 26, 2014
```scala
rdd.aggregate(Sum('val))
```
is just shorthand for
```scala
rdd.groupBy()(Sum('val))
```
but seems be more natural than doing a groupBy with no grouping expressions when you really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to be lacking several other methods like groupBy() already -- leaving that cleanup for future patches.
Author: Aaron Davidson <aaron@databricks.com>
Closes#874 from aarondav/schemardd and squashes the following commits:
e9e68ee [Aaron Davidson] Add comment
db6afe2 [Aaron Davidson] Introduce SchemaRDD#aggregate() for simple aggregations
(cherry picked from commit c3576ff)
Signed-off-by: Reynold Xin <rxin@apache.org>pdeyhim pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
```scala
rdd.aggregate(Sum('val))
```
is just shorthand for
```scala
rdd.groupBy()(Sum('val))
```
but seems be more natural than doing a groupBy with no grouping expressions when you really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to be lacking several other methods like groupBy() already -- leaving that cleanup for future patches.
Author: Aaron Davidson <aaron@databricks.com>
Closesapache#874 from aarondav/schemardd and squashes the following commits:
e9e68ee [Aaron Davidson] Add comment
db6afe2 [Aaron Davidson] Introduce SchemaRDD#aggregate() for simple aggregationsxiliu82 pushed a commit
to xiliu82/spark
that referenced
this pull request
Sep 4, 2014
```scala
rdd.aggregate(Sum('val))
```
is just shorthand for
```scala
rdd.groupBy()(Sum('val))
```
but seems be more natural than doing a groupBy with no grouping expressions when you really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to be lacking several other methods like groupBy() already -- leaving that cleanup for future patches.
Author: Aaron Davidson <aaron@databricks.com>
Closesapache#874 from aarondav/schemardd and squashes the following commits:
e9e68ee [Aaron Davidson] Add comment
db6afe2 [Aaron Davidson] Introduce SchemaRDD#aggregate() for simple aggregations
wangyum pushed a commit
that referenced
this pull request
May 26, 2023
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
is just shorthand for
but seems be more natural than doing a groupBy with no grouping expressions when you really just want an aggregation over all rows.
Did not add a JavaSchemaRDD or Python API, as these seem to be lacking several other methods like groupBy() already -- leaving that cleanup for future patches.