Uh oh!
There was an error while loading. Please reload this page.
[SPARK-1914] [SQL] Simplify CountFunction not to traverse to evaluate all child expressions. - #861
Closed
ueshin wants to merge 2 commits into
Closed
[SPARK-1914] [SQL] Simplify CountFunction not to traverse to evaluate all child expressions.#861ueshin wants to merge 2 commits into
ueshin wants to merge 2 commits into
Conversation
AmplabJenkins
commented
May 23, 2014
Merged build triggered. |
AmplabJenkins
commented
May 23, 2014
Merged build started. |
AmplabJenkins
commented
May 23, 2014
Merged build finished. |
AmplabJenkins
commented
May 23, 2014
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15164/ |
Conflicts: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregates.scala
AmplabJenkins
commented
May 26, 2014
Merged build triggered. |
AmplabJenkins
commented
May 26, 2014
Merged build started. |
ueshin
commented
May 26, 2014
MemberAuthor
Merged master to fix conflicts. |
AmplabJenkins
commented
May 26, 2014
Merged build finished. All automated tests passed. |
AmplabJenkins
commented
May 26, 2014
All automated tests passed. |
liancheng
commented
May 26, 2014
Contributor
LGTM. Added steps to reproduce this bug in This should be a blocking issue for Spark 1.0 release. @rxin@marmbrus |
liancheng
commented
May 26, 2014
Contributor
And, good catch @ueshin, thanks very much! |
rxin
commented
May 26, 2014
Contributor
Thanks. I've merged this in master & branch-1.0. |
asfgit pushed a commit
that referenced
this pull request
May 26, 2014
… all child expressions. `CountFunction` should count up only if the child's evaluated value is not null. Because it traverses to evaluate all child expressions, even if the child is null, it counts up if one of the all children is not null. Author: Takuya UESHIN <ueshin@happy-camper.st> Closes#861 from ueshin/issues/SPARK-1914 and squashes the following commits: 3b37315 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-19142afa238 [Takuya UESHIN] Simplify CountFunction not to traverse to evaluate all child expressions. (cherry picked from commit d6395d8) Signed-off-by: Reynold Xin <rxin@apache.org>
pdeyhim pushed a commit
to pdeyhim/spark-1
that referenced
this pull request
Jun 25, 2014
… all child expressions. `CountFunction` should count up only if the child's evaluated value is not null. Because it traverses to evaluate all child expressions, even if the child is null, it counts up if one of the all children is not null. Author: Takuya UESHIN <ueshin@happy-camper.st> Closesapache#861 from ueshin/issues/SPARK-1914 and squashes the following commits: 3b37315 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-1914 2afa238 [Takuya UESHIN] Simplify CountFunction not to traverse to evaluate all child expressions.
xiliu82 pushed a commit
to xiliu82/spark
that referenced
this pull request
Sep 4, 2014
… all child expressions. `CountFunction` should count up only if the child's evaluated value is not null. Because it traverses to evaluate all child expressions, even if the child is null, it counts up if one of the all children is not null. Author: Takuya UESHIN <ueshin@happy-camper.st> Closesapache#861 from ueshin/issues/SPARK-1914 and squashes the following commits: 3b37315 [Takuya UESHIN] Merge branch 'master' into issues/SPARK-1914 2afa238 [Takuya UESHIN] Simplify CountFunction not to traverse to evaluate all child expressions.
agirish pushed a commit
to HPEEzmeral/apache-spark
that referenced
this pull request
May 5, 2022
Co-authored-by: Egor Krivokon <>
wangyum added a commit
that referenced
this pull request
May 26, 2023
…861) * [SPARK-36183][SQL][FOLLOWUP] Fix push down limit 1 through Aggregate ### What changes were proposed in this pull request? Use `Aggregate.aggregateExpressions` instead of `Aggregate.output` when pushing down limit 1 through Aggregate. For example: ```scala spark.range(10).selectExpr("id % 5 AS a", "id % 5 AS b").write.saveAsTable("t1") spark.sql("SELECT a, b, a AS alias FROM t1 GROUP BY a, b LIMIT 1").explain(true) ``` Before this pr: ``` == Optimized Logical Plan == GlobalLimit 1 +- LocalLimit 1 +- !Project [a#227L, b#228L, alias#226L] +- LocalLimit 1 +- Relation default.t1[a#227L,b#228L] parquet ``` After this pr: ``` == Optimized Logical Plan == GlobalLimit 1 +- LocalLimit 1 +- Project [a#227L, b#228L, a#227L AS alias#226L] +- LocalLimit 1 +- Relation default.t1[a#227L,b#228L] parquet ``` ### Why are the changes needed? Fix bug. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test. Closes#35286 from wangyum/SPARK-36183-2. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 9b12571)
udaynpusa pushed a commit
to mapr/spark
that referenced
this pull request
Jan 30, 2024
Co-authored-by: Egor Krivokon <>
mapr-devops pushed a commit
to mapr/spark
that referenced
this pull request
May 8, 2025
Co-authored-by: Egor Krivokon <>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CountFunctionshould count up only if the child's evaluated value is not null.Because it traverses to evaluate all child expressions, even if the child is null, it counts up if one of the all children is not null.