[SPARK-32816][SQL][3.0] Fix analyzer bug when aggregating multiple distinct DECIMAL columns by linhongliu-db · Pull Request #30053 · apache/spark

linhongliu-db · 2020-10-15T08:44:39Z

What changes were proposed in this pull request?

This PR fixes a conflict between RewriteDistinctAggregates and DecimalAggregates.
In some cases, DecimalAggregates will wrap the decimal column to UnscaledValue using
different rules for different aggregates.

This means, same distinct column with different aggregates will change to different distinct columns
after DecimalAggregates. For example:
avg(distinct decimal_col), sum(distinct decimal_col) may change to
avg(distinct UnscaledValue(decimal_col)), sum(distinct decimal_col)

We assume after RewriteDistinctAggregates, there will be at most one distinct column in aggregates,
but DecimalAggregates breaks this assumption. To fix this, we have to switch the order of these two
rules.

Why are the changes needed?

bug fix

Does this PR introduce any user-facing change?

no

How was this patch tested?

added test cases

Authored-by: Linhong Liu linhong.liu@databricks.com
Signed-off-by: Wenchen Fan wenchen@databricks.com
(cherry picked from commit 40ef5c9)

SparkQA · 2020-10-15T09:37:54Z

Test build #129826 has finished for PR 30053 at commit b043f94.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-15T10:20:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34429/

SparkQA · 2020-10-15T10:38:44Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34429/

linhongliu-db · 2020-10-16T02:38:58Z

This PR depends on #30052 to pass tests

…t DECIMAL columns This PR fixes a conflict between `RewriteDistinctAggregates` and `DecimalAggregates`. In some cases, `DecimalAggregates` will wrap the decimal column to `UnscaledValue` using different rules for different aggregates. This means, same distinct column with different aggregates will change to different distinct columns after `DecimalAggregates`. For example: `avg(distinct decimal_col), sum(distinct decimal_col)` may change to `avg(distinct UnscaledValue(decimal_col)), sum(distinct decimal_col)` We assume after `RewriteDistinctAggregates`, there will be at most one distinct column in aggregates, but `DecimalAggregates` breaks this assumption. To fix this, we have to switch the order of these two rules. bug fix no added test cases Closesapache#29673 from linhongliu-db/SPARK-32816. Authored-by: Linhong Liu <linhong.liu@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 40ef5c9)

SparkQA · 2020-10-16T05:46:59Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34481/

SparkQA · 2020-10-16T06:05:15Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34481/

SparkQA · 2020-10-16T07:01:06Z

Test build #129876 has finished for PR 30053 at commit 2634588.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

linhongliu-db · 2020-10-16T07:06:25Z

retest this please

SparkQA · 2020-10-16T07:46:10Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34486/

SparkQA · 2020-10-16T08:09:19Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34486/

SparkQA · 2020-10-16T08:12:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34487/

SparkQA · 2020-10-16T08:41:35Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34487/

SparkQA · 2020-10-16T09:06:50Z

Test build #129881 has finished for PR 30053 at commit 2634588.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-16T09:43:28Z

Test build #129882 has finished for PR 30053 at commit 96a0706.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

linhongliu-db · 2020-10-19T02:22:53Z

@cloud-fan This PR depends on another 2 fixes in the master branch. Should we cherry-pick them?
#29026
#27627

cloud-fan · 2020-10-22T07:18:32Z

ah, that two are hard to backport as they change streaming state store. I guess we can't backport this fix either.

linhongliu-db · 2020-10-22T10:04:52Z

got it. Let's close it for now

dongjoon-hyun added the SQL label Oct 15, 2020

linhongliu-db mentioned this pull request Oct 16, 2020
[SPARK-32761][SQL][3.0] Allow aggregating multiple foldable distinct expressions #30052
Closed

linhongliu-db force-pushed the SPARK-32816-3.0 branch from b043f94 to 2634588Compare October 16, 2020 05:02

empty commit
96a0706

linhongliu-db closed this Oct 22, 2020

Uh oh!

Conversation

linhongliu-db commented Oct 15, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Oct 15, 2020

Uh oh!

SparkQA commented Oct 15, 2020

Uh oh!

SparkQA commented Oct 15, 2020

Uh oh!

linhongliu-db commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

linhongliu-db commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

SparkQA commented Oct 16, 2020

Uh oh!

linhongliu-db commented Oct 19, 2020

Uh oh!

cloud-fan commented Oct 22, 2020

Uh oh!

linhongliu-db commented Oct 22, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants