Optimize DISTINCT, ORDER BY and DISTINCT ON when Aggregation without Group By. by avamingli · Pull Request #685 · apache/cloudberry

avamingli · 2024-10-24T13:28:16Z

For query which has Aggregation but without Group by clause, the DISTINCT/DISTINCT ON/ORDER BY clause could be removed as there would be one row returned at most.
And there is no necessary to do unique or sort.
This can simply the plan, and process less expressions like: Aggref nodes during planner.

DISTINCT

explain(verbose, costs off)
select distinctcount(a), sum(b) from t_distinct_sort ;
QUERY PLAN
------------------------------------------------------------------------
Unique
Output: (count(a)), (sum(b))
Group Key: (count(a)), (sum(b))
-> Sort
Output: (count(a)), (sum(b))
Sort Key: (count(t_distinct_sort.a)), (sum(t_distinct_sort.b))
-> Finalize Aggregate
Output: count(a), sum(b)
-> Gather Motion 3:1 (slice1; segments: 3)
Output: (PARTIAL count(a)), (PARTIAL sum(b))
-> Partial Aggregate
Output: PARTIAL count(a), PARTIAL sum(b)
-> Seq Scan onpublic.t_distinct_sort
Output: a, b, c
Settings: optimizer ='off'
Optimizer: Postgres query optimizer
(16 rows)

After this commit:

explain(verbose, costs off)
select distinctcount(a), sum(b) from t_distinct_sort ;
QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------
Finalize Aggregate
Output: count(a), sum(b)
-> Gather Motion 3:1 (slice1; segments: 3)
Output: (PARTIAL count(a)), (PARTIAL sum(b))
-> Partial Aggregate
Output: PARTIAL count(a), PARTIAL sum(b)
-> Seq Scan onpublic.t_distinct_sort
Output: a, b, c
Optimizer: Postgres query optimizer
(10 rows)

DISTINCT ON and ORDER BY

select distincton(count(b), count(c)) count(a), sum(b) from t_distinct_sort order bycount(c);
QUERY PLAN
--------------------------------------------------------------------
Unique
Output: (count(a)), (sum(b)), (count(c)), (count(b))
Group Key: (count(c)), (count(b))
-> Sort
Output: (count(a)), (sum(b)), (count(c)), (count(b))
Sort Key: (count(t_distinct_sort.c)),
(count(t_distinct_sort.b))
-> Finalize Aggregate
Output: count(a), sum(b), count(c), count(b)
-> Gather Motion 3:1 (slice1; segments: 3)
Output: (PARTIAL count(a)), (PARTIAL sum(b)),
(PARTIAL count(c)), (PARTIAL count(b))
-> Partial Aggregate
Output: PARTIAL count(a), PARTIAL sum(b),
PARTIAL count(c), PARTIAL count(b)
-> Seq Scan onpublic.t_distinct_sort
Output: a, b, c

After this commit:

select distincton(count(b), count(c)) count(a), sum(b) from t_distinct_sort order bycount(c);
QUERY PLAN
--------------------------------------------------------
Finalize Aggregate
Output: count(a), sum(b)
-> Gather Motion 3:1 (slice1; segments: 3)
Output: (PARTIAL count(a)), (PARTIAL sum(b))
-> Partial Aggregate
Output: PARTIAL count(a), PARTIAL sum(b)
-> Seq Scan onpublic.t_distinct_sort
Output: a, b, c
Optimizer: Postgres query optimizer

ORDER BY

explain(verbose, costs off)
selectcount(a), sum(b) from t_distinct_sort order bysum(a), count(c);
QUERY PLAN
--------------------------------------------------------------------------------------------------
Sort
Output: (count(a)), (sum(b)), (sum(a)), (count(c))
Sort Key: (sum(t_distinct_sort.a)), (count(t_distinct_sort.c))
-> Finalize Aggregate
Output: count(a), sum(b), sum(a), count(c)
-> Gather Motion 3:1 (slice1; segments: 3)
Output: (PARTIAL count(a)), (PARTIAL sum(b)), (PARTIAL sum(a)), (PARTIAL count(c))
-> Partial Aggregate
Output: PARTIAL count(a), PARTIAL sum(b), PARTIAL sum(a), PARTIAL count(c)
-> Seq Scan onpublic.t_distinct_sort
Output: a, b, c
Settings: optimizer ='off'
Optimizer: Postgres query optimizer
(13 rows)

After this commit:

explain(verbose, costs off)
selectcount(a), sum(b) from t_distinct_sort order bysum(a), count(c);
QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------
Finalize Aggregate
Output: count(a), sum(b)
-> Gather Motion 3:1 (slice1; segments: 3)
Output: (PARTIAL count(a)), (PARTIAL sum(b))
-> Partial Aggregate
Output: PARTIAL count(a), PARTIAL sum(b)
-> Seq Scan onpublic.t_distinct_sort
Output: a, b, c
Optimizer: Postgres query optimizer
(10 rows)

DISTINCT and ORDER BY

select distinctcount(a), sum(b) from t_distinct_sort order bysum(b), count(a);
QUERY PLAN
------------------------------------------------------------------------
Unique
Output: (count(a)), (sum(b))
Group Key: (sum(b)), (count(a))
-> Sort
Output: (count(a)), (sum(b))
Sort Key: (sum(t_distinct_sort.b)), (count(t_distinct_sort.a))
-> Finalize Aggregate
Output: count(a), sum(b)
-> Gather Motion 3:1 (slice1; segments: 3)
Output: (PARTIAL count(a)), (PARTIAL sum(b))
-> Partial Aggregate
Output: PARTIAL count(a), PARTIAL sum(b)
-> Seq Scan onpublic.t_distinct_sort
Output: a, b, c
Settings: optimizer ='off'
Optimizer: Postgres query optimizer
(16 rows)

After this commit:

select distinctcount(a), sum(b) from t_distinct_sort order bysum(b), count(a);
QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------
Finalize Aggregate
Output: count(a), sum(b)
-> Gather Motion 3:1 (slice1; segments: 3)
Output: (PARTIAL count(a)), (PARTIAL sum(b))
-> Partial Aggregate
Output: PARTIAL count(a), PARTIAL sum(b)
-> Seq Scan onpublic.t_distinct_sort
Output: a, b, c
Optimizer: Postgres query optimizer
(10 rows)

Authored-by: Zhang Mingli avamingli@gmail.com

fix #ISSUE_Number

Change logs

Describe your change clearly, including what problem is being solved or what feature is being added.

If it has some breaking backward or forward compatibility, please clary.

Why are the changes needed?

Describe why the changes are necessary.

Does this PR introduce any user-facing change?

If yes, please clarify the previous behavior and the change this PR proposes.

How was this patch tested?

Please detail how the changes were tested, including manual tests and any relevant unit or integration tests.

Contributor's Checklist

Here are some reminders and checklists before/when submitting your pull request, please check them:

Make sure your Pull Request has a clear title and commit message. You can take git-commit template as a reference.
Sign the Contributor License Agreement as prompted for your first-time contribution(One-time setup).
Learn the coding contribution guide, including our code conventions, workflow and more.
List your communication in the GitHub Issues or Discussions (if has or needed).
Document changes.
Add tests for the change
Pass make installcheck
Pass make -C src/test installcheck-cbdb-parallel
Feel free to request cloudberrydb/dev team for review and approval when your PR is ready🥳

avamingli · 2024-10-25T01:42:54Z

Many plan diffs, will fix later.

avamingli · 2024-11-04T07:04:20Z

For query which has Aggregation but without Group by clause, the DISTINCT/DISTINCT ON/ORDER BY clause could be removed as there would be one row returned at most.

SRF will break the assumption.

selectcount(*), generate_series(1, 4) from t1;
count | generate_series
-------+-----------------3 | 13 | 23 | 33 | 4
(4 rows)

Fix it and Postgres' WITH ORDINALITY as well.

fanfuxiaoran · 2024-11-18T09:41:10Z

I took a look at orca, it has already optimized distinct function.

explain select distinct(count(a)) from foo;
QUERY PLAN
------------------------------------------------------------------------------------
Finalize Aggregate (cost=0.00..526.96 rows=1 width=8)
-> Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..526.96 rows=1 width=8)
-> Partial Aggregate (cost=0.00..526.96 rows=1 width=8)
-> Seq Scan on foo (cost=0.00..500.67 rows=3333334 width=4)
Optimizer: Pivotal Optimizer (GPORCA)
(5 rows)

Even if with group by , the distinct also can be removed

explain select distinct(count(a)) from foo group by a ;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..1395.69 rows=1000 width=8)
-> HashAggregate (cost=0.00..1395.66 rows=334 width=8)
Group Key: (count(a))
-> Redistribute Motion 3:3 (slice2; segments: 3) (cost=0.00..1395.62 rows=334 width=8)
Hash Key: (count(a))
-> Streaming HashAggregate (cost=0.00..1395.61 rows=334 width=8)
Group Key: count(a)
-> HashAggregate (cost=0.00..985.15 rows=3333334 width=8)
Group Key: a
Planned Partitions: 16
-> Redistribute Motion 3:3 (slice3; segments: 3) (cost=0.00..567.20 rows=3333334 width=4)
Hash Key: a
-> Seq Scan on foo (cost=0.00..500.67 rows=3333334 width=4)
Optimizer: Pivotal Optimizer (GPORCA)
(14 rows)

as distinct is a function which only works in a group.

The function called PexprRemoveSuperfluousDistinctInDQA in orca.

avamingli · 2024-11-21T08:06:12Z

I took a look at orca, it has already optimized distinct function.

explain select distinct(count(a)) from foo;
QUERY PLAN
------------------------------------------------------------------------------------
Finalize Aggregate (cost=0.00..526.96 rows=1 width=8)
-> Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..526.96 rows=1 width=8)
-> Partial Aggregate (cost=0.00..526.96 rows=1 width=8)
-> Seq Scan on foo (cost=0.00..500.67 rows=3333334 width=4)
Optimizer: Pivotal Optimizer (GPORCA)
(5 rows)

Even if with group by , the distinct also can be removed

explain select distinct(count(a)) from foo group by a ;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..1395.69 rows=1000 width=8)
-> HashAggregate (cost=0.00..1395.66 rows=334 width=8)
Group Key: (count(a))
-> Redistribute Motion 3:3 (slice2; segments: 3) (cost=0.00..1395.62 rows=334 width=8)
Hash Key: (count(a))
-> Streaming HashAggregate (cost=0.00..1395.61 rows=334 width=8)
Group Key: count(a)
-> HashAggregate (cost=0.00..985.15 rows=3333334 width=8)
Group Key: a
Planned Partitions: 16
-> Redistribute Motion 3:3 (slice3; segments: 3) (cost=0.00..567.20 rows=3333334 width=4)
Hash Key: a
-> Seq Scan on foo (cost=0.00..500.67 rows=3333334 width=4)
Optimizer: Pivotal Optimizer (GPORCA)
(14 rows)

as distinct is a function which only works in a group.

The function called PexprRemoveSuperfluousDistinctInDQA in orca.

Yeah, see #677 (reply in thread)

fanfuxiaoran · 2024-11-26T09:24:30Z

I took a look at orca, it has already optimized distinct function.

explain select distinct(count(a)) from foo;
QUERY PLAN
------------------------------------------------------------------------------------
Finalize Aggregate (cost=0.00..526.96 rows=1 width=8)
-> Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..526.96 rows=1 width=8)
-> Partial Aggregate (cost=0.00..526.96 rows=1 width=8)
-> Seq Scan on foo (cost=0.00..500.67 rows=3333334 width=4)
Optimizer: Pivotal Optimizer (GPORCA)
(5 rows)

Even if with group by , the distinct also can be removed

explain select distinct(count(a)) from foo group by a ;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
Gather Motion 3:1 (slice1; segments: 3) (cost=0.00..1395.69 rows=1000 width=8)
-> HashAggregate (cost=0.00..1395.66 rows=334 width=8)
Group Key: (count(a))
-> Redistribute Motion 3:3 (slice2; segments: 3) (cost=0.00..1395.62 rows=334 width=8)
Hash Key: (count(a))
-> Streaming HashAggregate (cost=0.00..1395.61 rows=334 width=8)
Group Key: count(a)
-> HashAggregate (cost=0.00..985.15 rows=3333334 width=8)
Group Key: a
Planned Partitions: 16
-> Redistribute Motion 3:3 (slice3; segments: 3) (cost=0.00..567.20 rows=3333334 width=4)
Hash Key: a
-> Seq Scan on foo (cost=0.00..500.67 rows=3333334 width=4)
Optimizer: Pivotal Optimizer (GPORCA)
(14 rows)

as distinct is a function which only works in a group.
The function called PexprRemoveSuperfluousDistinctInDQA in orca.

Yeah, see #677 (reply in thread)

Orca removed the distinct expression when it is used on the agg expression even if there is group by clause, do we need to consider that?

avamingli · 2024-11-26T09:31:36Z

Orca removed the distinct expression when it is used on the agg expression even if there is group by clause, do we need to consider that?

We can consider that type of optimization in the future.
In line with the goals of this PR, we have optimized statements like DISTINCT, DISTINCT ON, ORDER BY, and LIMIT.
However, given the presence of GROUP BY, the optimizations for ORDER BY and LIMIT may no longer apply.
With GROUP BY clause should be another topic for optimization.

For query which has Aggregation but without Group by clause, the DISTINCT/DISTINCT ON/ORDER BY clause could be removed as there would be one row returned at most. And there is no necessary to do unique or sort. This can simply the plan, and process less Aggref nodes during planner. select distinct on(count(b), count(c)) count(a), sum(b) from t_distinct_sort order by count(c); QUERY PLAN -------------------------------------------------------------------- Unique Output: (count(a)), (sum(b)), (count(c)), (count(b)) Group Key: (count(c)), (count(b)) -> Sort Output: (count(a)), (sum(b)), (count(c)), (count(b)) Sort Key: (count(t_distinct_sort.c)), (count(t_distinct_sort.b)) -> Finalize Aggregate Output: count(a), sum(b), count(c), count(b) -> Gather Motion 3:1 (slice1; segments: 3) Output: (PARTIAL count(a)), (PARTIAL sum(b)), (PARTIAL count(c)), (PARTIAL count(b)) -> Partial Aggregate Output: PARTIAL count(a), PARTIAL sum(b), PARTIAL count(c), PARTIAL count(b) -> Seq Scan on public.t_distinct_sort Output: a, b, c After this commit: select distinct on(count(b), count(c)) count(a), sum(b) from t_distinct_sort order by count(c); QUERY PLAN -------------------------------------------------------- Finalize Aggregate Output: count(a), sum(b) -> Gather Motion 3:1 (slice1; segments: 3) Output: (PARTIAL count(a)), (PARTIAL sum(b)) -> Partial Aggregate Output: PARTIAL count(a), PARTIAL sum(b) -> Seq Scan on public.t_distinct_sort Output: a, b, c Optimizer: Postgres query optimizer Authored-by: Zhang Mingli avamingli@gmail.com

fanfuxiaoran · 2024-12-04T02:12:00Z

LGTM!

avamingli · 2024-12-04T02:51:00Z

Many thanks @fanfuxiaoran for detailed review!

avamingli force-pushed the opt_dist_sort_on_agg branch from 4a01111 to 1d31349Compare November 4, 2024 06:58

avamingli requested a review from gfphoenix78 November 5, 2024 07:52

avamingli mentioned this pull request Nov 25, 2024
[AQUMV] Answer Aggregation Query Directly. #705
Merged
9 tasks

avamingli force-pushed the opt_dist_sort_on_agg branch from 1d31349 to 66614afCompare November 26, 2024 05:22

fanfuxiaoran reviewed Nov 26, 2024
View reviewed changes

Comment threadsrc/backend/optimizer/plan/transform.c Outdated
Comment threadsrc/backend/optimizer/plan/transform.c

avamingli force-pushed the opt_dist_sort_on_agg branch from 66614af to 96d4e42Compare November 27, 2024 07:11

fanfuxiaoran reviewed Nov 28, 2024
View reviewed changes

Comment threadsrc/backend/optimizer/plan/transform.c Outdated
Comment threadsrc/backend/optimizer/plan/transform.c Outdated

avamingli force-pushed the opt_dist_sort_on_agg branch from 96d4e42 to d53a163Compare December 3, 2024 09:51

fanfuxiaoran reviewed Dec 3, 2024
View reviewed changes

Comment threadsrc/backend/optimizer/plan/transform.c

fanfuxiaoran approved these changes Dec 4, 2024
View reviewed changes

my-ship-it approved these changes Dec 4, 2024
View reviewed changes

my-ship-it merged commit ca642bf into apache:mainDec 4, 2024

avamingli deleted the opt_dist_sort_on_agg branch December 4, 2024 02:56

Navigation Menu

Uh oh!

Optimize DISTINCT, ORDER BY and DISTINCT ON when Aggregation without Group By. - #685

Optimize DISTINCT, ORDER BY and DISTINCT ON when Aggregation without Group By.#685
my-ship-it merged 1 commit into
apache:mainfrom
avamingli:opt_dist_sort_on_agg

avamingli commented Oct 24, 2024

Uh oh!

avamingli commented Oct 25, 2024

Uh oh!

avamingli commented Nov 4, 2024•
edited
Loading

Uh oh!

fanfuxiaoran commented Nov 18, 2024•
edited
Loading

Uh oh!

avamingli commented Nov 21, 2024

Uh oh!

Uh oh!

Uh oh!

fanfuxiaoran commented Nov 26, 2024

Uh oh!

avamingli commented Nov 26, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fanfuxiaoran commented Dec 4, 2024

Uh oh!

avamingli commented Dec 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

avamingli commented Oct 24, 2024

DISTINCT

DISTINCT ON and ORDER BY

ORDER BY

DISTINCT and ORDER BY

Change logs

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Contributor's Checklist

Uh oh!

avamingli commented Oct 25, 2024

Uh oh!

avamingli commented Nov 4, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fanfuxiaoran commented Nov 18, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avamingli commented Nov 21, 2024

Uh oh!

Uh oh!

Uh oh!

fanfuxiaoran commented Nov 26, 2024

Uh oh!

avamingli commented Nov 26, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fanfuxiaoran commented Dec 4, 2024

Uh oh!

avamingli commented Dec 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

avamingli commented Nov 4, 2024•
edited
Loading

fanfuxiaoran commented Nov 18, 2024•
edited
Loading