[opt](nereids) support pushing down aggr distinct through join by xzj7019 · Pull Request #43380 · apache/doris

xzj7019 · 2024-11-06T13:00:27Z

What problem does this PR solve?

support pushing down agg distinct through join on one side.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No colde files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.
Release note
None

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-06T13:00:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xzj7019 · 2024-11-28T13:37:30Z

run buildall

xzj7019 · 2024-11-29T02:41:45Z

run buildall

feiniaofeiafei · 2024-11-29T04:19:33Z

+ .when(agg -> agg.child().child().getOtherJoinConjuncts().isEmpty())
+ .when(agg -> !agg.isGenerated())
+ .whenNot(agg -> agg.getAggregateFunctions().isEmpty())
+ .whenNot(agg -> agg.child().children().stream().anyMatch(p -> p instanceof LogicalAggregate))


agg -> agg.child().children().stream().anyMatch(p -> p instanceof LogicalAggregate) , this is always false.

thx, should be changed to .whenNot(agg -> agg.child()
.child(0).children().stream().anyMatch(p -> p instanceof LogicalAggregate))

feiniaofeiafei · 2024-11-29T04:20:39Z

+ leftFuncs.forEach(func -> {
+ Alias alias = func.alias("PDADT_" + func.getName());
+ leftSlotToOutput.put((Slot) func.child(0), alias);
+ });


map leftSlotToOutput value seems to be useless

xzj7019 · 2024-11-29T06:51:24Z

run buildall

github-actions · 2024-11-29T06:54:20Z

PR approved by anyone and no changes requested.

xzj7019 · 2024-11-29T08:16:44Z

run buildall

xzj7019 · 2024-11-29T11:35:50Z

run cloud_p0

xzj7019 · 2024-12-02T01:55:42Z

run cloud_p0

englefly · 2024-12-02T02:08:16Z

+ public List<Rule> buildRules() {
+ return ImmutableList.of(
+ logicalAggregate(logicalProject(innerLogicalJoin()))
+ .when(agg -> agg.child().isAllSlots())


agg.child().isAllSlots()
is this restriction too strong?

aggr with distinct pushing down has special rewrite logic and restrict this case carefully by safety

englefly · 2024-12-02T02:09:10Z

+ return ImmutableList.of(
+ logicalAggregate(logicalProject(innerLogicalJoin()))
+ .when(agg -> agg.child().isAllSlots())
+ .when(agg -> agg.child().child().getOtherJoinConjuncts().isEmpty())


"getOtherJoinConjuncts().isEmpty()"
how about remove this?

leave this clean up task for the whole series of aggr pushing down rules

englefly · 2024-12-02T02:18:51Z

why it is called oneSide? it could push down through both sides

xzj7019 · 2024-12-02T03:13:48Z

run cloud_p0

xzj7019 · 2024-12-02T03:29:37Z

run buildall

xzj7019 · 2024-12-02T09:22:39Z

run cloud_p0

xzj7019 · 2024-12-02T09:29:07Z

run buildall

xzj7019 · 2024-12-04T07:15:23Z

run buildall

xzj7019 · 2024-12-04T10:59:54Z

run p0

xzj7019 · 2024-12-04T11:25:30Z

run p0

github-actions · 2024-12-05T01:57:01Z

PR approved by at least one committer and no changes requested.

… join #43380 (#45067) Cherry-picked from #43380 Co-authored-by: xzj7019 <xiongzhongjian@selectdb.com>

…e#43380)

xzj7019 marked this pull request as draft November 6, 2024 13:00

xzj7019 marked this pull request as ready for review November 19, 2024 06:16

xzj7019 marked this pull request as draft November 19, 2024 06:16

xzj7019 force-pushed the pushdown_agg_distinct branch 2 times, most recently from 7a772d1 to 29c1de5Compare November 28, 2024 13:37

xzj7019 marked this pull request as ready for review November 28, 2024 13:37

xzj7019 changed the title ~~[opt](nereids) support pushdown agg distinct through join~~[opt](nereids) support pushing down aggr distinct through joinNov 28, 2024

xzj7019 force-pushed the pushdown_agg_distinct branch from 29c1de5 to 964514cCompare November 29, 2024 02:40

feiniaofeiafei reviewed Nov 29, 2024
View reviewed changes

xzj7019 force-pushed the pushdown_agg_distinct branch from 964514c to e99aa8cCompare November 29, 2024 06:50

feiniaofeiafei approved these changes Nov 29, 2024
View reviewed changes

github-actionsBot added the reviewed label Nov 29, 2024

xzj7019 force-pushed the pushdown_agg_distinct branch from e99aa8c to 8076eb0Compare November 29, 2024 08:16

englefly reviewed Dec 2, 2024
View reviewed changes

xzj7019 force-pushed the pushdown_agg_distinct branch from 8076eb0 to b243038Compare December 2, 2024 03:28

xzj7019 force-pushed the pushdown_agg_distinct branch from bf3e09d to 5e048dbCompare December 2, 2024 09:27

xzj7019 added 14 commits December 4, 2024 15:14

[opt](nereids) support pushdown agg distinct through join
d8aae1d

[opt](nereids) support pushdown agg distinct through join
0ed5e3f

[opt](nereids) support pushdown agg distinct through join
3fed78c

[opt](nereids) support pushdown agg distinct through join
2b9e6ad

[opt](nereids) support pushdown agg distinct through join
eafab7b

[opt](nereids) support pushdown agg distinct through join
f961f14

[opt](nereids) support pushdown agg distinct through join
02ba75d

[opt](nereids) support pushdown agg distinct through join
805c312

[opt](nereids) support pushdown agg distinct through join
67049f7

[opt](nereids) support pushdown agg distinct through join
2530ff0

[opt](nereids) support pushdown agg distinct through join
6d4e705

[opt](nereids) support pushdown agg distinct through join
2c4c431

[opt](nereids) support pushdown agg distinct through join
5687192

[opt](nereids) support pushdown agg distinct through join
9e4d53c

xzj7019 force-pushed the pushdown_agg_distinct branch from c85ccfa to 9e4d53cCompare December 4, 2024 07:15

englefly approved these changes Dec 5, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 5, 2024

morrySnow merged commit bf4ba66 into apache:masterDec 5, 2024

morrySnow added the dev/3.0.x label Dec 5, 2024

github-actionsBot pushed a commit that referenced this pull request Dec 5, 2024
[opt](nereids) support pushing down aggr distinct through join (#43380)
986af64

github-actionsBot mentioned this pull request Dec 5, 2024
branch-3.0: [opt](nereids) support pushing down aggr distinct through join #43380 #45067
Merged

morrySnow pushed a commit that referenced this pull request Dec 10, 2024
branch-3.0: [opt](nereids) support pushing down aggr distinct through…
243561d
… join #43380 (#45067) Cherry-picked from #43380 Co-authored-by: xzj7019 <xiongzhongjian@selectdb.com>

morrySnow added dev/3.0.4-merged and removed dev/3.0.x labels Dec 10, 2024

gavinchou mentioned this pull request Feb 18, 2025
Release Notes 3.0.4 #48013
Closed

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[opt](nereids) support pushing down aggr distinct through join (apach…
4c211f8
…e#43380)

Uh oh!

Conversation

xzj7019 commented Nov 6, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 6, 2024

Uh oh!

xzj7019 commented Nov 28, 2024

Uh oh!

xzj7019 commented Nov 29, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xzj7019 commented Nov 29, 2024

Uh oh!

github-actionsBot commented Nov 29, 2024

Uh oh!

xzj7019 commented Nov 29, 2024

Uh oh!

xzj7019 commented Nov 29, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

englefly commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 2, 2024

Uh oh!

xzj7019 commented Dec 4, 2024

Uh oh!

xzj7019 commented Dec 4, 2024

Uh oh!

xzj7019 commented Dec 4, 2024

Uh oh!

github-actionsBot commented Dec 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

xzj7019 commented Nov 6, 2024•
edited
Loading