Skip to content

[refactor](Nereids) mv top-n two phase read rule from post processor to rewriter - #22487

Merged
jackwener merged 1 commit into
apache:masterfrom
morrySnow:topn_phase1
Aug 3, 2023
Merged

[refactor](Nereids) mv top-n two phase read rule from post processor to rewriter#22487
jackwener merged 1 commit into
apache:masterfrom
morrySnow:topn_phase1

Conversation

@morrySnow

@morrySnowmorrySnow commented Aug 2, 2023

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

use three new plan node to represent defer materialize of TopN.
Example:

-- SQL
select * from t1 order by c1 limit 10;
-- PLAN
+------------------------------------------+
| Explain String |
+------------------------------------------+
| PhysicalDeferMaterializeResultSink |
| --PhysicalDeferMaterializeTopN |
| ----PhysicalDistribute |
| ------PhysicalDeferMaterializeTopN |
| --------PhysicalDeferMaterializeOlapScan |
+------------------------------------------+

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morrySnowmorrySnow changed the title [refactor] mv top-n two phase read rule from post processor to rewriter[refactor](Nereids) mv top-n two phase read rule from post processor to rewriterAug 2, 2023
@morrySnow

morrySnow commented Aug 2, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morrySnow

morrySnow commented Aug 2, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

hello-stephen commented Aug 2, 2023

Copy link
Copy Markdown
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.71 seconds
stream load tsv: 514 seconds loaded 74807831229 Bytes, about 138 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162257663 Bytes

@morrySnow

morrySnow commented Aug 2, 2023

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

hello-stephen commented Aug 2, 2023

Copy link
Copy Markdown
Contributor

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.6 seconds
stream load tsv: 517 seconds loaded 74807831229 Bytes, about 137 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.6 seconds inserted 10000000 Rows, about 337K ops/s
storage size: 17162131516 Bytes

@eldenmoon

eldenmoon commented Aug 2, 2023

Copy link
Copy Markdown
Member

run p0

eldenmoon
eldenmoon approved these changes Aug 2, 2023
View reviewed changes

@eldenmooneldenmoon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 2, 2023
@github-actions

github-actionsBot commented Aug 2, 2023

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Aug 2, 2023

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@jackwener
jackwener merged commit 3961b8d into apache:masterAug 3, 2023
@morrySnow
morrySnow deleted the topn_phase1 branch August 11, 2023 10:47
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Aug 11, 2023
@morrySnow
[refactor](Nereids) mv top-n two phase read rule from post processor …
3c4be62
…to rewriter (apache#22487)
use three new plan node to represent defer materialize of TopN.
Example:
```
-- SQL
select * from t1 order by c1 limit 10;
-- PLAN
+------------------------------------------+
| Explain String |
+------------------------------------------+
| PhysicalDeferMaterializeResultSink |
| --PhysicalDeferMaterializeTopN |
| ----PhysicalDistribute |
| ------PhysicalDeferMaterializeTopN |
| --------PhysicalDeferMaterializeOlapScan |
+------------------------------------------+
```
@morrySnowmorrySnow mentioned this pull request Aug 11, 2023
Merged
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Aug 11, 2023
@morrySnow
[refactor](Nereids) mv top-n two phase read rule from post processor …
1aaa0e3
…to rewriter (apache#22487)
use three new plan node to represent defer materialize of TopN.
Example:
```
-- SQL
select * from t1 order by c1 limit 10;
-- PLAN
+------------------------------------------+
| Explain String |
+------------------------------------------+
| PhysicalDeferMaterializeResultSink |
| --PhysicalDeferMaterializeTopN |
| ----PhysicalDistribute |
| ------PhysicalDeferMaterializeTopN |
| --------PhysicalDeferMaterializeOlapScan |
+------------------------------------------+
```
xiaokang pushed a commit that referenced this pull request Aug 11, 2023
@morrySnow
[refactor](Nereids) mv top-n two phase read rule from post processor …
4ef0f53
…to rewriter (#22487) (#22894)
use three new plan node to represent defer materialize of TopN.
Example:
```
-- SQL
select * from t1 order by c1 limit 10;
-- PLAN
+------------------------------------------+
| Explain String |
+------------------------------------------+
| PhysicalDeferMaterializeResultSink |
| --PhysicalDeferMaterializeTopN |
| ----PhysicalDistribute |
| ------PhysicalDeferMaterializeTopN |
| --------PhysicalDeferMaterializeOlapScan |
+------------------------------------------+
```
@morrySnowmorrySnow mentioned this pull request Mar 6, 2025
Merged
16 tasks
morrySnow added a commit that referenced this pull request Mar 20, 2025
@morrySnow
[opt](Nereids) support more pattern in DeferMaterializeTopNResult (#4…
947463d
…8747)
### What problem does this PR solve?
Related PR: #47661, #22487
Problem Summary:
after this PR we support:
1. TopN(Scan)
2. Project(TopN(Project(Scan)))
3. TopN(Filter(Scan))
4. TopN(Project(Filter(Scan)))
5. Project(TopN(Project(Filter(Scan))))
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@morrySnowkoarz
[opt](Nereids) support more pattern in DeferMaterializeTopNResult (ap…
bd2a11f
…ache#48747)
### What problem does this PR solve?
Related PR: apache#47661, apache#22487
Problem Summary:
after this PR we support:
1. TopN(Scan)
2. Project(TopN(Project(Scan)))
3. TopN(Filter(Scan))
4. TopN(Project(Filter(Scan)))
5. Project(TopN(Project(Filter(Scan))))
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jackwenerjackwenerjackwener approved these changes
@eldenmooneldenmooneldenmoon approved these changes
@starocean999starocean999Awaiting requested review from starocean999starocean999 is a code owner
@engleflyengleflyAwaiting requested review from engleflyenglefly is a code owner

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.1-mergedmerge_conflictreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morrySnow@hello-stephen@eldenmoon@jackwener@xiaokang