[feature](nereids)support window function by YangShaw · Pull Request #14397 · apache/doris

YangShaw · 2022-11-18T11:05:18Z

Proposed changes

Issue Number: close #xxx

Problem summary

support WindowFunction in nereids;

Checklist(Required)

Does it affect the original behavior:
- Yes
- No
- I don't know
Has unit tests been added:
- Yes
- No
- No Need
Has document been added or modified:
- Yes
- No
- No Need
Does it need to update dependencies:
- Yes
- No
Are there any changes that cannot be rolled back:
- Yes (If Yes, please explain WHY)
- No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

hello-stephen · 2022-12-28T13:38:24Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.21 seconds
stream load tsv: 462 seconds loaded 74807831229 Bytes, about 154 MB/s
stream load json: 39 seconds loaded 2358488459 Bytes, about 57 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 28 seconds loaded 861443392 Bytes, about 29 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230213131555_clickbench_pr_95395.html

morrySnow · 2023-02-01T07:08:32Z

+ /**
+ * translate WindowFrame to AnalyticWindow
+ */
+ default AnalyticWindow translateWindowFrame(WindowFrame windowFrame, PlanTranslatorContext context) {
+ FrameUnitsType frameUnits = windowFrame.getFrameUnits();
+ FrameBoundary leftBoundary = windowFrame.getLeftBoundary();
+ FrameBoundary rightBoundary = windowFrame.getRightBoundary();
+
+ AnalyticWindow.Type type = frameUnits == FrameUnitsType.ROWS
+ ? AnalyticWindow.Type.ROWS : AnalyticWindow.Type.RANGE;
+
+ AnalyticWindow.Boundary left = withFrameBoundary(leftBoundary, context);
+ AnalyticWindow.Boundary right = withFrameBoundary(rightBoundary, context);
+
+ return new AnalyticWindow(type, left, right);
+ }


i think it it better that put translate functions in translator

due to that AnalyticWindow doesn't extend Expr, it is hard to use visitWindowFrame() in ExpressionTranslator. And in consider of that the code related of translate windowFrame is long, I put these functions in Window interface rather than PhysicalPlanTranslator.

morrySnow · 2023-02-01T08:01:17Z

+ if (containsWindowExpressions(projects)) {
+ return new LogicalWindow<>(projects, input);
+ }
+ return new LogicalProject<>(projects, Collections.emptyList(), input, isDistinct);


maybe we should generate LogicalWindow after we do bind.
we chould just treat windowExpression as a scalar expression.
if we use window function without agg. Then window function will in Projections.
we could generate LogicalProject(LogicalWindow(LogicalProject)) to handle all case.
if we use window function in agg. Then window function will in Agg's output expressions. After normalize agg it will present in the Project topping on Aggregate. and then we could do the same thing as first scene.

morrySnow · 2023-02-13T10:11:50Z

+ * which is an UnboundFunction at first and will be analyzed as relevant BoundFunction
+ * (can be a WindowFunction or AggregateFunction) after BindFunction.
+ */
+public class WindowExpression extends Expression implements PropagateNullable {


Suggested change
publicclassWindowExpressionextendsExpressionimplementsPropagateNullable{
publicclassWindowExpressionextendsExpression {
@Override
publicbooleannullable() {
returnfunction.nullable();
}

morrySnow · 2023-02-13T10:12:08Z

+ @Override
+ public DataType getDataType() {
+ return IntegerType.INSTANCE;
+ }


remove this function

morrySnow · 2023-02-13T10:15:00Z

+ @Override
+ public DataType getDataType() {
+ return IntegerType.INSTANCE;
+ }


remove this

morrySnow · 2023-02-13T10:17:46Z

+ SELECT *, row_number() over(partition by c1)
+ FROM (
+ SELECT *, row_number() over(partition by c2)
+ FROM window_test


need order by in over to get a stable result

morrySnow · 2023-02-13T10:36:59Z

+ FunctionSignature.ret(LargeIntType.INSTANCE).args(LargeIntType.INSTANCE)
+ );
+
+ private Expression buckets;


bucket is not use anymore

github-actions · 2023-02-13T13:02:35Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-02-13T13:02:40Z

PR approved by anyone and no changes requested.

fix distinct window compute wrong result, introduced by apache#14397 ```sql select distinct sum(value) over(partition by id) from ( select 100 value, 1 id union all select 100, 2 )a; +----------------------------------+ | sum(value) over(partition by id) | +----------------------------------+ | 100 | | 100 | +----------------------------------+ ```

### What problem does this PR solve? Related PR: #21727 #14397 Problem Summary: 1. forgot to copy isChecked flag in LogicalWindow when do deep copy 2. implement LogicalWindow To PhyscialWindow should not check isChecked flag This PR: 1. check deep copy for all plan node 2. remove check isChecked in LogicalWindow To PhyscialWindow

Related PR: #21727 #14397 Problem Summary: 1. forgot to copy isChecked flag in LogicalWindow when do deep copy 2. implement LogicalWindow To PhyscialWindow should not check isChecked flag This PR: 1. check deep copy for all plan node 2. remove check isChecked in LogicalWindow To PhyscialWindow

fix distinct window compute wrong result, introduced by apache#14397 ```sql select distinct sum(value) over(partition by id) from ( select 100 value, 1 id union all select 100, 2 )a; +----------------------------------+ | sum(value) over(partition by id) | +----------------------------------+ | 100 | | 100 | +----------------------------------+ ```

### What problem does this PR solve? Related PR: apache#21727 apache#14397 Problem Summary: 1. forgot to copy isChecked flag in LogicalWindow when do deep copy 2. implement LogicalWindow To PhyscialWindow should not check isChecked flag This PR: 1. check deep copy for all plan node 2. remove check isChecked in LogicalWindow To PhyscialWindow

github-actionsBot added area/nereids area/planner Issues or PRs related to the query planner labels Nov 18, 2022

YangShaw force-pushed the analytic_node branch 3 times, most recently from 2deb44f to 04696e9Compare November 21, 2022 17:20

YangShaw force-pushed the analytic_node branch from f6b2091 to 4a05b19Compare December 28, 2022 08:11

YangShaw force-pushed the analytic_node branch from f8de731 to 8ba1a4fCompare January 16, 2023 10:55

YangShaw marked this pull request as ready for review January 18, 2023 05:49

github-actionsBot added the kind/test label Jan 18, 2023

YangShaw force-pushed the analytic_node branch from 3c4e44a to 06b2082Compare January 18, 2023 11:41

YangShaw force-pushed the analytic_node branch 3 times, most recently from 09c9cda to f81180bCompare January 31, 2023 03:33

morrySnow reviewed Feb 1, 2023
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java Outdated

YangShaw force-pushed the analytic_node branch 2 times, most recently from 780f3c9 to 3c3cd0fCompare February 10, 2023 01:35

fuyu29 added 8 commits February 10, 2023 16:46

window function
3f03c66

rebase
a2a8bc0

temp change pull_request_id
dac1046

fix window functions
277545e

fix window(agg)
1f9afbb

rebase
7da5744

fix
caa22e1

fix reg
81fb58d

YangShaw force-pushed the analytic_node branch from feb9aad to 81fb58dCompare February 10, 2023 09:07

fuyu29 added 2 commits February 11, 2023 13:23

fix
ffcda18

fix
423948a

morrySnow reviewed Feb 13, 2023
View reviewed changes

fuyu29 added 5 commits February 13, 2023 18:43

fix ntile & dense_rank
135f39d

fix reg
4ae669f

fix
9d496a9

fix
90a5cec

fix
809230d

morrySnow approved these changes Feb 13, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 13, 2023

github-actionsBot added the reviewed label Feb 13, 2023

morrySnow merged commit 77a3288 into apache:masterFeb 13, 2023

YangShaw added a commit to YangShaw/doris that referenced this pull request Feb 17, 2023
[feature](Nereids) support window function (apache#14397)
ba14e20

morrySnow mentioned this pull request Mar 10, 2025
[fix](Nereids) deep copy for LogicalWindow is wrong #48861
Merged
16 tasks

924060929 mentioned this pull request Mar 12, 2025
[fix](nereids) fix distinct window compute wrong result #48987
Closed
16 tasks

-publicclassWindowExpressionextendsExpressionimplementsPropagateNullable{
+publicclassWindowExpressionextendsExpression {
+@Override
+publicbooleannullable() {
+returnfunction.nullable();
+ }

Uh oh!

Conversation

YangShaw commented Nov 18, 2022• edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

hello-stephen commented Dec 28, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Feb 13, 2023

Uh oh!

github-actionsBot commented Feb 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

YangShaw commented Nov 18, 2022•
edited by morrySnow
Loading

hello-stephen commented Dec 28, 2022•
edited
Loading